What devices do you have between your server and the Internet and are you sure that your ISP gives you a public IP address? For example if you go to http://www.canyouseeme.org/ does that show the public IP address that you think it should, and if you test port 25565 from there (with your minecraft server running) does it show that port open or not. If not, you need to determine what you are overlooking.
And is your modem just a plain old bridge modem, or is it a modem/router or gateway (wireless/router/modem)? Because if you have a modem/router and a separate router, on your modem/router you would also need to port forward port 25565 to the WAN IP of your secondary router, besides port forwarding on your secondary router to LAN IP of your server.
For example when I checked my mobile data IP it appeared to be a public IP, but when I used whois to look up those IP addresses I got things like US Dept of Defense or UK Ministry of Defense. My first thought was that NSA is listening, but I found out that my carrier uses publicly assigned, but not publicly used IP addresses on their mobile network to avoid conflict with private IP address ranges used for Wi-Fi and that I was actually behind another NAT router at the phone company.
If you find that you do not really have a public IP, you could look into Hamachi (related to Logmein) that allows others to connect to your server with a VPN. You can control who connects to that Hamachi VPN with a main password, individual usernames and passwords, or that you have to individually approve before they can connect. Or you can leave it wide open to anyone with your Hamachi IP.