Some people can connect, while some can't.

Hello everyone.

Recently, I've been having an issue with some people being able to connect to my server, while some can't. It will even happen to me once in a while; I'll see that people are online via my website, and the console shows people talking and using commands. But for some reason I'm not able to connect for 10 minutes or so. Refreshing doesn't help, and I just have to wait. I've tried restarting my server and my Minecraft client.

Has anyone had this same issue? Does anyone know how to solve it?