IP bans are the biggest problem with me. Almost any staff will tell you that it's a bad idea for a network our size, and I don't blame them.
Other than that, I find the concept of using a special client to connect to a server incredibly inconvenient. Imagine it from a noob's perspective:
1) "Hey, this server looks pretty cool!"
2) *not allowed to connect* "Really?"
3) "Forget it, I'm not downloading some dumb mod to join one or two servers."
4) Our playerbase starts decreasing more and more.
Mojang just needs to code in a system to detect any changes to different files. This would stop gamma, >110 fov, and it would know when mods are active. Every mod would need a codename to work, which would be registered on minecraft.net and set into the .minecraft. Then when you log into a server, a whitelist would make sure you were using one of the allowed codenames.