First, check your white list (Server > Mail > White List tab) and make sure it doesn't say 127.0.0.1 / 8. The 127.0.0.1 part is okay, but the subnet mask should be 32.
Then in your preferences set up some sort of authorization requirement. Personally I use POP3 lock with a 20 minute time out. Seems to keep everyone happy yet keep the spammers at bay.
Lastly, sign up for a Feedback loop somewhere to make sure you get an early notification if someone does start spamming either through a script, their PC gets hacked or they're just idiots.
AOL offers one that's completely automated once you set it up. And since the spammers always target thousands of aol.com addresses you're sure to get an early heads up. Though you may get the occasional false positive too.