Any ssh whizzes in the house?

I noticed that someone / some thing is knocking on the sshd port of our home server lately. It looks like a script of some kind, checking for users root, admin, test, guest, and user.

I tightened up the sshd_config file a bit (turned off password authentication, relying entirely on keys). One thing which intrigues me is whether or not I can use host keys to further lock down connecting to the sshd server. My initial read of the various sshd references I could find seemed to indicate that this is possible but on testing I was able to connect with a client which I'd verified wasn't in /etc/ssh_known_hosts or ~/.ssh/known_hosts.

I can't do DNS or IP-address restrictions because I connect from various WiFi and Hotel broadband hookups and won't know the address or hostname until I actually connect.

Any ideas? (I'm looking at you, Chet).