ssh always asks for passphrase?

[Paul Spicer]

My linux machine at home is setup to only allow key authentication. I have a
copy of that key on my machine at work and it works great. However, when I
try to connect to one of the servers here on site that aren't setup for key
authentication, I'm forced to enter my passphrase every time before I can
enter the password. Is there any way around this? It even does it when I try
to connect to network resources through nautilus.