Controlling who can use my server as relayhost

[Nicolás]

Hi,

After maintaining an old version of Postfix for some longer time, I 
finally decided to jump to version 2.11 and currently I'm tuning it up. 
I'm having an issue with smtpd_relay_restrictions. At this time, the 
configuration is the default one:

    smtpd_relay_restrictions =
       permit_mynetworks
       permit_sasl_authenticated
       defer_unauth_destination

Having this configuration, anyone using my mail server as the relayhost 
is able to send mails to the domains that I handle (not outside), even 
without SASL. I guess that behavior is determined by 
'defer_unauth_destination', however, my aim is to specifically allow 
certain IPs to use my mail server as relayhost, otherwise reject the 
requests.

I tried putting a 'reject' line after 'defer_unauth_destination' and 
although it seemed to achieve the desired effect, this would block any 
incoming e-mails from any sender, logically.

So, here goes the question: Is there a way to only whitelist certain IPs 
to use this server as the relayhost and reject anyone else but without 
affecting incoming e-mails?

Thanks!