The problem you got, is that the encrypted content has already travelled the amateur frequencies even if you block/reject the mail. Thus the rules are already broken, thus you should deal with those users in a "AUP" way even if the mail gets blocked. Better might be to block this in firewall then. I guess you intend to operate a outgoing mail relay now.
You could use iptables to look for: "--BEGIN" "--END" "/signed" "/encrypted" "/pkcs7" "/pgp" Anywhere in the packet. In that case, you drop the connection, send a RST to source and target, and then you could ban the user account involved. You can look here how to do with IP-tables, both for blocking encrypted content, but also triggering some sort of ban. http://blog.nintechnet.com/how-to-block-w00tw00t-at-isc-sans-dfind-and-other -web-vulnerability-scanners/ You can also use Fail2Ban for this, and trigger ban based on user account. (NOTE: Signed content is technically encrypted content too) ----- If you instead operate a incoming mail relay (so internet users can send messages to amateur radio operators), you said this would only apply to certain users. Then its better to block this on IMAP/POP3 level, as the users in question might connect over public internet. Eg, encrypted mail is received and stored, but it will never Be delivered if the user is doing the fetch over radio frequencies (you could instead send some auto message like "You have 1 new encrypted message", but if the user happens to fetch over the public internet, then they get any encrypted content.
smime.p7s
Description: S/MIME Cryptographic Signature