On Mon, 26 Feb 2018 00:07:54 -0600 (CST)
David B Funk <dbf...@engineering.uiowa.edu> wrote:
> So my bet is that the spammers are crafty enough to check things like
> browser referrer, cookies, etc to detect/differentiate a browser vs a
> link-checker.
Yep. You need to fake your User-Agent (not hard) and put a limit on
the number of redirections you follow or you can end up in trouble.
I picked a likely-looking agent out of our logs and use this:
$ua->agent('Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64;
Trident/7.0; SLCC2;');
Might need revising later because it's pretty ancient!
Regards,
Dianne.
