Hello, we are having some issues when being collected by Oaister, and we suspect that their robot doesn't obey the Retry-After HTTP header. Before modifying our configuration or asking them if this Retry-After header is problematic for them, may we ask you why the default value is set to 10 seconds?
Is it safe to lower it to zero? Are you being successfully collected by Oaister? Thanks for your advices, Ferran
