t.co is not a crawler; Are you referring to the URL unpacking process or something else?
-john On Thu, Jun 10, 2010 at 11:46 PM, Ken <k...@cimas.ch> wrote: > If tco is to be the new three-letter agency and gatekeeper, we would > like to treat it nice and whitelist its crawler. If tco is > inadvertantly blocked, what happens? > > I do not know if we have already been checked by tco as I have not > sent or received a dm with one of our own URLs. > > What are the user-agent and IP addresses used by this crawler? Does it > check robots.txt? > > And since, for some, a tco thumbsdown could be a problem, is there a > (speedy) appeals process? >
