Konstantin Ryabitsev wrote:
> On Mon, Apr 01, 2024 at 01:21:45PM +, Eric Wong wrote:
> > Performance is still slow, and crawler traffic patterns tend to
> > do bad things with caches at all levels, so I've regretfully had
> > to experiment with robots.txt to mitigate performance problems.
>
On Mon, Apr 01, 2024 at 01:21:45PM +, Eric Wong wrote:
> Performance is still slow, and crawler traffic patterns tend to
> do bad things with caches at all levels, so I've regretfully had
> to experiment with robots.txt to mitigate performance problems.
This has been the source of grief for
