ure request, please create a CONNECTORS ticket that
> describes the functionality you think the connector should have.
>
> Karl
>
> On Wed, Jul 7, 2021 at 9:29 AM h0444xk8 wrote:
>
>> Hi,
>>
>> yes, that seems to be the reason. In:
>>
t;sitemaps" line, which was
likely not in the spec for robots when this connector was written.
Karl
On Wed, Jul 7, 2021 at 3:31 AM h0444xk8 wrote:
Hi,
I have a general question. Is the Web connector supporting sitemap
files
referenced by the robots.txt? In my use case the robots.tx
Hi,
I have a general question. Is the Web connector supporting sitemap files
referenced by the robots.txt? In my use case the robots.txt is stored in
the root of the website and is referencing two compressed sitemaps.
Example of robots.txt
User-Agent: *
Disallow:
