vincbeck commented on PR #761:
URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496345696

   I found a way to fetch all versions from providers and Airflow. I am using 
the file `packages-metadata.json`. That way we can apply some deterministic 
rules to decide whether a given version should be indexed. The current 
implementation disallows all versions but the last one. **I understand this is 
not what we want and just did that for example**. Here is the
   
[robots.txt](https://github.com/apache/airflow-site/files/11151148/robots.txt) 
file generated with this implementation.
   
   I would like to use this PR to discuss and decide on which condition a given 
version (for Airflow and providers) should be indexed by search engines.
   - Airflow: @potiuk suggested to have an upper limit (currently `2.3` and 
will potentially bump up to `2.4` in May). Everything **below** this upper 
limit would NOT be indexed by search engines. This upper limit is the minimum 
Airflow version needed to install latest provider releases
   - Providers. Hard question. My personal opinion is we should come up with a 
rule which applies to all providers. Having provider specific rules would be 
too complex/time spending to manage. That being said I dont have strong opinion 
on which rule we should have for providers. e.g. Only versions which belong the 
latest major version? Only versions which belong the 2 latest major versions? 
WDYT?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to