vincbeck commented on PR #761: URL: https://github.com/apache/airflow-site/pull/761#issuecomment-1496345696
I found a way to fetch all versions from providers and Airflow. I am using the file `packages-metadata.json`. That way we can apply some deterministic rules to decide whether a given version should be indexed. The current implementation disallows all versions but the last one. **I understand this is not what we want and just did that for example**. Here is the [robots.txt](https://github.com/apache/airflow-site/files/11151148/robots.txt) file generated with this implementation. I would like to use this PR to discuss and decide on which condition a given version (for Airflow and providers) should be indexed by search engines. - Airflow: @potiuk suggested to have an upper limit (currently `2.3` and will potentially bump up to `2.4` in May). Everything **below** this upper limit would NOT be indexed by search engines. This upper limit is the minimum Airflow version needed to install latest provider releases - Providers. Hard question. My personal opinion is we should come up with a rule which applies to all providers. Having provider specific rules would be too complex/time spending to manage. That being said I dont have strong opinion on which rule we should have for providers. e.g. Only versions which belong the latest major version? Only versions which belong the 2 latest major versions? WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
