Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk commented on PR #50716:
URL: https://github.com/apache/airflow/pull/50716#issuecomment-2888292193

   > cool Nice catch amogh
   
   Nice catch indeed - i had no idea that  you cannot pu t authorization in raw 
URL 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk commented on PR #50716:
URL: https://github.com/apache/airflow/pull/50716#issuecomment-2888297462

   Very good catch @amoghrajesh !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk commented on PR #50716:
URL: https://github.com/apache/airflow/pull/50716#issuecomment-2888297355

   I think we have other places in some bash scripts and yaml files where we 
are using `raw` urls -> we should  fix all of them in a similar way.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


github-actions[bot] commented on PR #50716:
URL: https://github.com/apache/airflow/pull/50716#issuecomment-2888297807

   ### Backport successfully created: v3-0-test
   
   
   
   Status
   Branch
   Result
   
   
   ✅
   v3-0-test
   https://github.com/apache/airflow/pull/50721";>https://img.shields.io/badge/PR-50721-blue"; alt="PR Link">
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


gopidesupavan commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094093104


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   @amoghrajesh  your good to go :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


amoghrajesh commented on PR #50716:
URL: https://github.com/apache/airflow/pull/50716#issuecomment-2888297566

   > I think we have other places in some bash scripts and yaml files where we 
are using `raw` urls -> we should fix all of them in a similar way.
   
   Yep, will do!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk merged PR #50716:
URL: https://github.com/apache/airflow/pull/50716


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


amoghrajesh commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094093040


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   This is the 3.0.1 constraints:
   ```
   
   #
   # This constraints file was automatically generated on 
2025-05-10T16:35:29.362244
   # via `uv pip install --resolution highest` for the "v3-0-test" branch of 
Airflow.
   # This variant of constraints install uses the HEAD of the branch version 
for 'apache-airflow' but installs
   # the providers from PIP-released packages at the moment of the constraint 
generation.
   #
   # Those constraints are actually those that regular users use to install 
released version of Airflow.
   # We also use those constraints after "apache-airflow" is released and the 
constraints are tagged with
   # "constraints-X.Y.Z" tag to build the production image for that version.
   #
   # This constraints file is meant to be used only in the "apache-airflow" 
installation command and not
   # in all subsequent pip commands. By using a constraints.txt file, we ensure 
that solely the Airflow
   # installation step is reproducible. Subsequent pip commands may install 
packages that would have
   # been incompatible with the constraints used in Airflow reproducible 
installation step. Finally, pip
   # commands that might change the installed version of apache-airflow should 
include "apache-airflow==X.Y.Z"
   # in the list of install targets to prevent Airflow accidental upgrade or 
downgrade.
   #
   # Typical installation process of airflow for Python 3.9 is (with random 
selection of extras and custom
   # dependencies added), usually consists of two steps:
   #
   # 1. Reproducible installation of airflow with selected providers (note 
constraints are used):
   #
   # pip install 
"apache-airflow[celery,cncf.kubernetes,google,amazon,snowflake]==X.Y.Z" \
   # --constraint \
   #
"https://raw.githubusercontent.com/apache/airflow/constraints-X.Y.Z/constraints-3.9.txt";
   #
   # 2. Installing own dependencies that are potentially not matching the 
constraints (note constraints are not
   #used, and apache-airflow==X.Y.Z is used to make sure there is no 
accidental airflow upgrade/downgrade.
   #
   # pip install "apache-airflow==X.Y.Z" 
"snowflake-connector-python[pandas]=N.M.O"
   #
   APScheduler==3.11.0
   Authlib==1.3.1
   Deprecated==1.2.18
   Events==0.5
   Flask-AppBuilder==4.5.3
   Flask-Babel==2.0.0
   Flask-JWT-Extended==4.7.1
   Flask-Limiter==3.11.0
   Flask-Login==0.6.3
   Flask-SQLAlchemy==2.5.1
   Flask-Session==0.5.0
   Flask-WTF==1.2.2
   Flask==2.2.5
   GitPython==3.1.44
   JayDeBeApi==1.2.3
   Jinja2==3.1.6
   Js2Py==0.74
   Mako==1.3.10
   Markdown==3.8
   MarkupSafe==3.0.2
   PyAthena==3.13.0
   PyGithub==2.6.1
   PyHive==0.7.0
   PyJWT==2.10.1
   PyMySQL==1.1.1
   PyNaCl==1.5.0
   PyYAML==6.0.2
   Pygments==2.19.1
   SQLAlchemy-JSONField==1.0.2
   SQLAlchemy-Utils==0.41.2
   SQLAlchemy==1.4.54
   SecretStorage==3.3.3
   Sphinx==7.4.7
   WTForms==3.2.1
   Werkzeug==2.2.3
   a2wsgi==1.10.8
   adal==1.2.7
   adlfs==2024.12.0
   aiobotocore==2.22.0
   aiofiles==24.1.0
   aiohappyeyeballs==2.6.1
   aiohttp-cors==0.8.1
   aiohttp==3.11.18
   aioitertools==0.12.0
   aiologic==0.14.0
   aiomysql==0.2.0
   aioresponses==0.7.8
   aiosignal==1.3.2
   aiosqlite==0.21.0
   airbyte-api==0.52.2
   alabaster==0.7.16
   alembic==1.15.2
   alibabacloud-adb20211201==3.0.3
   alibabacloud-credentials-api==1.0.0
   alibabacloud-credentials==1.0.2
   alibabacloud-tea==0.4.3
   alibabacloud_endpoint_util==0.0.3
   alibabacloud_gateway_spi==0.0.3
   alibabacloud_openapi_util==0.2.2
   alibabacloud_tea_openapi==0.3.15
   alibabacloud_tea_util==0.3.13
   alibabacloud_tea_xml==0.0.2
   aliyun-python-sdk-core==2.16.0
   aliyun-python-sdk-kms==2.16.5
   amqp==5.3.1
   analytics-python==1.2.9
   annotated-types==0.7.0
   ansicolors==1.1.8
   anyio==4.9.0
   apache-airflow-providers-airbyte==5.0.2
   apache-airflow-providers-alibaba==3.0.3
   apache-airflow-providers-amazon==9.7.0
   apache-airflow-providers-apache-beam==6.0.4
   apache-airflow-providers-apache-cassandra==3.7.2
   apache-airflow-providers-apache-drill==3.0.1
   apache-airflow-providers-apache-druid==4.1.1
   apache-airflow-providers-apache-flink==1.6.2
   apache-airflow-providers-apache-hdfs==4.8.1
   apache-airflow-providers-apache-hive==9.0.6
   apache-airflow-providers-apache-iceberg==1.2.1
   apache-airflow-providers-apache-impala==1.6.1
   apache-airflow-providers-apache-kafka==1.8.1
   apache-airflow-providers-apache-kylin==3.8.3
   apache-airflow-providers-apache-livy==4.3.2
   apache-airflow-providers-apache-pig==4.6.1
   apache-airflow-providers-apache

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


amoghrajesh commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094093003


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   Just checked, all good on that front:
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094092994


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   Just checked all fine - we get the content with 
`application/vnd.github.v3.raw` and not base64 encoded :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


amoghrajesh commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094092742


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   The end result seems pretty fine too. For ex, the entry for:
   "cloudant" is:
   ```
   "4.1.1": {
   "associated_airflow_version": "3.0.1",
   "date_released": "2025-03-13T17:46:49Z"
   }
   ```
   
   Which is in line with other things



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


gopidesupavan commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094092871


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   Ah this should work `application/vnd.github.raw+json` :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


amoghrajesh commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094092742


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   The end result seems pretty fine too. For ex, the entry for:
   "cloudant" is:
   ```
   "4.1.1": {
   "associated_airflow_version": "3.0.1",
   "date_released": "2025-03-13T17:46:49Z"
   }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094092399


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   Ah `application/vnd.github.raw+json` - > this should return the content. Is 
it really base64 encoded @gopidesupavan @amoghrajesh - > have not checked it ;)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


amoghrajesh commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094092668


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   Delayed response. Yes that header shouls make sure we get a json instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


potiuk commented on code in PR #50716:
URL: https://github.com/apache/airflow/pull/50716#discussion_r2094091812


##
dev/breeze/src/airflow_breeze/utils/github.py:
##
@@ -55,21 +55,21 @@ def download_file_from_github(
 """
 import requests
 
-url = f"https://raw.githubusercontent.com/apache/airflow/{tag}/{path}";
+url = 
f"https://api.github.com/repos/apache/airflow/contents/{path}?ref={tag}";

Review Comment:
   Indeed @gopidesupavan  is right - this API  
https://docs.github.com/en/rest/repos/contents?apiVersion=2022-11-28 does not 
return the content itself, it returns JSON where you can retrieve download URL 
(with signed download_url that is time-limited) to download the actual content.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] Use GitHub API to download constraint files and avoid rate limiting [airflow]

2025-05-17 Thread via GitHub 


gopidesupavan commented on PR #50716:
URL: https://github.com/apache/airflow/pull/50716#issuecomment-2888180989

   I think the response we get it from github api is base64 , we need to decode 
that.?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org