Re: [PR] Build python from source for CI [airflow]
potiuk commented on PR #52265: URL: https://github.com/apache/airflow/pull/52265#issuecomment-3038498415 Hmmm.. We need to revert .. I found some issues with other versions of python :( -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Build python from source for CI [airflow]
github-actions[bot] commented on PR #52265: URL: https://github.com/apache/airflow/pull/52265#issuecomment-3038409436 ### Backport failed to create: v3-0-test. View the failure log Run details Status Branch Result ❌ v3-0-test https://github.com/apache/airflow/commit/d06a27ecfeb0e06dff701cb4c204ead3e64899b9";> You can attempt to backport this manually by running: ```bash cherry_picker d06a27e v3-0-test ``` This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking the files that need manual conflict resolution. After you have resolved the conflicts, you can continue the backport process by running: ```bash cherry_picker --continue ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on PR #52265: URL: https://github.com/apache/airflow/pull/52265#issuecomment-3038409302 Let me merge it now - we can always iterate on details ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk merged PR #52265: URL: https://github.com/apache/airflow/pull/52265 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186929934
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
It's just a bit more explicit and canonical
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186929015
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
Of course all those are NITs... It can go without it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186914612
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
You know .. there is this saying..
| You have problem, introduce regexp, now you have two problems.
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
You know .. there is this saying..
> You have problem, introduce regexp, now you have two problems.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186913489
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
(we still have to remove v possibly)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186909682
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
Also. We can do it better without earlier filtering:
```python
def is_valid_final_version(v):
try:
version = Version(v)
return version == version.base_version
except InvalidVersion:
return False
valid_versions = list(filter(is_valid_final_version, versions))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186909682
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
Also. We can do it better without earlier filtering:
```python
def is_valid_final_version(v):
try:
version = Version(v)
return version == version.base_version
except InvalidVersion:
return False
valid_final_versions = list(filter(is_valid_final_version, versions))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186909682
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
Also. We can do it better without earlier filtering:
```
def is_valid_final_version(v):
try:
version = Version(v)
return version == version.base_version
except InvalidVersion:
return False
valid_versions = list(filter(is_valid_final_version, versions))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186906211
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
> They're also using a regex internally, just a more complex one 🙃
He he. But it's THEM to manage it :). and they're canonical. Also it's
a bit far from the original filter to that place. How about:
``python
def is_valid_version(v):
try:
Version(v)
return True
except InvalidVersion:
return False
valid_versions = list(filter(is_valid_version, versions))
```
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
> They're also using a regex internally, just a more complex one 🙃
He he. But it's THEM to manage it :). and they're canonical. Also it's
a bit far from the original filter to that place. How about:
```python
def is_valid_version(v):
try:
Version(v)
return True
except InvalidVersion:
return False
valid_versions = list(filter(is_valid_version, versions))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on PR #52265: URL: https://github.com/apache/airflow/pull/52265#issuecomment-3038344503 > Is the redis test failure something we're seeing in other prs as well? I don't think I've touched anything to break redis 🤔 Yeah - we have a flaky redis tests that we need to address. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on PR #52265: URL: https://github.com/apache/airflow/pull/52265#issuecomment-3038323902 Is the redis test failure something we're seeing in other prs as well? I don't think I've touched anything to break redis 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186895370
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
works now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186894906
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
Yeah, I think that was it, I cleaned up my build cache and then it did
rebuild it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186890342
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
*must work* if you pass the value from ARG .. The thing is it will **not**
invalidate layer when you already built it before with that ARG and set that
ENV VAR before -- it will keep the built layer in cache and see that it already
built it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186887900
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
https://github.com/user-attachments/assets/79976bff-4fb8-45bb-886f-00408b8de15d";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186887477
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
https://github.com/user-attachments/assets/383c3fdb-b715-495b-b142-106e1ee06f87";
/>
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
https://github.com/user-attachments/assets/383c3fdb-b715-495b-b142-106e1ee06f87";
/>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186886474
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
I think I did try that, it still was not invalidating the cache 🤔 I can
recheck once more though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186885199
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
This is what we do in other places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
potiuk commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186884089
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
Define them as ENV before that will invalidate the layers before.
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
Define them as ENV before that will invalidate the layers after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186878941
##
Dockerfile.ci:
##
@@ -1279,8 +1308,14 @@ ENV DEV_APT_COMMAND=${DEV_APT_COMMAND} \
ADDITIONAL_DEV_APT_DEPS=${ADDITIONAL_DEV_APT_DEPS} \
ADDITIONAL_DEV_APT_COMMAND=${ADDITIONAL_DEV_APT_COMMAND}
-COPY --from=scripts install_os_dependencies.sh /scripts/docker/
-RUN bash /scripts/docker/install_os_dependencies.sh dev
+ARG AIRFLOW_PYTHON_VERSION=v3.10.18
+ARG GOLANG_MAJOR_MINOR_VERSION=1.24.4
+
+COPY --from=scripts install_os_dependencies_ci.sh /scripts/docker/
+
+RUN AIRFLOW_PYTHON_VERSION=$AIRFLOW_PYTHON_VERSION \
+GOLANG_MAJOR_MINOR_VERSION=${GOLANG_MAJOR_MINOR_VERSION} \
+bash /scripts/docker/install_os_dependencies_ci.sh ci
Review Comment:
One minor note here, I had to pass the args explicitly into the run command
because without them the layers weren't getting invalidated since docker tries
to be smart that way to reduce unnecessary builds.
cc: @potiuk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186871163
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
they don't seem to have a separate is valid method, I chose to do it with
regex myself because I wanted to exclude alpha and beta versions from our list
which would've been matched by their regex, so I think given our regex is a
subset of theirs it should be fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Build python from source for CI [airflow]
aritra24 commented on code in PR #52265:
URL: https://github.com/apache/airflow/pull/52265#discussion_r2186864139
##
scripts/ci/pre_commit/update_installers_and_pre_commit.py:
##
@@ -65,6 +66,31 @@ def get_latest_pypi_version(package_name: str) -> str:
return latest_version
+def get_latest_python_version(python_major_minor: str) -> str | None:
+latest_version = None
+version_match = re.compile(rf"^v{python_major_minor}.*\.\d+$")
+for i in range(5):
+response = requests.get(
+
f"https://api.github.com/repos/python/cpython/tags?per_page=100&page={i + 1}",
+headers={"User-Agent": "Python requests"},
+)
+response.raise_for_status() # Ensure we got a successful response
+data = response.json()
+versions = [str(tag["name"]) for tag in data if
version_match.match(tag.get("name", ""))]
+if versions:
+latest_version = sorted(versions, key=Version, reverse=True)[0]
Review Comment:
They're also using a regex internally, just a more complex one 🙃
```
v?
(?:
(?:(?P[0-9]+)!)? # epoch
(?P[0-9]+(?:\.[0-9]+)*) # release segment
(?P # pre-release
[-_\.]?
(?Palpha|a|beta|b|preview|pre|c|rc)
[-_\.]?
(?P[0-9]+)?
)?
(?P # post release
(?:-(?P[0-9]+))
|
(?:
[-_\.]?
(?Ppost|rev|r)
[-_\.]?
(?P[0-9]+)?
)
)?
(?P # dev release
[-_\.]?
(?Pdev)
[-_\.]?
(?P[0-9]+)?
)?
)
(?:\+(?P[a-z0-9]+(?:[-_\.][a-z0-9]+)*))? # local version
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
