gregwhorley opened a new issue, #36118: URL: https://github.com/apache/airflow/issues/36118
### Official Helm Chart version 1.11.0 (latest released) ### Apache Airflow version 2.7.3 ### Kubernetes Version 1.24 ### Helm Chart configuration ``` dags: gitSync: enabled: true repo: https://github.com/myOrg/myRepo.git branch: feature/enable-gitsync rev: HEAD depth: 1 # the number of consecutive failures allowed before aborting maxFailures: 0 # subpath within the repo where dags are located # should be "" if dags are at repo root subPath: "dataflow/airflow/dags" credentialsSecret: git-sync-key ``` ### Docker Image customizations ``` ARG AIRFLOW_VERSION ARG PYTHON_VERSION FROM --platform=linux/amd64 apache/airflow:slim-${AIRFLOW_VERSION}-${PYTHON_VERSION} ARG PIP_INDEX_URL="https://pypi.org/simple" ARG SPARK_VERSION USER root RUN apt-get update \ && ACCEPT_EULA=Y apt-get upgrade -y \ && apt-get install -y \ pkgconf \ openjdk-17-jre-headless \ procps \ libpq-dev \ default-libmysqlclient-dev \ gcc \ && export PIP_INDEX_URL=${PIP_INDEX_URL} \ && su airflow -c 'pip install psycopg2 mysqlclient' \ && apt-get autoremove gcc --purge -y \ && export SUDO_FORCE_REMOVE=yes \ && apt-get -y remove python3.9 sqlite3 ldap-utils openssh-client curl rsync sudo \ && apt -y autoremove \ && rm -rf /var/cache/apt/* \ && rm -rf /var/lib/apt/lists/* # NB: We switch back to our airflow user and install the remaining pip dependencies USER airflow COPY requirements.txt requirements.txt RUN export PIP_INDEX_URL=${PIP_INDEX_URL} RUN pip install --upgrade pip RUN pip install --user --no-cache pyspark==${SPARK_VERSION} apache-airflow[aws,postgres,slack,docker,mongo,mysql,spark,statsd]~=${AIRFLOW_VERSION} RUN pip install --user --no-cache -r requirements.txt RUN echo "JAVA_HOME=$(dirname $(dirname $(update-alternatives --list java)))" >> ${HOME}/.bashrc \ && echo "SPARK_HOME=/home/airflow/.local/lib/${PYTHON_VERSION}/site-packages/pyspark" >> ${HOME}/.bashrc ``` ``` # requirements.txt apache-airflow-providers-cncf-kubernetes~=7.5.0 boto3 docker kubernetes yoyo-migrations~=8.2.0 flask-appbuilder sqlalchemy authlib ``` ### What happened I am trying to get the gitsync feature enabled and working for my org's Airflow instance running in Kubernetes. I tried configuring it for ssh at first, following the steps in https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#mounting-dags-from-a-private-github-repo-using-git-sync-sidecar but I kept getting invalid key errors: ``` Cloning into '/git'... Load key "/etc/git-secret/ssh": invalid format my-org-u...@github.com: Permission denied (publickey). fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. ``` So I decided to switch to HTTPS for the time being just so I could prove it works. The airflow helm chart's values file shows the overrides to enable this: https://github.com/apache/airflow/blob/a315165719f950c5d13493b6138c6d6eba7929ae/chart/values.yaml#L2421-L2443 When I switch to this auth method, a new error occurs: ``` Cloning into '/git'... remote: Support for password authentication was removed on August 13, 2021. remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication. fatal: Authentication failed for 'https://github.com/my-org/my-repo.git/ ``` Not sure how this should be addressed other than removing the https auth option from the helm chart values and docs. ### What you think should happen instead Make the https auth option non-existent since Github does not support it anymore. ### How to reproduce - Spin up a k8s cluster - Create a values file with the following overrides ``` dags: gitSync: enabled: true repo: https://github.com/aRealRepo.git branch: feature/enable-gitsync rev: HEAD credentialsSecret: git-sync-key ``` - Create and apply a Kubernetes secret resource with the following data ``` apiVersion: v1 kind: Secret metadata: name: git-sync-key data: GIT_SYNC_USERNAME: <base64_encoded_git_username> GIT_SYNC_PASSWORD: <base64_encoded_git_password> ``` - Install airflow helm chart and include this values override file ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org