Jarek, I don't get what you are saying. This is a provider release vote. There is no earlier discussion. I don't know where you expected me to raise -1 prior to the vote? >From my point **at the time I casted the vote** there was no indication of the reasoning. I'm sorry but both you and Radu added a lot of information afterward. I am not a mind reader.
> So maybe the whole discussion, especially the "-1," was unnecessary? Maybe we should just - I do not know - talk about it? -1 vote is a tool. It's a tool to say - stop. I am really against it which is what I felt needed. I don't understand why you are focusing on this. It's OK to disagree with my vote choice but trying to say that I should have avoided casting a vote on a vote thread is a very odd position. > We could probably discuss it in the PR, but since those voting threads are records of the foundation - I think we should explain it here in detail. This way, board members reviewing our voting process will know we treat it seriously. They actually review our VOTE discussions in case you weren't aware :D) and sometimes ask questions about why something happened in a specific VOTE thread. VOTE is a legal act of foundation so we should treat it very seriously - and we should be very precise on why and how we vote here. I don't get this. I provided reasoning which you can agree or not but you can not say there was no reasoning. I am limited on time so I will respond to the other questions/comments later. On Thu, Apr 23, 2026 at 3:45 PM Jarek Potiuk <[email protected]> wrote: > Elad, I would really love to solve this in a way that avoids those kinds of > discussions and unnecessary "-1" in the future. So let's try to "solve" it > for the future. > And I am actually super-calm - no stress at all :) . > > I am actually glad you raised it, because it shows some deficiencies in our > process. I was just hugely surprised to see "-1" rather than discussion. > It's pretty strange to discuss this in the VOTE thread, but ... here we are > and the reason is your "-1" on those providers, Elad - nothing else. > > We could probably discuss it in the PR, but since those voting threads are > records of the foundation - I think we should explain it here in detail. > This way, board members reviewing our voting process will know we treat it > seriously. They actually review our VOTE discussions in case you weren't > aware :D) and sometimes ask questions about why something happened in a > specific VOTE thread. VOTE is a legal act of foundation so we should treat > it very seriously - and we should be very precise on why and how we vote > here. > > Radu: > > > I mentioned this in the PR, that maybe it's OK to modify the CHANGELOG to > say that paramiko was bumped because cryptography was bumped - and that > happened because of security reasons? > > Na ..... that was not really it. I looked it up for details: > > We had **many** issues trying to add Vespa. and it's been - to be honest - > one of the most difficult "cross-provider" interaction we had for a while. > We had similar cases where we needed to bump botocore in order to > avoid whole CI failing, we had cases where we had to bump 5 or 7 provider > min-dependnecies, to avoid resolution going for hours - but Vespa was > special one indeed. > > Here are the three errors we had: > > * We detected unrelated cargo failures, which were really caused by > "cargo + uv + installing several rust providers in the same > installation"—this caused changes in our Docker container (we added rustc > in the CI image) > * Cryptography bump was indeed needed > * Then we had this one: This is from the track of PRs, and the root > cause was this failure: > > https://github.com/apache/airflow/actions/runs/23802024315/job/69372913309?pr=63988#step:8:2556 > > > > ```__________________________________________________________________________ > TestSSHHook.test_tunnel_with_private_key_passphrase > ___________________________________________________________________________ > providers/ssh/tests/unit/ssh/hooks/test_ssh.py:436: in > test_tunnel_with_private_key_passphrase > hook = SSHHook( > providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:183: in __init__ > self.pkey = self._pkey_from_private_key(private_key, > passphrase=private_key_passphrase) > providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:435: in > _pkey_from_private_key > raise AirflowException( > E airflow.sdk.exceptions.AirflowException: Private key provided cannot be > read by paramiko.Ensure key provided is valid for one of the followingkey > formats: RSA, DSS, ECDSA, or Ed25519 > ___________________________________________________________________ > TestSSHHook.test_ssh_connection_with_private_key_passphrase_extra > ____________________________________________________________________ > providers/ssh/tests/unit/ssh/hooks/test_ssh.py:548: in > test_ssh_connection_with_private_key_passphrase_extra > hook = SSHHook( > providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:183: in __init__ > self.pkey = self._pkey_from_private_key(private_key, > passphrase=private_key_passphrase) > providers/ssh/src/airflow/providers/ssh/hooks/ssh.py:435: in > _pkey_from_private_key > raise AirflowException( > E airflow.sdk.exceptions.AirflowException: Private key provided cannot be > read by paramiko.Ensure key provided is valid for one of the followingkey > formats: RSA, DSS, ECDSA, or Ed25519``` > ``` > > The root cause was that after adding vespa, when `uv` resolved > lower-possible direct dependencies for ssh - because of the build > heuristics—it also decided to lower-down the paramiko library. And this is > difficult to assess - because we do not know `uv` heuristics and it's > NP-complete problem in general, so `uv` does a number of shortcuts to > figure out the right set of dependencies in as short time as possible, > > And it turned out that when you downgrade paramiko from 3.5.1 to 3.4.0 - > which was the lowest possible version back then: > > ``` > - paramiko==3.5.1 > + paramiko==3.4.0 > ``` > > It triggers unrelated issues with supported key formats. It wasn't caused > by "vespa" per-se - it was caused by the interaction between the SSH > provider, the SFTP provider, Vespa, and uv trying to find a good dependency > resolution—apparently, just the presence of vespa and its dependencies in > pyproject.toml, causes `uv` to choose this downgrade. > > And since paramiko 3.5.When version 1 was released back then, we figured > the best way to fix our CI and potentially protect our users (who - by > various combinations of installation steps could also trigger the same > scenario) - we decided to bump paramiko to 3.5.1. > > This was the "thought train". > > I don't see any better solution, or one that could take less time to figure > out. > > *Question Elad*: I understand you don't have a problem with the "solution," > I described. but with its description of it and process it arrived - can > you please confirm that ? Because I think discussion will be different if > you have the problem with solution itsefl (but I do not know curreently how > else it could be solved).. > > Now - the question of the "changelog": > > In this case, adding Vespa to our pyproject is the fact our pyproject.toml > triggered it. This is why we bumped it. This is the root cause. There is no > other. There is no direct interaction between "vespa" and "ssh" - the root > cause is that we have a workspace where we want to install all providers > togeether, and due to heuristics of how `uv` works, the issue has been > triggered. However - it was there before. If someone installed ssh provider > with paramiko==3.5.1 and then downgraded to "3.4.0" - The issue was > triggered. I reproduced it. So, a better description would be: > > "Bumping to minimum version of 3.5.1 because when the user installs the > provider with Paramiko 3.5.1 and then downgrades to 3.4.0, it triggers > "uknown key" bug" > > *Elad* - would you be ok if we change it like that in a follow up release > or do you have a better wording in mind? Maybe you can create a PR for > that? > > *Very near future: * > > Also, following what Shahar wrote, I'm about to add "Provider doc > preparation skill." That should completely replace (though I will leave it > there for now) our "breeze release management prepare-providers-release" > command. This is after learning from the "breeze pr auto-triage" > conversion—I literally discussed it with Shahar yesterday on Slack > And one of the things I planned there was to instruct the agent to search > for the root cause of the issue and present it in "user facing way"—letting > the LLM figure out the best way to explain the change to the user. And - to > be honest - I even planned to re-run on ALL our changelogs to improve them. > > So maybe the whole discussion, especially the "-1," was unnecessary? Maybe > we should just - I do not know - talk about it? > > J. > > > > > > On Thu, Apr 23, 2026 at 2:37 PM Shahar Epstein <[email protected]> wrote: > > > Just to put things into context from my perspective as a release manager: > > 1. I was aware of the changes to SFTP and SSH resulting from Vespa's PR > > while working on the release. I thought that it was an intended change - > I > > didn't go deeper into whether there was a discussion, as I assumed that > it > > had been agreed on during approval. > > 2. I asked the LLM to modify the titles in the CHANGELOG if they are > > unclear, and indeed it detected this issue and modified the entries on > its > > own. > > 3. If we had added the upper bound "<4.0.0" as part of this PR, it would > > have indeed been an issue - because users who have already upgraded > > paramiko, might not be able to roll back. However, the upper bound had > > already been set to "<4.0.0" *before* Vespa's PR, which just increased > the > > lower bound - and I agree here with Jarek that we do that all the time > > < > > > https://github.com/apache/airflow/commits/main/?author=dependabot%5Bbot%5D > > > > > (even > > without discussing it). > > Nonetheless: > > Specifically here I would have expected this matter to be discussed in > the > > PR as these changes seem unrelated, or even making them in separate PRs - > > it was a bit confusing for me as a release manager to figure out why > these > > changes were needed. > > > > I'll pay more attention to this matter in future releases and let you > know > > beforehand if I encounter similar cases. > > No changes to my vote. > > > > > > Shahar > > > > On Thu, Apr 23, 2026, 14:15 Jarek Potiuk <[email protected]> wrote: > > > > > Hey Elad, > > > > > > But it's - IMHO - cler explained in the changelog: > > > > > > > > > > > > https://airflow.staged.apache.org/docs/apache-airflow-providers-ssh/5.0.1/changelog.html > > > > > > > > > https://airflow.staged.apache.org/docs/apache-airflow-providers-sftp/stable/changelog.html > > > > > > SSH: > > > > > > 5.0.1¶ > > > Release Date: 2026-04-22 > > > > > > Misc¶ > > > Bump paramiko lower bound to >=3.5.1 due to adding Vespa provider > > (#63988) > > > > > > SFTP" > > > > > > Changelog¶ > > > 5.7.4¶ > > > Release Date: 2026-04-22 > > > > > > Misc¶ > > > Bump paramiko lower bound to >=3.5.1 due to adding Vespa provider > > (#63988) > > > > > > Why do you think it needs more explanation? We usually do not do more > > > explanation when we bump minimum version - and here the explanation is > > > quite clear ? > > > > > > J. > > > > > > > > > > > > On Thu, Apr 23, 2026 at 1:02 PM Elad Kalif <[email protected]> wrote: > > > > > > > -1 (binding) for ssh and sftp providers. > > > > the changes in these providers are bumping paramiko min version which > > > came > > > > from a PR that adds vespa provider > > > > https://github.com/apache/airflow/pull/63988#discussion_r3130238616 > > > > There is no indication into why version was bumped and seems like we > > > missed > > > > it during review. > > > > Even if the version needs to be bumped that is not normally how we do > > it > > > > and generally lets leave this to when > > > > https://github.com/apache/airflow/pull/64898 is resolved. > > > > > > > > On Thu, Apr 23, 2026 at 1:01 PM Wei Lee <[email protected]> wrote: > > > > > > > > > +1 (binding), checked: > > > > > > > > > > - SVN > > > > > - in Docker installation > > > > > - Reproducible package builds > > > > > - Licence > > > > > - Signature > > > > > - Checksums > > > > > > > > > > Best regards, > > > > > Wei Lee > > > > > > > > > > Shahar Epstein <[email protected]> 於 2026年4月23日週四 上午7:55寫道： > > > > > > > > > > > Hey all, > > > > > > > > > > > > I have just cut the new wave Airflow Providers packages with > > release > > > > > > preparation date 2026-04-21. This email is calling a vote on the > > > > release, > > > > > > which will last for 72 hours - which means that it will end on > > > > 2026-04-25 > > > > > > 23:54 UTC and until 3 binding +1 votes have been received. > > > > > > > > > > > > > > > > > > Consider this my (binding) +1. > > > > > > Please note that we have an RC for a new provider, Vespa - I'd > like > > > to > > > > > ask > > > > > > the stewards to test it well before shipping its first version > > > > (v0.1.0). > > > > > > > > > > > > > > > > > > Airflow Providers are available at: > > > > > > > > https://dist.apache.org/repos/dist/dev/airflow/providers/2026-04-21 > > > > > > > > > > > > *apache-airflow-providers-2026-04-21-source.tar.gz* is the full > > > source > > > > > > tarball of airflow repo - snapshot taken at the moment of > > provider's > > > > > > release. > > > > > > > > > > > > *apache-airflow-providers-<PROVIDER>-*.tar.gz* are the > convenience > > > > python > > > > > > "sdist" distributions that we publish in PyPI > > > > > > > > > > > > *apache_airflow_providers_<PROVIDER>-*.whl are the convenience > > Python > > > > > > "wheel" distributions that we publish in PyPI. > > > > > > > > > > > > The test procedure for PMC members is described in > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-pmc-members > > > > > > > > > > > > The test procedure for and Contributors who would like to test > this > > > RC > > > > is > > > > > > described in: > > > > > > > > > > > > > > > > > > > > > > > > > > > https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDERS.md#verify-the-release-candidate-by-contributors > > > > > > > > > > > > > > > > > > Public keys are available at: > > > > > > https://dist.apache.org/repos/dist/release/airflow/KEYS > > > > > > > > > > > > Please vote accordingly: > > > > > > > > > > > > [ ] +1 approve > > > > > > [ ] +0 no opinion > > > > > > [ ] -1 disapprove with the reason > > > > > > > > > > > > Only votes from PMC members are binding, but members of the > > community > > > > are > > > > > > encouraged to test the release and vote with "(non-binding)". > > > > > > > > > > > > Please note that the version number excludes the 'rcX' string. > > > > > > This will allow us to rename the artifact without modifying > > > > > > the artifact checksums when we actually release it. > > > > > > > > > > > > The status of testing the providers by the community is kept > here: > > > > > > https://github.com/apache/airflow/issues/65702 > > > > > > > > > > > > The issue is also the easiest way to see important PRs included > in > > > the > > > > RC > > > > > > candidates. > > > > > > Detailed changelog for the providers will be published in the > > > > > documentation > > > > > > after the > > > > > > RC candidates are released. > > > > > > > > > > > > You can find the RC packages in PyPI following these links: > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-amazon/9.26.0rc1/ > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-apache-kafka/1.13.3rc1/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-celery/3.19.0rc1/ > > > > > > > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/10.16.1rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-common-ai/0.1.1rc1/ > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-common-sql/1.35.0rc1/ > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-databricks/7.13.0rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-edge3/3.5.0rc1/ > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-elasticsearch/6.5.3rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-fab/3.6.2rc1/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-google/21.2.0rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-hashicorp/4.6.0rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-jdbc/5.4.4rc1/ > > > > > > > > > > > > > > > > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-microsoft-azure/13.1.2rc1/ > > > > > > > > > > > > https://pypi.org/project/apache-airflow-providers-openlineage/2.15.0rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-opensearch/1.9.1rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-papermill/3.13.0rc1/ > > > > > > > > https://pypi.org/project/apache-airflow-providers-sendgrid/4.2.3rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-sftp/5.7.4rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-smtp/3.0.0rc1/ > > > > > > > > > https://pypi.org/project/apache-airflow-providers-snowflake/6.12.2rc1/ > > > > > > https://pypi.org/project/apache-airflow-providers-ssh/5.0.1rc1/ > > > > > > > https://pypi.org/project/apache-airflow-providers-vespa/0.1.0rc1/ > > > > > > > > > > > > Cheers, > > > > > > Shahar Epstein > > > > > > > > > > > > > > > > > > > > >
