potiuk commented on PR #35473: URL: https://github.com/apache/airflow/pull/35473#issuecomment-1796836907
> Gotcha @potiuk (and thats summary of all of the above :-P ) I'm just confused that we are end-2-end testing this, including the db. It seems not to make sense to me, but then I would need to dive in. Yeah. they are likely a few of those that we do needlessly. I am not sure if dask executor is that used at all... And well.. maybe one day someone will trace and remove those tests. We have accumulated a lot of those I am now merely on a quest to keep them, running, optimizing their executions, tracking flakes etc. But if somoene would like to improve those - or even remove .... Welll. I am fully supportive. Side comment: That's why maybe suspending dask provider might be a good idea anyway - together with it's tests, regardless from results of the experiment. People will still be able to use the released latest provider if they want, we do not loose anything by disabling the code (and it's related tests) and maybe later even removing it. This is what I think we should start doing more and starting with Qubole being removed in the next week or so. The way how we have providers organized now - we could technically suspend and then later remove `ALL` providers that we are unsure of and wait for people to come to us and unsuspend those they use and want to release them. I am actually seriously considering this approach and be more aggressive in those suspensions (and later removals). The providers will remain relased in PyPI and continue working, if there is no maintenance work, we should likely ... stop maintaining those and stop releasing them (and remove the code from our codebase eventually). This is what we've been discussing before I believe, and it's now entirely possible, with qubole it will be tested and we could generally proceed with it much more aggressively I think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
