Re: Donating code to add Common Workflow Language import to Airflow

2019-11-16 Thread Soma S Dhavala
On Sat, Nov 16, 2019 at 2:25 PM Jarek Potiuk wrote: > > > > 1) The more dags I have in a dags folder, the longer time it takes to > > parse them all. Taking into account that in my case I have also to parse > > CWL files, it takes even more time for such a simple operation. So I was > > wondering

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-16 Thread Misha Kotliar
Thanks a lot Jarek and Brian for your answers. I will learn from the links you provided and when I have other questions I will post them in the correspondent discussion threads. As for the main topic of this thread, I agree that currently, it might be more reasonable to keep it in as a separate

Re: [VOTE] 2nd attempt for AIP-21 update for Airflow 1.10.* backportability

2019-11-16 Thread Jarek Potiuk
Subtasks in https://issues.apache.org/jira/browse/AIRFLOW-4733 created and the table of operators is now part of the AIP-21

Re: [VOTE] 2nd attempt for AIP-21 update for Airflow 1.10.* backportability

2019-11-16 Thread Jarek Potiuk
So we have now 5 (+1) binding votes: Jarek Potiuk, Kaxil Naik, Felix Uelendall, Kamil BreguĊ‚a, Kevin Yang, One (+1) non-binding vote: Tomasz Urbaszek. The vote is "Passed" Thank you! It was a long discussion but I think we have a reasonable approach at the end :). I will update the AIP-21 appro

Re: [DISCUSS] Back to (some) dependency pinning

2019-11-16 Thread Jarek Potiuk
After some discussions and some tests by myself and Astronomer docker, I think I have finally a complete proposal for good, consistent solution for our dependency management. It uses well-known standards, does not introduce any new tools (like poetry) and I think it serves all the use cases we hav

Re: [Discuss] Airflow Summits 2020

2019-11-16 Thread Jarek Potiuk
Great Aizhamal, I love the proposal and will add my comments soon ! When do you think you will be submitting it ? I am talking to Polidea's Board and I would like to get them on board (pun intended ;)). And I need to tell them when they need to decide if/how Polidea as organisation is involved on

Re: Closing JIRA Issue for Merged PRs

2019-11-16 Thread Jarek Potiuk
And yes. I was one of the culprits - I saw :(. Sorry about that Kaxil. Just hope we can streamline this :). On Sat, Nov 16, 2019 at 10:12 AM Jarek Potiuk wrote: > Heartily agree with it ! > > I try always to close the PRs but sometimes I got distracted and forget to > resolve an issue - it happ

Re: Closing JIRA Issue for Merged PRs

2019-11-16 Thread Jarek Potiuk
Heartily agree with it ! I try always to close the PRs but sometimes I got distracted and forget to resolve an issue - it happend several times that I recalled it few hours later that I have forgotten to resolve it. I hope it happens rarely - I'd love to know if I was one of the culprits here :).

Re: Donating code to add Common Workflow Language import to Airflow

2019-11-16 Thread Jarek Potiuk
> > 1) The more dags I have in a dags folder, the longer time it takes to > parse them all. Taking into account that in my case I have also to parse > CWL files, it takes even more time for such a simple operation. So I was > wondering is there any common solution to approach this issue. Also, I wa