Re: Rewriting Breeze in Python ?

2020-11-11 Thread Daniel Imberman
I am all for rewriting breeze, but I think waiting until after 2.0 makes the most sense. Python could work, but let’s be intentional about the decision before we choose. via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.51&pv=10.15.7&source=email_footer_2] On Wed, Nov 11, 2020 a

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Deng Xiaodong
I agree with Kaxil’s point (or even a bit later, say when 2.0 gets relatively more “stable”). My aspect is more about to concentrate development/community focus. XD On Thu, Nov 12, 2020 at 00:05 Kaxil Naik wrote: > I think we should wait until 2.0 is out before discussing or even > gathering

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Kaxil Naik
I think we should wait until 2.0 is out before discussing or even gathering feedback. As I am sure any feedback will trigger a discussion. On Wed, Nov 11, 2020 at 5:52 PM Jarek Potiuk wrote: > Andrew, > > Thanks for chiming in - just to answer your questions and clarify the > scope of the discus

Re: Stack Overflow as valuable source of information about Airflow ?

2020-11-11 Thread Tomasz Urbaszek
My only concern about connecting SO to Slack is that it may impact the history of slack (free tier, here we go again). However, I think it's worth it if we will promote SO for Q&A and Discussions for discussion (and slack for chat). Tomek On Wed, Nov 11, 2020 at 11:40 PM Leah Cole wrote: > Jare

Re: Stack Overflow as valuable source of information about Airflow ?

2020-11-11 Thread Leah Cole
Jarek I'm glad you suggested connecting it to a slack channel - I think that automating the integration into we currently have will be crucial to gaining adoption. My only nit would be to spell out #stackoverflow-questions in the channel name just so that it's immediately obvious what it is at firs

Re: Stack Overflow as valuable source of information about Airflow ?

2020-11-11 Thread Ry Walker
i like all those ideas, Jarek On Wed, Nov 11, 2020 at 2:39 PM Jarek Potiuk wrote: > I read the docs (thanks Martijn!). Looks like SO would be great if we use > more of it. > > We are promoting it very lightly, though. At our > https://airflow.apache.org/community/ page we have "Stack Overflow" l

Re: [NEWSLETTER] Apache Airflow newsletter #01

2020-11-11 Thread Aizhamal Nurmamat kyzy
This is so nice! Thanks Karolina for driving this! On Wed, Nov 11, 2020 at 7:15 AM Paola Peraza Calderon wrote: > This is great! Awesome work putting this together, Karolina. > > Paola > > On Wed, Nov 11, 2020 at 3:30 AM Shivam Kumar > wrote: > >> Thanks karolina 😊 >> >> >> On Wed, 11 Nov 2020,

Re: Stack Overflow as valuable source of information about Airflow ?

2020-11-11 Thread Jarek Potiuk
I read the docs (thanks Martijn!). Looks like SO would be great if we use more of it. We are promoting it very lightly, though. At our https://airflow.apache.org/community/ page we have "Stack Overflow" lin kas one of the options (which is good, according to those SO guidelines). But I think we ha

Re: Measuring community state

2020-11-11 Thread Jarek Potiuk
I think this is a very, very good time to do this. I am all for it. For sure there are those questions I would like to add: Airflow deployment: * which version of Airflow you use in production * which version of Airflow you use in staging/dev * what is the database you are using as the metadata

Re: Removing Code QL security analysis CI jobs out of PRs

2020-11-11 Thread Jarek Potiuk
Calling for lazy consensus here as well. Again if there are no objections till the end of the weekend, I will leave CodeQL only in the master. I believe checking v1-10 is indeed not something we must do now when we are switching our focus to 2.0. Though if someone has some doubts here, please raise

Re: Proposal: Change default to LocalExecutor and (eventually) remove SequentialExecutor

2020-11-11 Thread Deng Xiaodong
Sounds good to me to enforce LocalExecutor (and ok to fail hard to me), when (and ONLY when) it’s confirmed user is using MySQL/Postgres. XD On Wed, Nov 11, 2020 at 19:19 Jarek Potiuk wrote: > How about this? Are we doing it for 2.0 :)? Changing default /failing hard > when Postgres/MySQL is u

Re: Proposal: Change default to LocalExecutor and (eventually) remove SequentialExecutor

2020-11-11 Thread Jarek Potiuk
How about this? Are we doing it for 2.0 :)? Changing default /failing hard when Postgres/MySQL is used? WDYT ? J. On Sat, Oct 24, 2020 at 10:01 AM Jarek Potiuk wrote: > One more proposal on that. Why don't we fail hard Airflow in > Postgres/MySQL when Sequential Executor is used? > > I think

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Jarek Potiuk
Andrew, Thanks for chiming in - just to answer your questions and clarify the scope of the discussion: Breeze is for developing Airflow itself, it's purpose is not to develop and run DAGs. It was never intended to be used by the "users" of Airflow or DAG development or testing the DAGs. And while

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Jarek Potiuk
Tomek: I started the discussion here, so just everyone is aware of it even if they are not watching GH issues. I now created the GH Issue https://github.com/apache/airflow/issues/12282 so that I can gather together people with some interest and I think it's best to continue the discussion there.

Re: [NEWSLETTER] Apache Airflow newsletter #01

2020-11-11 Thread Paola Peraza Calderon
This is great! Awesome work putting this together, Karolina. Paola On Wed, Nov 11, 2020 at 3:30 AM Shivam Kumar wrote: > Thanks karolina 😊 > > > On Wed, 11 Nov 2020, 12:10 pm Ace Haidrey, > wrote: > >> This is really great. Thanks team for putting this together. >> >> On Nov 10, 2020, at 7:36

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Andrew Harmon
I would agree as an end user, I’m not really sure what Breeze does. Is it for CI or is it a way to quickly spin up a containerized env for local development. I do think it would be great to have something similar to Puckel that uses official airflow images. Very easy to quickly get started with

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Tomasz Urbaszek
+1 for using python. > I would also say: make breeze do less. Right now it is three major things: > * A local development environment > * CI runner > * It's recently grown the ability to run airflow for developing dags. My first thought was similar - breeze does too much now. However, I think the

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Jarek Potiuk
I also created issue for it: https://github.com/apache/airflow/issues/12282 Anyone interested in taking part - please comment there! On Wed, Nov 11, 2020 at 11:59 AM Jarek Potiuk wrote: > You screamed (among many others) and I listened :). And I think the time > is now to act. > > I believe the

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Jarek Potiuk
You screamed (among many others) and I listened :). And I think the time is now to act. I believe the scope of "Breeze 2" should be part of the design discussion, where we will hear other's opinions (especially the first time or fresh contributors). For now, my vision is quite a bit different tha

Re: Rewriting Breeze in Python ?

2020-11-11 Thread Ash Berlin-Taylor
Omg yes. I have been screaming out for this for months. $ find scripts -name '*.sh' | xargs egrep -v '^#' | wc -l 6911 That's entirely too much bash for my liking by about an order of magnitude ;) I would also say: make breeze do less. Right now it is three major things: * A local development env

Rewriting Breeze in Python ?

2020-11-11 Thread Jarek Potiuk
Hello Everyone, TL; DR; I was thinking for quite a while on this and I think this is the right time to raise that subject. It's been asked several times, why Breeze is not written in something else than Bash since it is "that big" or some people said "monstrous" :). I think it's the right time to

Re: [NEWSLETTER] Apache Airflow newsletter #01

2020-11-11 Thread Shivam Kumar
Thanks karolina 😊 On Wed, 11 Nov 2020, 12:10 pm Ace Haidrey, wrote: > This is really great. Thanks team for putting this together. > > On Nov 10, 2020, at 7:36 AM, Karolina Rosół > wrote: > > Apache Airflow newsletter for anyone too busy to read the devlist. > *1st Apache Airflow newsletter* >