Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Kamil Breguła
Inline comment On Fri, Apr 19, 2019 at 12:23 AM Arthur Wiedmer wrote: > > > > > > Case #5 Are we talking about the class name or the file name? The > > class > > > > > name is fine to me, but we could remove _operator from the file > name. > > > > > > > > > > > > > Case #2 describes the change o

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Arthur Wiedmer
> > You can not put this document on the mailing list because the document > contains advanced formatting. > > I will try to put this document in the wiki in the near future to the > archival purpose. I would like to point out that this is not the final > document and contains the final recommendat

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Ash Berlin-Taylor
> I think that the operator should describe the behavior(verb), hook should > describe a service(noun). These are other parts of speech. > In my opinion, HiveOperator should be named HiveExecuteQuery. Hook will > be > called Hive. Then there will be no name conflicts I'm not sure I like Hive as a

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Jarek Potiuk
Congrats Ash !. Principal Software Engineer Phone: +48660796129 czw., 18 kwi 2019, 22:45 użytkownik Tao Feng napisał: > Congrats Ash and the Astronomer team! > > On Thu, Apr 18, 2019 at 11:37 AM Ash Berlin-Taylor wrote: > > > Hi everyone! > > > > I've hinted at this to a few people in Slack, a

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Kamil Breguła
Comments inline On Thu, Apr 18, 2019 at 10:18 PM Arthur Wiedmer wrote: > Comments inline > > On Thu, Apr 18, 2019, 11:10 Kamil Breguła > wrote: > > > On Thu, Apr 18, 2019 at 7:30 PM Arthur Wiedmer > > > wrote: > > > > > Hi Kamil, > > > > > > Thank you for this doc, it was a lot of good work to

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Tao Feng
Congrats Ash and the Astronomer team! On Thu, Apr 18, 2019 at 11:37 AM Ash Berlin-Taylor wrote: > Hi everyone! > > I've hinted at this to a few people in Slack, and it's now official. I'm > super excited to be able to say that I've joined Astronomer as a full time > employee, and a good portion

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Arthur Wiedmer
Comments inline On Thu, Apr 18, 2019, 11:10 Kamil Breguła wrote: > On Thu, Apr 18, 2019 at 7:30 PM Arthur Wiedmer > wrote: > > > Hi Kamil, > > > > Thank you for this doc, it was a lot of good work to put all of this > > together. Could you open the doc for comment? > > May I recommend that once

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Ben Tallman
Very excited! Especially as we are leveraging Astronomer more every day! Thanks, Ben From: James Meickle Sent: Thursday, April 18, 2019 12:48 PM To: dev@airflow.apache.org Subject: Re: New job: I'm joining Astronomer.io to work on Airflow That's great news! Cong

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread James Meickle
That's great news! Congratulations. On Thu, Apr 18, 2019 at 2:37 PM Ash Berlin-Taylor wrote: > Hi everyone! > > I've hinted at this to a few people in Slack, and it's now official. I'm > super excited to be able to say that I've joined Astronomer as a full time > employee, and a good portion of

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Aizhamal Nurmamat kyzy
Congratulations, Ash! Happy to know you will keep working on Airflow as part of your new job! Good luck in a new place, hope you will enjoy it a lot:) On Thu, Apr 18, 2019 at 11:43 AM Bas Harenslak < basharens...@godatadriven.com> wrote: > Congratulations with the new job! > > Seems you’re alread

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Kevin Yang
Wow! Oooh yeah, win win win for you, Astronomer and us the community! :D Congratulations! Shame on me not knowing the podcast before, now I know and will follow ;) Looking forward to discuss the roadmap with you esp. about DAG serialization effort. Cheers, Kevin Y On Thu, Apr 18, 2019 at 11:43

Re: New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Bas Harenslak
Congratulations with the new job! Seems you’re already spending lots of time outside working hours on Airflow so looking forward to the future! Bas On 18 Apr 2019, at 20:37, Ash Berlin-Taylor mailto:a...@apache.org>> wrote: Hi everyone! I've hinted at this to a few people in Slack, and it's

New job: I'm joining Astronomer.io to work on Airflow

2019-04-18 Thread Ash Berlin-Taylor
Hi everyone! I've hinted at this to a few people in Slack, and it's now official. I'm super excited to be able to say that I've joined Astronomer as a full time employee, and a good portion of my time will be still be devoted to working on the core Airflow project itself! Hopefully the amount

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Kamil Breguła
On Thu, Apr 18, 2019 at 7:30 PM Arthur Wiedmer wrote: > Hi Kamil, > > Thank you for this doc, it was a lot of good work to put all of this > together. Could you open the doc for comment? > May I recommend that once comments have been added to eventually port the > doc to the wiki and keep trace o

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Arthur Wiedmer
Hi Kamil, Thank you for this doc, it was a lot of good work to put all of this together. Could you open the doc for comment? May I recommend that once comments have been added to eventually port the doc to the wiki and keep trace of the history there as well. Here are my choices Case #1 B or A wi

Re: [2.0 spring cleaning] Deprecate contrib folder?

2019-04-18 Thread Kamil Breguła
The discussion about solving this problem and similar has been started in the thread: https://lists.apache.org/thread.html/c880ef89f8cb4a0240c404f9372615b998c4a4eeca342651927d596c@%3Cdev.airflow.apache.org%3E On Fri, Apr 12, 2019 at 10:03 AM Kaxil Naik wrote: > > Contrib folder was used when it w

Re: schedule_interval question

2019-04-18 Thread Chen Tong
Do not set to datetime.now(). You could set to 2019-04-18 and it will start scheduling at 2019-04-18 2 AM. Chen On Thu, Apr 18, 2019, 08:55 Pawel Bartoszek wrote: > Ash, If I omit start_date it I get the error > Task is missing the start_date parameter > > What should I set it to then? > > On T

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Ash Berlin-Taylor
My preference: Case 1: (contrib vs not) A (move all) or B (move all that are "well" tested). Case 2 irflow.operators.foo_operator.FooOperator could become A: airflow.operators.foo.FooOperator Case 3: airflow.contrib.operators.gcp_bigtable_operator.BigTableOperator could become A: airflow.contr

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Chen Tong
Thanks for collecting all related in google doc. I think rethinking the import path is a good start point for all other discussion including deprecate the contrib, separate pluings into packages. Since we have multiple threads discussing this topic, maybe we could choose one thread as the main plac

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Kamil Breguła
Yes. This document also describes this problem. I wanted to collect all related problems. On Thu, Apr 18, 2019 at 4:13 PM Ash Berlin-Taylor wrote: > This has some overlap with the "Deprecate contrib folder" right? > > -ash > > On 18 Apr 2019, at 14:57, Kamil Breguła > wrote: > > > > Hello all,

Re: [2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Ash Berlin-Taylor
This has some overlap with the "Deprecate contrib folder" right? -ash > On 18 Apr 2019, at 14:57, Kamil Breguła wrote: > > Hello all, > > During the 4-year project development, the community made many > decisions about the structure and principles of module creation. Some > decisions have been

[2.0 spring cleaning] Changes in import paths

2019-04-18 Thread Kamil Breguła
Hello all, During the 4-year project development, the community made many decisions about the structure and principles of module creation. Some decisions have been made by Airbnb and they no longer apply. Other recommendations were rejected because of the low recognition of the community. I have

Re: schedule_interval question

2019-04-18 Thread Pawel Bartoszek
Ash, If I omit start_date it I get the error Task is missing the start_date parameter What should I set it to then? On Thu, Apr 18, 2019 at 1:03 PM Ash Berlin-Taylor wrote: > Do not set start_date to now. That will _always_ be wrong. > https://airflow.apache.org/faq.html#what-s-the-deal-with-st

Re: schedule_interval question

2019-04-18 Thread Ash Berlin-Taylor
Do not set start_date to now. That will _always_ be wrong. https://airflow.apache.org/faq.html#what-s-the-deal-with-start-date > On 18 Apr 2019, at 12:13, Pawel Bartoszek > wrote: > > Hi, > > When I set start_date to datetime.now() ie > > DAG( >dag_id="dag", >start_date=datet

Re: schedule_interval question

2019-04-18 Thread Pawel Bartoszek
Hi, When I set start_date to datetime.now() ie DAG( dag_id="dag", start_date=datetime.now(), schedule_interval="0 2 * * *", default_view="graph", orientation="TB", concurrency=1, max_active_runs=1, catchup=False ) I get following in