Re: Medium series: Airflow for Google Cloud

Looks like you don't have an account.. once you create one.. let me know and I will grant you admin perms on the wiki. -s On Fri, Jan 20, 2017 at 6:08 PM, siddharth anand wrote: > I've added it to https://cwiki.apache.org/confluence/display/AIRFLOW/ > Airflow+Links > > Feel free to add future po

Re: Medium series: Airflow for Google Cloud

I've added it to https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Links Feel free to add future posts to this page. You should have access. -s On Fri, Jan 20, 2017 at 3:23 PM, Alex Van Boxel wrote: > Hey all, > > now that 1.8 is nearing release. I finally started writing about Airflo

Article: The Rise of the Data Engineer

Hey I just published an article about the "Data Engineer" role in modern organizations and thought it could be of interest to this community. https://medium.com/@maximebeauchemin/the-rise-of-the-data-engineer-91be18f1e603#.5rkm4htnf Max

Re: Airflow 1.8.0 BETA 1

I completely understand what your trying to achieve, but I'm just not sure your get that result by using a 1.7 scheduler with a 1.8 worker, exactly because the contract is so simple and the worker itself doesn’t do too much (although the dependency engine has changed as well, so the “why isn’t m

Medium series: Airflow for Google Cloud

Hey all, now that 1.8 is nearing release. I finally started writing about Airflow. As it's me writing, I'll be focussing on the Google Cloud integration. Today's post is about BigQuery https://medium.com/google-cloud/airflow-for-google-cloud-part-1-d7da9a048aa4#.qe6f0gldf Next one will be about

Re: Airflow 1.8.0 BETA 1

The benefit is really just to limit the scope of the errors as we proceed cautiously, progressively with more confidence. As in we upgrade one small low SLA queue first (set of workers), find some worker-related bugs, web server bugs, fix them. Rinse and repeat until all workers are on 1.8.0. Then

Re: Airflow 1.8.0 BETA 2

Installing in production on Monday as well. Alpha's run ok till now. On Fri, Jan 20, 2017 at 7:39 PM Chris Riccomini wrote: > Installed in dev. Prod will go on Monday. Will keep you posted. > > On Fri, Jan 20, 2017 at 9:35 AM, Bolke de Bruin wrote: > > > Yes > > > > Sent from my iPhone > > > >

Re: Experiences with 1.8.0

Bolke, I will tackle logging because I started integrating with Google StackDriver logging. I have a separate branch, but this will be for after 1.8. It will be configurable and context aware and everyone will benefit (not only stackdriver logging). On Fri, Jan 20, 2017 at 11:55 PM Bolke de Bruin

Re: Experiences with 1.8.0

Will do. And thanks. Adding another issue: * Some of our DAGs are not getting scheduled for some unknown reason. Need to investigate why. Related but not root cause: * Logging is so chatty that it gets really hard to find the real issue Bolke. > On 20 Jan 2017, at 23:45, Dan Davydov wrote: >

Re: Airflow 1.8.0 BETA 1

Hi Max, Interesting idea. I agree with your assumption that the contract between the scheduler and the worker is pretty simple and it may work for upgrades where this contract hasn’t been altered. However, between plain 1.7.1 and 1.8.0 this contract has significantly changed. The handover to wo

Re: Experiences with 1.8.0

I'd be happy to lend a hand fixing these issues and hopefully some others are too. Do you mind creating jiras for these since you have the full context? I have created a JIRA for (1) and have assigned it to myself: https://issues.apache.org/jira/browse/AIRFLOW-780 On Fri, Jan 20, 2017 at 1:01 AM,

Re: Airflow 1.8.0 BETA 1

Hi all, I need some input around this progressive upgrade idea I had recently. At Airbnb we have many queues of workers, and I was entertaining the idea of rolling out 1.8.0beta in production on a per worker or per-queue basis to minimize the risks around upgrading. This of course assumes that h

Re: Airflow Meetup @ Paypal (San Jose)

I think if we hold it in the evening, there is no requirement to buy a ticket to come to the meetup. Let me verify. On Fri, Jan 20, 2017 at 12:45 PM, Jayesh Senjaliya wrote: > Hi Russell, > > Sure, Strata will have its own flavor of visitors, but the tickets are > kind of expensive too for every

Re: Airflow Meetup @ Paypal (San Jose)

Hi Russell, Sure, Strata will have its own flavor of visitors, but the tickets are kind of expensive too for everybody to join. I agree on turnouts though, so we can try for Strata first and fallback to regular meetup in March end or even April if we dont get space in Strata. or we can just do b

Re: Airflow Meetup @ Paypal (San Jose)

As I mentioned in the other thread, I am available to speak on Predictive Analytics with Airflow and PySpark. Mid march has been suggested. What about the evening of Tuesday, 3/14 - the first day of sessions at Strata? We could promote the meetup with the conference, get it listed as an evening ev

Re: Airflow Meetup in NYC @ Blue Apron

Cool, would you have remote joining setup (hangout?, adobe?) or recording for this for the folks not in NYC ? Thanks for hosting ! On Fri, Jan 20, 2017 at 10:37 AM, Joseph Napolitano < joseph.napolit...@blueapron.com.invalid> wrote: > Hi all! > > I want to officially announce a Meetup for Airflo

Re: Airflow Meetup in NYC @ Blue Apron

Great to hear. Yes, please do set up an official meet-up page. You are welcome to add a few of the committer or other contributors as co-admins of the meet-up page. (e.g. https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/ ) Once the meet-up page and the event is up, we can tweet the

Airflow Meetup @ Paypal (San Jose)

Hello Airflow community ! I am Jayesh from Paypal, and at last meetup we briefly talked about hosting next one and I offered to host at Paypal office in San Jose. If we can come up with some dates, I can talk to facilities to reserve space accordingly. so that it dont become short notice for the

Re: Airflow 1.8.0 BETA 2

Installed in dev. Prod will go on Monday. Will keep you posted. On Fri, Jan 20, 2017 at 9:35 AM, Bolke de Bruin wrote: > Yes > > Sent from my iPhone > > > On 20 Jan 2017, at 18:20, Boris Tyukin wrote: > > > > just to make sure this is the latest one, right? > > https://dist.apache.org/repos/dis

Airflow Meetup in NYC @ Blue Apron

Hi all! I want to officially announce a Meetup for Airflow in NYC! I'm looking forward to meeting other community members to share knowledge and network. We may create an official Meetup page, but in the meantime please signup here: https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf- u1uh3Ile

Re: NYC Meetup?

Hi All, I wanted to bump this thread again. I sent out another email about a meetup in NYC, so look for that one. It took a long time to get approved over the holidays, so I hope we can still generate interest in a short time. Cheers! On Thu, Dec 29, 2016 at 3:01 PM, Joseph Napolitano < joseph.

Re: Airflow 1.8.0 BETA 2

Yes Sent from my iPhone > On 20 Jan 2017, at 18:20, Boris Tyukin wrote: > > just to make sure this is the latest one, right? > https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz > >> On Fri, Jan 20, 2017 at 10:57 AM, Bolke de Bruin wrote: >> >>

Re: New book covers Airflow with PySpark: Agile Data Science 2.0 (O'Reilly, 2017) AND Airflow Meetup?

Let me email about this with its own email subject. On Thu, Jan 19, 2017 at 10:54 PM Jayesh Senjaliya wrote: > Hi Siddharth, > > I am Jayesh from Paypal, and at last meetup we briefly talked about > hosting next one and I offered to host next Airflow meetup at Paypal > office. > > If we can com

Re: Airflow 1.8.0 BETA 2

just to make sure this is the latest one, right? https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b2+apache.incubating.tar.gz On Fri, Jan 20, 2017 at 10:57 AM, Bolke de Bruin wrote: > Hi All, > > I have made the SECOND beta of Airflow 1.8.0 available at: > https://dist.apach

Airflow 1.8.0 BETA 2

Hi All, I have made the SECOND beta of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys are available at https://dist.apache.org/repos/dist/release/incubator/airflow/

Re: How to learn more about deprecation warnings?

Awesome, thanks Jeremiah! On Fri, Jan 20, 2017 at 8:20 AM, Jeremiah Lowin wrote: > Hi Laura, > > The error is raised if an unused argument is passed to BaseOperator -- > basically if there is anything in either args or kwargs. The original issue > was that in a number of cases arguments were mis

Re: How to learn more about deprecation warnings?

Hi Laura, The error is raised if an unused argument is passed to BaseOperator -- basically if there is anything in either args or kwargs. The original issue was that in a number of cases arguments were misspelled or misused by Operator subclasses and instead of raising an error, they were just pas

Experiences with 1.8.0 (updated)

— continued accidentally pressed send — This is to report back on some of the (early) experiences we have with Airflow 1.8.0 (beta 1 at the moment): 1. The UI does not show faulty DAG, leading to confusion for developers. When a faulty dag is placed in the dags folder the UI would report a pars

Experiences with 1.8.0

This is to report back on some of the (early) experiences we have with Airflow 1.8.0 (beta 1 at the moment): 1. The UI does not show faulty DAG, leading to confusion for developers. When a faulty dag is placed in the dags folder the UI would report a parsing error. Now it doesn’t due to the sep

Re: Airflow 1.8.0 BETA 1

1. Always do backups 2. Your airflow.cfg will work, but you might want to adjust some settings that are new 3. Pip install https://dist.apache.org/repos/dist/dev/incubator/airflow/airflow-1.8.0b1+apache.incubating.tar.gz should work. > On 19 Jan 2017, at 23:25, Boris Tyukin wrote: > > I'd lik