Re: Pulling XCom from parent DAG in a sub DAG template

[Maxime Beauchemin]

I think `dag.parent_dag.dag_id` should work, either in a template of
through the context in a PythonOperator.

Max

On Wed, Feb 1, 2017 at 4:23 PM, Daniel Huang  wrote:

> Hey everyone,
> I currently have a parent DAG that generates a filename based on the
> current timestamp. That same filename needs to be used down the line in
> multiple operators, including some subdag operators. So I was hoping to use
> XComs to pass this filename along. It works fine for operators within the
> parent DAG, but not for any operators in the sub DAGs because the dag_id
> arg on xcom_pull() defaults to self.dag_id. I'm pulling this XCom value
> from templates, so I don't want to have to hardcode a parent's dag id in it
> because it should work for different parent dags. The sub dag template
> would still have to make assumptions about what is available in xcom under
> a task id, but I think that's alright?
>
> So my question is if there is a way to programmatically specify the parent
> dag id in an xcom pull from within a template or if there's a better way of
> doing what I'm trying to achieve. Note, I also have sub dags within sub
> dags that require the same filename.
>
> If there is no way to this already, I did come across
> https://issues.apache.org/jira/browse/AIRFLOW-54 and wondered if we need
> something similar for the dag_id arg, like ti.xcom_pull(dag_id='@parent',
> task_ids='foo').
>
> Thanks,
> Daniel
>

Re: Airflow Meetup @ Paypal (San Jose)

[siddharth anand]

IMHO, I'd publish the meet-up. You still have 6 weeks to find a 3rd
speaker. If Bolke and Alex are traveling all the way for Strata, perhaps
one of them can speak :-)

-s

On Wed, Feb 1, 2017 at 1:48 PM, Russell Jurney 
wrote:

> Maybe start a new thread with a title "Call for Speakers for Meetup on Mar
> 14" ?
>
> On Wed, Feb 1, 2017 at 11:59 AM Jayesh Senjaliya 
> wrote:
>
> > Yes, we are still waiting for more speakers.
> >
> > can anybody from Airbnb present ?
> >
> > anybody else ?
> >
> >
> > - Jayesh
> >
> > On Tue, Jan 31, 2017 at 8:16 PM, siddharth anand 
> > wrote:
> >
> > > Jayesh,
> > > Looks good. No need to vote. Just publish a new event with details on
> the
> > > meet-up page:
> > > https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/
> > >
> > > Please add a short abstract as well for the talks and find a 3rd
> speaker.
> > > Please be sure to record the meet-up so that we can publish it. Once
> the
> > > meet-up event is up, please respond to this email! We can help promote
> > it.
> > > I suggest picking a start time after the Strata talks end but not super
> > > late either.
> > >
> > > -s
> > >
> > > On Tue, Jan 31, 2017 at 9:19 AM, Jayesh Senjaliya  >
> > > wrote:
> > >
> > > > HI All,
> > > >
> > > > √ I have approval from Paypal to host Airflow meetup.  How about
> March
> > > 14th
> > > > ? Please vote.
> > > >
> > > > √ we will have food and drinks.
> > > > Please let me know if anybody has any special request, I will try to
> > > > accommodate :)
> > > >
> > > > For presentations:
> > > >  1) Disk recommission using airflow with overall automation of
> "Hadoop
> > > Node
> > > > and Disk Remediation". - Jayesh Senjaliya ( Paypal )
> > > >  2) Predictive Analytics with Airflow and PySpark - ( Russell Jurney
> )
> > > >
> > > >
> > > > Please send request to present to this email thread if you are
> > interested
> > > > in presenting.
> > > >
> > > > Thanks
> > > > Jayesh
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jan 26, 2017 at 4:08 PM, Russell Jurney <
> > > russell.jur...@gmail.com>
> > > > wrote:
> > > >
> > > > > Cool!
> > > > >
> > > > > On Wed, Jan 25, 2017 at 11:23 PM Jayesh Senjaliya <
> > jhsonl...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Russell,
> > > > > >
> > > > > > yes, I will be presenting from Paypal side.
> > > > > > Once i have official approval from Paypal, I will sent out email.
> > > > > > I am basically going by the steps what Siddharth outlined earlier
> > in
> > > > the
> > > > > > thread.
> > > > > >
> > > > > > Thanks
> > > > > > Jayesh
> > > > > >
> > > > > > On Wed, Jan 25, 2017 at 7:50 PM, Russell Jurney <
> > > > > russell.jur...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Is someone from Paypal likely to speak? Should we start a new
> > > thread
> > > > > > with a
> > > > > > > call for another speaker? There was mention of three being
> > needed.
> > > > > > >
> > > > > > > On Wed, Jan 25, 2017 at 5:33 PM Jayesh Senjaliya <
> > > > jhsonl...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yes I am waiting for response from facilities about it, most
> > > likely
> > > > > by
> > > > > > > > early next week.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Jayesh
> > > > > > > >
> > > > > > > > On Wed, Jan 25, 2017 at 4:52 PM, Russell Jurney <
> > > > > > > russell.jur...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Boris, would you be able to attend an evening meetup on the
> > > > nights
> > > > > of
> > > > > > > > 3/15
> > > > > > > > > or 3/16? I think attendance would be better on one of those
> > > days,
> > > > > as
> > > > > > > many
> > > > > > > > > people don't attend the tutorial days.
> > > > > > > > >
> > > > > > > > > Paypal sounds awesome as a venue. Would they handle food
> and
> > > > drink
> > > > > as
> > > > > > > > well?
> > > > > > > > >
> > > > > > > > > On Wed, Jan 25, 2017 at 11:28 AM, Boris Tyukin <
> > > > > > bo...@boristyukin.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > it would be great!
> > > > > > > > > >
> > > > > > > > > > On Wed, Jan 25, 2017 at 1:26 PM, siddharth anand <
> > > > > > san...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Paypal is quite close (11 minute drive on local streets
> > per
> > > > > > google
> > > > > > > > > Maps :
> > > > > > > > > > > https://goo.gl/maps/otUpve9StxJ2) to the Strata venue,
> > so
> > > it
> > > > > > would
> > > > > > > > > make
> > > > > > > > > > > sense to hold the meet-up at Paypal during Strata week.
> > > > > > > > > > >
> > > > > > > > > > > -s
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jan 25, 2017 at 5:48 AM, Boris Tyukin <
> > > > > > > bo...@boristyukin.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > any way to schedule it during Strata week? would love
> > to
> > > 

Re: Airflow Meetup in NYC @ Blue Apron

[siddharth anand]

Also, if you record a video, we'd be happy to place it on the wiki and
promote it via our twitter feed, etc...
-s

On Mon, Jan 30, 2017 at 5:34 PM, Boris Tyukin  wrote:

> i hope you guys can share presentation slides at least for all of us who
> are not in NYC
>
> On Mon, Jan 30, 2017 at 7:33 PM, Maxime Beauchemin <
> maximebeauche...@gmail.com> wrote:
>
> > I'd love to watch, is there any way you guys can livecast or share a
> video
> > after the event?
> >
> > Looking forward to it!
> >
> > Max
> >
> > On Mon, Jan 30, 2017 at 1:56 PM, Joseph Napolitano <
> > joseph.napolit...@blueapron.com.invalid> wrote:
> >
> > > Hi All!
> > >
> > > We are excited to host an Airflow Meetup in NYC.  We will have a guest
> > > speaker from Spotify!
> > >
> > > The Meetup is in 2 days, on Feb 1st @ 6:30pm at Blue Apron's
> > headquarters.
> > >
> > > In Summary:
> > > Date: Feb 1st
> > > Time 6:30 - 9pm EST
> > > Location: 40 W 23rd St. New York, NY 10010
> > > https://www.google.com/maps/place/40+W+23rd+St,+New+York,+NY
> > > +10010/@40.7420885,-73.9938457,17z/data=!3m1!4b1!4m5!3m4!
> > > 1s0x89c259a46471d2a1:0xc2517d92b1b68bba!8m2!3d40.
> > > 7420845!4d-73.9916517?hl=en
> > >
> > > Schedule:
> > > 6:30 - 7:15 Meet and greet
> > > 7:15 - ? Presentations from Blue Apron and Spotify
> > >
> > > It's not too late to signup for a presentation.  We will stick around
> as
> > > late as 9pm.
> > >
> > > We don't have an official Meetup page, so please sign up here :)
> > > The signup sheet is available here:
> > > https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-u1uh3I
> > > leeHy8QTwaJ4BkkSkVM-X1E/edit?usp=sharing
> > >
> > > Feel free to share the signup sheet with other parties.
> > >
> > > As mentioned, we're on the 5th floor.  You need to check in with
> security
> > > in the building lobby, and again when you reach the fifth floor to get
> a
> > > name tag.
> > >
> > > Thanks, and looking forward to meeting everyone!
> > >
> > > Cheers,
> > > Joe Nap
> > >
> > >
> > >
> > > On Fri, Jan 20, 2017 at 1:37 PM, Joseph Napolitano <
> > > joseph.napolit...@blueapron.com> wrote:
> > >
> > > > Hi all!
> > > >
> > > > I want to officially announce a Meetup for Airflow in NYC!  I'm
> looking
> > > > forward to meeting other community members to share knowledge and
> > > network.
> > > >
> > > > We may create an official Meetup page, but in the meantime please
> > signup
> > > > here:
> > > > https://docs.google.com/spreadsheets/d/1WmfgZeExSVdLf-u1uh3I
> > > > leeHy8QTwaJ4BkkSkVM-X1E/edit?usp=sharing
> > > >
> > > > I have a confirmed date of February 1st @ 6:30 at Blue Apron's
> > > > headquarters.
> > > >
> > > > In Summary:
> > > > Date: Feb 1st
> > > > Time 6:30 - 9pm EST
> > > > Location: 40 W 23rd St. New York, NY 10010
> > > > https://www.google.com/maps/place/40+W+23rd+St,+New+York,+NY
> > > > +10010/@40.7420885,-73.9938457,17z/data=!3m1!4b1!4m5!3m4!
> > > > 1s0x89c259a46471d2a1:0xc2517d92b1b68bba!8m2!3d40.7420845!4d-
> > > > 73.9916517?hl=en
> > > >
> > > > We're on the 5th floor.  You need to check in with security in the
> > > > building lobby, and again when you reach the fifth floor to get a
> name
> > > tag.
> > > >
> > > > Food & drink will be provided!
> > > >
> > > > Let me know if you would like to present.  We'd love to hear about
> your
> > > > architecture and war stories.  We will have a large projector and PA
> > > system
> > > > setup.
> > > >
> > > > Sorry about the short notice, but it took a while to get approved
> over
> > > the
> > > > holidays and new year.  If we can't generate enough interest we can
> > > > certainly push it back a month.
> > > >
> > > > Thanks, and Bon Appétite!
> > > >
> > > > --
> > > > *Joe Napolitano *| Sr. Data Engineer
> > > > www.blueapron.com | 5 Crosby Street, New York, NY 10013
> > > >
> > >
> > >
> > >
> > > --
> > > *Joe Napolitano *| Sr. Data Engineer
> > > www.blueapron.com | 5 Crosby Street, New York, NY 10013
> > >
> >
>

Pulling XCom from parent DAG in a sub DAG template

[Daniel Huang]

Hey everyone,
I currently have a parent DAG that generates a filename based on the
current timestamp. That same filename needs to be used down the line in
multiple operators, including some subdag operators. So I was hoping to use
XComs to pass this filename along. It works fine for operators within the
parent DAG, but not for any operators in the sub DAGs because the dag_id
arg on xcom_pull() defaults to self.dag_id. I'm pulling this XCom value
from templates, so I don't want to have to hardcode a parent's dag id in it
because it should work for different parent dags. The sub dag template
would still have to make assumptions about what is available in xcom under
a task id, but I think that's alright?

So my question is if there is a way to programmatically specify the parent
dag id in an xcom pull from within a template or if there's a better way of
doing what I'm trying to achieve. Note, I also have sub dags within sub
dags that require the same filename.

If there is no way to this already, I did come across
https://issues.apache.org/jira/browse/AIRFLOW-54 and wondered if we need
something similar for the dag_id arg, like ti.xcom_pull(dag_id='@parent',
task_ids='foo').

Thanks,
Daniel

Re: Flow-based Airflow?

[Jeremiah Lowin]

Great point. I think the best solution is to solve this for all XComs by
checking object size before adding it to the DB. I don't see a built in way
of handling it (though apparently MySQL is internally limited to 64kb).
I'll look into a PR that would enforce a similar limit for all databases.

On Wed, Feb 1, 2017 at 4:52 PM Maxime Beauchemin 
wrote:

I'm not sure about XCom being the default, it seems pretty dangerous. It
just takes one person that is not fully aware of the size of the data, or
one day with an outlier and that could put the Airflow db in jeopardy.

I guess it's always been an aspect of XCom, and it could be good to have
some explicit gatekeeping there regardless of this PR/feature. Perhaps the
DB itself has protection against large blobs?

Max

On Wed, Feb 1, 2017 at 12:42 PM, Jeremiah Lowin  wrote:

> Yesterday I began converting a complex script to a DAG. It turned out to
be
> a perfect test case for the dataflow model: a big chunk of data moving
> through a series of modification steps.
>
> So I have built an extensible dataflow extension for Airflow on top of
XCom
> and the existing dependency engine:
> https://issues.apache.org/jira/browse/AIRFLOW-825
> https://github.com/apache/incubator-airflow/pull/2046 (still waiting for
> tests... it will be quite embarrassing if they don't pass)
>
> The philosophy is simple:
> Dataflow objects represent the output of upstream tasks. Downstream tasks
> add Dataflows with a specific key. When the downstream task runs, the
> (optionally indexed) upstream result is available in the downstream
context
> under context['dataflows'][key]. In addition, PythonOperators receive the
> data as a keyword argument.
>
> The basic Dataflow serializes the data through XComs, but is trivially
> extended to alternative storage via subclasses. I have provided (in
> contrib) implementations of a local filesystem-based Dataflow as well as a
> Google Cloud Storage dataflow.
>
> Laura, I hope you can have a look and see if this will bring some of your
> requirements in to Airflow as first-class citizens.
>
> Jeremiah
>

Re: Flow-based Airflow?

[Maxime Beauchemin]

I'm not sure about XCom being the default, it seems pretty dangerous. It
just takes one person that is not fully aware of the size of the data, or
one day with an outlier and that could put the Airflow db in jeopardy.

I guess it's always been an aspect of XCom, and it could be good to have
some explicit gatekeeping there regardless of this PR/feature. Perhaps the
DB itself has protection against large blobs?

Max

On Wed, Feb 1, 2017 at 12:42 PM, Jeremiah Lowin  wrote:

> Yesterday I began converting a complex script to a DAG. It turned out to be
> a perfect test case for the dataflow model: a big chunk of data moving
> through a series of modification steps.
>
> So I have built an extensible dataflow extension for Airflow on top of XCom
> and the existing dependency engine:
> https://issues.apache.org/jira/browse/AIRFLOW-825
> https://github.com/apache/incubator-airflow/pull/2046 (still waiting for
> tests... it will be quite embarrassing if they don't pass)
>
> The philosophy is simple:
> Dataflow objects represent the output of upstream tasks. Downstream tasks
> add Dataflows with a specific key. When the downstream task runs, the
> (optionally indexed) upstream result is available in the downstream context
> under context['dataflows'][key]. In addition, PythonOperators receive the
> data as a keyword argument.
>
> The basic Dataflow serializes the data through XComs, but is trivially
> extended to alternative storage via subclasses. I have provided (in
> contrib) implementations of a local filesystem-based Dataflow as well as a
> Google Cloud Storage dataflow.
>
> Laura, I hope you can have a look and see if this will bring some of your
> requirements in to Airflow as first-class citizens.
>
> Jeremiah
>

Re: Changelog 1.8

[Bolke de Bruin]

Perfect!

Sent from my iPhone

> On 1 Feb 2017, at 22:32, Alex Van Boxel  wrote:
> 
> Yes, I will take care of it tomorrow morning. Need sleep now... I'll add
> the new addition to it.
> 
>> On Wed, Feb 1, 2017 at 8:59 PM Bolke de Bruin  wrote:
>> 
>> Hey Alex,
>> 
>> Can you finalize the changelog for 1.8? I can then make 1.8 rc 1 available.
>> 
>> Bolke
>> 
>> Sent from my iPhone
>> 
> -- 
>  _/
> _/ Alex Van Boxel

Re: Changelog 1.8

[Alex Van Boxel]

Yes, I will take care of it tomorrow morning. Need sleep now... I'll add
the new addition to it.

On Wed, Feb 1, 2017 at 8:59 PM Bolke de Bruin  wrote:

> Hey Alex,
>
> Can you finalize the changelog for 1.8? I can then make 1.8 rc 1 available.
>
> Bolke
>
> Sent from my iPhone
>
-- 
  _/
_/ Alex Van Boxel

Re: Flow-based Airflow?

[Jeremiah Lowin]

Yesterday I began converting a complex script to a DAG. It turned out to be
a perfect test case for the dataflow model: a big chunk of data moving
through a series of modification steps.

So I have built an extensible dataflow extension for Airflow on top of XCom
and the existing dependency engine:
https://issues.apache.org/jira/browse/AIRFLOW-825
https://github.com/apache/incubator-airflow/pull/2046 (still waiting for
tests... it will be quite embarrassing if they don't pass)

The philosophy is simple:
Dataflow objects represent the output of upstream tasks. Downstream tasks
add Dataflows with a specific key. When the downstream task runs, the
(optionally indexed) upstream result is available in the downstream context
under context['dataflows'][key]. In addition, PythonOperators receive the
data as a keyword argument.

The basic Dataflow serializes the data through XComs, but is trivially
extended to alternative storage via subclasses. I have provided (in
contrib) implementations of a local filesystem-based Dataflow as well as a
Google Cloud Storage dataflow.

Laura, I hope you can have a look and see if this will bring some of your
requirements in to Airflow as first-class citizens.

Jeremiah

Re: Airflow Meetup @ Paypal (San Jose)

[Jayesh Senjaliya]

Yes, we are still waiting for more speakers.

can anybody from Airbnb present ?

anybody else ?


- Jayesh

On Tue, Jan 31, 2017 at 8:16 PM, siddharth anand  wrote:

> Jayesh,
> Looks good. No need to vote. Just publish a new event with details on the
> meet-up page:
> https://www.meetup.com/Bay-Area-Apache-Airflow-Incubating-Meetup/
>
> Please add a short abstract as well for the talks and find a 3rd speaker.
> Please be sure to record the meet-up so that we can publish it. Once the
> meet-up event is up, please respond to this email! We can help promote it.
> I suggest picking a start time after the Strata talks end but not super
> late either.
>
> -s
>
> On Tue, Jan 31, 2017 at 9:19 AM, Jayesh Senjaliya 
> wrote:
>
> > HI All,
> >
> > √ I have approval from Paypal to host Airflow meetup.  How about March
> 14th
> > ? Please vote.
> >
> > √ we will have food and drinks.
> > Please let me know if anybody has any special request, I will try to
> > accommodate :)
> >
> > For presentations:
> >  1) Disk recommission using airflow with overall automation of "Hadoop
> Node
> > and Disk Remediation". - Jayesh Senjaliya ( Paypal )
> >  2) Predictive Analytics with Airflow and PySpark - ( Russell Jurney )
> >
> >
> > Please send request to present to this email thread if you are interested
> > in presenting.
> >
> > Thanks
> > Jayesh
> >
> >
> >
> >
> > On Thu, Jan 26, 2017 at 4:08 PM, Russell Jurney <
> russell.jur...@gmail.com>
> > wrote:
> >
> > > Cool!
> > >
> > > On Wed, Jan 25, 2017 at 11:23 PM Jayesh Senjaliya  >
> > > wrote:
> > >
> > > > Hi Russell,
> > > >
> > > > yes, I will be presenting from Paypal side.
> > > > Once i have official approval from Paypal, I will sent out email.
> > > > I am basically going by the steps what Siddharth outlined earlier in
> > the
> > > > thread.
> > > >
> > > > Thanks
> > > > Jayesh
> > > >
> > > > On Wed, Jan 25, 2017 at 7:50 PM, Russell Jurney <
> > > russell.jur...@gmail.com>
> > > > wrote:
> > > >
> > > > > Is someone from Paypal likely to speak? Should we start a new
> thread
> > > > with a
> > > > > call for another speaker? There was mention of three being needed.
> > > > >
> > > > > On Wed, Jan 25, 2017 at 5:33 PM Jayesh Senjaliya <
> > jhsonl...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Yes I am waiting for response from facilities about it, most
> likely
> > > by
> > > > > > early next week.
> > > > > >
> > > > > > Thanks
> > > > > > Jayesh
> > > > > >
> > > > > > On Wed, Jan 25, 2017 at 4:52 PM, Russell Jurney <
> > > > > russell.jur...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Boris, would you be able to attend an evening meetup on the
> > nights
> > > of
> > > > > > 3/15
> > > > > > > or 3/16? I think attendance would be better on one of those
> days,
> > > as
> > > > > many
> > > > > > > people don't attend the tutorial days.
> > > > > > >
> > > > > > > Paypal sounds awesome as a venue. Would they handle food and
> > drink
> > > as
> > > > > > well?
> > > > > > >
> > > > > > > On Wed, Jan 25, 2017 at 11:28 AM, Boris Tyukin <
> > > > bo...@boristyukin.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > it would be great!
> > > > > > > >
> > > > > > > > On Wed, Jan 25, 2017 at 1:26 PM, siddharth anand <
> > > > san...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Paypal is quite close (11 minute drive on local streets per
> > > > google
> > > > > > > Maps :
> > > > > > > > > https://goo.gl/maps/otUpve9StxJ2) to the Strata venue, so
> it
> > > > would
> > > > > > > make
> > > > > > > > > sense to hold the meet-up at Paypal during Strata week.
> > > > > > > > >
> > > > > > > > > -s
> > > > > > > > >
> > > > > > > > > On Wed, Jan 25, 2017 at 5:48 AM, Boris Tyukin <
> > > > > bo...@boristyukin.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > any way to schedule it during Strata week? would love to
> > > attend
> > > > > one
> > > > > > > of
> > > > > > > > > > airflow meetups but I am in Florida. 03/13 or 03/14 would
> > > work
> > > > > the
> > > > > > > best
> > > > > > > > > > because first two days of Strata are training days and
> not
> > > very
> > > > > > busy
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 24, 2017 at 10:33 PM, Russell Jurney <
> > > > > > > > > russell.jur...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Unfortunately, Strata has no room for us :( Paypal
> sounds
> > > > like
> > > > > a
> > > > > > > > great
> > > > > > > > > > > option.
> > > > > > > > > > >
> > > > > > > > > > > Jayesh, sounds like you're driving? :)
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 24, 2017 at 12:04 PM, siddharth anand <
> > > > > > > san...@apache.org
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Russell,
> > > > > > > > > > > > Let us know what you learn about Strata.
> > > > > > > > > > > >
> > 

Changelog 1.8

[Bolke de Bruin]

Hey Alex,

Can you finalize the changelog for 1.8? I can then make 1.8 rc 1 available. 

Bolke

Sent from my iPhone

Re: Airflow Logging Updates

[Miller, Robin]

Hi Alex,


Thanks for your interest in this.


Hopefully this will  be a good first step for any further logging changes in 
the future, so will be helpful with logging to Google Cloud.


To be clear, I wasn't expecting this go into the 1.8 release at this stage. 
Having it in for the next release after that would be nice, though.


Thanks,

Robin Miller
OLIVER WYMAN
robin.mil...@affiliate.oliverwyman.com
www.oliverwyman.com


From: Alex Van Boxel 
Sent: 01 February 2017 12:55:42
To: dev@airflow.incubator.apache.org
Subject: Re: Airflow Logging Updates

Hey Robin,

I also have an interest in logging, but to get Google Cloud logging to
work. I'm happy to work with you on this. But *first* the 1.8 needs to go
out.

But I don't think this will go on the patch branch, but maybe for the next
release. I don't know what the other people think.

On Wed, Feb 1, 2017 at 11:58 AM Miller, Robin <
robin.mil...@affiliate.oliverwyman.com> wrote:

> Hi All,
>
>
> A while ago we took over this issue:
> https://issues.apache.org/jira/browse/AIRFLOW-409, which revolved around
> avoiding making changes to the root python logging configuration so as to
> avoid any accidental side effects that this could produce in other python
> applications.
>
>
> The approach we decided on was to avoid use of the root logger entirely
> and produce a single configuration point for the logging to allow it to be
> configured pretty much however is desired in one place (with currently
> fairly simple options, but this could be made as complex or configurable as
> desired with changes to only a single file).
>
>
> As you might expect this affects a lot of the code, as it requires
> changing every log statement that's contains "logging.Info",
> "logging.Error", etc.
>
>
> The pull request for this,
> https://github.com/apache/incubator-airflow/pull/1921, has been open for
> almost 2 months now and unsurprisingly now has merge conflicts. We'll
> happily clean these up, but would prefer to do so at the time this is being
> reviewed, otherwise more will inevitably appear.
>
>
> So I'm wondering, is this issue/pull request likely to get any attention
> soon? Or is this change simply unwanted?
>
>
> Thanks,
>
> Robin Miller
> OLIVER WYMAN
> robin.mil...@affiliate.oliverwyman.com robin.mil...@affiliate.oliverwyman.com>
> www.oliverwyman.com
>
>
> 
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>
--
 _/
_/ Alex Van Boxel


This e-mail and any attachments may be confidential or legally privileged. If 
you received this message in error or are not the intended recipient, you 
should destroy the e-mail message and any attachments or copies, and you are 
prohibited from retaining, distributing, disclosing or using any information 
contained herein. Please inform us of the erroneous delivery by return e-mail. 
Thank you for your cooperation.

Re: Airflow Logging Updates

[Alex Van Boxel]

Hey Robin,

I also have an interest in logging, but to get Google Cloud logging to
work. I'm happy to work with you on this. But *first* the 1.8 needs to go
out.

But I don't think this will go on the patch branch, but maybe for the next
release. I don't know what the other people think.

On Wed, Feb 1, 2017 at 11:58 AM Miller, Robin <
robin.mil...@affiliate.oliverwyman.com> wrote:

> Hi All,
>
>
> A while ago we took over this issue:
> https://issues.apache.org/jira/browse/AIRFLOW-409, which revolved around
> avoiding making changes to the root python logging configuration so as to
> avoid any accidental side effects that this could produce in other python
> applications.
>
>
> The approach we decided on was to avoid use of the root logger entirely
> and produce a single configuration point for the logging to allow it to be
> configured pretty much however is desired in one place (with currently
> fairly simple options, but this could be made as complex or configurable as
> desired with changes to only a single file).
>
>
> As you might expect this affects a lot of the code, as it requires
> changing every log statement that's contains "logging.Info",
> "logging.Error", etc.
>
>
> The pull request for this,
> https://github.com/apache/incubator-airflow/pull/1921, has been open for
> almost 2 months now and unsurprisingly now has merge conflicts. We'll
> happily clean these up, but would prefer to do so at the time this is being
> reviewed, otherwise more will inevitably appear.
>
>
> So I'm wondering, is this issue/pull request likely to get any attention
> soon? Or is this change simply unwanted?
>
>
> Thanks,
>
> Robin Miller
> OLIVER WYMAN
> robin.mil...@affiliate.oliverwyman.com robin.mil...@affiliate.oliverwyman.com>
> www.oliverwyman.com
>
>
> 
> This e-mail and any attachments may be confidential or legally privileged.
> If you received this message in error or are not the intended recipient,
> you should destroy the e-mail message and any attachments or copies, and
> you are prohibited from retaining, distributing, disclosing or using any
> information contained herein. Please inform us of the erroneous delivery by
> return e-mail. Thank you for your cooperation.
>
-- 
  _/
_/ Alex Van Boxel

Airflow Logging Updates

[Miller, Robin]

Hi All,


A while ago we took over this issue: 
https://issues.apache.org/jira/browse/AIRFLOW-409, which revolved around 
avoiding making changes to the root python logging configuration so as to avoid 
any accidental side effects that this could produce in other python 
applications.


The approach we decided on was to avoid use of the root logger entirely and 
produce a single configuration point for the logging to allow it to be 
configured pretty much however is desired in one place (with currently fairly 
simple options, but this could be made as complex or configurable as desired 
with changes to only a single file).


As you might expect this affects a lot of the code, as it requires changing 
every log statement that's contains "logging.Info", "logging.Error", etc.


The pull request for this, 
https://github.com/apache/incubator-airflow/pull/1921, has been open for almost 
2 months now and unsurprisingly now has merge conflicts. We'll happily clean 
these up, but would prefer to do so at the time this is being reviewed, 
otherwise more will inevitably appear.


So I'm wondering, is this issue/pull request likely to get any attention soon? 
Or is this change simply unwanted?


Thanks,

Robin Miller
OLIVER WYMAN
robin.mil...@affiliate.oliverwyman.com
www.oliverwyman.com



This e-mail and any attachments may be confidential or legally privileged. If 
you received this message in error or are not the intended recipient, you 
should destroy the e-mail message and any attachments or copies, and you are 
prohibited from retaining, distributing, disclosing or using any information 
contained herein. Please inform us of the erroneous delivery by return e-mail. 
Thank you for your cooperation.

RE: nvd3 is an external resource

[Сёмочкин Максим Викторович]

I apologize for the delay in response.
We tested and everything was fine!
Thx you!

-Original Message-
From: Bolke de Bruin [mailto:bdbr...@gmail.com] 
Sent: Tuesday, January 31, 2017 9:59 PM
To: dev@airflow.incubator.apache.org
Subject: Re: nvd3 is an external resource

Can you please test https://github.com/apache/incubator-airflow/pull/2035 and 
+1 if it works for you.

Thanks
Bolke

> On 30 Jan 2017, at 12:04, Сёмочкин Максим Викторович  wrote:
> 
> I think I found a way to prevent the generation of unnecessary headers in the 
> html code.
> Maybe instead of a chart.buildhtml() will call the method buildcontent()?
> 
> See my example below which shows how it is implemented now in the 
> Airflow and how this can be implemented
> 
> First, prepare a random graph (example taken from the documentation of the 
> module):
> =
> from nvd3 import lineChart
> chart = lineChart(name="lineChart", x_is_date=False, 
> x_axis_format="AM_PM")
> 
> xdata = range(24)
> ydata = [0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 4, 3, 3, 5, 7, 5, 3, 16, 6, 
> 9, 15, 4, 12]
> ydata2 = [9, 8, 11, 8, 3, 7, 10, 8, 6, 6, 9, 6, 5, 4, 3, 10, 0, 6, 3, 
> 1, 0, 0, 0, 1]
> 
> extra_serie = {"tooltip": {"y_start": "There are ", "y_end": " 
> calls"}} chart.add_serie(y=ydata, x=xdata, name='sine', 
> extra=extra_serie) extra_serie = {"tooltip": {"y_start": "", "y_end": 
> " min"}} chart.add_serie(y=ydata2, x=xdata, name='cose', 
> extra=extra_serie) 
> =
> 
> So now use the module in Airflow:
> =
> chart.buildhtml()
> str(chart)
> 
> Out[62]: '\n\n\n charset="utf-8" />\n href="https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.css; 
> rel="stylesheet" />
> 
> 
> 
> \n'
> =
> 
> But this way you can generate html code which is guaranteed not to contain we 
> do not need the headers:
> =
> chart.buildcontent()
> str(chart.content)
> 
> Out[64]: '\n style="height:450px;">\n\n\n\n\n\n\n
> data_linechart=[{"values": [{"x": 0, "y": 0}, {"x": 1, "y": 0}, {"x": 2, "y": 
> 1}, {"x": 3, "y": 1}, {"x": 4, "y": 0}, {"x": 5, "y": 0}, {"x": 6, "y": 0}, 
> {"x": 7, "y": 0}, {"x": 8, "y": 1}, {"x": 9, "y": 0}, {"x": 10, "y": 0}, 
> {"x": 11, "y": 4}, {"x": 12, "y": 3}, {"x": 13, "y": 3}, {"x": 14, "y": 5},
> 
> 
> 
> \n'
> =
> 
> -Original Message-
> From: Bolke de Bruin [mailto:bdbr...@gmail.com]
> Sent: Monday, January 30, 2017 12:52 PM
> To: dev@airflow.incubator.apache.org
> Subject: Re: nvd3 is an external resource
> 
> This is in python-nvd3, NVD3Chart.py. Can you please open an issue with them? 
> It seems we can override the setting in Airflow so we can workaround it, but 
> they should fix this.
> 
> Bolke
> 
>> On 30 Jan 2017, at 10:50, Сёмочкин Максим Викторович  wrote:
>> 
>> I checked again and it seems I saw the problem.
>> In the Airflow (more precisely in the Flask) the necessary files are loaded 
>> from the directory static no problem with that.
>> But at some point in the code the method is called chart.buildhtml(), then a 
>> representation of a chart object will become an html that among other things 
>> contains the lines:
>>   > href="https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.css; 
>> rel="stylesheet" />
>>   > src="https://cdnjs.cloudflare.com/ajax/libs/d3/3.5.5/d3.min.js";>
>>   > src="https://cdnjs.cloudflare.com/ajax/libs/nvd3/1.7.0/nv.d3.min.js";>
>> 
>> 
>> -Original Message-
>> From: Bolke de Bruin [mailto:bdbr...@gmail.com]
>> Sent: Monday, January 30, 2017 11:55 AM
>> To: dev@airflow.incubator.apache.org
>> Subject: Re: nvd3 is an external resource
>> 
>> Did you double check btw? The nvd3 libraries are pulled from a local 
>> resources as far as I can see.
>> 
>> - Bolke
>> 
>>> On 30 Jan 2017, at 09:45, Bolke de Bruin  wrote:
>>> 
>>> That is indeed unfortunate. python-nvd3 pulls its in this way (thus not 
>>> Airflow itself). It might be a config thing with them.
>>> 
>>> Bolke
>>> 
 On 30 Jan 2017, at 09:42, Сёмочкин Максим Викторович  
 wrote:
 
 Hello!
 Did I understand correctly that version 1.8 is a library for drawing 
 charts (nvd3) is loaded from external resource - cloudflare?
 Our problem is that the Airflow is installed in a closed network 
 segment that has no Internet access
>>> 
>> 
>