Re: how to capture sqoop mapreduce counters

2017-01-26 Thread Boris Tyukin
thanks Jayesh, replied via github On Thu, Jan 26, 2017 at 7:29 PM, Jayesh Senjaliya wrote: > Hi Boris, > > looks like bash_operator has same bug that ssh_execute_operator has, which > is it does not capture multi line output > > I have put up the fix for bash_oeprator as

Re: Airflow Meetup @ Paypal (San Jose)

2017-01-26 Thread Russell Jurney
Cool! On Wed, Jan 25, 2017 at 11:23 PM Jayesh Senjaliya wrote: > Hi Russell, > > yes, I will be presenting from Paypal side. > Once i have official approval from Paypal, I will sent out email. > I am basically going by the steps what Siddharth outlined earlier in the >

Airflow 1.8.0 BETA 4

2017-01-26 Thread Bolke de Bruin
Hi All, I have made the FOURTH beta of Airflow 1.8.0 available at: https://dist.apache.org/repos/dist/dev/incubator/airflow/ , public keys are available at https://dist.apache.org/repos/dist/release/incubator/airflow/

Re: Flow-based Airflow?

2017-01-26 Thread Jeremiah Lowin
Arthur, That's an excellent point. I think it touches on some ambiguity in how we conceptualize these two types of workflows (task-driven and data-driven) and so I think we should develop a more concrete vocabulary for describing them. Here's my attempt. We are all comfortable with a

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Bolke de Bruin
Sorry, it was working due to my unclean build environment. Please try https://github.com/apache/incubator-airflow/pull/2023 , +1 if it works and I will make Beta 4 available. Bolke > On 26 Jan 2017, at 13:27, Alex Van Boxel wrote: > > so, yes. doesn't work: > > I did: > -

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Alex Van Boxel
so, yes. doesn't work: I did: - create new conda environment - python setup.py install - rm airflow.db (sqllite) - airflow initdb same error: self._revision_map File "/Users/alexvanboxel/miniconda2/lib/python2.7/site-packages/alembic-0.8.10-py2.7.egg/alembic/util/langhelpers.py", line 240,

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Bolke de Bruin
I’ll have a look at it, and if needed release Beta 4. - Bolke > On 26 Jan 2017, at 10:53, Alex Van Boxel wrote: > > About the database error: starting from scratch also gives the same error: > > Fresh install. Delete airflow.db sqllite db. And the : airflow initdb > > same

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Alex Van Boxel
About the database error: starting from scratch also gives the same error: Fresh install. Delete airflow.db sqllite db. And the : airflow initdb same error as above. On Thu, Jan 26, 2017 at 10:12 AM Alex Van Boxel wrote: > Not directly one I can share. I'll spend some time

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Alex Van Boxel
Not directly one I can share. I'll spend some time looking at it. I'll try to create a unittest of it. On Thu, Jan 26, 2017 at 9:47 AM Bolke de Bruin wrote: > Do you have a example dag for this? That makes it easier to work on. > > Bolke > > Sent from my iPhone > > > On 26

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Bolke de Bruin
Do you have a example dag for this? That makes it easier to work on. Bolke Sent from my iPhone > On 26 Jan 2017, at 08:36, Alex Van Boxel wrote: > > Another thing that I noticed (but observed it in beta 2 as well). Is the > following: > > - The following trigger should not

Re: Airflow 1.8.0 BETA 3

2017-01-26 Thread Bolke de Bruin
It is only an issue when you go from beta 2 to beta 3. Not if you upgrade from 1.7.1. Sent from my iPhone > On 26 Jan 2017, at 08:38, Alex Van Boxel wrote: > > The downgrade is ok for testing, but we can't release with this change > (can't expect people to install a beta