Re: How Airflow import modules as it executes the tasks

[alireza . khoshkbari]

Thanks for the explanation, really helpful.

Cheers,
Ali

On 2018/05/16 03:27:27, Ruiqin Yang  wrote: 
> You are right, but that's within the same process. The way each operator
> gets executed is that one `airflow run` command get generated and sent to
> the local executor, executor spun up subprocesses to run `airflow run
> --raw` (which parses the file again and calls the operator.execute()). Thus
> each task would have its own process that parses the *.py file and import
> the module multiple times.
> 
> Hope this helped, cheers
> Kevin Y
> 
> On Tue, May 15, 2018 at 7:57 PM, alireza.khoshkb...@gmail.com <
> alireza.khoshkb...@gmail.com> wrote:
> 
> > Thanks Kevin. Yes, I'm importing db in different operators. That said, my
> > understanding is if a module has already been imported, it's not loaded
> > again even if you try to import it again (and I reckon this is why in
> > Python Singleton is not commonly used). Is that right?
> >
> > On 2018/05/16 02:34:18, Ruiqin Yang  wrote:
> > > Not exactly answering your question but the reason db.py is loaded in
> > each
> > > task might be because you have something like `import db` in each of your
> > > *.py file, and Airflow spun up one process to parse one *.py file, thus
> > > your db.py was loaded multiple time.
> > >
> > > I'm not sure how you can share the connection pool if it is created
> > within
> > > the same process your operator is in, since Airflow would spun up one
> > > process for each task even it is LocalExecutor. You might have to make
> > the
> > > connection pool available to outside processes (this part Idk how it can
> > be
> > > done) to be able to share it.
> > >
> > > Cheers,
> > > Kevin Y
> > >
> > > On Tue, May 15, 2018 at 6:21 PM, alireza.khoshkb...@gmail.com <
> > > alireza.khoshkb...@gmail.com> wrote:
> > >
> > > > To start off, here is my project structure:
> > > > ├── dags
> > > > │   ├── __init__.py
> > > > │   ├── core
> > > > │   │   ├── __init__.py
> > > > │   │   ├── operators
> > > > │   │   │   ├── __init__.py
> > > > │   │   │   ├── first_operator.py
> > > > │   │   └── util
> > > > │   │   ├── __init__.py
> > > > │   │   ├── db.py
> > > > │   ├── my_dag.py
> > > >
> > > > Here is the versions and details of the airflow docker setup:
> > > >
> > > > In my dag in different tasks I'm connecting to db (not Airflow db).
> > I've
> > > > setup db connection pooling,  I expected that my db.py would be be
> > loaded
> > > > once across the DagRun. However, in the log I can see that each task
> > > > imports the module and new db connections made by each and every task.
> > I
> > > > can see that db.py is loaded in each task by having the line below in
> > db.py:
> > > >
> > > > logging.info("I was loaded {}".format(random.randint(0,100)))
> > > >
> > > > I understand that each operator can technically be run in a separate
> > > > machine and it does make sense that each task runs sort of
> > independently.
> > > > However, not sure that if this does apply in case of using
> > LocalExecutor.
> > > > Now the question is, how I can share the resources (db connections)
> > across
> > > > tasks using LocalExecutor.
> > > >
> > >
> >
> 

Python3 and sensors module

[Cindy Rottinghuis]

Hi,

Are there any plans to update the HDFS_hook.py script to remove the reference 
to the snakebite python library? I’d like to run airflow on python3, and this 
is causing some issues.   The hdfs_hook script is referenced in the sensors 
module.

Any suggestions?

Thanks,
Cindy

Re: How Airflow import modules as it executes the tasks

[Ruiqin Yang]

You are right, but that's within the same process. The way each operator
gets executed is that one `airflow run` command get generated and sent to
the local executor, executor spun up subprocesses to run `airflow run
--raw` (which parses the file again and calls the operator.execute()). Thus
each task would have its own process that parses the *.py file and import
the module multiple times.

Hope this helped, cheers
Kevin Y

On Tue, May 15, 2018 at 7:57 PM, alireza.khoshkb...@gmail.com <
alireza.khoshkb...@gmail.com> wrote:

> Thanks Kevin. Yes, I'm importing db in different operators. That said, my
> understanding is if a module has already been imported, it's not loaded
> again even if you try to import it again (and I reckon this is why in
> Python Singleton is not commonly used). Is that right?
>
> On 2018/05/16 02:34:18, Ruiqin Yang  wrote:
> > Not exactly answering your question but the reason db.py is loaded in
> each
> > task might be because you have something like `import db` in each of your
> > *.py file, and Airflow spun up one process to parse one *.py file, thus
> > your db.py was loaded multiple time.
> >
> > I'm not sure how you can share the connection pool if it is created
> within
> > the same process your operator is in, since Airflow would spun up one
> > process for each task even it is LocalExecutor. You might have to make
> the
> > connection pool available to outside processes (this part Idk how it can
> be
> > done) to be able to share it.
> >
> > Cheers,
> > Kevin Y
> >
> > On Tue, May 15, 2018 at 6:21 PM, alireza.khoshkb...@gmail.com <
> > alireza.khoshkb...@gmail.com> wrote:
> >
> > > To start off, here is my project structure:
> > > ├── dags
> > > │   ├── __init__.py
> > > │   ├── core
> > > │   │   ├── __init__.py
> > > │   │   ├── operators
> > > │   │   │   ├── __init__.py
> > > │   │   │   ├── first_operator.py
> > > │   │   └── util
> > > │   │   ├── __init__.py
> > > │   │   ├── db.py
> > > │   ├── my_dag.py
> > >
> > > Here is the versions and details of the airflow docker setup:
> > >
> > > In my dag in different tasks I'm connecting to db (not Airflow db).
> I've
> > > setup db connection pooling,  I expected that my db.py would be be
> loaded
> > > once across the DagRun. However, in the log I can see that each task
> > > imports the module and new db connections made by each and every task.
> I
> > > can see that db.py is loaded in each task by having the line below in
> db.py:
> > >
> > > logging.info("I was loaded {}".format(random.randint(0,100)))
> > >
> > > I understand that each operator can technically be run in a separate
> > > machine and it does make sense that each task runs sort of
> independently.
> > > However, not sure that if this does apply in case of using
> LocalExecutor.
> > > Now the question is, how I can share the resources (db connections)
> across
> > > tasks using LocalExecutor.
> > >
> >
>

答复: How Airflow import modules as it executes the tasks

[Song Liu]

I think there might be two ways:

1. Setup the connections via. the Airflow UI: 
http://airflow.readthedocs.io/en/latest/configuration.html#connections, I guess 
this could be done in your code also.

2. Put your connection setup into a operator at the begin of your dag


发件人: alireza.khoshkbari@ 
发送时间: 2018年5月16日 1:21
收件人: d...@airflow.apache.org
主题: How Airflow import modules as it executes the tasks

To start off, here is my project structure:
├── dags
│   ├── __init__.py
│   ├── core
│   │   ├── __init__.py
│   │   ├── operators
│   │   │   ├── __init__.py
│   │   │   ├── first_operator.py
│   │   └── util
│   │   ├── __init__.py
│   │   ├── db.py
│   ├── my_dag.py

Here is the versions and details of the airflow docker setup:

In my dag in different tasks I'm connecting to db (not Airflow db). I've setup 
db connection pooling,  I expected that my db.py would be be loaded once across 
the DagRun. However, in the log I can see that each task imports the module and 
new db connections made by each and every task. I can see that db.py is loaded 
in each task by having the line below in db.py:

logging.info("I was loaded {}".format(random.randint(0,100)))

I understand that each operator can technically be run in a separate machine 
and it does make sense that each task runs sort of independently. However, not 
sure that if this does apply in case of using LocalExecutor. Now the question 
is, how I can share the resources (db connections) across tasks using 
LocalExecutor.

Re: How Airflow import modules as it executes the tasks

[Ruiqin Yang]

Not exactly answering your question but the reason db.py is loaded in each
task might be because you have something like `import db` in each of your
*.py file, and Airflow spun up one process to parse one *.py file, thus
your db.py was loaded multiple time.

I'm not sure how you can share the connection pool if it is created within
the same process your operator is in, since Airflow would spun up one
process for each task even it is LocalExecutor. You might have to make the
connection pool available to outside processes (this part Idk how it can be
done) to be able to share it.

Cheers,
Kevin Y

On Tue, May 15, 2018 at 6:21 PM, alireza.khoshkb...@gmail.com <
alireza.khoshkb...@gmail.com> wrote:

> To start off, here is my project structure:
> ├── dags
> │   ├── __init__.py
> │   ├── core
> │   │   ├── __init__.py
> │   │   ├── operators
> │   │   │   ├── __init__.py
> │   │   │   ├── first_operator.py
> │   │   └── util
> │   │   ├── __init__.py
> │   │   ├── db.py
> │   ├── my_dag.py
>
> Here is the versions and details of the airflow docker setup:
>
> In my dag in different tasks I'm connecting to db (not Airflow db). I've
> setup db connection pooling,  I expected that my db.py would be be loaded
> once across the DagRun. However, in the log I can see that each task
> imports the module and new db connections made by each and every task. I
> can see that db.py is loaded in each task by having the line below in db.py:
>
> logging.info("I was loaded {}".format(random.randint(0,100)))
>
> I understand that each operator can technically be run in a separate
> machine and it does make sense that each task runs sort of independently.
> However, not sure that if this does apply in case of using LocalExecutor.
> Now the question is, how I can share the resources (db connections) across
> tasks using LocalExecutor.
>

Re: Cannot access https://cms.apache.org/incubator/publish

[Naik Kaxil]

Hi Sid,

I think Jenkins trigger a build on incubator svn repo daily as seen here 
https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/426/console

When I checked http://incubator.apache.org/projects/airflow.html the page has 
already been updated.

However, I still tried to follow the steps in the last email but I got lost of 
jbake errors.

Regards,
Kaxil


On 15/05/2018, 14:51, "siddharth anand"  wrote:

Kaxil,
Can you try these steps and update the airflow wiki (committer guide) based 
on your findings?

-s

Sent from Sid's iPhone 



Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com
Begin forwarded message:

> From: Martin Gainty 
> Date: May 15, 2018 at 4:21:13 AM PDT
> To: "san...@apache.org" 
> Subject: Re: Cannot access https://cms.apache.org/incubator/publish
> 
> Hi Anand
> 
> apparently the new URL is incubator.apache.org as I could not find any 
references to cms ..here is jbake readme
> # Apache Incubator Website
> 2
> 3
>  ## Prerequisites
> 4
> 5
>  The website is built using JBake and a Groovy template.  The builds for 
the website do require internet access.
> 6
> 7
>  - Install JBake from http://jbake.org/download.html
> 8
>  - Create an environment variable `JBAKE_HOME` pointing to your JBake 
installation
> 9
>  - Ensure that you have a JVM locally, e.g. 
[OpenJDK](http://openjdk.java.net/install/)
> 10
> 11
>  ## Building & Running the site
> 12
> 13
>  There is a custom `bake.sh` file that is used to build the website.  You 
can call it with any of the [arguments you would pass to 
jbake](http://jbake.org/docs/2.5.1/#bake_command).
> 14
>  The easiest way to use it is to run `./bake.sh -b -s` this will start up 
JBake in a watching mode as you make changes it will refresh after a short 
period of time.
> 15
>  While working with it locally, you'll notice that the site URLs redirect 
to `incubator.apache.org`, to change this edit `jbake.properties` and uncomment 
the line referencing `localhost`
> 16
> 17
>  ## Jenkins Setup
> 18
> 19
>  Commits to the `jbake-site` branch are automatically checked out and 
built using `build_site.sh`.  Once this goes live those commits will go against 
`master`.  The jenkins job can be found at 
[https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/](https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/)
> 20
>  The result of the commits are pushed to the `asf-site` branch which are 
then published using `gitwcsub`
> 21
> 22
>  ## Asciidoctor
> 23
> 24
>  Most of the pages in the site are written using Asciidoctor.  While it 
is a form of asciidoc it does have some [syntax differences that are worth 
reviewing](http://asciidoctor.org/docs/asciidoc-syntax-quick-reference/)
> 25
> 26
>  ## Groovy Templates
> 27
> 28
>  The site templates are written in groovy scripts.  Even though the files 
end with `.gsp` they are not GSP files and do not have access to tag libraries. 
 You can run custom code in them, similar to what is done in 
[homepage.gsp](templates/homepage.gsp)
> 
> if you have hard requirement to accessing cms.apache.org 
> write a request for username/password to cms.apache.org to site admin: 
d...@apache.org
> 
> Good Luck!
> Martin 
> __ 
>  
> 
> 
> From: John D. Ament 
> Sent: Monday, May 14, 2018 10:01 PM
> To: gene...@incubator.apache.org
> Cc: san...@apache.org
> Subject: Re: Cannot access https://cms.apache.org/incubator/publish
>  
> The Incubator website is no longer managed via CMS.  Please review
> https://incubator.apache.org/guides/website.html
> Updating the top-level Incubator website
> incubator.apache.org
> The Incubator website is generated by the incubator git repository. The 
primary document format is asciidoc, templates are based on gsp, and we use 
jbake to build it. You can edit files directly on github and raise a pull 
request or just checkout the repository at https://git-wip-us.apache.org/repos 
...
> 
> 
> 
> John
> 
> On Mon, May 14, 2018 at 9:17 PM Martin Gainty  wrote:
> 
> > Hi Sid
> >
> >
> > as long as you have JavaScript enabled in the browser
> > and you gave the same issue with curl then AFAIK its a permissions error
> >
> >
> > can you contract admin or webmaster and have them email you valid
> > username/password
> >
> > ?
> >
> > Martin
> > __

Re: Airflow with Celery

[David Capwell]

What I find is that when celery rejects we hit this.  For us we don't do
work on the hosts so solve by over provisioning tasks in celery

On Tue, May 15, 2018, 6:30 AM Andy Cooper  wrote:

> I have had very similar issues when there was a problem with the connection
> string pointing to the message broker. Triple check those connection
> strings and attempt to connect outside of airflow.
>
> On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa 
> wrote:
>
> > Hi all,
> >
> > I have been using airflow with Celery executor in the background
> >
> > https://hastebin.com/sipecovomi.ini --> airflow.cfg
> >
> > https://hastebin.com/urutokuvoq.py   --> The dag I have been using
> >
> >
> >
> > This shows that the dag is always in running state.
> >
> >
> >
> >
> > Airflow flower shows nothing in the tasks or in the broker.
> >
> >
> > Did I miss anything can anyone help me in this regard.
> >
> >
> > --
> > Cheers !!!
> > Goutham Pratapa
> >
>

Re: Cannot access https://cms.apache.org/incubator/publish

[Naik Kaxil]

Sure I will do this and update the wiki

On 15/05/2018, 14:51, "siddharth anand"  wrote:

Kaxil,
Can you try these steps and update the airflow wiki (committer guide) based 
on your findings?

-s

Sent from Sid's iPhone 



Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com
Begin forwarded message:

> From: Martin Gainty 
> Date: May 15, 2018 at 4:21:13 AM PDT
> To: "san...@apache.org" 
> Subject: Re: Cannot access https://cms.apache.org/incubator/publish
> 
> Hi Anand
> 
> apparently the new URL is incubator.apache.org as I could not find any 
references to cms ..here is jbake readme
> # Apache Incubator Website
> 2
> 3
>  ## Prerequisites
> 4
> 5
>  The website is built using JBake and a Groovy template.  The builds for 
the website do require internet access.
> 6
> 7
>  - Install JBake from http://jbake.org/download.html
> 8
>  - Create an environment variable `JBAKE_HOME` pointing to your JBake 
installation
> 9
>  - Ensure that you have a JVM locally, e.g. 
[OpenJDK](http://openjdk.java.net/install/)
> 10
> 11
>  ## Building & Running the site
> 12
> 13
>  There is a custom `bake.sh` file that is used to build the website.  You 
can call it with any of the [arguments you would pass to 
jbake](http://jbake.org/docs/2.5.1/#bake_command).
> 14
>  The easiest way to use it is to run `./bake.sh -b -s` this will start up 
JBake in a watching mode as you make changes it will refresh after a short 
period of time.
> 15
>  While working with it locally, you'll notice that the site URLs redirect 
to `incubator.apache.org`, to change this edit `jbake.properties` and uncomment 
the line referencing `localhost`
> 16
> 17
>  ## Jenkins Setup
> 18
> 19
>  Commits to the `jbake-site` branch are automatically checked out and 
built using `build_site.sh`.  Once this goes live those commits will go against 
`master`.  The jenkins job can be found at 
[https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/](https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/)
> 20
>  The result of the commits are pushed to the `asf-site` branch which are 
then published using `gitwcsub`
> 21
> 22
>  ## Asciidoctor
> 23
> 24
>  Most of the pages in the site are written using Asciidoctor.  While it 
is a form of asciidoc it does have some [syntax differences that are worth 
reviewing](http://asciidoctor.org/docs/asciidoc-syntax-quick-reference/)
> 25
> 26
>  ## Groovy Templates
> 27
> 28
>  The site templates are written in groovy scripts.  Even though the files 
end with `.gsp` they are not GSP files and do not have access to tag libraries. 
 You can run custom code in them, similar to what is done in 
[homepage.gsp](templates/homepage.gsp)
> 
> if you have hard requirement to accessing cms.apache.org 
> write a request for username/password to cms.apache.org to site admin: 
d...@apache.org
> 
> Good Luck!
> Martin 
> __ 
>  
> 
> 
> From: John D. Ament 
> Sent: Monday, May 14, 2018 10:01 PM
> To: gene...@incubator.apache.org
> Cc: san...@apache.org
> Subject: Re: Cannot access https://cms.apache.org/incubator/publish
>  
> The Incubator website is no longer managed via CMS.  Please review
> https://incubator.apache.org/guides/website.html
> Updating the top-level Incubator website
> incubator.apache.org
> The Incubator website is generated by the incubator git repository. The 
primary document format is asciidoc, templates are based on gsp, and we use 
jbake to build it. You can edit files directly on github and raise a pull 
request or just checkout the repository at https://git-wip-us.apache.org/repos 
...
> 
> 
> 
> John
> 
> On Mon, May 14, 2018 at 9:17 PM Martin Gainty  wrote:
> 
> > Hi Sid
> >
> >
> > as long as you have JavaScript enabled in the browser
> > and you gave the same issue with curl then AFAIK its a permissions error
> >
> >
> > can you contract admin or webmaster and have them email you valid
> > username/password
> >
> > ?
> >
> > Martin
> > __
> >
> >
> >
> > 
> > From: Sid Anand 
> > Sent: Monday, May 14, 2018 9:12 PM
> > To: Martin Gainty
> > Cc: gene...@incubator.apache.org
> > Subject: Re: Cannot access https://cms.apache.org/incubator/publish
> >
> > So, perhaps I (sanand) don't 

Fwd: Cannot access https://cms.apache.org/incubator/publish

[siddharth anand]

Kaxil,
Can you try these steps and update the airflow wiki (committer guide) based on 
your findings?

-s

Sent from Sid's iPhone 

Begin forwarded message:

> From: Martin Gainty 
> Date: May 15, 2018 at 4:21:13 AM PDT
> To: "san...@apache.org" 
> Subject: Re: Cannot access https://cms.apache.org/incubator/publish
> 
> Hi Anand
> 
> apparently the new URL is incubator.apache.org as I could not find any 
> references to cms ..here is jbake readme
> # Apache Incubator Website
> 2
> 3
>  ## Prerequisites
> 4
> 5
>  The website is built using JBake and a Groovy template.  The builds for the 
> website do require internet access.
> 6
> 7
>  - Install JBake from http://jbake.org/download.html
> 8
>  - Create an environment variable `JBAKE_HOME` pointing to your JBake 
> installation
> 9
>  - Ensure that you have a JVM locally, e.g. 
> [OpenJDK](http://openjdk.java.net/install/)
> 10
> 11
>  ## Building & Running the site
> 12
> 13
>  There is a custom `bake.sh` file that is used to build the website.  You can 
> call it with any of the [arguments you would pass to 
> jbake](http://jbake.org/docs/2.5.1/#bake_command).
> 14
>  The easiest way to use it is to run `./bake.sh -b -s` this will start up 
> JBake in a watching mode as you make changes it will refresh after a short 
> period of time.
> 15
>  While working with it locally, you'll notice that the site URLs redirect to 
> `incubator.apache.org`, to change this edit `jbake.properties` and uncomment 
> the line referencing `localhost`
> 16
> 17
>  ## Jenkins Setup
> 18
> 19
>  Commits to the `jbake-site` branch are automatically checked out and built 
> using `build_site.sh`.  Once this goes live those commits will go against 
> `master`.  The jenkins job can be found at 
> [https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/](https://builds.apache.org/view/H-L/view/Incubator/job/Incubator%20Site/)
> 20
>  The result of the commits are pushed to the `asf-site` branch which are then 
> published using `gitwcsub`
> 21
> 22
>  ## Asciidoctor
> 23
> 24
>  Most of the pages in the site are written using Asciidoctor.  While it is a 
> form of asciidoc it does have some [syntax differences that are worth 
> reviewing](http://asciidoctor.org/docs/asciidoc-syntax-quick-reference/)
> 25
> 26
>  ## Groovy Templates
> 27
> 28
>  The site templates are written in groovy scripts.  Even though the files end 
> with `.gsp` they are not GSP files and do not have access to tag libraries.  
> You can run custom code in them, similar to what is done in 
> [homepage.gsp](templates/homepage.gsp)
> 
> if you have hard requirement to accessing cms.apache.org 
> write a request for username/password to cms.apache.org to site admin: 
> d...@apache.org
> 
> Good Luck!
> Martin 
> __ 
>  
> 
> 
> From: John D. Ament 
> Sent: Monday, May 14, 2018 10:01 PM
> To: gene...@incubator.apache.org
> Cc: san...@apache.org
> Subject: Re: Cannot access https://cms.apache.org/incubator/publish
>  
> The Incubator website is no longer managed via CMS.  Please review
> https://incubator.apache.org/guides/website.html
> Updating the top-level Incubator website
> incubator.apache.org
> The Incubator website is generated by the incubator git repository. The 
> primary document format is asciidoc, templates are based on gsp, and we use 
> jbake to build it. You can edit files directly on github and raise a pull 
> request or just checkout the repository at 
> https://git-wip-us.apache.org/repos ...
> 
> 
> 
> John
> 
> On Mon, May 14, 2018 at 9:17 PM Martin Gainty  wrote:
> 
> > Hi Sid
> >
> >
> > as long as you have JavaScript enabled in the browser
> > and you gave the same issue with curl then AFAIK its a permissions error
> >
> >
> > can you contract admin or webmaster and have them email you valid
> > username/password
> >
> > ?
> >
> > Martin
> > __
> >
> >
> >
> > 
> > From: Sid Anand 
> > Sent: Monday, May 14, 2018 9:12 PM
> > To: Martin Gainty
> > Cc: gene...@incubator.apache.org
> > Subject: Re: Cannot access https://cms.apache.org/incubator/publish
> >
> > So, perhaps I (sanand) don't have the necessary permissions?
> > -s
> >
> > On Mon, May 14, 2018 at 6:11 PM, Sid Anand  wrote:
> >
> > > I get the same error (I've hidden my password).
> > >
> > > sianand@LM-SJN-21002367:~ $ curl  --user sanand:
> > > https://cms.apache.org/incubator/publish
> > >
> > > 
> > >
> > > 
> > >
> > > 404 Not Found
> > >
> > > 
> > >
> > > Page Not Found
> > >
> > > The requested URL was not found on this server. If you are trying to
> > > edit a CMS-driven page, your local working copy may have been pruned.
> > >
> > > Please go to cms.apache.org,
> > find
> > > your project, and click on 'force new working copy' to create a new
> > >
> > > working 

Re: Airflow with Celery

[Andy Cooper]

I have had very similar issues when there was a problem with the connection
string pointing to the message broker. Triple check those connection
strings and attempt to connect outside of airflow.

On Tue, May 15, 2018 at 9:27 AM Goutham Pratapa 
wrote:

> Hi all,
>
> I have been using airflow with Celery executor in the background
>
> https://hastebin.com/sipecovomi.ini --> airflow.cfg
>
> https://hastebin.com/urutokuvoq.py   --> The dag I have been using
>
>
>
> This shows that the dag is always in running state.
>
>
>
>
> Airflow flower shows nothing in the tasks or in the broker.
>
>
> Did I miss anything can anyone help me in this regard.
>
>
> --
> Cheers !!!
> Goutham Pratapa
>

Airflow with Celery

[Goutham Pratapa]

Hi all,

I have been using airflow with Celery executor in the background

https://hastebin.com/sipecovomi.ini --> airflow.cfg

https://hastebin.com/urutokuvoq.py   --> The dag I have been using



This shows that the dag is always in running state.




Airflow flower shows nothing in the tasks or in the broker.


Did I miss anything can anyone help me in this regard.


-- 
Cheers !!!
Goutham Pratapa

Re: Facing SSL issue with dev/airflow-pr tool

[Sumit Maheshwari]

Thanks for the reply Kaxil.

Yeah, I too figured out that issue is with my laptop's OpenSSL lib and
python installed on it. So upgraded OpenSSL and installed python3.6, and
created a new virtualenv for now.



On Tue, May 15, 2018 at 1:07 PM, Naik Kaxil  wrote:

> Hi Sumit,
>
> I think it is unrelated to the PR tool. Please check the below link:
>
> https://stackoverflow.com/questions/44316292/ssl-
> sslerror-tlsv1-alert-protocol-version
>
> Try running: pip install requests[security]
>
> Regards,
> Kaxil
>
> On 15/05/2018, 07:46, "Sumit Maheshwari"  wrote:
>
> Hi folks,
>
> I am getting *SSL: TLSV1_ALERT_PROTOCOL_VERSION* errors when running
> dev/airflow-pr tool. My pip version is 10.0.1 and python version is
> 2.7.10.
> Please let me know if anyone has any pointers to fix that.
>
>
>
> $ python
> Python 2.7.10 (default, Feb  7 2017, 00:08:15)
> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin
>
> $ pip -V
> pip 10.0.1 from
> /Users/sumitm/work/incubator-airflow/venv/lib/python2.7/
> site-packages/pip
> (python 2.7)
>
> $ pip list |grep urllib3
> urllib3   1.22
>
>
> $ dev/airflow-pr merge 3300
> >> Running command: git rev-parse --abbrev-ref HEAD
> Resetting git to remove any changes
> >> Running command: git reset --hard
> Restoring head pointer to master
> >> Running command: git checkout master
> Already on 'master'
> >> Running command: git branch
> Traceback (most recent call last):
>   File "dev/airflow-pr", line 1084, in 
> cli()
>   File
> "/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/
> site-packages/click-6.7-py2.7.egg/click/core.py",
> line 722, in __call__
> return self.main(*args, **kwargs)
>   File
> "/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/
> site-packages/click-6.7-py2.7.egg/click/core.py",
> line 697, in main
> rv = self.invoke(ctx)
>   File
> "/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/
> site-packages/click-6.7-py2.7.egg/click/core.py",
> line 1066, in invoke
> return _process_result(sub_ctx.command.invoke(sub_ctx))
>   File
> "/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/
> site-packages/click-6.7-py2.7.egg/click/core.py",
> line 895, in invoke
> return ctx.invoke(self.callback, **ctx.params)
>   File
> "/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/
> site-packages/click-6.7-py2.7.egg/click/core.py",
> line 535, in invoke
> return callback(*args, **kwargs)
>   File "dev/airflow-pr", line 1018, in merge
> main(pr_num, local=False)
>   File "dev/airflow-pr", line 829, in main
> branches = get_json("%s/branches" % GITHUB_API_BASE)
>   File "dev/airflow-pr", line 115, in get_json
> response = urllib.urlopen(request).read().decode('utf-8')
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/urllib2.py",
> line 154, in urlopen
> return opener.open(url, data, timeout)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/urllib2.py",
> line 431, in open
> response = self._open(req, data)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/urllib2.py",
> line 449, in _open
> '_open', req)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/urllib2.py",
> line 409, in _call_chain
> result = func(*args)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/urllib2.py",
> line 1240, in https_open
> context=self._context)
>   File
> "/System/Library/Frameworks/Python.framework/Versions/2.7/
> lib/python2.7/urllib2.py",
> line 1197, in do_open
> raise URLError(err)
> urllib2.URLError:  tlsv1
> alert protocol version (_ssl.c:590)>
>
>
>
> Thanks,
> Sumit
>
>
>
>
> Kaxil Naik
>
> Data Reply
> 2nd Floor, Nova South
> 160 Victoria Street, Westminster
> London SW1E 5LB - UK
> phone: +44 (0)20 7730 6000
> k.n...@reply.com
> www.reply.com
>

Re: Facing SSL issue with dev/airflow-pr tool

[Naik Kaxil]

Hi Sumit,

I think it is unrelated to the PR tool. Please check the below link:

https://stackoverflow.com/questions/44316292/ssl-sslerror-tlsv1-alert-protocol-version

Try running: pip install requests[security]

Regards,
Kaxil

On 15/05/2018, 07:46, "Sumit Maheshwari"  wrote:

Hi folks,

I am getting *SSL: TLSV1_ALERT_PROTOCOL_VERSION* errors when running
dev/airflow-pr tool. My pip version is 10.0.1 and python version is 2.7.10.
Please let me know if anyone has any pointers to fix that.



$ python
Python 2.7.10 (default, Feb  7 2017, 00:08:15)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin

$ pip -V
pip 10.0.1 from
/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/site-packages/pip
(python 2.7)

$ pip list |grep urllib3
urllib3   1.22


$ dev/airflow-pr merge 3300
>> Running command: git rev-parse --abbrev-ref HEAD
Resetting git to remove any changes
>> Running command: git reset --hard
Restoring head pointer to master
>> Running command: git checkout master
Already on 'master'
>> Running command: git branch
Traceback (most recent call last):
  File "dev/airflow-pr", line 1084, in 
cli()
  File

"/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py",
line 722, in __call__
return self.main(*args, **kwargs)
  File

"/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py",
line 697, in main
rv = self.invoke(ctx)
  File

"/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py",
line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
  File

"/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py",
line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
  File

"/Users/sumitm/work/incubator-airflow/venv/lib/python2.7/site-packages/click-6.7-py2.7.egg/click/core.py",
line 535, in invoke
return callback(*args, **kwargs)
  File "dev/airflow-pr", line 1018, in merge
main(pr_num, local=False)
  File "dev/airflow-pr", line 829, in main
branches = get_json("%s/branches" % GITHUB_API_BASE)
  File "dev/airflow-pr", line 115, in get_json
response = urllib.urlopen(request).read().decode('utf-8')
  File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 154, in urlopen
return opener.open(url, data, timeout)
  File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 431, in open
response = self._open(req, data)
  File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 449, in _open
'_open', req)
  File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 409, in _call_chain
result = func(*args)
  File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 1240, in https_open
context=self._context)
  File

"/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py",
line 1197, in do_open
raise URLError(err)
urllib2.URLError: 



Thanks,
Sumit




Kaxil Naik 

Data Reply
2nd Floor, Nova South
160 Victoria Street, Westminster
London SW1E 5LB - UK 
phone: +44 (0)20 7730 6000
k.n...@reply.com
www.reply.com