Re: [Tutor] threading tutorial

[Cameron Simpson]

On 25May2017 11:52, Michael C  wrote:

Right now all i need is to grab 3 values from 3 variables before killing a
thread, like this:

def stuff():
  do stuff,
   get values, (x,y,d)

# main code
startthread(stuff(), blah)
# if else need to sleep or kill the thread, and because I'll restart the
thread later, I'd like to get the values from the thread, say x,y,d in
order to restart the thread.
loop.

Therefore, how do I get a few values from a few variables from the thread
and then close it?

Threading is very new to me, so I have to be very diligent.


You always need to be diligent with threads :-)

Can you explain why you need to use a thread for this? The first cut of your 
program looks like you could do it with a ordinary function:


 def stuff():
   ... compute x, y, z ...
   return x, y, z

 def main():
   x, y, z = stuff()

OTOH, your later description suggests that you want to kick off a thread to 
work on something, and have your main program let it run, or pause it. The 
implication is that x, y, z represent the thread state allowing you to restart 
it from scratch at the same point where you paused/stopped things.


There are a few different ways to manage that scenario. But first, write 
yourself a small Thread based program to familiarise yourself with threads. For 
example (untested):


 from __future__ import print_function
 from time import sleep
 from threading import Thread

 def thread_main_body():
   print("thread started")
   for n in range(20):
 print("thread", n)
 sleep(0.3)
   print("thread done")

 def main():
   print("main")
   T = Thread(target=thread_main_body)
   print("main: Thread created but _not_ started")
   sleep(1)
   T.start()
   print("thread started")
   for n in range(10):
 print("main", n)
 sleep(0.4)
   print("main program waiting for thread")
   T.join()
   print("main program done")

You should see the main thread and your subthread outputs interleaved. The 
sleeps are just to ensure some interleaving and to give good interactive feel.  
It should run for about 8 seconds overall. Make sure you're happy you 
understand what is happening, why, and when.


There are a few things you need to keep in mind with threads (in Python, and to 
a degree in other languages):


1: You can't kill/stop a Thread. Instead, the usual approach to to share some 
state with some kind of "running" flag, a boolean saying that the Thread's 
function should continue. Then the thread polls that regularly. Eg, if the 
thread function runs a main loop it might look like this:


 def fn():
   global run_me
   while run_me:
 ... do some work ...

and then elsewhere you go:

 global run_me
 run_me = True
 ... create and start the Thread ...
 ... later ...
 run_me = False
 T.join()

so effectively you ask the Thread to stop, and it obeys when it notices the 
change to "run_me". Using a global for this is pretty crube, and not the 
general approach, BTW.


2: Like any other function, the local varaibles to the thread function are not 
available outside. Thus the "global" hack above. So to share state you would 
usually make some kind of object with the state, and pass it in to the Thread 
when you create and start it:


 def fn(state):
   while state.run_me:
 ... do stuff, keep important things like results in "state" ...
 state.x = 1
 state.y = whatever

 class State(object):
   pass

 def main():
   state = State()
   state.run_me = True
   T = Thread(target=fn, args=(state,))
   T.start()
   for n in range(10):
 print("main", n, "x =", state.x, "y =", state.y)
 sleep(0.3)
   state.run_me = False
   T.join()

As I remarked, there are a few ways to approach your scenario. The above should 
get you started on one approach. Pausing can be done in a few ways, either by 
starting and stopping individual threads, one after another, or by starting one 
thread and using a mutex of some kind to cause it to suspend activity when 
needed. Yet another approach is corroutines, but I'd recommend getting 
threading understood first to avoid confusion.


Come back woith some functioning code and more questions.

Cheers,
Cameron Simpson 
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] threading tutorial

[Michael C]

Right now all i need is to grab 3 values from 3 variables before killing a
thread, like this:

def stuff():
   do stuff,
get values, (x,y,d)


# main code
startthread(stuff(), blah)
# if else need to sleep or kill the thread, and because I'll restart the
thread later, I'd like to get the values from the thread, say x,y,d in
order to restart the thread.
loop.


Therefore, how do I get a few values from a few variables from the thread
and then close it?

Threading is very new to me, so I have to be very diligent.

Thanks!




On Thu, May 25, 2017 at 11:06 AM Michael C 
wrote:

> Hi all:
>
>  I tried to google for tutorials of threading, but they are all
> equally confusing.
>  Does someone know of a good page or a book that talk about threading?
>
>
> thanks!
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] threading tutorial

[Michael C]

Hi all:

 I tried to google for tutorials of threading, but they are all equally
confusing.
 Does someone know of a good page or a book that talk about threading?


thanks!
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] airflow dag

[Peter Otten]

Alan Gauld via Tutor wrote:

>>dag=dag
> 
> I'm not sure what you think the line above does but
> in normal Python it would have zero effect.

The context makes it a keyword argument.

dag = DAG(...)
...
SimpleHttpOperator(
...
dag=dag
)


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] airflow dag

[Peter Otten]

shubham goyal wrote:

> He guys,
> 
> I want to ask that can we pass the parameters as commandline arguments in
> airflow when we are triggering the dag and access them inside the dag's
> python script/file.
> script:
 
> like this here i am trying to create a cluster but i need to pass password
> as cli arguments when i trigger the dag. can we do that. please help.

Have a look at argparse:



Provided

(1) you are OK with the password being echoed when you type it
(2) the password doesn't require escaping to be a valid part of the
`endpoint` argument

the code might look like this:

> from airflow import DAG
> from datetime import datetime,timedelta

  import argparse

  parser = argparse.ArgumentParser()
  parser.add_argument("passwd")
  args = parser.parse_args()

> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime.now(),
> 'email': ['airf...@airflow.com'],
> 'email_on_failure': False,
> 'email_on_retry': False
> }
> MAIN_DAG='check_dag'
> dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
> schedule_interval=None)
> 
> with open(file, "r") as f:
> payload = f.read()  # Reading the json data from a file
> SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
> task_id='cluster_create',
> method='POST',
> http_conn_id='qubole_default',
> # for directing to https://qa.qubole.net/api

  endpoint='/v2/clusters?auth_token=%s' % args.passwd,

> data=payload,
> headers={"Content-Type": "application/json"},

  params={'auth_token': args.passwd},

> response_check=lambda response: True if response.status_code ==
> 200
> else False,
> dag=dag
> )
> 

To address (1) there is 
(2) can be solved with 



___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] airflow dag

[Alan Gauld via Tutor]

On 25/05/17 13:15, shubham goyal wrote:
> He guys,
> 
> I want to ask that can we pass the parameters as commandline arguments in
> airflow when we are triggering the dag and access them inside the dag's
> python script/file.

I've no idea what a dag is.
This list is for people learning Python as a programming
language but it has no knowledge(necessarily) of other
environments that use Python. So you need to tell us more.

In generic terms Python can read command line arguments
in the sys.argv list. So you could try adding a line like

import sys
...
print sys.argv


And see if your arguments appear there.
Otherwise you'll need to find a dsag forum and try asking there.
Or get very lucky and find someone on this list that speaks dag.

If they are present in sys.aergv then there are several modules
that can parse the arguments looking for option flags etc.
argparse is the current official recommendation.


> script:
> 
> from airflow import DAG
> from datetime import datetime,timedelta
> default_args = {
> 'owner': 'airflow',
> 'depends_on_past': False,
> 'start_date': datetime.now(),
> 'email': ['airf...@airflow.com'],
> 'email_on_failure': False,
> 'email_on_retry': False
> }
> MAIN_DAG='check_dag'
> dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
> schedule_interval=None)
> 
> with open(file, "r") as f:
> payload = f.read()  # Reading the json data from a file
> SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
> task_id='cluster_create',
> method='POST',
> http_conn_id='qubole_default',
> # for directing to https://qa.qubole.net/api
> endpoint='/v2/clusters?auth_token=%s' % (passwd),
> data=payload,
> headers={"Content-Type": "application/json"},
> params={'auth_token': passwd},
> response_check=lambda response: True if response.status_code == 200
> else False,
> dag=dag

I'm not sure what you think the line above does but
in normal Python it would have zero effect.

> like this here i am trying to create a cluster but i need to pass password
> as cli arguments when i trigger the dag. can we do that. please help.

You need to explain what you mean bearing in mind we don't
know anything about dag, or clusters.

-- 
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/
http://www.amazon.com/author/alan_gauld
Follow my photo-blog on Flickr at:
http://www.flickr.com/photos/alangauldphotos


___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

[Tutor] airflow dag

[shubham goyal]

He guys,

I want to ask that can we pass the parameters as commandline arguments in
airflow when we are triggering the dag and access them inside the dag's
python script/file.
script:

from airflow import DAG
from datetime import datetime,timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)

with open(file, "r") as f:
payload = f.read()  # Reading the json data from a file
SimpleHttpOperator(  # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)

like this here i am trying to create a cluster but i need to pass password
as cli arguments when i trigger the dag. can we do that. please help.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor