Re: [Tutor] (no subject)
Use this link to install python package and make sure to add the path in environment variables (just click the option it asks when you run the setup, Add Python to path or something). Recent python packages include pip also so you will get your problem solved. https://www.python.org/ftp/python/3.6.4/python-3.6.4-amd64.exe In your case pip module is not installed or not added to windows environment variables thats why its showing pip not recognized as a command. do as mentioned and you should able to install whatever. On Wed, Mar 28, 2018 at 11:38 AM, wrote: > I want to install 3rd party module, ex openpyxl. > And I executed the following command in windows command prompt as follows: > pip install openpyxl > But pip is not recognized as executable command at windows. > I also tried the same way in python command line. > But the result is the same. > SyntaxError: invalid syntax > > What should I do to use openpyxl ? > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Not able to import Tensorflow in python script
Hey Alan, I found this issue on internet. It turns out that if you have some file named string.py in the same folder as script which is using tensorflow, it give this kind of error. I don't know why but this was happening. Now it's working . Thank you for your help. On Oct 26, 2017 1:58 AM, "Alan Gauld via Tutor" wrote: > On 25/10/17 19:24, shubham goyal wrote: > > I was trying to run this check script of tensorflow after installing the > > tensorflow successfully. > > Its not part of the standard library so I'll need to assume you > are correct and it installed correctly. However from some of the > errors below it looks as if you \also need numpy/SciPy installed > - have you done that too? > > If so then I hope somebody else can assist. > > You might find a tensorflow help forum or email address you > can ask too? > > > > "/home/goyal/tensorflow/lib/python3.5/site-packages/ > tensorflow/python/__init__.py", > > line 47, in > > import numpy as np > > File > > "/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/__init__.py", > > line 142, in > > from . import add_newdocs > > File > > "/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/ > add_newdocs.py", > > line 13, in > > from numpy.lib import add_newdoc > > File > > "/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/ > lib/__init__.py", > > line 8, in > > from .type_check import * > > File > > "/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/ > lib/type_check.py", > > line 11, in > > import numpy.core.numeric as _nx > > File > > "/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/ > core/__init__.py", > > line 74, in > > from numpy.testing.nosetester import _numpy_tester > > File > > "/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/ > testing/__init__.py", > > line 10, in > > from unittest import TestCase > > File "/usr/lib/python3.5/unittest/__init__.py", line 59, in > > from .case import (TestCase, FunctionTestCase, SkipTest, skip, > skipIf, > > File "/usr/lib/python3.5/unittest/case.py", line 6, in > > import logging > > File "/usr/lib/python3.5/logging/__init__.py", line 28, in > > from string import Template > > ImportError: cannot import name 'Template' > > Do you have the right version of numpy for your module? > > > -- > Alan G > Author of the Learn to Program web site > http://www.alan-g.me.uk/ > http://www.amazon.com/author/alan_gauld > Follow my photo-blog on Flickr at: > http://www.flickr.com/photos/alangauldphotos > > > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Not able to import Tensorflow in python script
Hello all,
I was trying to run this check script of tensorflow after installing the
tensorflow successfully. but it is not able to import tensorflow properly i
think. but sometimes in python
shell it gets imported. it is giving the error "can't import name template"
script
import tensorflow as tf
hello = tf.constant('Hello, TensorFlow!')
sess = tf.Session()
print(sess.run(hello))
errors:
[0, 1, 2, 3, 4]
at 0x7fc0298652b0>
0
1
2
3
4
Traceback (most recent call last):
File "a.py", line 1, in
import tensorflow as tf
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/tensorflow/__init__.py",
line 24, in
from tensorflow.python import *
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/tensorflow/python/__init__.py",
line 47, in
import numpy as np
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/__init__.py",
line 142, in
from . import add_newdocs
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/add_newdocs.py",
line 13, in
from numpy.lib import add_newdoc
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/lib/__init__.py",
line 8, in
from .type_check import *
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/lib/type_check.py",
line 11, in
import numpy.core.numeric as _nx
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/core/__init__.py",
line 74, in
from numpy.testing.nosetester import _numpy_tester
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/testing/__init__.py",
line 10, in
from unittest import TestCase
File "/usr/lib/python3.5/unittest/__init__.py", line 59, in
from .case import (TestCase, FunctionTestCase, SkipTest, skip, skipIf,
File "/usr/lib/python3.5/unittest/case.py", line 6, in
import logging
File "/usr/lib/python3.5/logging/__init__.py", line 28, in
from string import Template
ImportError: cannot import name 'Template'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in
apport_excepthook
from apport.fileutils import likely_packaged, get_recent_crashes
File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in
from apport.report import Report
File "/usr/lib/python3/dist-packages/apport/report.py", line 21, in
from urllib.request import urlopen
File "/usr/lib/python3.5/urllib/request.py", line 88, in
import http.client
File "/usr/lib/python3.5/http/client.py", line 71, in
import email.parser
File "/usr/lib/python3.5/email/parser.py", line 12, in
from email.feedparser import FeedParser, BytesFeedParser
File "/usr/lib/python3.5/email/feedparser.py", line 27, in
from email import message
File "/usr/lib/python3.5/email/message.py", line 16, in
from email import utils
File "/usr/lib/python3.5/email/utils.py", line 40, in
from email.charset import Charset
File "/usr/lib/python3.5/email/charset.py", line 15, in
import email.quoprimime
File "/usr/lib/python3.5/email/quoprimime.py", line 44, in
from string import ascii_letters, digits, hexdigits
ImportError: cannot import name 'ascii_letters'
Original exception was:
Traceback (most recent call last):
File "a.py", line 1, in
import tensorflow as tf
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/tensorflow/__init__.py",
line 24, in
from tensorflow.python import *
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/tensorflow/python/__init__.py",
line 47, in
import numpy as np
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/__init__.py",
line 142, in
from . import add_newdocs
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/add_newdocs.py",
line 13, in
from numpy.lib import add_newdoc
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/lib/__init__.py",
line 8, in
from .type_check import *
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/lib/type_check.py",
line 11, in
import numpy.core.numeric as _nx
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/core/__init__.py",
line 74, in
from numpy.testing.nosetester import _numpy_tester
File
"/home/goyal/tensorflow/lib/python3.5/site-packages/numpy/testing/__init__.py",
line 10, in
from unittest import TestCase
File "/usr/lib/python3.5/unittest/__init__.py", line 59, in
from .case import (TestCase, FunctionTestCase, SkipTest, skip, skipIf,
File "/usr/lib/python3.5/unittest/case.py", line 6, in
import logging
File "/usr/lib/python3.5/logging/__init__.py", line 28, in
from string import Template
ImportError: cannot import name 'Template'
please tell me what to do.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] dictionaries are same but returning false
On Jul 5, 2017 11:09 PM, "shubham goyal" wrote:
> Yha that's very smart. Only experience can drive you this way. Otherwise I
> was checking value by value.
>
> On Jul 5, 2017 10:21 PM, "Ashfaq" wrote:
>
> Hi Peter,
> The way you find the issue is really cool! Very cool! :)
>
> On Wed, Jul 5, 2017 at 6:10 PM, shubham goyal
> wrote:
>
>> Thank you Peter.
>> Silly mistakes 😀
>>
>> On Jul 5, 2017 5:10 PM, "Peter Otten" <__pete...@web.de> wrote:
>>
>> > shubham goyal wrote:
>> >
>> > > null=None
>> > > x={'_udp_options': None, '_icmp_options': None, 'attribute_map':
>> > > {'icmp_options': 'icmpOptions', 'protocol': 'protocol', 'source':
>> > > {'source',
>> > > 'tcp_options': 'tcpOptions', 'is_stateless': 'isStateless',
>> > 'udp_options':
>> > > 'udpOptions'}, '_is_stateless': False, 'swagger_types':
>> {'icmp_options':
>> > > 'IcmpOptions', 'protocol': 'str', 'source': 'str', 'tcp_options':
>> > > 'TcpOptions', 'is_stateless': 'bool', 'udp_options': 'UdpOptions'},
>> > > '_protocol': '6', '_source': '0.0.4.0/24', '_tcp_options': {
>> > > "destination_port_range": {
>> > > "max": "22",
>> > > "min": "22"
>> > > },
>> > > "source_port_range": null
>> > > }}
>> > >
>> > > y={'_udp_options': None, '_icmp_options': None, 'attribute_map':
>> > > {'icmp_options': 'icmpOptions', 'protocol': 'protocol', 'source':
>> > > {'source',
>> > > 'tcp_options': 'tcpOptions', 'is_stateless': 'isStateless',
>> > 'udp_options':
>> > > 'udpOptions'}, '_is_stateless': False, 'swagger_types':
>> {'icmp_options':
>> > > 'IcmpOptions', 'protocol': 'str', 'source': 'str', 'tcp_options':
>> > > 'TcpOptions', 'is_stateless': 'bool', 'udp_options': 'UdpOptions'},
>> > > '_protocol': '6', '_source': '0.0.4.0/24', '_tcp_options': {
>> > > "destination_port_range": {
>> > > "max": 22,
>> > > "min": 22
>> > > },
>> > > "source_port_range": null
>> > > }}
>> > > if x==y:
>> > > print "true"
>> > > else:
>> > > print "false"
>> > >
>> > >
>> > > These dictionaries are same exactly. but its returning false. i don't
>> > > understand
>> > > what to do?
>> >
>> > Let's narrow down the problem, with the help of the interactive
>> > interpreter:
>> >
>> > >>> changed = [k for k in x if x[k] != y[k]]
>> > >>> changed
>> > ['_tcp_options']
>> > >>> k, = changed
>> >
>> > A closer look:
>> >
>> > >>> x[k]
>> > {'source_port_range': None, 'destination_port_range': {'max': '22',
>> 'min':
>> > '22'}}
>> > >>> y[k]
>> > {'source_port_range': None, 'destination_port_range': {'max': 22, 'min':
>> > 22}}
>> >
>> > So x uses strings for min/max while y uses integers, and those do not
>> > compare equal in Python:
>> >
>> > >>> 22 == "22"
>> > False
>> >
>> > Once you fix this
>> >
>> > >>> x[k]["destination_port_range"]["max"] = 22
>> > >>> x[k]["destination_port_range"]["min"] = 22
>> >
>> > you get the expected result:
>> >
>> > >>> x == y
>> > True
>> >
>> >
>> > ___
>> > Tutor maillist - Tutor@python.org
>> > To unsubscribe or change subscription options:
>> > https://mail.python.org/mailman/listinfo/tutor
>> >
>> ___
>> Tutor maillist - Tutor@python.org
>> To unsubscribe or change subscription options:
>> https://mail.python.org/mailman/listinfo/tutor
>>
>
>
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] dictionaries are same but returning false
Thank you Peter.
Silly mistakes 😀
On Jul 5, 2017 5:10 PM, "Peter Otten" <__pete...@web.de> wrote:
> shubham goyal wrote:
>
> > null=None
> > x={'_udp_options': None, '_icmp_options': None, 'attribute_map':
> > {'icmp_options': 'icmpOptions', 'protocol': 'protocol', 'source':
> > {'source',
> > 'tcp_options': 'tcpOptions', 'is_stateless': 'isStateless',
> 'udp_options':
> > 'udpOptions'}, '_is_stateless': False, 'swagger_types': {'icmp_options':
> > 'IcmpOptions', 'protocol': 'str', 'source': 'str', 'tcp_options':
> > 'TcpOptions', 'is_stateless': 'bool', 'udp_options': 'UdpOptions'},
> > '_protocol': '6', '_source': '0.0.4.0/24', '_tcp_options': {
> > "destination_port_range": {
> > "max": "22",
> > "min": "22"
> > },
> > "source_port_range": null
> > }}
> >
> > y={'_udp_options': None, '_icmp_options': None, 'attribute_map':
> > {'icmp_options': 'icmpOptions', 'protocol': 'protocol', 'source':
> > {'source',
> > 'tcp_options': 'tcpOptions', 'is_stateless': 'isStateless',
> 'udp_options':
> > 'udpOptions'}, '_is_stateless': False, 'swagger_types': {'icmp_options':
> > 'IcmpOptions', 'protocol': 'str', 'source': 'str', 'tcp_options':
> > 'TcpOptions', 'is_stateless': 'bool', 'udp_options': 'UdpOptions'},
> > '_protocol': '6', '_source': '0.0.4.0/24', '_tcp_options': {
> > "destination_port_range": {
> > "max": 22,
> > "min": 22
> > },
> > "source_port_range": null
> > }}
> > if x==y:
> > print "true"
> > else:
> > print "false"
> >
> >
> > These dictionaries are same exactly. but its returning false. i don't
> > understand
> > what to do?
>
> Let's narrow down the problem, with the help of the interactive
> interpreter:
>
> >>> changed = [k for k in x if x[k] != y[k]]
> >>> changed
> ['_tcp_options']
> >>> k, = changed
>
> A closer look:
>
> >>> x[k]
> {'source_port_range': None, 'destination_port_range': {'max': '22', 'min':
> '22'}}
> >>> y[k]
> {'source_port_range': None, 'destination_port_range': {'max': 22, 'min':
> 22}}
>
> So x uses strings for min/max while y uses integers, and those do not
> compare equal in Python:
>
> >>> 22 == "22"
> False
>
> Once you fix this
>
> >>> x[k]["destination_port_range"]["max"] = 22
> >>> x[k]["destination_port_range"]["min"] = 22
>
> you get the expected result:
>
> >>> x == y
> True
>
>
> ___
> Tutor maillist - Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
[Tutor] dictionaries are same but returning false
null=None
x={'_udp_options': None, '_icmp_options': None, 'attribute_map':
{'icmp_options': 'icmpOptions', 'protocol': 'protocol', 'source': 'source',
'tcp_options': 'tcpOptions', 'is_stateless': 'isStateless', 'udp_options':
'udpOptions'}, '_is_stateless': False, 'swagger_types': {'icmp_options':
'IcmpOptions', 'protocol': 'str', 'source': 'str', 'tcp_options':
'TcpOptions', 'is_stateless': 'bool', 'udp_options': 'UdpOptions'},
'_protocol': '6', '_source': '0.0.4.0/24', '_tcp_options': {
"destination_port_range": {
"max": "22",
"min": "22"
},
"source_port_range": null
}}
y={'_udp_options': None, '_icmp_options': None, 'attribute_map':
{'icmp_options': 'icmpOptions', 'protocol': 'protocol', 'source': 'source',
'tcp_options': 'tcpOptions', 'is_stateless': 'isStateless', 'udp_options':
'udpOptions'}, '_is_stateless': False, 'swagger_types': {'icmp_options':
'IcmpOptions', 'protocol': 'str', 'source': 'str', 'tcp_options':
'TcpOptions', 'is_stateless': 'bool', 'udp_options': 'UdpOptions'},
'_protocol': '6', '_source': '0.0.4.0/24', '_tcp_options': {
"destination_port_range": {
"max": 22,
"min": 22
},
"source_port_range": null
}}
if x==y:
print "true"
else:
print "false"
These dictionaries are same exactly. but its returning false. i don't
understand
what to do?
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Query regarding output
Thanks all Great place to learn Python. On Jun 29, 2017 7:24 PM, "shubham goyal" wrote: Thankyou all. Great place to learn. On Jun 29, 2017 5:55 PM, "Mats Wichmann" wrote: > On 06/29/2017 03:02 AM, Alan Gauld via Tutor wrote: > > On 29/06/17 03:14, shubham goyal wrote: > > > >> This Question is asked in some exam. i am not able to figure it out. > >> > >> a = [0, 1, 2, 3] > >> for a[-1] in a: > >> print(a[-1]) > >> > >> its giving output 0 1 2 2 > >> > >> it should be 3 3 3 3 as a[-1] belongs to 3. > >> can anyone help me figuring it out. > > > > This is quite subtle and it took me a few minutes to figure > > it out myself. > > > > It might be clearer if we print all of 'a' instead > > of a[-1]: > > > >>>> for a[-1] in a: > > ...print(a) > > ... > > [0, 1, 2, 0] > > [0, 1, 2, 1] > > [0, 1, 2, 2] > > [0, 1, 2, 2] > > > > What is happening is that a[-1] is being assigned the value > > of each item in a in turn. The final iteration assigns a[-1] > > to itself, thus we repeat the 2. > > > Ugh. I guess on an exam where they're trying to see if you can tease it > out, as you may one day have to debug such a sequence... but don't write > code like that. One of the reasons people like "functional programming" > is it avoids these kind of side effects. > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] Query regarding output
Hello all, This Question is asked in some exam. i am not able to figure it out. a = [0, 1, 2, 3] for a[-1] in a: print(a[-1]) its giving output 0 1 2 2 it should be 3 3 3 3 as a[-1] belongs to 3. can anyone help me figuring it out. Thanks ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] airflow dag
Hello,
See this is the code:
from airflow import DAG
from datetime import datetime,timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
So this code use simplehttpopeator(
https://airflow.incubator.apache.org/code.html) which is used to create the
cluster or post the information on this url(qa.qubole.net/api) this is same
as requests.post() in python.
As you can see i am using passwd here
>endpoint='/v2/clusters?auth_token=%s' % (passwd),
and i am making many dags like that which are connected and they all use
the password(auth_token) to send the data so i want to pass passwd as
command line argument.
Argument parser is not working here
i tried argument parser and also sys.argv but it doesn't work.
from airflow import DAG
from airflow.operators import SimpleHttpOperator
from datetime import datetime,timedelta
import argparse
import os
parser = argparse.ArgumentParser()
parser.add_argument("passwd")
args = parser.parse_args()
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
print args.passwd
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
file=os.path.join(os.path.dirname(os.path.abspath(__file__)),"check.json")
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (args.passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': args.passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
if i use this its not able to trigger the dag.
for triggering the dag we use
airflow trigger_dag dagid -r runid
argumentparser work in python
python script.py --passwd passwd #this work
but this is not a python script
airflow trigger_dag dagid -r runid --passwd passwd
sorry for my english.
On Sun, May 28, 2017 at 9:17 PM, shubham goyal wrote:
> Hello all,
>
>
> You are not understanding my point I understand command line argument
> ,sys.argv[],argparser to pass arguments via command line but I want to do
> these things when I trigger the dag
> And how I trigger the dag
> airflow trigger_dag dagname -s runid
>
> This triggering of dag doesn't support to pass command line argument but I
> want to pass authentication token as command line argument when I trigger
> the dag(you can see in airflow docs) so I am asking is there a way to do it.
>
> I tried argument parser but when I use that it's not able to trigger the
> dag.
>
> I want something like this (in cli)
>
> airflow trigger_dag dagname -r runid --passwd auth_token
>
> And want to access this authtoken in Python script.
>
>
> On May 28, 2017 8:18 PM, "Francois Dion" wrote:
>
>> My mailbox if full of similar stories: companies dumping airflow on their
>> ETL (or similar) group. Those who knew Python succeeded, those who didn't
>> failed, and some even moved to other companies because they couldn't cope
>> with all this complexity dumped on them all at once.
>>
>> Moral of the story, it is best to learn python first, become good at it,
>> then get into specific tools.
>>
>> And, of course, this is not specific to a relatively straightforward
>> module
>> / applic
Re: [Tutor] airflow dag
Does anybody have answer? On May 27, 2017 1:53 PM, "Cameron Simpson" wrote: > On 25May2017 18:02, Alan Gauld wrote: > >> On 25/05/17 13:15, shubham goyal wrote: >> >>> I want to ask that can we pass the parameters as commandline arguments in >>> airflow when we are triggering the dag and access them inside the dag's >>> python script/file. >>> >> >> I've no idea what a dag is. >> > > It's a directed acyclic graph, a very standard computer science term. > > Determining that that is what AirFlow means when it says "DAG" was > surprisingly hard, but here we are: > > http://airflow.apache.org/ > > (So the hardness comes from a web search not dropping me on their front > page:-) > > Looks like a tool for data driven computation using a graph of data > pipelines. By restricting it to a DAG on can be certain the computation > will finish and the termination condition is trivially satisfied. No > feedback loops. > > Cheers, > Cameron Simpson > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] airflow dag
He guys,
I want to ask that can we pass the parameters as commandline arguments in
airflow when we are triggering the dag and access them inside the dag's
python script/file.
script:
from airflow import DAG
from datetime import datetime,timedelta
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'start_date': datetime.now(),
'email': ['airf...@airflow.com'],
'email_on_failure': False,
'email_on_retry': False
}
MAIN_DAG='check_dag'
dag = DAG(dag_id=MAIN_DAG, default_args=default_args,
schedule_interval=None)
with open(file, "r") as f:
payload = f.read() # Reading the json data from a file
SimpleHttpOperator( # creating cluster using SimpleHttpOperator
task_id='cluster_create',
method='POST',
http_conn_id='qubole_default',
# for directing to https://qa.qubole.net/api
endpoint='/v2/clusters?auth_token=%s' % (passwd),
data=payload,
headers={"Content-Type": "application/json"},
params={'auth_token': passwd},
response_check=lambda response: True if response.status_code == 200
else False,
dag=dag
)
like this here i am trying to create a cluster but i need to pass password
as cli arguments when i trigger the dag. can we do that. please help.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
[Tutor] creating a cluster
Hello tutors,
here i am trying to create a cluster on cloud using json data but its
giving some error
can you walk me through it. thankyou
import urllib2
import requests
data="""{
"spark_version": null,
"presto_version": null,
"label": [
"test"
],
"disallow_cluster_termination": false,
"force_tunnel": false,
"enable_ganglia_monitoring": false,
"node_bootstrap_file": null,
"tunnel_server_ip": null,
"ec2_settings": {
"aws_preferred_availability_zone": "Any",
"aws_region": "us-east-1",
"compute_validated": false,
"vpc_id": null,
"subnet_id": null,
"bastion_node_public_dns": null,
"master_elastic_ip": null,
"instance_tenancy": null,
"compute_secret_key": "",
"compute_access_key": "AKIAITFMYWE7A3BG5BDQ",
"use_account_compute_creds": true
},
"hadoop_settings": {
"use_hbase": null,
"use_spark": false,
"custom_config": null,
"use_hadoop2": false,
"use_qubole_placement_policy": true,
"is_ha": null,
"enable_rubix": false,
"fairscheduler_settings": {
"default_pool": null
}
},
"node_configuration": {
"master_instance_type": "m1.large",
"slave_instance_type": "m1.xlarge",
"initial_nodes": 1,
"max_nodes": 1,
"idle_cluster_timeout": null,
"slave_request_type": "spot",
"fallback_to_ondemand": null,
"spot_instance_settings": {
"maximum_bid_price_percentage": 100,
"timeout_for_request": 10,
"maximum_spot_instance_percentage": 50
},
"cluster_name": "qa_qbol_acc5572_cl67507"
},
"security_settings": {
"encrypted_ephemerals": false
},
"presto_settings": {
"enable_presto": false,
"custom_config": null
},
"spark_settings": {
"custom_config": null
},
"errors": [
"invalid_credentials"
],
"datadog_settings": {
"datadog_api_token": null,
"datadog_app_token": null
},
"spark_s3_package_name": null,
"zeppelin_s3_package_name": null,
"engine_config": {},
"zeppelin_interpreter_mode": null
}"""
u='auth_token'
p='d3d62d04584d49bbaf2a2430a453f3314fb03b38adcd4e58ac671f40638ac6cd'
url='https://qa.qubole.net/api/v1.3/clusters'
print url
headers={'u':'p'}
req = requests.post(url, data,headers)
#base64string = base64.encodestring('%s:%s' % (u, p)).replace('\n', '')
#req.add_header("Authorization", "Basic %s" %base64string)
for line in req:
print line
#f.close()
Output:
https://qubole.net/api/v1.3/clusters
Traceback (most recent call last):
File "createCluster.py", line 83, in
f = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError:
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] sorted function
Thankyou. got it. On Sat, Apr 15, 2017 at 5:44 AM, Steven D'Aprano wrote: > On Fri, Apr 14, 2017 at 11:59:25PM +0530, shubham goyal wrote: > > > sorted(ls) > > sorted(ls1) > > Here you sort ls and throw the result away, then you do the same to ls1. > > sorted() makes a copy of the list and sorts it. You need to write: > > ls = sorted(ls) > ls1 = sorted(ls1) > > but even better would be to sort in place: > > ls.sort() > ls1.sort() > > which doesn't make a copy. > > > > -- > Steve > ___ > Tutor maillist - Tutor@python.org > To unsubscribe or change subscription options: > https://mail.python.org/mailman/listinfo/tutor > ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] sorted function
Dear mentors, sorted function is not working when i am trying to sort the list of strings but list.sort() is working. can you please help me understand.In this question i was trying to sort the list but first sorting the letter starting from x and taking them first. def front_x(words): # +++your code here+++ ls=[] ls1=[] for str in words: if str[0]=='x': ls.append(str) else: ls1.append(str); print ls print ls1 sorted(ls) sorted(ls1) ls.extend(ls1) return ls front_x(['bbb', 'ccc', 'axx', 'xzz', 'xaa']) Output:['xzz', 'xaa', 'bbb', 'ccc', 'axx'] Output should be: ['xaa', 'xzz', 'axx', 'bbb', 'ccc'] Thankyou. Sorry for bad writing skills. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
[Tutor] (no subject)
Hello, I am a c++ programmer and wants to learn python for internship purposes. How can i learn the advanced python fast mostly data science packages, I know basic python. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
