Re: [Tutor] get aws path from argParser

[ashish makani]

+1 to what Alan said.

Its not clear what you are asking & if you are asking something at all.

( to me, it seems like you are answering someone's query & accidentally
posted here on the python tutor mailing list)

sent from mobile device ;
excuse typos & auto-correct errors
On Jan 9, 2016 00:17, "Alan Gauld"  wrote:

> On 08/01/16 14:04, sebastian cheung via Tutor wrote:
> > * take an s3 destination path as an argument optionally containing the
> string ++DATE++ as a placeholder (e.g. s3://my-bucket/objects/++DATE++/,
> s3://my-bucket/objects/++DATE++/file-++DATE++.txt and
> s3://my-bucket/objects/ should all be valid)
> > I already have something for something more simple, but for s3 maybe use
> awscli etc? Thanks Seb
>
> I have no idea what you are asking about (other than I
> assume its something related to AWS?). This is the python
> tutor list for answering questions about the Python language
> and its standard library. Did you mean to post here?
>
> If so you need to give us a bit more information about
> what you are doing and what exactly you want help with.
>
> > def dateType(string):
> > """
> > Convert a date string to a date object
> > """
> > try:
> > date = datetime.datetime.strptime(string, '%Y-%m-%d').date()
> > except ValueError:
> > msg = "%r is not a valid date" % string
> > raise argparse.ArgumentTypeError(msg)
> > return date
> >
> >
> > def is_valid_file(parser, arg):
> > if not os.path.exists(arg):
> > parser.error("The file %s does not exist!" % arg)
> > else:
> > return open(arg, 'r')parser = argparse.ArgumentParser(
> > description="Take CLI options called start-date and end-date,
> which must be formatted -MM-DD. "
> > "These should default to today if not supplied",
> > epilog="See http://bitbucket.org/niceseb/ for details about the
> Project Time Tracker.")
> > parser.add_argument('-e', '--end-date', metavar='DATE', type=dateType,
> default=datetime.date.today(),
> > help='the date tracking data should start at,
> inclusive in the format -MM-DD (defaults to today)')
> > parser.add_argument('-s', '--start-date', metavar='DATE', type=dateType,
> default=datetime.date.today(),
> > help='the date tracking data should end at,
>  inclusive in the format -MM-DD (defaults to today)')
> > parser.add_argument('-v', action='version', version='%(prog)s 1.0')
> > parser.add_argument('-i', dest="filename", required=False, help="input
> file name", metavar="FILE",
> > type=lambda x: is_valid_file(parser, x))
>
>
> --
> Alan G
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
> http://www.amazon.com/author/alan_gauld
> Follow my photo-blog on Flickr at:
> http://www.flickr.com/photos/alangauldphotos
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> https://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Creating a torrent file & associated tracker through a django web app

[ashish makani]

Clarifying query inline .

On Wed, Nov 21, 2012 at 3:53 PM, Timo  wrote:

> Op 21-11-12 08:54, ashish makani schreef:
>
>
>> Hi Tutor folks
>>
>> I am trying to achieve the following :
>>
>> The user should log into a web app, select a file & the web app should
>> generate a .torrent file & a private 
>> tracker(http://IP_ADDRESS<http://ip_address/>
>> :**PORT_NUMBER/announce) for that .torrent file.
>>
>> Basically, i want to programmatically create a .torrent file, on the
>> server side.
>>
>> I found two libraries mktorrent[1] & py3createtorrent [2], but i need
>> help with how to create a torrent from a django web app.
>>
>> Any suggestions for django modules or libraries which can do this ?
>>
>> You don't need Django specific modules, just use the ones you linked and
> install them on your server. It doesn't matter if you call it from a script
> on your desktop or a Django webapp.
> Looking at the example on Wikipedia [1], it's not hard to create it
> manually. It even looks like a JSON file, you'll just need a way to bencode
> the data.
>
> Timo
>

Thanks a ton Timo for your prompt reply.
I am a django newbie, so can you please clarify the following for me :

I can create a torrent using py3createtorrent locally, but am unable to
figure how it will work via a django web app.

To make it work locally, i specified the local location of the folder, i
wanted to create a torrent for.

In the web app scenario, the user selects the content present on the user's
local machine,
after that, how do i use py3createtorrent to create a torrent on the server
end, since the content is not present on the server .

>
> [1] 
> http://en.wikipedia.org/wiki/**Torrent_file#Single_file<http://en.wikipedia.org/wiki/Torrent_file#Single_file>
>
> Any suggestions/pointers/ideas/**links will be greatly appreciated
>>
>> Thanks a ton,
>>
>> cheers,
>>
>> ashish
>>
>>  1. http://mktorrent.sourceforge.**net/<http://mktorrent.sourceforge.net/>
>>  2. 
>> http://www.robertnitsch.de/**projects/py3createtorrent<http://www.robertnitsch.de/projects/py3createtorrent>
>>
>>
>>
>> __**_
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/**mailman/listinfo/tutor<http://mail.python.org/mailman/listinfo/tutor>
>>
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Creating a torrent file & associated tracker through a django web app

[ashish makani]

Hi Tutor folks

I am trying to achieve the following :

The user should log into a web app, select a file & the web app should
generate a .torrent file & a private
tracker(http://IP_ADDRESS:PORT_NUMBER/announce)
for that .torrent file.

Basically, i want to programmatically create a .torrent file, on the server
side.

I found two libraries mktorrent[1] & py3createtorrent [2], but i need help
with how to create a torrent from a django web app.

Any suggestions for django modules or libraries which can do this ?

Any suggestions/pointers/ideas/links will be greatly appreciated

Thanks a ton,

cheers,

ashish

   1. http://mktorrent.sourceforge.net/
   2. http://www.robertnitsch.de/projects/py3createtorrent
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Passing arguments to & running a python script on a remote machine from a python script on local machine .

[ashish makani]

On Thu, Sep 20, 2012 at 3:43 AM, eryksun  wrote:

> On Wed, Sep 19, 2012 at 2:47 PM, ashish makani 
> wrote:
> >
> > I tried this
> >>>> import os
> >>>> os.system ("ssh remoteuser@remote python remote.py arg1 arg2 arg3")
> >
> > This worked, but if the arguments i tried to pass, had spaces, i was not
> > able to 'escape' the spaces.
>
> Presuming "remote" has an SSH server running, and that a firewall
> isn't blocking port 22, and that you've enabled logging in without a
> password (i.e. ssh-keygen), try the following:
>
> import sys
>> import subprocess
>> user = remoteuser
>> hostname = remote
>> cmd = 'python remote.py "%s" "%s" "%s"' % (arg1, arg2, arg3)
>> try:
>> out = subprocess.check_output(['ssh', '%s@%s' % (user,
>> hostname), cmd])
>> except subprocess.CalledProcessError as e:
>> print >>sys.stderr, str(e)   # handle/log the error, retry
>
>
This worked like a charm.
Thank you so much.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Passing arguments to & running a python script on a remote machine from a python script on local machine .

[ashish makani]

Thanks a ton for the prompt reply & the great suggestions, Dave.

1.   A colleague gave this exact same suggestion
os.system ('ssh remoteuser@remote python remote.py arg1 "arg 2 has spaces"
arg3')

I was thinking spaces is my problem, so i initially tested the following
(no ssh)
os.system('python remote.py arg1 "arg 2 has spaces" arg3')  & it works :)

But sadly,  os.system ('ssh remoteuser@remote python remote.py arg1 "arg 2
has spaces" arg3')   does not :(

2. Also, you mentioned os.exec
Which would you recommend ?
os.system or os.exec ?

Thanks a lot,

cheers
ashish



On Thu, Sep 20, 2012 at 12:54 AM, Dave Angel  wrote:

> On 09/19/2012 02:47 PM, ashish makani wrote:
> > Hi PyTutor Folks
> >
> > Here is my situation
> >
> > 1. I have two machines. Lets call them *local & remote.*
> > Both run ubuntu & both have python installed
> >
> > 2. I have a python script, *local.py*, running on *local *which needs to
> > pass arguments ( 3/4 string arguments, containing whitespaces like
> spaces,
> > etc ) to a python script, *remote.py* running on *remote* (the remote
> > machine).
> >
> > I have the following questions:
> >
> > 1. What's the best way to accomplish my task ?
> > I have researched quite a bit & so far found really conflicting & complex
> > workarounds.
> >
> > I googled & found people using several libraries to accomplish ssh to
> > remote machine & execute a command on remote machine.
> > paramiko <http://www.lag.net/paramiko/> ( now forked into the ssh
> > moduke<https://github.com/bitprophet/ssh>),
> > fabric <http://docs.fabfile.org/en/1.4.3/index.html>,
> > pushy<http://packages.python.org/pushy/>,etc
> >
> > People who have used any of these libraries, which one would you
> recommend,
> > as the most apt (simple & easy to use, lightweight, best performance,
> etc)
> > for my situation ?
> >
> > 2. I would prefer a solution, which does NOT require the installation of
> > extra libraries on the local & remote machines.
> > If installing external librar
> >
> > 3. Has anybody been able to do this using os.system ?
> >
> > I tried this
> >>>> import os
> >>>> *os.system ("ssh remoteuser@remote python remote.py arg1 arg2 arg3")*
> > *
> > *
> > This worked, but if the arguments i tried to pass, had spaces, i was not
> > able to 'escape' the spaces.
> >
> > Any & all explanations/links/code
> > snippets/thoughts/ideas/suggestions/feedback/comments/ of the Python
> tutor
> > community would be greatly appreciated.
> >
> > Thanks a ton
> >
> > cheers
> > ashish
> >
> > email :
> > ashish.makani
> > domain:gmail.com
> >
> > *“The only way to do great work is to love what you do. If you haven’t
> > found it yet, keep looking. Don’t settle. As with all matters of the
> heart,
> > you’ll know when you find it.” - Steve Jobs (1955 - 2011)*
> >
> >
>
> Since you prefer not installing any optional libraries, I'll just talk
> about your os.system() mechanism. Is that adequate for you, other than
> the problem with spaces?
>
> If so, then why not test it with the real ssh, to figure out where the
> quotes need to be to handle the spaces. Then once you have it working,
> use single quotes around the whole thing when calling os.exec().
>
> Something like (all untested):
>
> os.system ('ssh remoteuser@remote python remote.py arg1 "arg 2 has
> spaces" arg3')
>
>
> Or, more likely, build the arguments in separate variables, and use
>
> os.system( 'ssh remoteuser@remote python remote.py "%s" "%s" "%s"' %
> (arg1, arg2, arg3) )
>
> that will fail if the argn already has quotes in it. You can get much
> fancier with encoding, which will add backslashes in front of some
> characters, etc. But keep it simple if you can.
>
> I ought to give the obligatory: use the multiprocess module instead of
> os.exec, but if something works for you, I'm not going to argue.
>
>
>
>
> --
>
> DaveA
>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Passing arguments to & running a python script on a remote machine from a python script on local machine .

[ashish makani]

Hi PyTutor Folks

Here is my situation

1. I have two machines. Lets call them *local & remote.*
Both run ubuntu & both have python installed

2. I have a python script, *local.py*, running on *local *which needs to
pass arguments ( 3/4 string arguments, containing whitespaces like spaces,
etc ) to a python script, *remote.py* running on *remote* (the remote
machine).

I have the following questions:

1. What's the best way to accomplish my task ?
I have researched quite a bit & so far found really conflicting & complex
workarounds.

I googled & found people using several libraries to accomplish ssh to
remote machine & execute a command on remote machine.
paramiko  ( now forked into the ssh
moduke),
fabric ,
pushy,etc

People who have used any of these libraries, which one would you recommend,
as the most apt (simple & easy to use, lightweight, best performance, etc)
for my situation ?

2. I would prefer a solution, which does NOT require the installation of
extra libraries on the local & remote machines.
If installing external librar

3. Has anybody been able to do this using os.system ?

I tried this
>>> import os
>>> *os.system ("ssh remoteuser@remote python remote.py arg1 arg2 arg3")*
*
*
This worked, but if the arguments i tried to pass, had spaces, i was not
able to 'escape' the spaces.

Any & all explanations/links/code
snippets/thoughts/ideas/suggestions/feedback/comments/ of the Python tutor
community would be greatly appreciated.

Thanks a ton

cheers
ashish

email :
ashish.makani
domain:gmail.com

*“The only way to do great work is to love what you do. If you haven’t
found it yet, keep looking. Don’t settle. As with all matters of the heart,
you’ll know when you find it.” - Steve Jobs (1955 - 2011)*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to send email from a gmail a/c using smtp when port 587(smtp) is blocked

[ashish makani]

Walter, Marc,

Thanks for your helpful suggestions & super quick replies.

As a noobie, i often run into brick walls, thinking something(problem i am
stuck at) is not possible to do in python.
I love struggling against that problem & figuring a way out.

I have posted to the tutor mailing list in the past & the community here
has always been amazingly welcoming & super helpful, so as a n00b, i was
just a little taken aback &surprised at Emil's not particularly helpful ,
yet completely accurate response :)

Marc,
your 3rd point,

> you could establish a VPN tunnel to some server outside of the
> university's network and send from port 587 on THAT machine.  Complicated,
> weird, and not horribly secure.  But doable.

Could you point me to a good link/resource on how to do this ?

Walter,
you suggested writing a web service running somewhere else (e.g. Google
Apps, AWS, etc) which i could request from my python code, which in turn
would do the emailing.
Can you point me to any good links/example code which might explain writing
a simple SOAP/ReST web service using port80(http) or preferably,
 443(https).
I know nothing about web services, but could this web service essentially
be the python email code, i mentioned in my original post, if i run it on
Google App Engine ?

Thanks a ton,

Best,
ashish

On Wed, Sep 12, 2012 at 3:29 AM, Walter Prins  wrote:

> Hi Ashish,
>
> On 11 September 2012 22:44, ashish makani  wrote:
> > These emails are automated diagnostic emails sent to a group of a few
> > admins, so we get notified when a python heartbeat script, detects a
> failure
> > in things like n/w connectivity, router status, etc.
> > We dont use university email, we use gmail.
> >
> > Emile,
> > Please don't presume people's intentions (that we are sending spam) &
> judge
> > people without knowing anything about them.
> > We are a tiny startup trying to connect rural communities using voice &
> ivr
> > systems - http://gramvaani.org/
>
> OK, well I'm sure you can see how an apparent newbie asking to get out
> of a university network without any explanation can be seem um,
> suspect, so I think Emile's response was reasonable.   I must further
> note that I can't actually see how/where your request actually fits
> under the projects listed by that site.  So, colour me still a bit
> sceptical, but I'll give you the benefit of the doubt.
>
> So then, given that you can only get out on port 80, your only real
> option the way I see it is to write a small web service, maybe a SOAP
> or preferably ReST service, to run on e.g. Google APP engine that will
> do the emailing for you.Of course, you'll have to consider whether
> to implement some security yourself if you use port 80 as the data
> going over the wire will be sent unencrypted.  It may not be a problem
> but then again it may.  Note, alternatively you can perhaps also use
> https (port 443), if that's also open as that will give you end-to-end
> encryption for free. (But I have no idea and suspect that this may
> also introduce a boatload of other complications...)
>
> Walter
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] How to send email from a gmail a/c using smtp when port 587(smtp) is blocked

[ashish makani]

Folks,

some more context so people presume we are spammers :)

These emails are automated diagnostic emails sent to a group of a few
admins, so we get notified when a python heartbeat script, detects a
failure in things like n/w connectivity, router status, etc.
We dont use university email, we use gmail.

Emile,
Please don't presume people's intentions (that we are sending spam) & judge
people without knowing anything about them.
We are a tiny startup trying to connect rural communities using voice & ivr
systems - http://gramvaani.org/

Best,
ashish


On Wed, Sep 12, 2012 at 2:56 AM, Emile van Sebille  wrote:

> On 9/11/2012 2:19 PM ashish makani said...
>
>> Hi Python Tutor folks
>>
>> I am stuck with an issue, so am coming to the Pythonistas who rescue me
>> everytime :)
>>
>> I am trying to send out email programmatically, from a gmail a/c, using
>> smtplib, using the following chunk of code (b/w [ & ] below)
>>
>> [
>>
>> import smtplib
>> from email.mime.text import MIMEText
>>
>> #uname, pwd are username & password of gmail a/c i am trying to send from
>>
>> server = smtplib.SMTP('smtp.gmail.com:**587 <http://smtp.gmail.com:587> <
>> http://smtp.gmail.com:587/>')
>>
>> server.starttls() # get response(220, '2.0.0 Ready to start TLS')
>> server.login(uname,pwd)  # get response(235, '2.7.0 Accepted')
>>
>> toaddrs  = ['x...@gmail.com <mailto:x...@gmail.com>', 'y...@gmail.com
>> <mailto:y...@gmail.com>' ] # list of To email addresses
>>
>> msg = MIMEText('email body')
>> msg['Subject'] = 'email subject'
>> server.sendmail(fromaddr, toaddrs, msg.as_string())
>>
>>
>> ]
>>
>> The code above works perfectly fine on my local machine, but fails on
>> the production server at the university where i work( all ports other
>> than port 80 are blocked) :(
>>
>
> Good -- the university is taking steps to block spam.
>
> You should send mail using the university mail system and not
> smtp.gmail.com.
>
> Emile
>
>
> __**_
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/**mailman/listinfo/tutor<http://mail.python.org/mailman/listinfo/tutor>
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] How to send email from a gmail a/c using smtp when port 587(smtp) is blocked

[ashish makani]

Hi Python Tutor folks

I am stuck with an issue, so am coming to the Pythonistas who rescue me
everytime :)

I am trying to send out email programmatically, from a gmail a/c, using
smtplib, using the following chunk of code (b/w [ & ] below)

[

import smtplib
from email.mime.text import MIMEText

#uname, pwd are username & password of gmail a/c i am trying to send from

server = smtplib.SMTP('smtp.gmail.com:587')
server.starttls() # get response(220, '2.0.0 Ready to start TLS')
server.login(uname,pwd)  # get response(235, '2.7.0 Accepted')

toaddrs  = ['x...@gmail.com', 'y...@gmail.com' ] # list of To email addresses
msg = MIMEText('email body')
msg['Subject'] = 'email subject'
server.sendmail(fromaddr, toaddrs, msg.as_string())


]

The code above works perfectly fine on my local machine, but fails on the
production server at the university where i work( all ports other than port
80 are blocked) :(

So , when i try to run the 2 py statements (in bold below) on a python
prompt from the production server, which has port 587 blocked, i get the
following error

{
*>>> import smtplib*
*>>> server = smtplib.SMTP('smtp.gmail.com:587')*
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.6/smtplib.py", line 239, in __init__
(code, msg) = self.connect(host, port)
  File "/usr/lib/python2.6/smtplib.py", line 295, in connect
self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/lib/python2.6/smtplib.py", line 273, in _get_socket
return socket.create_connection((port, host), timeout)
  File "/usr/lib/python2.6/socket.py", line 514, in create_connection
raise error, msg
socket.error: [Errno 101] Network is unreachable

}


1. How can i overcome this ?
A friend suggested , that i could use something called smtp relay, which
would solve my problem, but would be time-consuming & a pain to set up.
I did some cursory googling & searching on stackoverflow but could not find
any good, well explained results.
I dont know anything about SMTP.
Anybody has any recommendations on a good explanation on smtp relay & how
to set it up for sending email from a gmail a/c using a python script ?

2. Also, is there a more elegant/cleaner/graceful solution to my problem
than using an smtp relay ?

Any & all explanations/links/code
snippets/thoughts/ideas/suggestions/feedback/comments/ of the Python tutor
community would be greatly appreciated.

Thanks a ton

cheers
ashish

email :
ashish.makani
domain:gmail.com
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] What made Python differ from other Languages?

[ashish makani]

An excellent recent article on hn(hacker news) on why python is important

http://blaag.haard.se/Why-Python-is-important-for-you/

(via  http://news.ycombinator.com/item?id=3579847  )

cheers
ashish

On Tue, Feb 21, 2012 at 1:13 AM, Mark Lawrence wrote:

> On 20/02/2012 16:43, Sunil Tech wrote:
>
>> *I am Beginner (very little i know), i want to know what are new things i
>> can find in Python.*
>>
>>
>>
>>
>>
>> __**_
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/**mailman/listinfo/tutor
>>
>
> This sums up the Python philosophy.
>
> C:\Users\Mark\cpython\PCbuild>**py -3.2 -c "import this"
> The Zen of Python, by Tim Peters
>
> Beautiful is better than ugly.
> Explicit is better than implicit.
> Simple is better than complex.
> Complex is better than complicated.
> Flat is better than nested.
> Sparse is better than dense.
> Readability counts.
> Special cases aren't special enough to break the rules.
> Although practicality beats purity.
> Errors should never pass silently.
> Unless explicitly silenced.
> In the face of ambiguity, refuse the temptation to guess.
> There should be one-- and preferably only one --obvious way to do it.
> Although that way may not be obvious at first unless you're Dutch.
> Now is better than never.
> Although never is often better than *right* now.
> If the implementation is hard to explain, it's a bad idea.
> If the implementation is easy to explain, it may be a good idea.
> Namespaces are one honking great idea -- let's do more of those!
>
> --
> Cheers.
>
> Mark Lawrence.
>
>
> __**_
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/**mailman/listinfo/tutor
>
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] vim as a python editor

[ashish makani]

@Alan, maybe hitting the actual "Esc" key requires too much "reach"
Mapping CapsLock to escape maybe lets u type faster, perhaps

I am just guessing though..maybe Paul can explain

On vim, just saw this on hnews today morning...might be of interest

http://news.ycombinator.com/item?id=2080342
http://vim.runpaint.org/

Happy Friday

cheers
ashish


On Fri, Jan 7, 2011 at 1:25 PM, Alan Gauld wrote:

> "Paul Griffiths"  wrote
>
>> I've learned that:
>> ...
>>
>> - re-configuring the Caps Lock to be an extra Esc saves time
>>
>
> Huh? How do you use that? Its a new one on me. Why would two escape keys be
> useful?
>
>
>  - I must learn how to  'fold' functions
>>
>
> Read the vim help.
>
> Also read about ctags - they work with python code too.
> Combining ctags with vim gives you some very powerful navigation tools - ie
> put the cursor on a function name and hit the tags key to go directly to the
> function definition. Even if it's in another file...
>
> HTH,
>
> --
> Alan Gauld
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
>
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>



--
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] [OT] Bay area meetups for python &/ {linux, embedded, mobile, wireless ?}

[ashish makani]

>
> OT(off_topic): I moved to the bay area recently & am passionate about
> technology in general & linux, python, c, embedded, mobile, wireless
> stuff,.
> I was wondering if any of you guys, are part of some bay area python( or
> other tech) meetup ( as in do you guys meetup, in person) for like a tech
> talk / discussion / brainstorming/ hack nights ?
>


> If yes, i would love to know more & be a part of it
>


Thanks Tino for the info

Any other inputs from other folks ?
OT(off_topic): I moved to the bay area recently & am passionate about
technology in general & linux, python, c, embedded, mobile, wireless
stuff,.
I was wondering if any of you guys, are part of some bay area python( or
other tech) meetup ( as in do you guys meetup, in person) for like a tech
talk / discussion / brainstorming/ hack nights ?
If yes, i would love to know more & be a part of it

I am aware of  BayPiggies
http://www.baypiggies.net/
http://mail.python.org/mailman/listinfo/baypiggies


On Mon, Dec 20, 2010 at 9:10 PM, Tino Dai  wrote:

> Hi Ashish,
>
> Check out Noisebridge ( 
> https://www.*noisebridge*.net/) in
> SF. I think you will find there are like minded tech people there. It also
> has Mitch Altman (   
> http://en.wikipedia.org/wiki/*Mitch*_*Altman*
> ) is one of the founders/original members.
>
> -Tino
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
>

*"We act as though comfort and luxury were the chief requirements of life,
when all that we need to make us happy is something to be enthusiastic
about."
-- Albert Einstein*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Trying to parse a HUGE(1gb) xml file in python

[ashish makani]

Chris

This block of code made my day - especially yummydataaddrs & "here's your
stupid data"

> for start,end in yummydataaddrs:
>fd.seek(start)
>print "here's your stupid data:", fd.read(end-start+1)


Nothing is more impressive than solid code, with a good sense of humor.

Thanks for the code & especially since i am in a time crunch, this approach,
might get me what i need more quickly.

Thanks also for Knuth's awesome quote & reminded me of my stanford friend
who told me that Prof. Knuth, still holds a christmas tree lecture every
year...unfortunately inspite of being in the bay area this year, i missed it
:(
http://stanford-online.stanford.edu/seminars/knuth/101206-knuth-500.asx

Thanks a ton

cheers
ashish

p.s. To everybody

OT(off_topic): I moved to the bay area recently & am passionate about
technology in general & linux, python, c, embedded, mobile, wireless
stuff,.
I was wondering if any of you guys, are part of some bay area python( or
other tech) meetup ( as in do you guys meetup, in person) for like a tech
talk / discussion / brainstorming/ hack nights ?
If yes, i would love to know more & be a part of it

On Mon, Dec 20, 2010 at 9:27 PM, Chris Fuller  wrote:

>
> This isn't XML, it's an abomination of XML.  Best to not treat it as XML.
> Good thing you're only after one class of tags.  Here's what I'd do.  I'll
> give a general solution, but there are two parameters / four cases that
> could
> make the code simpler, I'll just point them out at the end.
>
> Iterate over the file descriptor, reading in line-by-line.  This will be
> slow
> on a huge file, but probably not so bad if you're only doing it once.  It
> makes
> the rest easier.  Knuth has some sage advice on this point (*) :)  Some
> feedback on progress to the user can be helpful here, if it is slow.
>
> Keep track of your offset into the file.  There are two ways: use the
> tell()
> method of the file descriptor (but you will have to subtract the length of
> the
> current line), or just add up the line lengths as you process them.
>
> Scan each line for the open tag.  Add the offset to the tag to the offset
> within
> the file of  the current line, and push that to a stack.  Scan for the end
> tag,
> when you find one, pop an address from the stack, and put the two
> (start/end)
> addresses a list for later.  Keep doing this until you run out of file.
>
> Now, take that list, and pull off the address-pairs; seek() and read() them
> directly.  Lather, rinse, repeat.
>
> Some off-the-cuff untested code:
>
> stk = []
> yummydataaddrs = []
>
> fileoff = 0
>
> fd = open('ginormous.xml', 'r')
> for line in fd:
>lineoff = line.index(start_tag)
>if fileoff != -1:
>stk.append(fileoff+lineoff)
>
>lineoff = line.index(end_tag)
>if lineoff != -1:
>yummydataaddr.append( (stk.pop(-1), fileoff+lineoff) )
>
>fileoff += len(line)
>
> for start,end in yummydataaddrs:
>fd.seek(start)
>print "here's your stupid data:", fd.read(end-start+1)
>
>
> You can simplify a bit if the tags are one a line by themselves, since you
> don't have to keep track of the offset with the line of the tag.  The other
> simplification is if they aren't nested.  You don't need to mess around
> with a
> stack in this case.
>
>
> (*) "Premature optimization is the root of all evil."
>
>
> Cheers
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>



*"We act as though comfort and luxury were the chief requirements of life,
when all that we need to make us happy is something to be enthusiastic
about."
-- Albert Einstein*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Trying to parse a HUGE(1gb) xml file in python

[ashish makani]

Thanks Luke, Steve, Brett, Lloyd & Alan
for your prompt responses & sharing your wisdom.

I <3 the python community... You(We ?) folks are AWESOME

I cross-posted this query on comp.lang.python
I bet most of you hang @ c.l.p too, but just in case, here is the link to
the discussion at c.l.p
https://groups.google.com/d/topic/comp.lang.python/i816mDMSoXM/discussion

Thanks again for the amazing help & advice

cheers
ashish


On Mon, Dec 20, 2010 at 5:13 PM, Alan Gauld wrote:

> "ashish makani"  wrote
>
>  I am looking for a specific element..there are several 10s/100s
>> occurrences
>> of that element in the 1gb file.
>>
>> I need to detect them & then for each 1, i need to copy all the content
>> b/w
>> the element's start & end tags & create a smaller xml
>>
>
> This is exactly what sax and its kin are for. If you wanted to manipulate
> the xml data and recreate the original file tree based is better but for
> this
> kind of one shot processing SAX will be much much faster.
>
> The concept is simple enough if you have ever used awk to process
> text files. (or the Python HTMLParser) You define a function that gets
> triggered when the parser detects a matching tag.
>
>
>  My hardware setup : I have a win7 pro box with 8gb of RAM & intel core2
>> quad
>> cpuq9400.
>> On this i am running sun virtualbox(3.2.12), with ubuntu 10.10(maverick)
>> as
>> guest os, with 23gb disk space & 2gb(2048mb) ram, assigned to the guest
>> ubuntu os.
>>
>
> Obviously running the code in the virtuial machjine is limiting your
> ability to deal with the data but in this case you would be pushing
> hard to build the entire tree in RAM anyway so it probably doesn't
> matter.
>
>
>  4. I then investigated some streaming libraries, but am confused - there
>> is
>> SAX[http://en.wikipedia.org/wiki/Simple_API_for_XML] ,
>>
>
>  Which one is the best for my situation ?
>>
>
> I've only used sax - I tried minidom once but couldn't get it to work
> as I wanted so went back to sax... There are lots of examples of
> xml parsing using sax, both in Python and Java - just google.
>
>
>  Should i instead just open the file, & use reg ex to look for the element
>> i
>> need ?
>>
>
> Unless the xml is very simple you would probably find yourself
> creating a bigger problem. regex's are not good at handling the
> kinds of recursive data structures as can be found in SGML
> based languages.
>
> HTH,
>
>
> --
> Alan Gauld
> Author of the Learn to Program web site
> http://www.alan-g.me.uk/
>
>
>
> ___
> Tutor maillist  -  Tutor@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>


*

"We act as though comfort and luxury were the chief requirements of life,
when all that we need to make us happy is something to be enthusiastic
about."
-- Albert Einstein*
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Trying to parse a HUGE(1gb) xml file in python

[ashish makani]

Hi Python Tutor folks

This is a rather long post, but i wanted to include all the details &
everything i have tried so far myself, so please bear with me & read the
entire boringly long post.

Goal : I am trying to parse a ginormous ( ~ 1gb) xml file.

I am looking for a specific element..there are several 10s/100s occurrences
of that element in the 1gb file.

I need to detect them & then for each 1, i need to copy all the content b/w
the element's start & end tags & create a smaller xml


0. I am a python & xml n00b, s& have been relying on the excellent beginner
book DIP(Dive_Into_Python3 by MP(Mark Pilgrim) Mark , if u are readng
this, you are AWESOME & so is your witty & humorous writing style)

My hardware setup : I have a win7 pro box with 8gb of RAM & intel core2 quad
cpuq9400.
On this i am running sun virtualbox(3.2.12), with ubuntu 10.10(maverick) as
guest os, with 23gb disk space & 2gb(2048mb) ram, assigned to the guest
ubuntu os.

1. Almost all exmaples pf parsing xml in python, i have seen, start off with
these 4 lines of code.

import xml.etree.ElementTree as etree
tree = etree.parse('*path_to_ginormous_xml*')
root = tree.getroot()  #my huge xml has 1 root at the top level
print root

2. In the 2nd line of code above, as Mark explains in DIP, the parse
function builds & returns a tree object, in-memory(RAM), which represents
the entire document.
I tried this code, which works fine for a small ( ~ 1MB), but when i run
this simple 4 line py code in a terminal for my HUGE target file (1GB),
nothing happens.
In a separate terminal, i run the top command, & i can see a python process,
with memory (the VIRT column) increasing from 100MB , all the way upto
2100MB.

I am guessing, as this happens (over the course of 20-30 mins), the tree
representing is being slowly built in memory, but even after 30-40 mins,
nothing happens.
I dont get an error, seg fault or out_of_memory exception.

3. I also tried using lxml, but an lxml tree is much more expensive, as it
retains more info about a node's context, including references to it's
parent.

[http://www.ibm.com/developerworks/xml/library/x-hiperfparse/]

When i ran the same 4line code above, but with lxml's elementree ( using the
import below in line1of the code above)
import lxml.etree as lxml_etree

i can see the memory consumption of the python process(which is running the
code) shoot upto ~ 2700mb & then, python(or the os ?) kills the process as
it nears the total system memory(2gb)

I ran the code from 1 terminal window (screenshot :
http://imgur.com/ozLkB.png)
& ran top from another terminal (http://imgur.com/HAoHA.png)

4. I then investigated some streaming libraries, but am confused - there is
SAX[http://en.wikipedia.org/wiki/Simple_API_for_XML] , the iterparse
interface[http://effbot.org/zone/element-iterparse.htm], & several otehr
options ( minidom)

Which one is the best for my situation ?

Should i instead just open the file, & use reg ex to look for the element i
need ?


Any & all code_snippets/wisdom/thoughts/ideas/suggestions/feedback/comments/
of the Python tutor community would be greatly appreciated.
Plz feel free to email me directly too.

thanks a ton

cheers
ashish

email :
ashish.makani
domain:gmail.com

p.s.
Other useful links on xml parsing in python
0. http://diveintopython3.org/xml.html
1.
http://stackoverflow.com/questions/1513592/python-is-there-an-xml-parser-implemented-as-a-generator
2. http://codespeak.net/lxml/tutorial.html
3.
https://groups.google.com/forum/?hl=en&lnk=gst&q=parsing+a+huge+xml#!topic/comp.lang.python/CMgToEnjZBk
4. http://www.ibm.com/developerworks/xml/library/x-hiperfparse/
5.http://effbot.org/zone/element-index.htm
http://effbot.org/zone/element-iterparse.htm
6. SAX : http://en.wikipedia.org/wiki/Simple_API_for_XML
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor