Re: The pty module, reading from a pty, and Python 2/3

[David Hutto]

in other words, the last two lines of your function should be:

 statinfo = os.stat(filename)
 return child_pty.read(statinfo.st_size)


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The pty module, reading from a pty, and Python 2/3

[David Hutto]

#better coded for you to understand

import sys
import pty
import os

def get_text(filename):

try:
( child_pid, fd ) = pty.fork()  # OK
except OSError as e:
print(str(e))
sys.exit(1)

if child_pid == 0:
try:
with open("log.txt", "w") as f:
f.write("about to execlp")
os.execlp("cat", "cat", filename)
except:
with open("log.txt", "w") as f:
f.write("could not spawn process")
print("Could not spawn")
sys.exit(1)

child_pty = os.fdopen(fd)
#you need to find the file size, and place it as an integer in read
below, where you return the value
#you have to input into read, how many characters you want read in
with statinfo.st_size
statinfo = os.stat(filename)
return child_pty.read(statinfo.st_size)


if __name__ == "__main__":
print(get_text("my-pty-test.py"))

-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The pty module, reading from a pty, and Python 2/3

[David Hutto]

import sys
import pty
import os

def get_text(filename):
#you need to find the file size, and place it as an integer in read
below, where you return the value
statinfo = os.stat(filename)

try:
( child_pid, fd ) = pty.fork()  # OK
except OSError as e:
print(str(e))
sys.exit(1)

if child_pid == 0:
try:
with open("log.txt", "w") as f:
f.write("about to execlp")
os.execlp("cat", "cat", filename)
except:
with open("log.txt", "w") as f:
f.write("could not spawn process")
print("Could not spawn")
sys.exit(1)

child_pty = os.fdopen(fd)
#you have to input into read, how many characters you want read in. if
you place a random integer in, it will read to that integer within the
file
return child_pty.read(statinfo.st_size)


if __name__ == "__main__":
print(get_text("my-pty-test.py"))


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[Mark Lawrence]

On 24/10/2012 06:46, Alain Ketterlin wrote:

satyam  writes:


I have a text file like this

A1980JE3937 2732 4195 12.527000
A1980JE3937 3465 9720 22.00
A1980JE3937 2732 9720 18.00
A1980KK18700010 130 303 4.985000
A1980KK18700010 7 4915 0.435000

[...]

I want to split the file and get multiple files like
A1980JE3937.txt and A1980KK18700010.txt, where each file will
contain column2, 3 and 4.


Sorry for being completely off-topic here, but awk has a very convenient
feature to deal with this. Simply use:

 awk '{ print $2,$3,$4 > $1".txt"; }' /path/to/your/file

-- Alain.



Although practicality beats purity :)

--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: turn list of letters into an array of integers

[Chris Rebert]

On Tue, Oct 23, 2012 at 10:23 PM, seektime  wrote:
> Here's some example code. The input is a list which is a "matrix" of letters:
>a  b  a
>b  b  a
>
> and I'd like to turn this into a Python array:

You mean a Python list. The datatype Python calls an `array` is very
different and relatively uncommonly used.
Although, confusingly, Python's lists are implemented using C arrays
rather than linked lists.

>   1 2 1
>   2 2 1
>
> so 1 replaces a, and 2 replaces b. Here's the code I have so far:
>
 L=['a b a\n','b b a\n']

 seq
> '1 2 1\n 2 2 1\n'
>
> My question is how can I turn "seq" into a python array?

I'd say you're asking the wrong question. The better question is "Why
wasn't the result a list in the first place?". Many transformations
are cumbersome to express over just strings, which is why the first
job of most programs is to parse their input into a more convenient
structure that is suited to their main task(s).

This (along with some other improvements) leads to a better, somewhat
different program/algorithm:

letter2number = {'a': 1, 'b': 2}
with open("path/to/file.txt", "r") as f:
result = [[letter2number[letter] for letter in
line.strip().split()] for line in f]

If it's safe to assume that the correspondence between the letters and
numbers isn't completely arbitrary, some further improvements are also
possible.

Some relevant docs:
http://docs.python.org/library/stdtypes.html#string-methods
http://docs.python.org/tutorial/datastructures.html#list-comprehensions

Cheers,
Chris

P.S.: I'm guessing you obtained `L` from file.readlines() or similar;
it is worth noting for future reference that the readlines() method is
considered somewhat deprecated.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The pty module, reading from a pty, and Python 2/3

[David Hutto]

On Tue, Oct 23, 2012 at 11:41 PM, Evan Driscoll  wrote:
> OK, one more self-reply. :-)
>
> I found http://bugs.python.org/issue5380 which seems to be relevant. Sounds
> like the OS is returning an error, and that's naturally being propagated
> through Python. And further testing reveals the problem only surfaces when
> the child actually exits -- if the child writes data to the pipe and is more
> enthusiastic about living, then the parent can read it successfully. While
> it'd be nice if my example worked, for my actual purpose I think that's good
> enough (I just won't be able to test *quite* as easily for a while).
>
> I am still curious if anyone know why it worked in 2 though.
>
> Evan
>
>
>
>
> On 10/23/2012 10:03 PM, Evan Driscoll wrote:
>>
>> Oh, and a little more information:
>>
>> The log.txt file I create has the message that it's "about to execlp", and
>> the exec() *does* actually happen -- the IOError is raised after the child
>> process quits.
>>
>> Evan
>>
>>
>>
>> On 10/23/2012 09:59 PM, Evan Driscoll wrote:
>>>
>>> I have the following program. Everything is sunshine and rainbows when I
>>> run in in Python 2, but when I run it under Python 3 I get an IOError. 2to3
>>> only reports one dumb suggestion re. a print call (which I can get rid of by
>>> importing __future__'s print_function, and then it just suggests removing
>>> that import).
>>>
>>> Can anyone shed any light? I am on Ubuntu Linux with Python 2.7.3 and
>>> 3.2.3.
>>>
>>>
>>> (Just for the record, I figured out that it ran under Python 2 by
>>> accident as I was reducing it for a "why doesn't this run?" email. :-) I'm
>>> not super-familiar with Py3 as I've mostly only worked with 2.)
>>>
>>> I'm not 100% sure how this will come through, so I've also put it at
>>> http://pastebin.com/60wjXSF3.
>>>
>>> Evan
>>>
>>>
>>> import sys
>>> import pty
>>> import os
>>>
>>> def get_text(filename):
>>> try:
>>> ( child_pid, fd ) = pty.fork()# OK
>>> except OSError as e:
>>> print(str(e))
>>> sys.exit(1)
>>>
>>> if child_pid == 0:
>>> try:
>>> with open("log.txt", "w") as f:
>>> f.write("about to execlp")
>>> os.execlp("cat", "cat", filename)
>>> except:
>>> with open("log.txt", "w") as f:
>>> f.write("could not spawn process")
>>> print("Could not spawn")
>>> sys.exit(1)
>>>
>>> child_pty = os.fdopen(fd)
>>> return child_pty.read()
>>>
>>>
>>> if __name__ == "__main__":
>>> print(get_text("my-pty-test.py"))
>>>
>>>
>>> The read error I get is
>>>
>>> Traceback (most recent call last):
>>>   File "my-pty-test.py", line 28, in 
>>> print(get_text("my-pty-test.py"))
>>>   File "my-pty-test.py", line 24, in get_text
>>> return child_pty.read()
>>> IOError: [Errno 5] Input/output error

at first glance, you have the file open for writing('w'), not
reading('r'), but may not be that.

I'll check if I get a few minutes.



-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: PIL and requests don't get along

[Kushal Kumaran]

On 23 Oct 2012 14:06:59 -0400, r...@panix.com (Roy Smith) wrote:
> I have a url from which I can get an image.  I want to use PIL to
> manipulate that image.  Getting the image is easy:
> 
> >>> import requests
> >>> r = requests.get(url)
> 
> There's a bunch of factory functions for Image, but none of them seem
> to take anything that requests is willing to give you.  Image.new()
> requires that you pass it the image size.  Image.open() takes a file
> object, but
> 
> >>> Image.open(r.raw)
> 
> doesn't work because r.raw gives you a socket which doesn't support
> seek().  I end up doing:
> 
> >>> r = requests.get(url)
> >>> data = cStringIO.StringIO(r.content)
> >>> image = Image.open(data)
> 
> which works, but it's gross.  Is there something I'm missing here?

That is pretty much what the requests module documentation says here:

http://docs.python-requests.org/en/latest/user/quickstart/#binary-response-content

-- 
regards,
kushal
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: turn list of letters into an array of integers

[Demian Brecht]

> Of course, if you want these to be ints, then you can either change the 
> format of your int list, or map(int, list_) if you don't have control over it.


Ugh, I'm tired. Shouldn't map it, the conversion should be done in the list 
comprehension to avoid a needless second list iteration.

K, I'm going to sleep now. :P

Demian Brecht
@demianbrecht
http://demianbrecht.github.com




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: turn list of letters into an array of integers

[David Hutto]

On Wed, Oct 24, 2012 at 1:23 AM, seektime  wrote:
> Here's some example code. The input is a list which is a "matrix" of letters:
>a  b  a
>b  b  a
>
> and I'd like to turn this into a Python array:
>
>   1 2 1
>   2 2 1
>
> so 1 replaces a, and 2 replaces b. Here's the code I have so far:
>
 L=['a b a\n','b b a\n']
 s=' '.join(L)
 seq1=('a','b')
 seq2=('1','2')
 d = dict(zip(seq1,seq2))
 # Define method to replace letters according to dictionary (got this from 
 http://gommeitputor.wordpress.com/2008/09/27/search-replace-multiple-words-or-characters-with-python/).
> ... def replace_all(text, dic):
> ... for i, j in dic.iteritems():
> ... text = text.replace(i, j)
> ... return text
> ...
>
 seq = replace_all(s,d)
 print seq
> 1 2 1
>  2 2 1
>
 seq
> '1 2 1\n 2 2 1\n'
>
I'd suggest, if this is what you're referring to:

x = seq.split('\n  ')
array_list = [ ]
next_3_d_array = []
range_of_seq = len(seq)
for num in range(0,range_of_seq):
   if num % 3 != 0:
   next_3_d_array.append(num)
   if num % 3 == 0:
   array_list.append(next_3_d_array)
   next_3_d_array = [ ]

-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: turn list of letters into an array of integers

[Demian Brecht]

On 2012-10-23, at 10:45 PM, Demian Brecht  wrote:

 list_ = [d[c] for c in s.strip('\n').split()]
 list_
> ['1', '2', '1', '2', '2', '1']


Of course, if you want these to be ints, then you can either change the format 
of your int list, or map(int, list_) if you don't have control over it.

Demian Brecht
@demianbrecht
http://demianbrecht.github.com




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[Alain Ketterlin]

satyam  writes:

> I have a text file like this
>
> A1980JE3937 2732 4195 12.527000
> A1980JE3937 3465 9720 22.00
> A1980JE3937 2732 9720 18.00
> A1980KK18700010 130 303 4.985000
> A1980KK18700010 7 4915 0.435000
[...]
> I want to split the file and get multiple files like
> A1980JE3937.txt and A1980KK18700010.txt, where each file will
> contain column2, 3 and 4.

Sorry for being completely off-topic here, but awk has a very convenient
feature to deal with this. Simply use:

awk '{ print $2,$3,$4 > $1".txt"; }' /path/to/your/file

-- Alain.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: turn list of letters into an array of integers

[Demian Brecht]

On 2012-10-23, at 10:23 PM, seektime  wrote:
> My question is how can I turn "seq" into a python array?


Something like this perhaps?:

>>> alpha = ('a', 'b')
>>> numeric = ('1', '2')
>>> L = ['a b a\n', 'b b a\n']
>>> s = ' '.join(L)
>>> d = dict(zip(alpha, numeric))
>>> list_ = [d[c] for c in s.strip('\n').split()]
>>> list_
['1', '2', '1', '2', '2', '1']

Demian Brecht
@demianbrecht
http://demianbrecht.github.com




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[Demian Brecht]

On 2012-10-23, at 10:24 PM, David Hutto  wrote:

> count = 0
Don't use count.

> for file_data in turn_text_to_txt:
Use enumerate: 

for count, file_data in enumerate(turn_text_to_txt):

> f = open('/home/david/files/%s_%s.txt' % (file_data.split(' ')[0], count), 
> 'w')
Use with: 

with open('file path', 'w') as f:
f.write('data')

Not only is it shorter, but it automatically closes the file once you've come 
out of the inner block, whether successfully or erroneously.


Demian Brecht
@demianbrecht
http://demianbrecht.github.com




-- 
http://mail.python.org/mailman/listinfo/python-list

turn list of letters into an array of integers

[seektime]

Here's some example code. The input is a list which is a "matrix" of letters:
   a  b  a
   b  b  a

and I'd like to turn this into a Python array:

  1 2 1
  2 2 1

so 1 replaces a, and 2 replaces b. Here's the code I have so far:

>>> L=['a b a\n','b b a\n']
>>> s=' '.join(L)
>>> seq1=('a','b')
>>> seq2=('1','2')
>>> d = dict(zip(seq1,seq2))
>>> # Define method to replace letters according to dictionary (got this from 
>>> http://gomputor.wordpress.com/2008/09/27/search-replace-multiple-words-or-characters-with-python/).
... def replace_all(text, dic):
... for i, j in dic.iteritems():
... text = text.replace(i, j)
... return text
... 

>>> seq = replace_all(s,d)
>>> print seq
1 2 1
 2 2 1

>>> seq
'1 2 1\n 2 2 1\n'

My question is how can I turn "seq" into a python array?

Thanks
Michael
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[David Hutto]

On Tue, Oct 23, 2012 at 11:01 PM, satyam  wrote:
> I have a text file like this
>
> A1980JE3937 2732 4195 12.527000
> A1980JE3937 3465 9720 22.00
> A1980JE3937 1853 3278 12.50
> A1980JE3937 2732 2732 187.50
> A1980JE3937 19 4688 3.619000
> A1980JE3937 2995 9720 6.667000
> A1980JE3937 1603 9720 30.00
> A1980JE3937 234 4195 42.416000
> A1980JE3937 2732 9720 18.00
> A1980KK18700010 130 303 4.985000
> A1980KK18700010 7 4915 0.435000
> A1980KK18700010 25 1620 1.722000
> A1980KK18700010 25 186 0.654000
> A1980KK18700010 50 130 3.199000
> A1980KK18700010 186 3366 4.78
> A1980KK18700010 30 186 1.285000
> A1980KK18700010 30 185 4.395000
> A1980KK18700010 185 186 9.00
> A1980KK18700010 25 30 3.493000
>
> I want to split the file and get multiple files like A1980JE3937.txt and 
> A1980KK18700010.txt, where each file will contain column2, 3 and 4.
> Thanks
> Satyam




#parse through the lines
turn_text_to_txt = ['A1980JE3937 2732 4195 12.527000',
'A1980JE3937 3465 9720 22.00',
'A1980JE3937 1853 3278 12.50',
'A1980JE3937 2732 2732 187.50',
'A1980JE3937 19 4688 3.619000',
'A1980KK18700010 30 186 1.285000',
'A1980KK18700010 30 185 4.395000',
'A1980KK18700010 185 186 9.00',
'A1980KK18700010 25 30 3.493000']
#then split and open a file for writing to create the file

#then start a count to add an extra number, because the files #you're
opening have the same name in some, which will #cause python to
overwrite the last file with that name.

#So I added an extra integer count after an underscore to #keep all
files, even if the have the first base number.

count = 0

for file_data in turn_text_to_txt:

#open the file for writing in 'w' mode so it creates the file, and
#adds in the appropriate data, including the extra count i#nteger just
in case there are files with the same name.

f = open('/home/david/files/%s_%s.txt' % (file_data.split(' ')[0], 
count), 'w')

#write the data to the file, however this is in list format, I could
go further, but need a little time for a few other things.

f.write( str(file_data.split(' ')[1:]))

#close the file 
f.close()

#increment the count for the next iteration, if necessary, and #again,
this is just in case the files have the same name, and #need an
additive.
#   count += 1


Full code from above, without comments:

turn_text_to_txt = ['A1980JE3937 2732 4195 12.527000',
'A1980JE3937 3465 9720 22.00',
'A1980JE3937 1853 3278 12.50',
'A1980JE3937 2732 2732 187.50',
'A1980JE3937 19 4688 3.619000',
'A1980KK18700010 30 186 1.285000',
'A1980KK18700010 30 185 4.395000',
'A1980KK18700010 185 186 9.00',
'A1980KK18700010 25 30 3.493000']
#then split and open a file for writing to create the file
count = 0

for file_data in turn_text_to_txt:

print '/home/david/files/%s.txt' % (file_data.split(' ')[0])

f = open('/home/david/files/%s_%s.txt' % (file_data.split(' ')[0], 
count), 'w')

f.write( str(file_data.split(' ')[1:]))

f.close()

count += 1




-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The pty module, reading from a pty, and Python 2/3

[Evan Driscoll]

OK, one more self-reply. :-)

I found http://bugs.python.org/issue5380 which seems to be relevant. 
Sounds like the OS is returning an error, and that's naturally being 
propagated through Python. And further testing reveals the problem only 
surfaces when the child actually exits -- if the child writes data to 
the pipe and is more enthusiastic about living, then the parent can read 
it successfully. While it'd be nice if my example worked, for my actual 
purpose I think that's good enough (I just won't be able to test *quite* 
as easily for a while).


I am still curious if anyone know why it worked in 2 though.

Evan



On 10/23/2012 10:03 PM, Evan Driscoll wrote:

Oh, and a little more information:

The log.txt file I create has the message that it's "about to execlp", 
and the exec() *does* actually happen -- the IOError is raised after 
the child process quits.


Evan



On 10/23/2012 09:59 PM, Evan Driscoll wrote:
I have the following program. Everything is sunshine and rainbows 
when I run in in Python 2, but when I run it under Python 3 I get an 
IOError. 2to3 only reports one dumb suggestion re. a print call 
(which I can get rid of by importing __future__'s print_function, and 
then it just suggests removing that import).


Can anyone shed any light? I am on Ubuntu Linux with Python 2.7.3 and 
3.2.3.



(Just for the record, I figured out that it ran under Python 2 by 
accident as I was reducing it for a "why doesn't this run?" email. 
:-) I'm not super-familiar with Py3 as I've mostly only worked with 2.)


I'm not 100% sure how this will come through, so I've also put it at 
http://pastebin.com/60wjXSF3.


Evan


import sys
import pty
import os

def get_text(filename):
try:
( child_pid, fd ) = pty.fork()# OK
except OSError as e:
print(str(e))
sys.exit(1)

if child_pid == 0:
try:
with open("log.txt", "w") as f:
f.write("about to execlp")
os.execlp("cat", "cat", filename)
except:
with open("log.txt", "w") as f:
f.write("could not spawn process")
print("Could not spawn")
sys.exit(1)

child_pty = os.fdopen(fd)
return child_pty.read()


if __name__ == "__main__":
print(get_text("my-pty-test.py"))


The read error I get is

Traceback (most recent call last):
  File "my-pty-test.py", line 28, in 
print(get_text("my-pty-test.py"))
  File "my-pty-test.py", line 24, in get_text
return child_pty.read()
IOError: [Errno 5] Input/output error





--
http://mail.python.org/mailman/listinfo/python-list

The pty module, reading from a pty, and Python 2/3

[Evan Driscoll]

I have the following program. Everything is sunshine and rainbows when I 
run in in Python 2, but when I run it under Python 3 I get an IOError. 
2to3 only reports one dumb suggestion re. a print call (which I can get 
rid of by importing __future__'s print_function, and then it just 
suggests removing that import).


Can anyone shed any light? I am on Ubuntu Linux with Python 2.7.3 and 
3.2.3.



(Just for the record, I figured out that it ran under Python 2 by 
accident as I was reducing it for a "why doesn't this run?" email. :-) 
I'm not super-familiar with Py3 as I've mostly only worked with 2.)


I'm not 100% sure how this will come through, so I've also put it at 
http://pastebin.com/60wjXSF3.


Evan


import sys
import pty
import os

def get_text(filename):
try:
( child_pid, fd ) = pty.fork()# OK
except OSError as e:
print(str(e))
sys.exit(1)

if child_pid == 0:
try:
with open("log.txt", "w") as f:
f.write("about to execlp")
os.execlp("cat", "cat", filename)
except:
with open("log.txt", "w") as f:
f.write("could not spawn process")
print("Could not spawn")
sys.exit(1)

child_pty = os.fdopen(fd)
return child_pty.read()


if __name__ == "__main__":
print(get_text("my-pty-test.py"))


The read error I get is

Traceback (most recent call last):
  File "my-pty-test.py", line 28, in 
print(get_text("my-pty-test.py"))
  File "my-pty-test.py", line 24, in get_text
return child_pty.read()
IOError: [Errno 5] Input/output error

--
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[satyam mukherjee]

Thanks I will take a look...My actual data is 2.5Gb in size.
Satyam

On Tue, Oct 23, 2012 at 10:43 PM, Jason Friedman wrote:

> On Tue, Oct 23, 2012 at 9:01 PM, satyam  wrote:
> > I have a text file like this
> >
> > A1980JE3937 2732 4195 12.527000
> > A1980JE3937 3465 9720 22.00
> > A1980JE3937 1853 3278 12.50
> > A1980JE3937 2732 2732 187.50
> > A1980JE3937 19 4688 3.619000
> > A1980KK18700010 30 186 1.285000
> > A1980KK18700010 30 185 4.395000
> > A1980KK18700010 185 186 9.00
> > A1980KK18700010 25 30 3.493000
> >
> > I want to split the file and get multiple files like A1980JE3937.txt
> and A1980KK18700010.txt, where each file will contain column2, 3 and 4.
>
> Unless your source file is very large this should be sufficient:
>
> $ cat source
> A1980JE3937 2732 4195 12.527000
> A1980JE3937 3465 9720 22.00
> A1980JE3937 1853 3278 12.50
> A1980JE3937 2732 2732 187.50
> A1980JE3937 19 4688 3.619000
> A1980JE3937 2995 9720 6.667000
> A1980JE3937 1603 9720 30.00
> A1980JE3937 234 4195 42.416000
> A1980JE3937 2732 9720 18.00
> A1980KK18700010 130 303 4.985000
> A1980KK18700010 7 4915 0.435000
> A1980KK18700010 25 1620 1.722000
> A1980KK18700010 25 186 0.654000
> A1980KK18700010 50 130 3.199000
> A1980KK18700010 186 3366 4.78
> A1980KK18700010 30 186 1.285000
> A1980KK18700010 30 185 4.395000
> A1980KK18700010 185 186 9.00
> A1980KK18700010 25 30 3.493000
>
> $ python3
> Python 3.2.3 (default, Sep 10 2012, 18:14:40)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> for line in open("source"):
> ... file_name, remainder = line.strip().split(None, 1)
> ... with open(file_name + ".txt", "a") as writer:
> ... print(remainder, file=writer)
> ...
> >>>
>
> $ ls *txt
> A1980JE3937.txt  A1980KK18700010.txt
>
> $ cat A1980JE3937.txt
> 2732 4195 12.527000
> 3465 9720 22.00
> 1853 3278 12.50
> 2732 2732 187.50
> 19 4688 3.619000
> 2995 9720 6.667000
> 1603 9720 30.00
> 234 4195 42.416000
> 2732 9720 18.00
>



-- 
---
WHEN LIFE GIVES U HUNDRED REASONS TO CRY,SHOW LIFE THAT U HAVE THOUSAND
REASONS TO SMILE :-)

satyam mukherjee
224-436-3672 (Mob)
847-491-7238 (Off)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[Jason Friedman]

On Tue, Oct 23, 2012 at 9:01 PM, satyam  wrote:
> I have a text file like this
>
> A1980JE3937 2732 4195 12.527000
> A1980JE3937 3465 9720 22.00
> A1980JE3937 1853 3278 12.50
> A1980JE3937 2732 2732 187.50
> A1980JE3937 19 4688 3.619000
> A1980KK18700010 30 186 1.285000
> A1980KK18700010 30 185 4.395000
> A1980KK18700010 185 186 9.00
> A1980KK18700010 25 30 3.493000
>
> I want to split the file and get multiple files like A1980JE3937.txt and 
> A1980KK18700010.txt, where each file will contain column2, 3 and 4.

Unless your source file is very large this should be sufficient:

$ cat source
A1980JE3937 2732 4195 12.527000
A1980JE3937 3465 9720 22.00
A1980JE3937 1853 3278 12.50
A1980JE3937 2732 2732 187.50
A1980JE3937 19 4688 3.619000
A1980JE3937 2995 9720 6.667000
A1980JE3937 1603 9720 30.00
A1980JE3937 234 4195 42.416000
A1980JE3937 2732 9720 18.00
A1980KK18700010 130 303 4.985000
A1980KK18700010 7 4915 0.435000
A1980KK18700010 25 1620 1.722000
A1980KK18700010 25 186 0.654000
A1980KK18700010 50 130 3.199000
A1980KK18700010 186 3366 4.78
A1980KK18700010 30 186 1.285000
A1980KK18700010 30 185 4.395000
A1980KK18700010 185 186 9.00
A1980KK18700010 25 30 3.493000

$ python3
Python 3.2.3 (default, Sep 10 2012, 18:14:40)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> for line in open("source"):
... file_name, remainder = line.strip().split(None, 1)
... with open(file_name + ".txt", "a") as writer:
... print(remainder, file=writer)
...
>>>

$ ls *txt
A1980JE3937.txt  A1980KK18700010.txt

$ cat A1980JE3937.txt
2732 4195 12.527000
3465 9720 22.00
1853 3278 12.50
2732 2732 187.50
19 4688 3.619000
2995 9720 6.667000
1603 9720 30.00
234 4195 42.416000
2732 9720 18.00
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Split single file into multiple files based on patterns

[Jason Friedman]

> I have a text file like this
>
> A1980JE3937 2732 4195 12.527000
> A1980JE3937 3465 9720 22.00
> A1980KK18700010 186 3366 4.78
> A1980KK18700010 30 186 1.285000
> A1980KK18700010 30 185 4.395000
> A1980KK18700010 185 186 9.00
> A1980KK18700010 25 30 3.493000
>
> I want to split the file and get multiple files like A1980JE3937.txt and 
> A1980KK18700010.txt, where each file will contain column2, 3 and 4.

The sample data above shows the data grouped by file name.  Will this
be true generally?
-- 
http://mail.python.org/mailman/listinfo/python-list

Split single file into multiple files based on patterns

[satyam]

I have a text file like this

A1980JE3937 2732 4195 12.527000
A1980JE3937 3465 9720 22.00
A1980JE3937 1853 3278 12.50
A1980JE3937 2732 2732 187.50
A1980JE3937 19 4688 3.619000
A1980JE3937 2995 9720 6.667000
A1980JE3937 1603 9720 30.00
A1980JE3937 234 4195 42.416000
A1980JE3937 2732 9720 18.00
A1980KK18700010 130 303 4.985000
A1980KK18700010 7 4915 0.435000
A1980KK18700010 25 1620 1.722000
A1980KK18700010 25 186 0.654000
A1980KK18700010 50 130 3.199000
A1980KK18700010 186 3366 4.78
A1980KK18700010 30 186 1.285000
A1980KK18700010 30 185 4.395000
A1980KK18700010 185 186 9.00
A1980KK18700010 25 30 3.493000

I want to split the file and get multiple files like A1980JE3937.txt and 
A1980KK18700010.txt, where each file will contain column2, 3 and 4.
Thanks
Satyam
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: The pty module, reading from a pty, and Python 2/3

[Evan Driscoll]

Oh, and a little more information:

The log.txt file I create has the message that it's "about to execlp", 
and the exec() *does* actually happen -- the IOError is raised after the 
child process quits.


Evan



On 10/23/2012 09:59 PM, Evan Driscoll wrote:
I have the following program. Everything is sunshine and rainbows when 
I run in in Python 2, but when I run it under Python 3 I get an 
IOError. 2to3 only reports one dumb suggestion re. a print call (which 
I can get rid of by importing __future__'s print_function, and then it 
just suggests removing that import).


Can anyone shed any light? I am on Ubuntu Linux with Python 2.7.3 and 
3.2.3.



(Just for the record, I figured out that it ran under Python 2 by 
accident as I was reducing it for a "why doesn't this run?" email. :-) 
I'm not super-familiar with Py3 as I've mostly only worked with 2.)


I'm not 100% sure how this will come through, so I've also put it at 
http://pastebin.com/60wjXSF3.


Evan


import sys
import pty
import os

def get_text(filename):
try:
( child_pid, fd ) = pty.fork()# OK
except OSError as e:
print(str(e))
sys.exit(1)

if child_pid == 0:
try:
with open("log.txt", "w") as f:
f.write("about to execlp")
os.execlp("cat", "cat", filename)
except:
with open("log.txt", "w") as f:
f.write("could not spawn process")
print("Could not spawn")
sys.exit(1)

child_pty = os.fdopen(fd)
return child_pty.read()


if __name__ == "__main__":
print(get_text("my-pty-test.py"))


The read error I get is

Traceback (most recent call last):
  File "my-pty-test.py", line 28, in 
print(get_text("my-pty-test.py"))
  File "my-pty-test.py", line 24, in get_text
return child_pty.read()
IOError: [Errno 5] Input/output error



--
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

On Tue, Oct 23, 2012 at 8:06 PM, Oscar Benjamin
 wrote:
> On 23 October 2012 15:31, Virgil Stokes  wrote:
>> I am working with some rather large data files (>100GB) that contain time
>> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
>> format. I perform various types of processing on these data (e.g. moving
>> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
>> manner and only a small number of these data need be stored in RAM when
>> being processed. When performing Kalman-filtering (forward in time pass, k =
>> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
>> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
>> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
>> variables saved to an external file from the forward pass, in reverse order
>> --- from last written to first written.
>>
>> Finally, to my question --- What is a fast way to write these variables to
>> an external file and then read them in backwards?
>
> You mentioned elsewhere that you are using numpy. I'll assume that the
> data you want to read/write are numpy arrays.

If that is the case always timeit. The following is an example of 3
functions, with repetitions of time that give an average:

import timeit
#3 dimensional matrix
x_dim = -1
y_dim = -1
z_dim = -1
s = """\

x_dim = -1
y_dim = -1
z_dim = -1
dict_1 = {}

for i in xrange(0,6):
x_dim = 1
y_dim = 1
z_dim = 1
dict_1['%s' % (i) ] = ['x = %i' % (x_dim), 'y = %i' % (y_dim),  'z =
%i' % (z_dim)]

"""

t = """\
import numpy
numpy.array([[ 1.,  0.,  0.],
   [ 0.,  1.,  2.]])
"""

u = """\
list_count = 0
an_array = []
for i in range(0,10):

if list_count > 3:
break

if i % 3 != 0:
an_array.append(i)

if i % 3 == 0:
list_count += 1

"""
print timeit.timeit(stmt=s, number=10)
print timeit.timeit(stmt=t, number=10)
print timeit.timeit(stmt=u, number=10)


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Tim Chase]

On 10/23/12 13:37, Virgil Stokes wrote:
> Yes, I do wish to inverse the order,  but the "forward in time"
> file will be in binary.

Your original post said:

> The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format

making it hard to know what sort of data is in this file.

So I guess it would help to have some sample data to work with, even
if it's just some dummy data and a raw processing loop without doing
anything inside it.  Something like the output of either of these

  $ xxd forward_data.txt | head -50 > forward_head.txt
  $ od forward_data.txt | head -50 > forward_head.txt

plus a basic loop to show how you're extracting the values:

  for line in file("forward_head.txt"):
data1, data2, data3 = process(line)

and how you want to reverse over them:

  for line in file("reversed.txt"):
if same_processing_as_forward_source:
  data1, data2, data3 = process(line)
else:
  data1, data2, data3 = other_process(line)

or do you want something more like

  for line in super_reverse_magic(file("forward_head.txt")):
data1, data2, data3 = process(line)

?

-tkc



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Oscar Benjamin]

On 23 October 2012 15:31, Virgil Stokes  wrote:
> I am working with some rather large data files (>100GB) that contain time
> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
> format. I perform various types of processing on these data (e.g. moving
> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
> manner and only a small number of these data need be stored in RAM when
> being processed. When performing Kalman-filtering (forward in time pass, k =
> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
> variables saved to an external file from the forward pass, in reverse order
> --- from last written to first written.
>
> Finally, to my question --- What is a fast way to write these variables to
> an external file and then read them in backwards?

You mentioned elsewhere that you are using numpy. I'll assume that the
data you want to read/write are numpy arrays.

Numpy arrays can be written very efficiently in binary form using
tofile/fromfile:

>>> import numpy
>>> a = numpy.array([1, 2, 5], numpy.int64)
>>> a
array([1, 2, 5])
>>> with open('data.bin', 'wb') as f:
...   a.tofile(f)
...

You can then reload the array with:

>>> with open('data.bin', 'rb') as f:
...   a2 = numpy.fromfile(f, numpy.int64)
...
>>> a2
array([1, 2, 5])

Numpy arrays can be reversed before writing or after reading using;

>>> a2
array([1, 2, 5])
>>> a2[::-1]
array([5, 2, 1])

Assuming you wrote the file forwards you can make an iterator to yield
the file in chunks backwards like so (untested):

def read_backwards(f, dtype, chunksize=1024 ** 2):
dtype = numpy.dtype(dtype)
nbytes = chunksize * dtype.itemsize
f.seek(0, 2)
fpos = f.tell()
while fpos > nbytes:
f.seek(fpos, 0)
yield numpy.fromfile(f, dtype, chunksize)[::-1]
fpos -= nbytes
yield numpy.fromfile(f, dtype)[::-1]


Oscar
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A desperate lunge for on-topic-ness

[Steven D'Aprano]

On Tue, 23 Oct 2012 17:24:34 -0600, Ian Kelly wrote:

> On Tue, Oct 23, 2012 at 4:34 PM, Steven D'Aprano
>  wrote:
>> On Tue, 23 Oct 2012 10:50:11 -0600, Ian Kelly wrote:
>>
 if someone is foolish enough to use the

 from xyz import *

 notation...
>>>
>>> It's already a SyntaxError to use a wildcard import anywhere other
>>> than the module level, so its use can only affect global variables.
>>
>> In Python 3.x.
>>
>> In Python 2.x, which includes the most recent version of three of the
>> four "big implementations" (PyPy, Jython, IronPython) it is still
>> legal, at least in theory.
> 
> If we're talking about making changes to the language, then we're
> clearly talking about Python 3.x and beyond.  There are no more major
> releases planned for 2.x.

In what way does "it is ALREADY a SyntaxError" [emphasis added] refer to 
making future changes to the language? :)

My point is that for probably 80% or more of Python users, it is not the 
case that wildcard imports in functions are already a syntax error. 
Anyone using CPython 2.x, PyPy, Jython or IronPython have such a syntax 
error to look forward to in the future, but not now. Until then, they 
have to deal with syntax warnings, implementation-dependent behaviour, 
and as far as I can see, an outright language bug in Jython, but no 
syntax errors.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

On Tue, Oct 23, 2012 at 7:35 PM, emile  wrote:
> On 10/23/2012 04:19 PM, David Hutto wrote:
>>
>> Whether this is fast enough, or not, I don't know:
>
>
> well, the OP's original post started with
>   "I am working with some rather large data files (>100GB)..."

Well, is this a dedicated system, and one that they have the budget to upgrade?

Data files have some sort of parsing, unless it's one huge dict, or
list, so there has to be an average size to the parse.

So big O notation should begin to refine without a full file.

>
>
>> filename = "data_file.txt"
>> f = open(filename, 'r')
>> forward =  [line.rstrip('\n') for line in f.readlines()]
>
>
> f.readlines() will be big(!) and have overhead... and forward results in
> something again as big.
>
Not if an average can be taken, and then refined as the actual gigs
are being iterated through.

>
>> backward =  [line.rstrip('\n') for line in reversed(forward)]
>
>
> and defining backward looks to me to require space to build backward and
> hold reversed(forward)
>
> So, let's see, at that point in time (building backward) you've got
> probably somewhere close to 400-500Gb in memory.
>
> My guess -- probably not so fast.  Thrashing is sure to be a factor on all
> but machines I'll never have a chance to work on.

But does the OP have access? They never stated their hardware, and
upgradable budget.

>
>
>> f.close()
>> print forward, "\n\n", "\n\n", backward, "\n"
>
>
>
> It's good to retain context.

Trying to practice good form ;).


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: regex function driving me nuts

[cyberdicks]

Stripping the line did it !!!
Thank you very much to all !!! 
Cheers! :-) 
Martin 


Le mardi 23 octobre 2012 16:36:44 UTC-4, Vlastimil Brom a écrit :
> 2012/10/23 MartinD. 
> 
> > Hi,
> 
> >
> 
> > I'm new to Python.
> 
> > Does someone has an idea what's wrong.  I tried everything. The only regex 
> > that is tested is the last one in a whole list of regex in keywords.txt
> 
> > Thanks!
> 
> > Martin
> 
> >
> 
> >
> 
> > 
> 
> > def checkKeywords( str, lstKeywords ):
> 
> >
> 
> > for regex in lstKeywords:
> 
> > match = re.search(regex, str,re.IGNORECASE)
> 
> > # If-statement after search() tests if it succeeded
> 
> > if match:
> 
> > print match.group() ##just debugging
> 
> > return match.group() ## 'found!
> 
> >
> 
> > return
> 
> >
> 
> > #
> 
> >
> 
> > keywords1 = [line for line in open('keywords1.txt')]
> 
> > resultKeywords1 = checkKeywords("string_to_test",keywords1)
> 
> > print resultKeywords1
> 
> >
> 
> > --
> 
> > http://mail.python.org/mailman/listinfo/python-list
> 
> 
> 
> Hi,
> 
> just a wild guess, as I don't have access to  containing the list of
> 
> potentially problematic regex patterns
> 
> does:
> 
> keywords1 = [line.strip() for line in open('keywords1.txt')]
> 
> possibly fix yout problem?
> 
> the lines of the file iterator also preserve newlines, which might not
> 
> be expected in your keywords, strip() removes (be default) any
> 
> starting and tryiling whitespace.
> 
> 
> 
> hth,
> 
>   vbr
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Paul Rubin]

Virgil Stokes  writes:
> Yes, I do wish to inverse the order,  but the "forward in time" file
> will be in binary.

I really think it will be simplest to just write the file in forward
order, then use mmap to read it one record at a time.  It might be
possible to squeeze out a little more performance with reordering tricks
but that's the first thing to try.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Getting a TimedRotatingFileHandler not to put two dates in the same file?

[Dave Angel]

On 10/23/2012 11:23 AM, David M Chess wrote:
> We have a TimedRotatingFileHandler with when='midnight'

You give us no clue what's in this class, or how it comes up with the
filenames used.

> . 
>
> This works great, splitting the log information across files by date, as 
> long as the process is actually up at midnight.
>
> But now the users have noticed that if the process isn't up at midnight, 
> they can end up with lines from two (or I guess potentially more) dates in 
> the same log file.
>
> Is there some way to fix this, either with cleverer arguments into the 
> TimedRotatingFileHandler, or by some plausible subclassing of it or its 
> superclass?

Why not use the date itself to derive the filename?  And check whether
the date has changed since the last update, and if so, close and
reopen.  Midnight is irrelevant.

> Or am I misinterpreting the symptoms somehow?
>
> Tx much!
> DC
>
>

-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[emile]

On 10/23/2012 04:19 PM, David Hutto wrote:

Whether this is fast enough, or not, I don't know:


well, the OP's original post started with
  "I am working with some rather large data files (>100GB)..."


filename = "data_file.txt"
f = open(filename, 'r')
forward =  [line.rstrip('\n') for line in f.readlines()]


f.readlines() will be big(!) and have overhead... and forward results in 
something again as big.



backward =  [line.rstrip('\n') for line in reversed(forward)]


and defining backward looks to me to require space to build backward and 
hold reversed(forward)


So, let's see, at that point in time (building backward) you've got
probably somewhere close to 400-500Gb in memory.

My guess -- probably not so fast.  Thrashing is sure to be a factor on 
all but machines I'll never have a chance to work on.




f.close()
print forward, "\n\n", "\n\n", backward, "\n"



It's good to retain context.

Emile

--
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

On Tue, Oct 23, 2012 at 6:53 PM, Steven D'Aprano
 wrote:
> On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:
>
>> On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes  wrote:
>>> I am working with some rather large data files (>100GB)
> [...]
>>> Finally, to my question --- What is a fast way to write these variables
>>> to an external file and then read them in backwards?
>>
>> Don't forget to use timeit for an average OS utilization.
>
> Given that the data files are larger than 100 gigabytes, the time
> required to process each file is likely to be in hours, not microseconds.
> That being the case, timeit is the wrong tool for the job, it is
> optimized for timings tiny code snippets. You could use it, of course,
> but the added inconvenience doesn't gain you any added accuracy.

It depends on the end result, and the fact that if the iterations
themselves are about the same time, then just using a segment of the
iterations could be scaled down, and a full run might be worth it, if
you have a second computer running optimization.

>
> Here's a neat context manager that makes timing long-running code simple:
>
>
> http://code.activestate.com/recipes/577896


I'll test this out for big O notation later. For the OP:

http://en.wikipedia.org/wiki/Big_O_notation





>
>
>
>> I'd suggest two list comprehensions for now, until I've reviewed it some
>> more:
>
> I would be very surprised if the poster will be able to fit 100 gigabytes
> of data into even a single list comprehension, let alone two.
Again, these can be scaled depending on the operations of the function
in question, and the average time of aforementioned function(s)

>
> This is a classic example of why the old external processing algorithms
> of the 1960s and 70s will never be obsolete. No matter how much memory
> you have, there will always be times when you want to process more data
> than you can fit into memory

This is a common misconception. You can engineer a device that
accommodates this if it's a direct experimental necessity.
>

-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A desperate lunge for on-topic-ness

[Ian Kelly]

On Tue, Oct 23, 2012 at 4:34 PM, Steven D'Aprano
 wrote:
> On Tue, 23 Oct 2012 10:50:11 -0600, Ian Kelly wrote:
>
>>> if someone is foolish enough to use the
>>>
>>> from xyz import *
>>>
>>> notation...
>>
>> It's already a SyntaxError to use a wildcard import anywhere other than
>> the module level, so its use can only affect global variables.
>
> In Python 3.x.
>
> In Python 2.x, which includes the most recent version of three of the
> four "big implementations" (PyPy, Jython, IronPython) it is still legal,
> at least in theory.

If we're talking about making changes to the language, then we're
clearly talking about Python 3.x and beyond.  There are no more major
releases planned for 2.x.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: How to do Table Reflection and MySQL?

[Nick Sabalausky]

On Tue, 23 Oct 2012 22:42:08 +
"Prasad, Ramit"  wrote:

> Nick Sabalausky wrote:
> > On Mon, 22 Oct 2012 14:35:23 -0700 (PDT)
> > darnold  wrote:
> > >
> > > i'm not brave enough to dig too deeply into SQLAlchemy, but maybe
> > > this will help? :
> > >
> > > http://kashififtikhar.blogspot.com/2010/07/using-sqlalchemy-reflection-with-pylons.html
> > >
> > > that came up from googling "sqlalchemy table reflection tutorial".
> > 
> > Thanks, your view of Google seems to be far better tailored for
> > Python than mine is, that doesn't come up for me anywhere on the
> > first five pages of results for that query.
> > 
> > Unfortunately the info on that page doesn't seem to work for me:
> > 
> > --
> > from sqlalchemy import *
> > from sqlalchemy.orm import sessionmaker
> > 
> > engine = create_engine(my connection string)
> > meta = MetaData()
> > meta.bind = engine
> > meta.reflect()
> > 
> > Session = sessionmaker(bind=engine)
> > session = Session()
> > 
> > res = session.query(user).filter(user.name=="bert").first()
> > print res.name
> > --
> > 
> > That just gives me:
> > 
> > NameError: name 'user' is not defined
> > 
> > (And yes, the code given on that page to print out the table info
> > *does* indicate a table named 'user' was found.)
> 
> This does not seem to be a SQLAlchemy problem. Instead it seems
> there is not a variable called `name`. 

Oops, yea, it's supposed to be:

meta.tables["user"].columns["name"]

Not:

meta.tables["user"].name

Works now, thanks all.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

Whether this is fast enough, or not, I don't know:


filename = "data_file.txt"
f = open(filename, 'r')
forward =  [line.rstrip('\n') for line in f.readlines()]
backward =  [line.rstrip('\n') for line in reversed(forward)]
f.close()
print forward, "\n\n", "\n\n", backward, "\n"


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Demian Brecht]

> This is a classic example of why the old external processing algorithms 
> of the 1960s and 70s will never be obsolete. No matter how much memory 
> you have, there will always be times when you want to process more data 
> than you can fit into memory.


But surely nobody will *ever* need more than 640k…

Right?

Demian Brecht
@demianbrecht
http://demianbrecht.github.com




-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Steven D'Aprano]

On Tue, 23 Oct 2012 17:50:55 -0400, David Hutto wrote:

> On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes  wrote:
>> I am working with some rather large data files (>100GB) 
[...]
>> Finally, to my question --- What is a fast way to write these variables
>> to an external file and then read them in backwards?
> 
> Don't forget to use timeit for an average OS utilization.

Given that the data files are larger than 100 gigabytes, the time 
required to process each file is likely to be in hours, not microseconds. 
That being the case, timeit is the wrong tool for the job, it is 
optimized for timings tiny code snippets. You could use it, of course, 
but the added inconvenience doesn't gain you any added accuracy.

Here's a neat context manager that makes timing long-running code simple:


http://code.activestate.com/recipes/577896



> I'd suggest two list comprehensions for now, until I've reviewed it some
> more:

I would be very surprised if the poster will be able to fit 100 gigabytes 
of data into even a single list comprehension, let alone two.

This is a classic example of why the old external processing algorithms 
of the 1960s and 70s will never be obsolete. No matter how much memory 
you have, there will always be times when you want to process more data 
than you can fit into memory.



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

> Missed the part about it being a file. Use:
>
> forward =  ["%i = %s" % (i,chr(i)) for i in range(33,126)]
> backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]
>
> print forward,backward
>
This was a dud, let me rework it real quick, I deleted what i had, and
accidentally wrote the wrong function.

>
>
>
> --
> Best Regards,
> David Hutto
> CEO: http://www.hitwebdevelopment.com



-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: SQLAlchemy: How to do Table Reflection and MySQL?

[Prasad, Ramit]

Nick Sabalausky wrote:
> On Mon, 22 Oct 2012 14:35:23 -0700 (PDT)
> darnold  wrote:
> >
> > i'm not brave enough to dig too deeply into SQLAlchemy, but maybe this
> > will help? :
> >
> > http://kashififtikhar.blogspot.com/2010/07/using-sqlalchemy-reflection-with-pylons.html
> >
> > that came up from googling "sqlalchemy table reflection tutorial".
> 
> Thanks, your view of Google seems to be far better tailored for Python
> than mine is, that doesn't come up for me anywhere on the first five
> pages of results for that query.
> 
> Unfortunately the info on that page doesn't seem to work for me:
> 
> --
> from sqlalchemy import *
> from sqlalchemy.orm import sessionmaker
> 
> engine = create_engine(my connection string)
> meta = MetaData()
> meta.bind = engine
> meta.reflect()
> 
> Session = sessionmaker(bind=engine)
> session = Session()
> 
> res = session.query(user).filter(user.name=="bert").first()
> print res.name
> --
> 
> That just gives me:
> 
> NameError: name 'user' is not defined
> 
> (And yes, the code given on that page to print out the table info
> *does* indicate a table named 'user' was found.)

This does not seem to be a SQLAlchemy problem. Instead it seems
there is not a variable called `name`. If you define 
the appropriate user it does it work? (From that page it seems 
like user should be a blank class like the following).

class user(object): pass

> 
> I also tried this which also fails:
> 
> res =
> session.query(meta.tables["user"]).filter(meta.tables["user"].name=="bert").first()
> 
> sqlalchemy.exc.ArgumentError: filter() argument must be of type
> sqlalchemy.sql.ClauseElement or string
> 
> The page you linked to appears to get around the matter by manually
> setting up tables filled with the reflected info, but that seems to
> defeat much of the point for me. I may as well just set up the tables
> manually without the reflection, which is what I'll probably do.
> 
> Maybe I just misunderstood what was meant in the SQLAlchemy docs here?:
> 
> "but note that SA can also “import” whole sets of Table objects
> automatically from an existing database (this process is called table
> reflection)."  --
> http://docs.sqlalchemy.org/en/rel_0_7/core/tutorial.html
> 
> It said that but then didn't say how and didn't link to any info on how.

Ramit Prasad



This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

> Don't forget to use timeit for an average OS utilization.
>
> I'd suggest two list comprehensions for now, until I've reviewed it some more:
>
> forward =  ["%i = %s" % (i,chr(i)) for i in range(33,126)]
> backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]
>
> for var in forward:
> print var
>
> for var in backward:
> print var
>
> You could also use a dict, and iterate through a straight loop that
> assigned a front and back to a dict_one =  {0 : [0.100], 1 : [1.99]}
> and the iterate through the loop, and call the first or second in the
> dict's var list for frontwards , or backwards calls.
>
>
> But there might be faster implementations, depending on other
> function's usage of certain lower level functions.
>

Missed the part about it being a file. Use:

forward =  ["%i = %s" % (i,chr(i)) for i in range(33,126)]
backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]

print forward,backward




-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A desperate lunge for on-topic-ness

[Steven D'Aprano]

On Tue, 23 Oct 2012 10:50:11 -0600, Ian Kelly wrote:

>> if someone is foolish enough to use the
>>
>> from xyz import *
>>
>> notation...
> 
> It's already a SyntaxError to use a wildcard import anywhere other than
> the module level, so its use can only affect global variables.

In Python 3.x. 

In Python 2.x, which includes the most recent version of three of the 
four "big implementations" (PyPy, Jython, IronPython) it is still legal, 
at least in theory.

I haven't tested PyPy, but IronPython 2.6 allows wildcard imports inside 
functions without even a warning. Bizarrely, Jython 2.5 *appears* to 
allow them with only a warning, but they don't take:

steve@runes:~$ jython
Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19)
[OpenJDK Client VM (Sun Microsystems Inc.)] on java1.6.0_18
Type "help", "copyright", "credits" or "license" for more information.
>>> def test():
... from math import *
... return cos
...
:2: SyntaxWarning: import * only allowed at module level
>>> test()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in test
NameError: global name 'cos' is not defined


So, legal or not, they're definitely something you want to avoid.


-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: SQLAlchemy: How to do Table Reflection and MySQL?

[Nick Sabalausky]

On Mon, 22 Oct 2012 14:35:23 -0700 (PDT)
darnold  wrote:
> 
> i'm not brave enough to dig too deeply into SQLAlchemy, but maybe this
> will help? :
> 
> http://kashififtikhar.blogspot.com/2010/07/using-sqlalchemy-reflection-with-pylons.html
> 
> that came up from googling "sqlalchemy table reflection tutorial".

Thanks, your view of Google seems to be far better tailored for Python
than mine is, that doesn't come up for me anywhere on the first five
pages of results for that query.

Unfortunately the info on that page doesn't seem to work for me:

--
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker

engine = create_engine(my connection string)
meta = MetaData()
meta.bind = engine
meta.reflect()

Session = sessionmaker(bind=engine)
session = Session()

res = session.query(user).filter(user.name=="bert").first()
print res.name
--

That just gives me:

NameError: name 'user' is not defined

(And yes, the code given on that page to print out the table info
*does* indicate a table named 'user' was found.)

I also tried this which also fails:

res =
session.query(meta.tables["user"]).filter(meta.tables["user"].name=="bert").first()

sqlalchemy.exc.ArgumentError: filter() argument must be of type
sqlalchemy.sql.ClauseElement or string

The page you linked to appears to get around the matter by manually
setting up tables filled with the reflected info, but that seems to
defeat much of the point for me. I may as well just set up the tables
manually without the reflection, which is what I'll probably do.

Maybe I just misunderstood what was meant in the SQLAlchemy docs here?:

"but note that SA can also “import” whole sets of Table objects
automatically from an existing database (this process is called table
reflection)."  --
http://docs.sqlalchemy.org/en/rel_0_7/core/tutorial.html

It said that but then didn't say how and didn't link to any info on how.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[David Hutto]

On Tue, Oct 23, 2012 at 10:31 AM, Virgil Stokes  wrote:
> I am working with some rather large data files (>100GB) that contain time
> series data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII
> format. I perform various types of processing on these data (e.g. moving
> median, moving average, and Kalman-filter, Kalman-smoother) in a sequential
> manner and only a small number of these data need be stored in RAM when
> being processed. When performing Kalman-filtering (forward in time pass, k =
> 0,1,...,N) I need to save to an external file several variables (e.g. 11*32
> bytes) for each (t_k, y(t_k)). These are inputs to the Kalman-smoother
> (backward in time pass, k = N,N-1,...,0). Thus, I will need to input these
> variables saved to an external file from the forward pass, in reverse order
> --- from last written to first written.
>
> Finally, to my question --- What is a fast way to write these variables to
> an external file and then read them in backwards?

Don't forget to use timeit for an average OS utilization.

I'd suggest two list comprehensions for now, until I've reviewed it some more:

forward =  ["%i = %s" % (i,chr(i)) for i in range(33,126)]
backward = ["%i = %s" % (i,chr(i)) for i in range(126,32,-1)]

for var in forward:
print var

for var in backward:
print var

You could also use a dict, and iterate through a straight loop that
assigned a front and back to a dict_one =  {0 : [0.100], 1 : [1.99]}
and the iterate through the loop, and call the first or second in the
dict's var list for frontwards , or backwards calls.


But there might be faster implementations, depending on other
function's usage of certain lower level functions.


-- 
Best Regards,
David Hutto
CEO: http://www.hitwebdevelopment.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: problem with ThreadingTCPServer Handler

[Miki Tebeka]

According to the docs 
(http://docs.python.org/library/socketserver.html#requesthandler-objects) 
there's self.server available.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: problem with ThreadingTCPServer Handler

[David M Chess]

> jorge 

> I'm programming a server that most send a message to each client 
> connected to it and nothing else. this is obviously a base of what i 
> want to do. the thing is, I made a class wich contains the Handler class 

> for the ThreadingTCPServer and starts the server but i don't know how 
> can i access the message variable contained in the class from the 
> Handler since I have not to instance the Handler by myself.

The information about the request is in attributes of the Handler object. 
>From the socketserver docs (
http://docs.python.org/library/socketserver.html ):

RequestHandler.handle() 
This function must do all the work required to service a request. The 
default implementation does nothing. Several instance attributes are 
available to it; the request is available as self.request; the client 
address asself.client_address; and the server instance as self.server, in 
case it needs access to per-server information. 

If that's what you meant by "the message variable contained in the class". 
 

If, on the other hand, you meant that you want to pass some specific data 
into the handler about what it's supposed to be doing, I've generally 
stashed that in the server, since the handler can see the server via 
self.server.

DC
 
-- 
http://mail.python.org/mailman/listinfo/python-list

ANN: Urwid 1.1.0 - Usability and Documentation

[Ian Ward]

Announcing Urwid 1.1.0
--

Urwid home page:
  http://excess.org/urwid/

Manual:
  http://excess.org/urwid/docs/

Package:
  http://pypi.python.org/pypi/urwid/1.1.0


About this release:
===

This is a major feature release for Urwid.

The first focus for this release has been on improving the usability of
container widgets by building a common API that works on all containers.
This API allows reading and updating the focus, contents and options across
most containers.  Urwid's container widgets have grown organically over
the years and many lack an API for updating their contents at all.  These
changes deprecate a number of existing methods and attributes, but continue
to support them.  Read about the new API here:

  http://excess.org/urwid/docs/manual/widgets.html#container-widgets

The second focus has been improving Urwid's documentation.  Urwid's
Tutorial, Manual and Reference have been converted from HTML and wiki pages
to Sphinx reStructured text.  The tutorial has been largely rewritten.
It now covers many more aspects of the library including common patterns
and the new container API.  The Manual has been heavily edited and most of
the class and function docstrings have been updated to look their best in
the new Reference.  The new documentation is available here:

  http://excess.org/urwid/docs/

See "New in 1.1.0" below for other changes.


Planned changes:


Urwid currently supports Python versions 2.4, 2.5, 2.6, 2.7, 3.2 and 3.3.
That's just too many versions.  I plan to drop support for Python 2.4 and
2.5 in the 1.2.x series to clean up some of the Python 3 compatibility code
and start taking advantage of the futuristic language features from the
year 2008.

I will make a greater effort than usual to back port fixes to the 1.1.x
series to support users that choose to continue using old versions of
Python.


New in 1.1.0:
=

 * New common container API: focus, focus_position, contents,
   options(), get_focus_path(), set_focus_path(), __getitem__,
   __iter__(), __reversed__() implemented across all included
   container widgets

   A full description doesn't fit here, see the Container Widgets
   section in the manual for details

 * New Sphinx-based documentation now included in source:
   Tutorial rewritten, manual revised and new reference based
   on updated docstrings (by Marco Giusti, Patrick Totzke)

 * New list walker SimpleFocusListWalker like SimpleListWalker but
   updates focus position as items are inserted or removed

 * New decoration widget WidgetDisable to disable interaction
   with the widgets it wraps

 * SelectableIcon selectable text widget used by button widgets is
   now documented (available since 0.9.9)

 * Columns widget now tries to keep column in focus visible, hiding
   columns on the left when necessary

 * Padding widget now defaults to ('relative', 100) instead of
   'pack' so that left and right parameters are more useful and more
   child widgets are supported

 * New list walker "API Version 2" that is simpler for many list
   walker uses; "API Version 1" will still continue to be supported

 * List walkers may now allow iteration from the absolute top or
   bottom of the list if they provide a positions() method

 * raw_display now erases to the end of the line with EL escape
   sequence to improve copy+paste behavior for some terminals

 * Filler now has top and bottom parameters like Padding's left and
   right parameters and accepts 'pack' instead of None as a height
   value for widgets that calculate their own number of rows

 * Pile and Columns now accepts 'pack' instead of 'flow' for widgets
   that calculate their own number of rows or columns

 * Pile and Columns now accept 'given' instead of 'fixed' for
   cases where the number of rows or columns are specified by the
   container options

 * Pile and Columns widgets now accept any iterable to their
   __init__() methods

 * Widget now has a default focus_position property that raises
   an IndexError when read to be consistent with new common container
   API

 * GridFlow now supports multiple cell widths within the same widget

 * BoxWidget, FlowWidget and FixedWidget are deprecated, instead
   use the sizing() function or _sizing attribute to specify the
   supported sizing modes for your custom widgets

 * Some new shift+arrow and numpad input sequences from RXVT and
   xterm are now recognized

 * Fix for alarms when used with a screen event loop (e.g.
   curses_display)

 * Fix for raw_display when terminal width is 1 column

 * Fixes for a Columns.get_cursor_coords() regression and a
   SelectableIcon.get_cursor_coords() bug

 * Fixes for incorrect handling of box columns in a number of
   Columns methods when that column is selectable

 * Fix for Terminal widget input handling with Python 3


About Urwid
===

Urwid is a console UI library for Python. It features fluid interface
resizing, Unicode support, multiple text

Re: regex function driving me nuts

[Vlastimil Brom]

2012/10/23 MartinD. :
> Hi,
>
> I'm new to Python.
> Does someone has an idea what's wrong.  I tried everything. The only regex 
> that is tested is the last one in a whole list of regex in keywords.txt
> Thanks!
> Martin
>
>
> 
> def checkKeywords( str, lstKeywords ):
>
> for regex in lstKeywords:
> match = re.search(regex, str,re.IGNORECASE)
> # If-statement after search() tests if it succeeded
> if match:
> print match.group() ##just debugging
> return match.group() ## 'found!
>
> return
>
> #
>
> keywords1 = [line for line in open('keywords1.txt')]
> resultKeywords1 = checkKeywords("string_to_test",keywords1)
> print resultKeywords1
>
> --
> http://mail.python.org/mailman/listinfo/python-list

Hi,
just a wild guess, as I don't have access to  containing the list of
potentially problematic regex patterns
does:
keywords1 = [line.strip() for line in open('keywords1.txt')]
possibly fix yout problem?
the lines of the file iterator also preserve newlines, which might not
be expected in your keywords, strip() removes (be default) any
starting and tryiling whitespace.

hth,
  vbr
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: regex function driving me nuts

[Ian Kelly]

On Tue, Oct 23, 2012 at 1:51 PM, MartinD.  wrote:
> Hi,
>
> I'm new to Python.
> Does someone has an idea what's wrong.  I tried everything. The only regex 
> that is tested is the last one in a whole list of regex in keywords.txt
> Thanks!
> Martin

How do you know that it's the only one being tested?  Your debugging
statement only prints one that actually matches, not each one that is
tried.

> keywords1 = [line for line in open('keywords1.txt')]

Note that each "keyword" will including the trailing newline, if any.
This is probably why you are only seeing the last keyword match:
because it is the only one without a trailing newline.

To remove the newlines, call the str.strip or str.rstrip method on
each line, or use the str.splitlines method to get the keywords:

keywords1 = open('keywords1.txt').read().splitlines()
-- 
http://mail.python.org/mailman/listinfo/python-list

RE: regex function driving me nuts

[Prasad, Ramit]

MartinD wrote:
> Hi,
> 
> I'm new to Python.
> Does someone has an idea what's wrong.  I tried everything. The only regex 
> that is tested is the last one in a
> whole list of regex in keywords.txt
> Thanks!
> Martin
> 
> 
> 
> def checkKeywords( str, lstKeywords ):
> 
>   for regex in lstKeywords:
>   match = re.search(regex, str,re.IGNORECASE)
>   # If-statement after search() tests if it succeeded
>   if match:
>   print match.group() ##just debugging
>   return match.group() ## 'found!
> 
>   return
> 
> #
> 
> keywords1 = [line for line in open('keywords1.txt')]
> resultKeywords1 = checkKeywords("string_to_test",keywords1)
> print resultKeywords1
> 

Hi Martin,
It is always helpful to provide python version, operating system version, full 
error message,
and input/expected output for the code. Now I can tell you are using Python 2.x 
but 
without having any clue what is in keywords1.txt it is impossible to figure out
what the problem might be. Other than using regular expressions that is. :)

Ramit Prasad


This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of
securities, accuracy and completeness of information, viruses,
confidentiality, legal privilege, and legal entity disclaimers,
available at http://www.jpmorgan.com/pages/disclosures/email.  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Joshua Landau]

On 23 October 2012 21:06, Joshua Landau  wrote:

> On 23 October 2012 21:03, Joshua Landau wrote:
>
>> On 23 October 2012 12:07, Jean-Michel Pichavant 
>> wrote:
>>
>>> - Original Message -
>>>
>>> > Thankyou.. but my problem is different than simply joining 2 lists
>>> > and it is done now :)
>>>
>>>
>>> A lot of people though you were asking for joining lists, you
>>> description was misleading.
>>>
>>> I'll take a guess: you want to flatten a list of list.
>>> "Nested" list comprehensions can do the trick.
>>>
>>> aList =[[1,5], [2,'a']]
>>> [item for sublist in aList for item in sublist]
>>>
>>> ...
>>> [1, 5, 2, 'a']
>>>
>>> I find it rather difficult to read though.
>>
>>
>> We have a library function for this, in the one-and-only itertools.
>>
>> >>> listoflists = [list(range(x, 2*x)) for x in range(5)]
>>> >>> listoflists
>>> [[], [1], [2, 3], [3, 4, 5], [4, 5, 6, 7]]
>>> >>> from itertools import chain
>>>  >>> list(chain.from_iterable(listoflists))
>>> [1, 2, 3, 3, 4, 5, 4, 5, 6, 7]
>>
>>
>> It does exactly what it says... fast and easy-to-read.
>
>
> Note that I think what he really wanted is to go from
>
> a, b, c = [list(x) for x in (range(10), range(11, 20), range(21, 30))]
>
> to
>
>> list(range(30))
>
>
UNDO! UNDO! UNDO!

I *meant *to say:

Note that I think what he really wanted is to go from

a, b, c = [list(x) for x in (range(10), range(11, 20), range(21, 30))]

to

> [a, b, c]
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[inshu chauhan]

ok I got it guys...

On Tue, Oct 23, 2012 at 10:08 PM, Joshua Landau
wrote:

>  On 23 October 2012 21:06, Joshua Landau wrote:
>
>>  On 23 October 2012 21:03, Joshua Landau wrote:
>>
>>> On 23 October 2012 12:07, Jean-Michel Pichavant 
>>> wrote:
>>>
 - Original Message -

 > Thankyou.. but my problem is different than simply joining 2 lists
 > and it is done now :)


 A lot of people though you were asking for joining lists, you
 description was misleading.

 I'll take a guess: you want to flatten a list of list.
 "Nested" list comprehensions can do the trick.

 aList =[[1,5], [2,'a']]
 [item for sublist in aList for item in sublist]

 ...
 [1, 5, 2, 'a']

 I find it rather difficult to read though.
>>>
>>>
>>> We have a library function for this, in the one-and-only itertools.
>>>
>>> >>> listoflists = [list(range(x, 2*x)) for x in range(5)]
 >>> listoflists
 [[], [1], [2, 3], [3, 4, 5], [4, 5, 6, 7]]
 >>> from itertools import chain
  >>> list(chain.from_iterable(listoflists))
 [1, 2, 3, 3, 4, 5, 4, 5, 6, 7]
>>>
>>>
>>> It does exactly what it says... fast and easy-to-read.
>>
>>
>> Note that I think what he really wanted is to go from
>>
>> a, b, c = [list(x) for x in (range(10), range(11, 20), range(21, 30))]
>>
>> to
>>
>>> list(range(30))
>>
>>
> UNDO! UNDO! UNDO!
>
> I *meant *to say:
>
>  Note that I think what he really wanted is to go from
>
> a, b, c = [list(x) for x in (range(10), range(11, 20), range(21, 30))]
>
> to
>
>> [a, b, c]
>
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Joshua Landau]

On 23 October 2012 21:03, Joshua Landau  wrote:

> On 23 October 2012 12:07, Jean-Michel Pichavant wrote:
>
>> - Original Message -
>>
>> > Thankyou.. but my problem is different than simply joining 2 lists
>> > and it is done now :)
>>
>>
>> A lot of people though you were asking for joining lists, you description
>> was misleading.
>>
>> I'll take a guess: you want to flatten a list of list.
>> "Nested" list comprehensions can do the trick.
>>
>> aList =[[1,5], [2,'a']]
>> [item for sublist in aList for item in sublist]
>>
>> ...
>> [1, 5, 2, 'a']
>>
>> I find it rather difficult to read though.
>
>
> We have a library function for this, in the one-and-only itertools.
>
> >>> listoflists = [list(range(x, 2*x)) for x in range(5)]
>> >>> listoflists
>> [[], [1], [2, 3], [3, 4, 5], [4, 5, 6, 7]]
>> >>> from itertools import chain
>>  >>> list(chain.from_iterable(listoflists))
>> [1, 2, 3, 3, 4, 5, 4, 5, 6, 7]
>
>
> It does exactly what it says... fast and easy-to-read.


Note that I think what he really wanted is to go from

a, b, c = [list(x) for x in (range(10), range(11, 20), range(21, 30))]

to

> list(range(30))
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Cousin Stanley]

Virgil Stokes wrote:

> Not sure about "tac" --- could you provide more details on this 
> and/or a simple example of how it could be used for fast reversed 
> "reading" of a data file ?

  tac is available as a command under linux  

  $ whatis tac
  tac (1) - concatenate and print files in reverse

  $ whereis tac
  tac: /usr/bin/tac /usr/bin/X11/tac /usr/share/man/man1/tac.1.gz

  $ man tac

  SYNOPSIS
tac [OPTION]... [FILE]...

  DESCRIPTION

Write each FILE to standard output, last line first.  

With no FILE, or when FILE is -, read standard input.


  I only know that the  tac  command exists
  but have never used it myself 


-- 
Stanley C. Kitching
Human Being
Phoenix, Arizona

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Joshua Landau]

On 23 October 2012 12:07, Jean-Michel Pichavant wrote:

> - Original Message -
>
> > Thankyou.. but my problem is different than simply joining 2 lists
> > and it is done now :)
>
>
> A lot of people though you were asking for joining lists, you description
> was misleading.
>
> I'll take a guess: you want to flatten a list of list.
> "Nested" list comprehensions can do the trick.
>
> aList =[[1,5], [2,'a']]
> [item for sublist in aList for item in sublist]
>
> ...
> [1, 5, 2, 'a']
>
> I find it rather difficult to read though.


We have a library function for this, in the one-and-only itertools.

>>> listoflists = [list(range(x, 2*x)) for x in range(5)]
> >>> listoflists
> [[], [1], [2, 3], [3, 4, 5], [4, 5, 6, 7]]
> >>> from itertools import chain
>  >>> list(chain.from_iterable(listoflists))
> [1, 2, 3, 3, 4, 5, 4, 5, 6, 7]


It does exactly what it says... fast and easy-to-read.
-- 
http://mail.python.org/mailman/listinfo/python-list

regex function driving me nuts

[MartinD.]

Hi, 

I'm new to Python. 
Does someone has an idea what's wrong.  I tried everything. The only regex that 
is tested is the last one in a whole list of regex in keywords.txt
Thanks! 
Martin 



def checkKeywords( str, lstKeywords ):

for regex in lstKeywords: 
match = re.search(regex, str,re.IGNORECASE)
# If-statement after search() tests if it succeeded
if match:  
print match.group() ##just debugging
return match.group() ## 'found!

return

#

keywords1 = [line for line in open('keywords1.txt')]
resultKeywords1 = checkKeywords("string_to_test",keywords1)
print resultKeywords1

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Virgil Stokes]

On 23-Oct-2012 19:56, Tim Chase wrote:

On 10/23/12 12:17, Virgil Stokes wrote:

On 23-Oct-2012 18:09, Tim Chase wrote:

Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in
backwards?

Am I missing something, or would the fairly-standard "tac"
utility do the reversal you want?  It should[*] be optimized to
handle on-disk files in a smart manner.

Not sure about "tac" --- could you provide more details on this
and/or a simple example of how it could be used for fast reversed
"reading" of a data file?

Well, if you're reading input.txt (and assuming it's one record per
line, separated by newlines), you can just use

   tac < input.txt > backwards.txt

which will create a secondary file that is the first file in reverse
order.  Your program can then process this secondary file in-order
(which would be backwards from your source).

I might have misunderstood your difficulty, but it _sounded_ like
you just want to inverse the order of a file.
Yes, I do wish to inverse the order,  but the "forward in time" file will be in 
binary.


--V

--
http://mail.python.org/mailman/listinfo/python-list

Re: PIL and requests don't get along

[Alex Clark]

On 2012-10-23 18:06:59 +, Roy Smith said:


I have a url from which I can get an image.  I want to use PIL to
manipulate that image.  Getting the image is easy:


import requests
r = requests.get(url)


There's a bunch of factory functions for Image, but none of them seem
to take anything that requests is willing to give you.  Image.new()
requires that you pass it the image size.  Image.open() takes a file
object, but


Image.open(r.raw)


doesn't work because r.raw gives you a socket which doesn't support
seek().  I end up doing:


r = requests.get(url)
data = cStringIO.StringIO(r.content)
image = Image.open(data)


which works, but it's gross.  Is there something I'm missing here?



No idea but you can open a ticket here if you think it's appropriate:  
https://github.com/python-imaging/Pillow/issues



--
Alex Clark · https://www.gittip.com/aclark4life/


--
http://mail.python.org/mailman/listinfo/python-list

PIL and requests don't get along

[Roy Smith]

I have a url from which I can get an image.  I want to use PIL to
manipulate that image.  Getting the image is easy:

>>> import requests
>>> r = requests.get(url)

There's a bunch of factory functions for Image, but none of them seem
to take anything that requests is willing to give you.  Image.new()
requires that you pass it the image size.  Image.open() takes a file
object, but

>>> Image.open(r.raw)

doesn't work because r.raw gives you a socket which doesn't support
seek().  I end up doing:

>>> r = requests.get(url)
>>> data = cStringIO.StringIO(r.content)
>>> image = Image.open(data)

which works, but it's gross.  Is there something I'm missing here?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Tim Chase]

On 10/23/12 12:17, Virgil Stokes wrote:
> On 23-Oct-2012 18:09, Tim Chase wrote:
>>> Finally, to my question --- What is a fast way to write these
>>> variables to an external file and then read them in
>>> backwards?
>> Am I missing something, or would the fairly-standard "tac"
>> utility do the reversal you want?  It should[*] be optimized to
>> handle on-disk files in a smart manner.
> Not sure about "tac" --- could you provide more details on this
> and/or a simple example of how it could be used for fast reversed
> "reading" of a data file?

Well, if you're reading input.txt (and assuming it's one record per
line, separated by newlines), you can just use

  tac < input.txt > backwards.txt

which will create a secondary file that is the first file in reverse
order.  Your program can then process this secondary file in-order
(which would be backwards from your source).

I might have misunderstood your difficulty, but it _sounded_ like
you just want to inverse the order of a file.

-tkc





-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Style help for a Smalltalk-hack

[Ian Kelly]

On Tue, Oct 23, 2012 at 9:13 AM, Travis Griggs  wrote:
>
> On Oct 22, 2012, at 6:33 PM, MRAB  wrote:
>
>> Another way you could do it is:
>>
>> while True:
>>chunk = byteStream.read(4)
>>if not chunk:
>>break
>>...
>>
>> And you could fetch multiple signatures in one read:
>>
>> signatures = list(struct.unpack('>{}I'.format(valveCount), byteStream.read(4 
>> * valueCount)))
>
> Thanks, both great ideas. Still does the read/decode slightly different 
> between the different sites, but at least it's localized better. Much 
> appreciated.

Another small optimization would be to read the gap and the valveCount together:

chunk = byteStream.read(8)
...
gap, valveCount = struct.unpack('>2l', chunk)
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Virgil Stokes]

On 23-Oct-2012 18:35, Dennis Lee Bieber wrote:

On Tue, 23 Oct 2012 16:31:17 +0200, Virgil Stokes 
declaimed the following in gmane.comp.python.general:


Finally, to my question --- What is a fast way to write these variables to an
external file and then read them in backwards?


Stuff them into an SQLite3 database and retrieve using a descending
sort?
Have never worked with a database; but, could be worth a try (at least to 
compare I/O times).


Thanks Dennis :-)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Virgil Stokes]

On 23-Oct-2012 18:17, Paul Rubin wrote:

Virgil Stokes  writes:

Finally, to my question --- What is a fast way to write these
variables to an external file and then read them in backwards?

Seeking backwards in files works, but the performance hit is
significant.  There is also a performance hit to scanning pointers
backwards in memory, due to cache misprediction.  If it's something
you're just running a few times, seeking backwards the simplest
approach.  If you're really trying to optimize the thing, you might
buffer up large chunks (like 1 MB) before writing.  If you're writing
once and reading multiple times, you might reverse the order of records
within the chunks during the writing phase.

I am writing (forward) once and reading (backward) once.


You're of course taking a performance bath from writing the program in
Python to begin with (unless using scipy/numpy or the like), enough that
it might dominate any effects of how the files are written.

I am currently using SciPy/NumPy


Of course (it should go without saying) that you want to dump in a
binary format rather than converting to decimal.

Yes, I am doing this (but thanks for "underlining" it!)

Thanks Paul :-)
--
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Virgil Stokes]

On 23-Oct-2012 18:09, Tim Chase wrote:

On 10/23/12 09:31, Virgil Stokes wrote:

I am working with some rather large data files (>100GB) that contain time series
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform
various types of processing on these data (e.g. moving median, moving average,
and Kalman-filter, Kalman-smoother) in a sequential manner and only a small
number of these data need be stored in RAM when being processed. When performing
Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an
external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These
are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0).
Thus, I will need to input these variables saved to an external file from the
forward pass, in reverse order --- from last written to first written.

Finally, to my question --- What is a fast way to write these variables to an
external file and then read them in backwards?

Am I missing something, or would the fairly-standard "tac" utility
do the reversal you want?  It should[*] be optimized to handle
on-disk files in a smart manner.
Not sure about "tac" --- could you provide more details on this and/or a simple 
example of how it could be used for fast reversed "reading" of a data file?


Otherwise, if you can pad the record-lengths so they're all the
same, and you know the total number of records, you can seek to
Total-(RecSize*OneBasedOffset) and write the record,optionally
padding if you need/can.  At least on *nix-like OSes, you can seek
into a sparse-file with no problems (untested on Win32).
The records lengths will all be the same and yes seek could be used; but, I was 
hoping for a faster method.


Thanks Tim! :-)


-tkc



[*]
Just guessing here. Would be disappointed if it *wasn't*.


--
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Paul Rubin]

Tim Chase  writes:
> Again, the conversion to/from decimal hasn't been a great cost in my
> experience, as it's overwhelmed by the I/O cost of shoveling the
> data to/from disk.

I've found that cpu costs both for processing and conversion are
significant.  Also, using a binary format makes the file a lot smaller,
which decreases the i/o cost as well eliminating the conversion cost.
And, the conversion can introduce precision loss, another thing to be
avoided.  The famous "butterfly effect" was serendipitously discovered
that way.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Tim Chase]

On 10/23/12 11:17, Paul Rubin wrote:
> Virgil Stokes  writes:
>> Finally, to my question --- What is a fast way to write these
>> variables to an external file and then read them in backwards?
> 
> Seeking backwards in files works, but the performance hit is
> significant.  There is also a performance hit to scanning pointers
> backwards in memory, due to cache misprediction.  If it's something
> you're just running a few times, seeking backwards the simplest
> approach.  If you're really trying to optimize the thing, you might
> buffer up large chunks (like 1 MB) before writing.  If you're writing
> once and reading multiple times, you might reverse the order of records
> within the chunks during the writing phase.

I agree with Paul here, it's been a while since I did it, and my
dataset was small enough (and passed through once) so I just let it
run.  Writing larger chunks is definitely a good way to go.

> You're of course taking a performance bath from writing the program in
> Python to begin with (unless using scipy/numpy or the like), enough that
> it might dominate any effects of how the files are written.

I usually find that the I/O almost always overwhelms the actual
processing.

> Of course (it should go without saying) that you want to dump in a
> binary format rather than converting to decimal.

Again, the conversion to/from decimal hasn't been a great cost in my
experience, as it's overwhelmed by the I/O cost of shoveling the
data to/from disk.

-tkc


-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A desperate lunge for on-topic-ness

[Ian Kelly]

On Mon, Oct 22, 2012 at 7:39 PM, Dennis Lee Bieber
 wrote:
> On Mon, 22 Oct 2012 16:02:34 -0600, Ian Kelly 
> declaimed the following in gmane.comp.python.general:
>
>> On my wishlist for Python is a big, fat SyntaxError for any variable
>> that could be interpreted as either local or nonlocal and is not
>> explicitly declared as either.  It would eliminate this sort of
>> confusion entirely and make code that shadows nonlocal variables much
>> more readable.
>>
> Which now makes code dependent upon changes to some imported modules
> if someone is foolish enough to use the
>
> from xyz import *
>
> notation...

It's already a SyntaxError to use a wildcard import anywhere other
than the module level, so its use can only affect global variables.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: get each pair from a string.

[Ian Kelly]

On Tue, Oct 23, 2012 at 12:47 AM,   wrote:
> The latest Python version is systematically slower
> than the previous ones as soon as one uses non "ascii
> strings".

No, it isn't.  You've previously demonstrated a *microbenchmark* where
3.3 is slower than 3.2.  This is a far cry from demonstrating that 3.3
is "systematically slower", and that larger claim is simply false.

In any case, have you been following the progress of issue 16061?
There is a patch for the str.replace regression that you identified,
which results in better performance across the board than 3.2.  So
although 3.3 is slower than 3.2 on this one microbenchmark, 3.4 will
not be.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Compiling extension module (linker error)

[MRAB]

On 2012-10-23 08:00, Paul Volkov wrote:

2012/10/22 MRAB :

By the way, the recommendation is for module names to be lowercase with
underscores, so "fund_rose" instead of "FundRose".

Try this code:



I tried as you suggested, but the linker error (unresolved external)
is still there.


I found a reference to PyModule_Create2TraceRefs in include\modsupport.h.


I'm sure python33.lib is properly added because if I remove it, there
are 3 errors, not 1.
By the way, what is the difference between python3.lib and python33.lib?


I don't know.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Paul Rubin]

Paul Rubin  writes:
> Seeking backwards in files works, but the performance hit is
> significant.  There is also a performance hit to scanning pointers
> backwards in memory, due to cache misprediction.  If it's something
> you're just running a few times, seeking backwards the simplest
> approach. 

Oh yes, I should have mentioned, it may be simpler and perhaps a little
bit faster to use mmap rather than seeking.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Paul Rubin]

Virgil Stokes  writes:
> Finally, to my question --- What is a fast way to write these
> variables to an external file and then read them in backwards?

Seeking backwards in files works, but the performance hit is
significant.  There is also a performance hit to scanning pointers
backwards in memory, due to cache misprediction.  If it's something
you're just running a few times, seeking backwards the simplest
approach.  If you're really trying to optimize the thing, you might
buffer up large chunks (like 1 MB) before writing.  If you're writing
once and reading multiple times, you might reverse the order of records
within the chunks during the writing phase.  

You're of course taking a performance bath from writing the program in
Python to begin with (unless using scipy/numpy or the like), enough that
it might dominate any effects of how the files are written.

Of course (it should go without saying) that you want to dump in a
binary format rather than converting to decimal.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Fast forward-backward (write-read)

[Tim Chase]

On 10/23/12 09:31, Virgil Stokes wrote:
> I am working with some rather large data files (>100GB) that contain time 
> series 
> data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I 
> perform 
> various types of processing on these data (e.g. moving median, moving 
> average, 
> and Kalman-filter, Kalman-smoother) in a sequential manner and only a small 
> number of these data need be stored in RAM when being processed. When 
> performing 
> Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an 
> external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). 
> These 
> are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). 
> Thus, I will need to input these variables saved to an external file from the 
> forward pass, in reverse order --- from last written to first written.
> 
> Finally, to my question --- What is a fast way to write these variables to an 
> external file and then read them in backwards?

Am I missing something, or would the fairly-standard "tac" utility
do the reversal you want?  It should[*] be optimized to handle
on-disk files in a smart manner.

Otherwise, if you can pad the record-lengths so they're all the
same, and you know the total number of records, you can seek to
Total-(RecSize*OneBasedOffset) and write the record,optionally
padding if you need/can.  At least on *nix-like OSes, you can seek
into a sparse-file with no problems (untested on Win32).

-tkc



[*]
Just guessing here. Would be disappointed if it *wasn't*.
-- 
http://mail.python.org/mailman/listinfo/python-list

Getting a TimedRotatingFileHandler not to put two dates in the same file?

[David M Chess]

We have a TimedRotatingFileHandler with when='midnight'. 

This works great, splitting the log information across files by date, as 
long as the process is actually up at midnight.

But now the users have noticed that if the process isn't up at midnight, 
they can end up with lines from two (or I guess potentially more) dates in 
the same log file.

Is there some way to fix this, either with cleverer arguments into the 
TimedRotatingFileHandler, or by some plausible subclassing of it or its 
superclass?

Or am I misinterpreting the symptoms somehow?

Tx much!
DC
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Style help for a Smalltalk-hack

[Travis Griggs]

On Oct 22, 2012, at 6:33 PM, MRAB  wrote:

> By the way, in Python the recommended style for variable names (well,
> what you'd call a 'variable' in other languages :-)) is lowercase with
> underscores, e.g. "byte_stream".

We went with the 

"...mixedCase is allowed only in contexts where that's already the prevailing 
style (e.g. threading.py), to retain backwards compatibility…"

escape clause. :)

The group of us here are working in multiple languages, and all use it 
(mixedCase) across all of them. Because PEP 8 makes it clear up front that 
"internal consistency is preferred",  we felt justified in marching on.

--
Travis Griggs
"A vital ingredient of success is not knowing that what you're attempting can't 
be done." -Terry Pratchett

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Style help for a Smalltalk-hack

[Travis Griggs]

On Oct 22, 2012, at 6:33 PM, MRAB  wrote:

> Another way you could do it is:
> 
> while True:
>chunk = byteStream.read(4)
>if not chunk:
>break
>...
> 
> And you could fetch multiple signatures in one read:
> 
> signatures = list(struct.unpack('>{}I'.format(valveCount), byteStream.read(4 
> * valueCount)))

Thanks, both great ideas. Still does the read/decode slightly different between 
the different sites, but at least it's localized better. Much appreciated.

--
Travis Griggs
"History has a habit of changing the people who think they are changing it." 
-Terry Pratchett

-- 
http://mail.python.org/mailman/listinfo/python-list

Fast forward-backward (write-read)

[Virgil Stokes]

I am working with some rather large data files (>100GB) that contain time series 
data. The data (t_k,y(t_k)), k = 0,1,...,N are stored in ASCII format. I perform 
various types of processing on these data (e.g. moving median, moving average, 
and Kalman-filter, Kalman-smoother) in a sequential manner and only a small 
number of these data need be stored in RAM when being processed. When performing 
Kalman-filtering (forward in time pass, k = 0,1,...,N) I need to save to an 
external file several variables (e.g. 11*32 bytes) for each (t_k, y(t_k)). These 
are inputs to the Kalman-smoother (backward in time pass, k = N,N-1,...,0). 
Thus, I will need to input these variables saved to an external file from the 
forward pass, in reverse order --- from last written to first written.


Finally, to my question --- What is a fast way to write these variables to an 
external file and then read them in backwards?



--
http://mail.python.org/mailman/listinfo/python-list

problem with ThreadingTCPServer Handler

[jorge]

I'm programming a server that most send a message to each client 
connected to it and nothing else. this is obviously a base of what i 
want to do. the thing is, I made a class wich contains the Handler class 
for the ThreadingTCPServer and starts the server but i don't know how 
can i access the message variable contained in the class from the 
Handler since I have not to instance the Handler by myself.

here is the base code of what I'm doing.

import SocketServer
import socket
from threading import Thread

class CustomTCPServer:
msg = "no message"
outport = 8080

class Handler(SocketServer.BaseRequestHandler):
def handle(self):
self.request.sendall('')


def __init__(self,msg,outport):
self.msg = server
self.outport = outport

def runServer(self):
addr = ('',self.outport)
s = SocketServer.ThreadingTCPServer(addr,self.Handler)
r = Thread(target=s.serve_forever).start()
print '> Server running running',self.server,self.outport


CustomTCPServer('msg1',1212).runServer()
CustomTCPServer('msg2',1213).runServer()

can anyone please help me?

10mo. ANIVERSARIO DE LA CREACION DE LA UNIVERSIDAD DE LAS CIENCIAS 
INFORMATICAS...
CONECTADOS AL FUTURO, CONECTADOS A LA REVOLUCION

http://www.uci.cu
http://www.facebook.com/universidad.uci
http://www.flickr.com/photos/universidad_uci
--
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Dave Angel]

On 10/23/2012 03:36 AM, inshu chauhan wrote:
> can we append a list with another list in Python ? using the normal routine
> syntax but with a for loop ??
>
>

To combine list a with list b, you can do any of:

c = a + b
a += b
a.extend(b)

Obviously, the first leaves a unmodified, while the other two modify it
in-place.



-- 

DaveA

-- 
http://mail.python.org/mailman/listinfo/python-list

RE: Fastest template engine

[Andriy Kornatskyy]

Python template engines offer high reusability of markup code and the following 
features are used by content developers most of the time:

* Includes: useful to incorporate some snippets of content that in most cases 
are common to the site, e.g. footer, scripts, styles, etc.

* Extends: useful to define a master layout for the majority of the site 
content with placeholders, e.g. sidebar, horizontal menu, content, etc. The 
content developers extend the master layout by substituting available 
placeholders.

* Widgets: usually small snippets of highly reusable markup, e.g. list item, 
button, etc. The content developers use widgets to increase readability and 
enforce consistency of design.

All mentioned features above are examined for various template engines (django, 
jinja2, mako, tornado and wheezy.template) in the following post:

http://mindref.blogspot.com/2012/10/python-templates-benchmark.html

The test is executed in isolated environment using CPython 2.7 but can be run 
for Python 3.3 and/or PyPy 1.9. Source code is here:

https://bitbucket.org/akorn/helloworld

Comments or suggestions are welcome.

Thanks.

Andriy



> From: andriy.kornats...@live.com
> To: python-list@python.org
> Subject: RE: Fastest template engine
> Date: Fri, 19 Oct 2012 11:34:41 +0300
>
>
> Per community request cheetah has been added to benchmark. Post updated, just 
> in case:
>
> http://mindref.blogspot.com/2012/07/python-fastest-template.html
>
> Comments or suggestions are welcome.
>
> Andriy
>
>
> 
> > From: andriy.kornats...@live.com
> > To: python-list@python.org
> > Subject: RE: Fastest template engine
> > Date: Wed, 26 Sep 2012 16:21:21 +0300
> >
> >
> > The post has been updated with the following template engines added (per 
> > community request):
> >
> > 1. chameleon
> > 2. django
> > 3. web2py
> >
> > Here is a link:
> >
> > http://mindref.blogspot.com/2012/07/python-fastest-template.html
> >
> > Comments or suggestions are welcome.
> >
> > Thanks.
> >
> > Andriy
> >
> >
> > 
> > > From: andriy.kornats...@live.com
> > > To: python-list@python.org
> > > Subject: Fastest template engine
> > > Date: Sun, 23 Sep 2012 12:24:36 +0300
> > >
> > >
> > > I have run recently a benchmark of a trivial 'big table' example for 
> > > various python template engines (jinja2, mako, tenjin, tornado and 
> > > wheezy.template) run on cpython2.7 and pypy1.9.. you might find it 
> > > interesting:
> > >
> > > http://mindref.blogspot.com/2012/07/python-fastest-template.html
> > >
> > > Comments or suggestions are welcome.
> > >
> > > Thanks.
> > >
> > > Andriy Kornatskyy
> > > --
> > > http://mail.python.org/mailman/listinfo/python-list
> >
>
> --
> http://mail.python.org/mailman/listinfo/python-list
  
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: get each pair from a string.

[Mark Lawrence]

On 23/10/2012 07:47, wxjmfa...@gmail.com wrote:


Why bother with speeed?

The latest Python version is systematically slower
than the previous ones as soon as one uses non "ascii
strings".

Python users are discussing "code optimizations" without
realizing the tool they are using, has killed itself its
own performances.

(Replace 'apple' with 'ap需')

jmf



Please stop giving blatant lies on this list about Python speed.

--
Cheers.

Mark Lawrence.

--
http://mail.python.org/mailman/listinfo/python-list

Re: get each pair from a string.

[Neil Cerutti]

On 2012-10-23, wxjmfa...@gmail.com  wrote:
> Why bother with speeed?

Because the differences between O(N), O(log(N)) and O(N ** 2)
operations are often relevant.

A Python string replace function experiencing a slow-down from
previous versions doesn't absolve me from making informed choices
of algorithm and implentation.

-- 
Neil Cerutti
-- 
http://mail.python.org/mailman/listinfo/python-list

dirty tara lynn takes on multiple cocks

[Constantine]

dirty tara lynn takes on multiple cocks 
http://www.google.com/search?hl=en&q=dirty+tara+lynn+takes+on+multiple+cocks+site:wxwusqxeh470.blogspot.com&btnI=I%27m+Feeling+Lucky
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to programmatically turn on remote registry?

[Kevin Holleran]

Tim,

I am re-implementing in my script now.  I'll let you know how it goes... I
just realized that the key that I sent over was completely wrong I am
not sure how I even got it as it is the combination of two different keys
from two different scripts... must have been working too long and
everything blending together... :)

Thanks for your help!

Kevin


On Tue, Oct 23, 2012 at 4:07 AM, Tim Golden  wrote:

> On 22/10/2012 21:01, Kevin Holleran wrote:
> > Tim,
> >
> > I am looking here:
> >
> >
> SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{BF9F6FB0-C999-4D19-BED0-144F77E2A9D6}
> >
> > Enumerating the keys for a BusType == 5, then grabbing the values of
> > DriverDesc, DriverDate, & DriverVersion.
> >
> > So I am doing this:
>
> [... snip querying uninstallers ...]
>
> I don't have that particular uninstaller key but the code below, using
> the wmi module to hide the plumbing, queries all the installers and
> should give you enough of an idea, hopefully. For brevilty, I've only
> bothered with extracting string values; it would be easy to extract
> other datatypes.
>
> To perform the same query on another computer, just pass the other
> computer name (or IP address) as the first parameter to the wmi.WMI call
> (or use the named param "computer").
>
> 
> import _winreg as winreg
> import wmi
>
> HKLM = winreg.HKEY_LOCAL_MACHINE
> UNINSTALLERS = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall"
>
> registry = wmi.WMI(namespace="default").StdRegProv
> _, names = registry.EnumKey(HKLM, UNINSTALLERS)
> for name in names:
> print name
> uninstaller = UNINSTALLERS + "\\" + name
> _, value_names, value_types = registry.EnumValues(HKLM, uninstaller)
> for value_name, value_type in zip(value_names, value_types):
> if value_type == winreg.REG_SZ:
> _, value = registry.GetStringValue(
>   HKLM, uninstaller, value_name
> )
> else:
> value = "(Non-string value)"
> print u"  ", value_name, u"=>", value
>
> 
>
>
> TJG
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Jean-Michel Pichavant]

- Original Message - 

> Thankyou.. but my problem is different than simply joining 2 lists
> and it is done now :)


A lot of people though you were asking for joining lists, you description was 
misleading.

I'll take a guess: you want to flatten a list of list.
"Nested" list comprehensions can do the trick.

aList =[[1,5], [2,'a']]
[item for sublist in aList for item in sublist]

...
[1, 5, 2, 'a']
   
I find it rather difficult to read though.

Cheers,

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Making `logging.basicConfig` log to *both* `sys.stderr` and `sys.stdout`?

[Daniel Dehennin]

Hello,

>> Unfortunately this setup makes `logging.basicConfig` pretty useless.
>> However, I believe that this is something that more people could
>> benefit from. I also believe, that it just "makes sense" to send
>> warnings (and above) to `stderr`, the rest to `stdout`.

[...]

> Python 2.x is closed to feature changes, and Python 2.7 and Python 3.2
> already support flexible configuration using dictConfig() - see
> 
> http://docs.python.org/library/logging.config.html#logging.config.dictConfig
> 
> Also, Python 3.3 will support passing a list of handlers to
> basicConfig(): see
> 
> http://plumberjack.blogspot.com/2011/04/added-functionality-for-basicconfig=
> -in.html
> 
> which will allow you to do what you want quite easily.

I tried to setup the same for my scripts with
"logging.config.dictConfig()" without much success, I want to limit
output to stdout to INFO, everything bellow INFO should be sent to
stderr instead.

Any hints?

Here is my configuration:

#+begin_src python
def init_logging(options):
# TODO: color stderr messages by level if sys.stderr.isatty()
config = { 'version' : 1,
   'formatters' : { 'stdout' : { 'format' : '%(message)s',
 'datefmt' : '', },
'stderr' : { 'format' : '%(message)s',
 'datefmt' : '', },
},
   'handlers' : { 'stdout' : { 'class' : 'logging.StreamHandler',
   'stream' : 'ext://sys.stdout',
   'level' : 'INFO',
   'formatter' : 'stdout', },
  'stderr' : { 'class' : 'logging.StreamHandler', 
   'stream' : 'ext://sys.stderr',
   'level' : 'WARNING', 
   'formatter' : 'stderr', }
},
   'root' : { 'level' : options.log_level.upper(),
  'handlers' : ['stdout', 'stderr'],
},
}

logging.config.dictConfig( config )
return logging.getLogger()
#+end_src

Regards.
-- 
Daniel Dehennin
EOLE


pgpP128hCWd2L.pgp
Description: PGP signature
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[inshu chauhan]

Thankyou.. but my problem is different than simply joining 2 lists and it
is done now :)

On Tue, Oct 23, 2012 at 11:38 AM, Joshua Landau
wrote:

> On 23/10/2012, inshu chauhan  wrote:
> > can we append a list with another list in Python ? using the normal
> routine
> > syntax but with a for loop ??
>
> I assume you want to join two lists.
>
> You are corrrect that we can do:
>
> >>> start = [1, 2, 3, 4]
> >>> end = [5, 6, 7, 8]
> >>>
> >>> for end_item in end:
> >>> start.append(end_item)
> >>>
> >>> print(start)
> [1, 2, 3, 4, 5, 6, 7, 8]
> >>>
>
> However, it is markedly repetitive, no?
> This is a common enough operation that there is a shortcut to find out
> about it.
>
> If you want to find out what methods there are, try "help(...)". I
> can't stress this enough.
>
> >>> help(start)
> Help on list object:
>
> class list(object)
>  |  list() -> new empty list
>  |  list(iterable) -> new list initialized from iterable's items
>  |
>  |  Methods defined here:
>  |
>  |  __add__(...)
>  |  x.__add__(y) <==> x+y
> ...
>  |  append(...)
>  |  L.append(object) -- append object to end
> ...
>  |  extend(...)
>  |  L.extend(iterable) -- extend list by appending elements from
> the iterable
> ...
>
> So list.extend seems to do exactly this!
>
> You can always check the documentation
> .
> An lo! The documentation says "start.extend(end)" is _equivilant_ to
> "start[len(start):] = end".
>
> Why?
>
> Well, this uses the slicing syntax.
>
> >>> start[:3]
> [1, 2, 3]
> >>> start[3:]
> [4]
> >>> start[2:3]
> [3]
>
> Wonderously, all these really say are "ranges" in the list. Hence, you
> can "put" lists in their place.
>
> "start[len(start):] = end" means "start[-1:] = end", so what you're
> doing is saying "the empty end part of the list is actually this new
> list". Hopefully that makes sense.
>
> Finally, there is another method. Instead of *changing* the list, you
> can make a new list which is equal to the others "added" together.
>
> >>> new = start + end
>
> ___
>
> Theses methods all have their own upsides. If you want to change the
> list, use .extend(). If you want to change the list, but by putting
> the new list somewhere inside the "old" one, use slicing:
>
> >>> start = [1, 2, 3, 4]
> >>> end = [5, 6, 7, 8]
> >>>
> >>> start[2:2] = end
> >>> print(start)
> [1, 2, 5, 6, 7, 8, 3, 4]
>
> Looping is good for when you want to generate the extra items as you go
> along.
>
> Finally, if you want to keep the old list or use these "inline", use "+".
>
> ___
>
> Note that, being in the unfortunate position of "away from an
> interpreter", none of my examples are copy-pastes. Hence they may be
> wrong :/
>
> # Not checked for errors, typos and my "friends" messing with it.
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Joshua Landau]

On 23/10/2012, inshu chauhan  wrote:
> can we append a list with another list in Python ? using the normal routine
> syntax but with a for loop ??

I assume you want to join two lists.

You are corrrect that we can do:

>>> start = [1, 2, 3, 4]
>>> end = [5, 6, 7, 8]
>>>
>>> for end_item in end:
>>> start.append(end_item)
>>>
>>> print(start)
[1, 2, 3, 4, 5, 6, 7, 8]
>>>

However, it is markedly repetitive, no?
This is a common enough operation that there is a shortcut to find out about it.

If you want to find out what methods there are, try "help(...)". I
can't stress this enough.

>>> help(start)
Help on list object:

class list(object)
 |  list() -> new empty list
 |  list(iterable) -> new list initialized from iterable's items
 |
 |  Methods defined here:
 |
 |  __add__(...)
 |  x.__add__(y) <==> x+y
...
 |  append(...)
 |  L.append(object) -- append object to end
...
 |  extend(...)
 |  L.extend(iterable) -- extend list by appending elements from
the iterable
...

So list.extend seems to do exactly this!

You can always check the documentation
.
An lo! The documentation says "start.extend(end)" is _equivilant_ to
"start[len(start):] = end".

Why?

Well, this uses the slicing syntax.

>>> start[:3]
[1, 2, 3]
>>> start[3:]
[4]
>>> start[2:3]
[3]

Wonderously, all these really say are "ranges" in the list. Hence, you
can "put" lists in their place.

"start[len(start):] = end" means "start[-1:] = end", so what you're
doing is saying "the empty end part of the list is actually this new
list". Hopefully that makes sense.

Finally, there is another method. Instead of *changing* the list, you
can make a new list which is equal to the others "added" together.

>>> new = start + end

___

Theses methods all have their own upsides. If you want to change the
list, use .extend(). If you want to change the list, but by putting
the new list somewhere inside the "old" one, use slicing:

>>> start = [1, 2, 3, 4]
>>> end = [5, 6, 7, 8]
>>>
>>> start[2:2] = end
>>> print(start)
[1, 2, 5, 6, 7, 8, 3, 4]

Looping is good for when you want to generate the extra items as you go along.

Finally, if you want to keep the old list or use these "inline", use "+".

___

Note that, being in the unfortunate position of "away from an
interpreter", none of my examples are copy-pastes. Hence they may be
wrong :/

# Not checked for errors, typos and my "friends" messing with it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: python scripts for web

[chip9munk]

On Friday, October 19, 2012 12:32:48 PM UTC+2, Gilles wrote:
> In that case, are you sure a web script is a good idea? If you're
> thinking web to make it easy for people to upload data, click on a
> button, and get the results back, you might want to write the UI in
> Python but write the number crunching part in a compiled language.

well actually I would like to separate the web interface with this API...
that is why I would like to work on the server side and not think about the 
interface side.



-- 
http://mail.python.org/mailman/listinfo/python-list

Fwd: can we append a list with another list in Python ?

[inshu chauhan]

-- Forwarded message --
From: Zero Piraeus 
Date: Tue, Oct 23, 2012 at 11:14 AM
Subject: Re: can we append a list with another list in Python ?
To: inshu chauhan 


:

On 23 October 2012 05:01, inshu chauhan  wrote:
> this is because this makes a single list out of 2 lists.. but I want to
retain both lists as lists within list... I hope u got it ??
>
> my problem is for example :
>
> x = [1,2,3]
> y = [10, 20, 30]
>
> I want the output to be :
>
> [[1,2,3], [10, 20, 30]] .. a list within list
>
> then i want to process each list in the big list using another function


For the example above, you'd just do:

>>> x = [1, 2, 3]
>>> y = [10, 20, 30]
>>> z = [x, y]
>>> z
[[1, 2, 3], [10, 20, 30]]

... but presumably there'll be more than two, and you're creating them
in some kind of loop? In that case, append is fine. For example:

>>> result = []
>>> for word in "squee", "kapow", "vroom":
... seq = list(word)  # or whatever you're doing to create the list
... result.append(seq)
...
>>> result
[['s', 'q', 'u', 'e', 'e'], ['k', 'a', 'p', 'o', 'w'], ['v', 'r', 'o',
'o', 'm']]

By the way - you only replied to me. Do you mind if I forward this
back to the list?

 -[]z.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Adnan Sadzak]

> ... but I'm not sure what you mean by "... with a for loop"?
>
>  -[]z.
>
>
Maybe this? :
-
x = [1,2,3]
y = [10,20,30]
for z in y:
 x.append(z)

print x
-

But list.extend should be right way.

Cheers
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Zero Piraeus]

:

On 23 October 2012 03:36, inshu chauhan  wrote:
> can we append a list with another list in Python ? using the normal routine
> syntax but with a for loop ??

The standard way to append the contents of one list to another is list.extend:

>>> a = [1, 2, 3]
>>> b = [4, 5, 6]
>>> a.extend(b)
>>> a
[1, 2, 3, 4, 5, 6]

... but I'm not sure what you mean by "... with a for loop"?

 -[]z.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Daniel Fetchinson]

> can we append a list with another list in Python ? using the normal routine
> syntax but with a for loop ??

x = [1,2,3]
y = [10,20,30]
x.extend( y )
print x

this will give you [1,2,3,10,20,30] which I guess is what you want.

Cheers,
Daniel



-- 
Psss, psss, put it down! - http://www.cafepress.com/putitdown
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Is there a way to programmatically turn on remote registry?

[Tim Golden]

On 22/10/2012 21:01, Kevin Holleran wrote:
> Tim,
> 
> I am looking here:
> 
> SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{BF9F6FB0-C999-4D19-BED0-144F77E2A9D6}
> 
> Enumerating the keys for a BusType == 5, then grabbing the values of
> DriverDesc, DriverDate, & DriverVersion.
> 
> So I am doing this:

[... snip querying uninstallers ...]

I don't have that particular uninstaller key but the code below, using
the wmi module to hide the plumbing, queries all the installers and
should give you enough of an idea, hopefully. For brevilty, I've only
bothered with extracting string values; it would be easy to extract
other datatypes.

To perform the same query on another computer, just pass the other
computer name (or IP address) as the first parameter to the wmi.WMI call
(or use the named param "computer").


import _winreg as winreg
import wmi

HKLM = winreg.HKEY_LOCAL_MACHINE
UNINSTALLERS = "SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Uninstall"

registry = wmi.WMI(namespace="default").StdRegProv
_, names = registry.EnumKey(HKLM, UNINSTALLERS)
for name in names:
print name
uninstaller = UNINSTALLERS + "\\" + name
_, value_names, value_types = registry.EnumValues(HKLM, uninstaller)
for value_name, value_type in zip(value_names, value_types):
if value_type == winreg.REG_SZ:
_, value = registry.GetStringValue(
  HKLM, uninstaller, value_name
)
else:
value = "(Non-string value)"
print u"  ", value_name, u"=>", value




TJG
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: can we append a list with another list in Python ?

[Alec Taylor]

Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit
(AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> [1,2,3,4,5] + [99,88,77,66,0]
[1, 2, 3, 4, 5, 99, 88, 77, 66, 0]

On Tue, Oct 23, 2012 at 6:36 PM, inshu chauhan  wrote:
> can we append a list with another list in Python ? using the normal routine
> syntax but with a for loop ??
>
> --
> http://mail.python.org/mailman/listinfo/python-list
>
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: A desperate lunge for on-topic-ness

[Joshua Landau]

On 23/10/2012, Dennis Lee Bieber  wrote:
> On Mon, 22 Oct 2012 16:02:34 -0600, Ian Kelly 
> declaimed the following in gmane.comp.python.general:
>
>> On my wishlist for Python is a big, fat SyntaxError for any variable
>> that could be interpreted as either local or nonlocal and is not
>> explicitly declared as either.  It would eliminate this sort of
>> confusion entirely and make code that shadows nonlocal variables much
>> more readable.
>>
>   Which now makes code dependent upon changes to some imported modules
> if someone is foolish enough to use the
>
>   from xyz import *
>
> notation...
>
>   I'd be very displeased if working code with local names suddenly
> fails because some third-party package was updated.
>
>   Yes, I prefer not to use the "from...*" notation, but how many
> tutorials (especially of GUI toolkits, with their dozens of constants)
> illustrate using the wildcard?

I'm not particularly fond (or disliking) of the proposal, but we
already make changes to the structure of locals/globals and so forth
when someone does "from  import *". Disabling checks when
it is used is totally reasonable.

Additionally, "SyntaxError: import * only allowed at module level".
This means, as far as I grasp, one should never *manage* to create an
ambiguity here. Ian already stated this idea should (due to
neccessity) be disabled for possible globals.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Compiling extension module (linker error)

[Paul Volkov]

2012/10/22 MRAB :
> By the way, the recommendation is for module names to be lowercase with
> underscores, so "fund_rose" instead of "FundRose".
>
> Try this code:
>

I tried as you suggested, but the linker error (unresolved external)
is still there.
I'm sure python33.lib is properly added because if I remove it, there
are 3 errors, not 1.
By the way, what is the difference between python3.lib and python33.lib?
-- 
http://mail.python.org/mailman/listinfo/python-list