Re: [Tutor] how to extract data only after a certain condition is met
On 10/6/2010 11:58 AM Joel Goldstick said...
On Wed, Oct 6, 2010 at 2:50 PM, Emile van Sebille wrote:
On 10/6/2010 9:25 AM Eduardo Vieira said...
Of course this solution is simpler:
extracted = a[a.index("i")+1:]
But I didn't want to build a list in memory with "readlines()" in the
case of a file.
This is what I do unless the files are _really big_
For-me-really-big-is-over-200Mb-ish-ly y'rs,
Emile
Why not loop with readline() and then the slice. That way only one line at
time in memory
Because I'd consider that a premature optimization. I don't commonly
worry about managing the memory footprint until there's a reason to.
I've found that you can work to minimize the footprint, but as it's
often indeterminate, you can't really control it. So I don't.
Emile
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how to extract data only after a certain condition is met
On Wed, Oct 6, 2010 at 2:50 PM, Emile van Sebille wrote:
> On 10/6/2010 9:25 AM Eduardo Vieira said...
>
>
>
> Of course this solution is simpler:
>> extracted = a[a.index("i")+1:]
>> But I didn't want to build a list in memory with "readlines()" in the
>> case of a file.
>>
>
> This is what I do unless the files are _really big_
>
> For-me-really-big-is-over-200Mb-ish-ly y'rs,
>
> Emile
>
> Why not loop with readline() and then the slice. That way only one line at
> time in memory
>
--
Joel Goldstick
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how to extract data only after a certain condition is met
On 10/6/2010 9:25 AM Eduardo Vieira said...
Of course this solution is simpler:
extracted = a[a.index("i")+1:]
But I didn't want to build a list in memory with "readlines()" in the
case of a file.
This is what I do unless the files are _really big_
For-me-really-big-is-over-200Mb-ish-ly y'rs,
Emile
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how to extract data only after a certain condition is met
"Eduardo Vieira" wrote The other day I was writing a script to extract data from a file from the line where a text is found to the end of the file. The standard pattern here is to use a sentinel, in pseudo code: def checkLine(line, start='',end=''): if (start in line) or (end in line): return True else: return False startPattern = 'some string (or regex)' endPattern = 'a concluding string or regex' sentinel = False while True read line from file sentinel = checkLine(line, startPattern, endPattern) if sentinel: processLine(line) You can simplify or complexify that in many ways, and you can add a break check to speed it up if you only expect to process a few lines. And checkLine can be as simple or as complex as you like. HTH, -- Alan Gauld Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Scrapy vs. Beautiful Soup
"Crusier" wrote I have also found that there is something called Scrapy. Please kindly comment on it. Which one is easier to use compared with Beautiful Soup? Well I can safely say that Beautiful Soup will be exactly the same as Beauitiful Soup to use! :-) As for scrapy I'd never heard of it, but a quick look at the tutorial suggests its a different kind of bbeast entirely. For a start you don't run it from Python you run it under scrapy - which is no doubt some kind of wrap around python, but exactly how you would integrate scrapy code within a bigger python project is not immediately clear. Whereas BS is just another module you can import as usual. Now it is possible that if all you want to do is scrape data from a web site then scrapy may be all you need and it may be easier than BS - it certainly didn't look too hard provided you are compfortable defining XPath statements - but thats not exactly a common skill! If you altready know XPath then scrapy would definitely be worth a try, if not I'd stick with BS. Just my opinion based on a very quick glance at the scrapy site. HTH, -- Alan Gauld Author of the Learn to Program web site http://www.alan-g.me.uk/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how to extract data only after a certain condition is met
Am 06.10.2010 18:25, schrieb Eduardo Vieira: The other day I was writing a script to extract data from a file from the line where a text is found to the end of the file. The same functionality is this sed script: '1,/regexp/'d I couldn't put my head to work around this and came up with a solution using list slicing. But how can I do that? I was experimenting with a simple list and I came up with this. I wonder if I shouldn't you a "while" statement, but how? a = ['m', 'a', 'r', 'i', 'g', 'o', 'l', 'd'] b = True for letter in a: if letter != 'i' and b: continue elif letter == 'i': b = False else: print letter Ok. This works, but I wonder if I shouldn't you a "while" statement, but how? Why would you want to use a while-loop? You would need to somehow stop the iteration (by catching some EOF Exception or the like). I think it's fine to use a for-loop as you have a predefined fixed number of iterations. I think your approach is OK. Easy to understand. But what if there's a second "i" after the first? In your solution all "i" are skipped. Also, I would choose clearer names: letters = ['m', 'a', 'r', 'i', 'g', 'o', 'l', 'd', 'i', 'n', 'i', 'o'] skip_letter = True for letter in letters: if letter == 'i' and skip_letter: skip_letter = False continue # if you don't want the first occurrence of "i" if not skip_letter: print letter Cheers, Jan ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] pymssql and encoding
Hi tutors,
I'm trying to fetch data from an MS SQL Server database with pymssql, but
non-ascii results get garbled.
>>> import pymssql
>>> conn = pymssql.connect(user='sa', password=myPassword, host=server,
database=targetDatabase, as_dict=True)
>>> curs = conn.cursor()
>>> curs.execute("select CardName from OCRD where CardCode = 'C00056'")
>>> resultAsDict = curs.fetchall()[0]
>>> customerName = resultAsDict['CardName']
>>> print customerName
ImmobiliŠre (whatever)
>>> customerName
'Immobili\x8are (whatever)'
There should be a small E with a grave accent (è) instead of the capital S
with a caron (Š) I'm getting.
I've tried applying various encodings, but to no avail:
>>> print customerName.decode('latin-1')
Immobili?re (whatever)
>>> print customerName.decode('utf-8')
Traceback (most recent call last):
File "", line 1, in
print customerName.decode('utf-8')
File "D:\Python26\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8a in position 8:
unexpected code byte
>>> print customerName.decode('utf-7')
Traceback (most recent call last):
File "", line 1, in
print customerName.decode('utf-7')
File "D:\Python26\lib\encodings\utf_7.py", line 12, in decode
return codecs.utf_7_decode(input, errors, True)
UnicodeDecodeError: 'utf7' codec can't decode byte 0x8a in position 8:
unexpected special character
When executed from MS SQL Server's Management Studio, the same query returns
"Immobilière (whatever)", with an 'è', as it should.
The field in question is of type nvarchar, with collation
SQL_Latin1_General_CP850_CI_AS.
Do you have an idea what the actual encoding of the string might be?
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] how to extract data only after a certain condition is met
Eduardo Vieira wrote: > The other day I was writing a script to extract data from a file from > the line where a text is found to the end of the file. The same > functionality is this sed script: > '1,/regexp/'d > I couldn't put my head to work around this and came up with a solution > using list slicing. But how can I do that? I was experimenting with a > simple list and I came up with this. I wonder if I shouldn't you a > "while" statement, but how? > > a = ['m', 'a', 'r', 'i', 'g', 'o', 'l', 'd'] > b = True > > for letter in a: > if letter != 'i' and b: > continue > elif letter == 'i': > b = False > else: > print letter > > Ok. This works, but I wonder if I shouldn't you a "while" statement, but > how? I would use two for loops: >>> a = ['m', 'a', 'r', 'i', 'g', 'o', 'l', 'd'] >>> ai = iter(a) # make a list iterator >>> for letter in ai: ... if letter == "i": break ... >>> for letter in ai: ... print letter ... g o l d Normally a list iterator is created implicitly by writing for item in some_list: ... but here you have to make one explicitly because you want to reuse it in the second loop. Alternatively, the itertools module has the building blocks for this and similar problems: >>> from itertools import dropwhile, islice >>> def not_an_i(letter): ... return letter != "i" ... >>> for letter in dropwhile(not_an_i, a): ... print letter ... i g o l d OK, let's shave off the first item in the dropwhile(...) sequence: >>> for letter in islice(dropwhile(not_an_i, a), 1, None): ... print letter ... g o l d Peter ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] how to extract data only after a certain condition is met
The other day I was writing a script to extract data from a file from
the line where a text is found to the end of the file. The same
functionality is this sed script:
'1,/regexp/'d
I couldn't put my head to work around this and came up with a solution
using list slicing. But how can I do that? I was experimenting with a
simple list and I came up with this. I wonder if I shouldn't you a
"while" statement, but how?
a = ['m', 'a', 'r', 'i', 'g', 'o', 'l', 'd']
b = True
for letter in a:
if letter != 'i' and b:
continue
elif letter == 'i':
b = False
else:
print letter
Ok. This works, but I wonder if I shouldn't you a "while" statement, but how?
Of course this solution is simpler:
extracted = a[a.index("i")+1:]
But I didn't want to build a list in memory with "readlines()" in the
case of a file.
Thanks for your guidance,
Eduardo
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] EXECUTING PYTHON AND SQL STAMENTS
hello
On Tue, Oct 5, 2010 at 5:57 PM, Susana Iraiis Delgado Rodriguez
wrote:
> Hello, I already solved the problem, I change all the code, instead of using
> os.system I changed to subprocess.Popen() and it worked fine:
> import shlex, subprocess
> def process():
> print "Ingresa en el siguiente orden:"
> print "Nombre del nuevo mapa.shp Nombre de la capa Nombre del mapa
> original"
> command_line = raw_input()
> args = shlex.split(command_line)
> p = subprocess.Popen(['C:/Archivos de
> programa/FWTools2.4.7/bin/ogr2ogr', args[0], '-where', args[1], args[2]])
> if p:
> print "Mapa generado"
> process()
>
> Now the user has to enter 3 arguments an finally it worked. I have a
> question, how can I tell the user if p execute ok? because even thouhg I
> entered wrong parameters, it prints "Mapa generado". This line should only
> appears if the arguments are acceptable.
it is better to validate the user input arguments before you execute
the subprocess
>
> 2010/10/5 Susana Iraiis Delgado Rodriguez
>>
>> Hello Norman:
>>
>> Thank you for taking the time to answer. I already changed my os.system()
>> for your code. I got an error, when I executed this:
>> os.system(" 'C:/Archivos de programa/FWTools2.4.7/bin/ogr2ogr.exe ' "+arg1
>> +" -where "+arg2 +" " +arg3)
>> it throws me that "C:/Archivos" is not recognized as an executable
>> external or internal command, programm or file.
>> If you really have other opton to fix my problem I'll be thankful because
>> I don't have any idea to make this code useful.
>> Thank you
>
> ___
> Tutor maillist - tu...@python.org
> To unsubscribe or change subscription options:
> http://mail.python.org/mailman/listinfo/tutor
>
>
--
˙uʍop ǝpısdn p,uɹnʇ pןɹoʍ ǝɥʇ ǝǝs noʎ 'ʇuǝɯɐן sǝɯıʇ ǝɥʇ puɐ 'ʇuǝʇuoɔ
ǝq s,ʇǝן ʇǝʎ
%>>> "".join( [ {'*':'@','^':'.'}.get(c,None) or
chr(97+(ord(c)-83)%26) for c in ",adym,*)&uzq^zqf" ] )
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] cgi help
Got it fixed. Thanks y'all. Regards, --Original Message-- From: Dave Angel To: Dipo Elegbede Cc: tutor Subject: Re: [Tutor] cgi help Sent: Oct 6, 2010 12:30 On 2:59 PM, Dipo Elegbede wrote: > here is the code: > > http://pastebin.com/K2Fxq6ac > > kindly help me figure out my mistake. > > It is supposed to return a table of environmental variables but it returns > only the header and a blank page. > > thanks > Care to define "return" ? How are you running this code? How are you trapping and displaying the exception it gets when run? Can you show us the traceback? Are you looking at the output in a web browser? Which one? And are you doing a view-Source, or whatever your browser offers? A CGI program is intended to run on a web server, and debugging it can be very difficult. Are you sure you want this as a beginner project? If so, then I'd start by saying you should run it locally, till you get rid of all the errors you'll see that way. By inspection, I can see that you should get more than the header, you should also get the element, then it'll crash on the bogus assignment of the uninitialized variable rowNumber. When I run it locally, I get: Content-type: text/html http://www.w3.org/1999/xhtml";> Environmental Variables Traceback (most recent call last): File "M:\cgitest.py", line 27, in rowNumber += 1 NameError: name 'rowNumber' is not defined You don't initialize the global variable rowNumber to anything, but just increment it. You do initialize a local variable in your function with the same name, but nothing is ever done with that, and of course it goes away when the function ends. Same with the backgroundColor local. To fix this problem you'll need to initialize rowNumber before the for statement. DaveA Sent from my BlackBerry wireless device from MTN ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] cgi help
On 2:59 PM, Dipo Elegbede wrote: here is the code: http://pastebin.com/K2Fxq6ac kindly help me figure out my mistake. It is supposed to return a table of environmental variables but it returns only the header and a blank page. thanks Care to define "return" ? How are you running this code? How are you trapping and displaying the exception it gets when run? Can you show us the traceback? Are you looking at the output in a web browser? Which one? And are you doing a view-Source, or whatever your browser offers? A CGI program is intended to run on a web server, and debugging it can be very difficult. Are you sure you want this as a beginner project? If so, then I'd start by saying you should run it locally, till you get rid of all the errors you'll see that way. By inspection, I can see that you should get more than the header, you should also get the element, then it'll crash on the bogus assignment of the uninitialized variable rowNumber. When I run it locally, I get: Content-type: text/html http://www.w3.org/1999/xhtml";> Environmental Variables Traceback (most recent call last): File "M:\cgitest.py", line 27, in rowNumber += 1 NameError: name 'rowNumber' is not defined You don't initialize the global variable rowNumber to anything, but just increment it. You do initialize a local variable in your function with the same name, but nothing is ever done with that, and of course it goes away when the function ends. Same with the backgroundColor local. To fix this problem you'll need to initialize rowNumber before the for statement. DaveA ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] Scrapy vs. Beautiful Soup
Hi Emile, I have also found that there is something called Scrapy. Please kindly comment on it. Which one is easier to use compared with Beautiful Soup? Thanks in advance. Cheers, Hank ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] CGI HELP
Yes, so I did a pastebin and resent to the group. Thanks. Sent from my BlackBerry wireless device from MTN -Original Message- From: Joel Goldstick Sender: tutor-bounces+delegbede=dudupay@python.org Date: Wed, 6 Oct 2010 05:54:03 To: tutor Subject: Re: [Tutor] CGI HELP ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] CGI HELP
On Wed, Oct 6, 2010 at 5:25 AM, Dipo Elegbede wrote:
> Hi all,
>
> I wrote this code as an example from a book:
>
> #!c:\Python26\python.exe
> # Program displaying CGI environment variables.
>
> import os
> import cgi
>
> def printHeader(title):
> print """Content-type: text/html
>
>
> "-//W3C//DTD XHTML 1.0 Strict//EN"
> "DTD/xhtml1-strict.dtd">
> http://www.w3.org/1999/xhtml";>
> %s
>
> """ % title
>
> rowNumber = 0
> backgroundColour = "white"
>
> printHeader("Environmental Variables")
> print ""
>
> # print table of cgi variables and values
> for item in os.environ.keys():
> rowNumber += 1
>
> if rowNumber % 2 == 0: # even row numbers are white
> backgroundColour = "white"
> else: # odd row numbers are grey
> backgroundColour = "lightgrey"
>
> print """
> %s%s""" %(backgroundColour,
> cgi.escape(item), cgi.escape(os.environ[item]))
> print ""
>
> It was supposed to return a list of environmental variables in a given
> order.
> Sadly only the header displayed.
> I have looked through over and over, please what am i doing wrong.
> This is from Deitel How to program python, Chapter 6.
>
> thanks and best regards.
> --
>
did you lose the indentation in copying code to email?
--
Joel Goldstick
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
[Tutor] cgi help
here is the code: http://pastebin.com/K2Fxq6ac kindly help me figure out my mistake. It is supposed to return a table of environmental variables but it returns only the header and a blank page. thanks -- Elegbede Muhammed Oladipupo OCA +2348077682428 +2347042171716 www.dudupay.com Mobile Banking Solutions | Transaction Processing | Enterprise Application Development ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
[Tutor] CGI HELP
Hi all,
I wrote this code as an example from a book:
#!c:\Python26\python.exe
# Program displaying CGI environment variables.
import os
import cgi
def printHeader(title):
print """Content-type: text/html
http://www.w3.org/1999/xhtml";>
%s
""" % title
rowNumber = 0
backgroundColour = "white"
printHeader("Environmental Variables")
print ""
# print table of cgi variables and values
for item in os.environ.keys():
rowNumber += 1
if rowNumber % 2 == 0: # even row numbers are white
backgroundColour = "white"
else: # odd row numbers are grey
backgroundColour = "lightgrey"
print """
%s%s""" %(backgroundColour,
cgi.escape(item), cgi.escape(os.environ[item]))
print ""
It was supposed to return a list of environmental variables in a given
order.
Sadly only the header displayed.
I have looked through over and over, please what am i doing wrong.
This is from Deitel How to program python, Chapter 6.
thanks and best regards.
--
Elegbede Muhammed Oladipupo
OCA
+2348077682428
+2347042171716
www.dudupay.com
Mobile Banking Solutions | Transaction Processing | Enterprise Application
Development
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] specifying precision with scientific notation
> But something like the "%.%de " %n is exactly what I am looking for - if I >could get it to work. Sorry my bad, I missed a % sign: >>> n=5 >>> "%%.%de" % n '%.5e' You need two %% to create a % in the output. Alan G. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: http://mail.python.org/mailman/listinfo/tutor
