Re: CSV reader ignore brackets
Skip Montanaro writes:
> How about just replacing *\(([^)]*)\)* with *"\1"* in a wrapper class's
> line reading method? (I think I have the re syntax approximately right.)
> The csv reader will "just work". Again, nesting parens not allowed.
>
> Skip
here is some working code:
def PReader(csvfile):
import re
for line in csvfile:
line = re.sub(r'\(.*?\)', '"\g<0>"', line)
yield line
import csv
with open('testcsv.csv') as csvfile:
reader = csv.reader(PReader(csvfile), quotechar='"')
for row in reader:
print(row)
--
Piet van Oostrum
WWW: http://piet.vanoostrum.org/
PGP key: [8DAE142BE17999C4]
--
https://mail.python.org/mailman/listinfo/python-list
Re: CSV reader ignore brackets
> Besides, the point isn't the shortest code but to illustrate the idea > of handling special syntax. In my defense, I was typing on my phone while watching a show on Netflix. I was hardly in a position to test any code. :-) As you indicated though, the problem is under-specified (nesting?, presence of quotation marks?, newlines between balanced parens? input size?, etc). It probably does little good to try and cook up a comprehensive solution to such a problem. Better to just toss out some ideas for the OP and let them mull it over, maybe try to solve the problem themselves. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: CSV reader ignore brackets
On 24Sep2019 19:02, Skip Montanaro wrote: How about just replacing *\(([^)]*)\)* with *"\1"* in a wrapper class's line reading method? Will that work if the OP's (TEST1,TEST2) term itself contains quotes? Not that his example data did, but example data are usually incomplete :-) Also, that would match FOO(TEST1,TEST2)BAH as well (making FOO"(TEST1,TEST2)"BAH. Which might be wanted, or be not wanted or be bad data (including but not restricted to csv module unparsable data). I was deliberately being very conservative and kind of treating brackets like quotes (needest at start and end) but not trying to hit things in one go. Better to match exactly the special case you expect and then scour of mismatches than to incorrectly match and have that mistake buried in the data. (I think I have the re syntax approximately right.) The csv reader will "just work". Again, nesting parens not allowed. Otherwise, a neat idea. Besides, the point isn't the shortest code but to illustrate the idea of handling special syntax. Cheers, Cameron Simpson -- https://mail.python.org/mailman/listinfo/python-list
Re: CSV reader ignore brackets
How about just replacing *\(([^)]*)\)* with *"\1"* in a wrapper class's line reading method? (I think I have the re syntax approximately right.) The csv reader will "just work". Again, nesting parens not allowed. Skip -- https://mail.python.org/mailman/listinfo/python-list
Re: CSV reader ignore brackets
On 2019-09-25 00:09, Cameron Simpson wrote:
On 24Sep2019 15:55, Mihir Kothari wrote:
I am using python 3.4. I have a CSV file as below:
ABC,PQR,(TEST1,TEST2)
FQW,RTE,MDE
Really? No quotes around the (TEST1,TEST2) column value? I would have
said this is invalid data, but that does not help you.
Basically comma-separated rows, where some rows have a data in column which
is array like i.e. in brackets.
So I need to read the file and treat such columns as one i.e. do not
separate based on comma if it is inside the bracket.
In short I need to read a CSV file where separator inside the brackets
needs to be ignored.
Output:
Column: 1 23
Row1:ABC PQR (TEST1,TEST2)
Row2:FQW RTE MDE
Can you please help with the snippet?
I would be reaching for a regular expression. If you partition your
values into 2 types: those starting and ending in a bracket, and those
not, you could write a regular expression for the former:
\([^)]*\)
which matches a string like (.) (with, importantly, no embedded
brackets, only those at the beginning and end.
And you can write a regular expression like:
[^,]*
for a value containing no commas i.e. all the other values.
Test the bracketed one first, because the second one always matches
something.
Then you would not use the CSV module (which expects better formed data
than you have) and instead write a simple parser for a line of text
which tries to match one of these two expressions repeatedly to consume
the line. Something like this (UNTESTED):
bracketed_re = re.compile(r'\([^)]*\)')
no_commas_re = re.compile(r'[^,]*')
def split_line(line):
line = line.rstrip() # drop trailing whitespace/newline
fields = []
offset = 0
while offset < len(line):
m = bracketed_re.match(line, offset)
if m:
field = m.group()
else:
m = no_commas_re.match(line, offset) # this always matches
field = m.group()
fields.append(field)
offset += len(field)
if line.startswith(',', offset):
# another column
offset += 1
elif offset < len(line):
raise ValueError(
"incomplete parse at offset %d, line=%r" % (offset, line))
return fields
Then read the lines of the file and split them into fields:
row = []
with open(datafilename) as f:
for line in f:
fields = split_line(line)
rows.append(fields)
So basicly you're writing a little parser. If you have nested brackets
things get harder.
You can simplify that somewhat to this:
import re
rows = []
with open(datafilename) as f:
for line in f:
rows.append(re.findall(r'(\([^)]*\)|(?=.)[^,\n]*),?', line))
--
https://mail.python.org/mailman/listinfo/python-list
Re: CSV reader ignore brackets
On 24Sep2019 15:55, Mihir Kothari wrote:
I am using python 3.4. I have a CSV file as below:
ABC,PQR,(TEST1,TEST2)
FQW,RTE,MDE
Really? No quotes around the (TEST1,TEST2) column value? I would have
said this is invalid data, but that does not help you.
Basically comma-separated rows, where some rows have a data in column which
is array like i.e. in brackets.
So I need to read the file and treat such columns as one i.e. do not
separate based on comma if it is inside the bracket.
In short I need to read a CSV file where separator inside the brackets
needs to be ignored.
Output:
Column: 1 23
Row1:ABC PQR (TEST1,TEST2)
Row2:FQW RTE MDE
Can you please help with the snippet?
I would be reaching for a regular expression. If you partition your
values into 2 types: those starting and ending in a bracket, and those
not, you could write a regular expression for the former:
\([^)]*\)
which matches a string like (.) (with, importantly, no embedded
brackets, only those at the beginning and end.
And you can write a regular expression like:
[^,]*
for a value containing no commas i.e. all the other values.
Test the bracketed one first, because the second one always matches
something.
Then you would not use the CSV module (which expects better formed data
than you have) and instead write a simple parser for a line of text
which tries to match one of these two expressions repeatedly to consume
the line. Something like this (UNTESTED):
bracketed_re = re.compile(r'\([^)]*\)')
no_commas_re = re.compile(r'[^,]*')
def split_line(line):
line = line.rstrip() # drop trailing whitespace/newline
fields = []
offset = 0
while offset < len(line):
m = bracketed_re.match(line, offset)
if m:
field = m.group()
else:
m = no_commas_re.match(line, offset) # this always matches
field = m.group()
fields.append(field)
offset += len(field)
if line.startswith(',', offset):
# another column
offset += 1
elif offset < len(line):
raise ValueError(
"incomplete parse at offset %d, line=%r" % (offset, line))
return fields
Then read the lines of the file and split them into fields:
row = []
with open(datafilename) as f:
for line in f:
fields = split_line(line)
rows.append(fields)
So basicly you're writing a little parser. If you have nested brackets
things get harder.
Cheers,
Cameron Simpson
--
https://mail.python.org/mailman/listinfo/python-list
CSV reader ignore brackets
Hi Team, I am using python 3.4. I have a CSV file as below: ABC,PQR,(TEST1,TEST2) FQW,RTE,MDE Basically comma-separated rows, where some rows have a data in column which is array like i.e. in brackets. So I need to read the file and treat such columns as one i.e. do not separate based on comma if it is inside the bracket. In short I need to read a CSV file where separator inside the brackets needs to be ignored. Output: Column: 1 23 Row1:ABC PQR (TEST1,TEST2) Row2:FQW RTE MDE Can you please help with the snippet? Regards, Mihir. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 30, 10:35 pm, rusi wrote: > On Jan 30, 6:31 pm, bansi wrote: > > Isn't it possible to implement your suggestion without installing > > Visual C++ 2008 . > > http://code.google.com/p/pyodbc/wiki/Building#Windows Well... This is what the official site says... On second thoughts I wonder: Would it not be possible to compile python +pyodbc from source and use gcc/ming for that? Someone who knows more of the windows build process may say something about the issues involved. -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 30, 6:31 pm, bansi wrote: > On Jan 28, 4:22 pm, Benjamin Kaplan wrote: > > > You'll need to have Visual C++ 2008 (not 2010) installed for this to > > work. You can get it for free > > fromhttp://www.microsoft.com/express/Downloads/if > > you don't already have it. > > > Thanks Benjamin. Wondering why i need to Visual C++ 2008 . What it has > to do with Python? > Isn't it possible to implement your suggestion without installing > Visual C++ 2008 . http://code.google.com/p/pyodbc/wiki/Building#Windows -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 28, 4:22 pm, Benjamin Kaplan wrote: > On Fri, Jan 28, 2011 at 3:42 PM, bansi wrote: > > On Jan 28, 1:52 pm, Benjamin Kaplan wrote: > >> On Fri, Jan 28, 2011 at 1:33 PM, bansi wrote: > >> > On Jan 28, 9:46 am, bansi wrote: > >> >> On Jan 26, 8:31 pm, MRAB wrote: > > >> >> > On 27/01/2011 00:57, bansi wrote: > > >> >> > > On Jan 26, 6:25 pm, Ethan Furman wrote: > >> >> > >> bansi wrote: > > >> >> > >> > First namelookupWrapper.py running under Python 2.6 accept > >> >> > >> arguments > >> >> > >> > from stdin and uses csv reader object to read it i.e. > >> >> > >> > r=csv.reader(sys.stdin) > > >> >> > >> > And then it has to pass csv reader object to another python > >> >> > >> script > >> >> > >> > namelookup.py running under Python 2.7 because it uses pyodbc > >> >> > >> to > >> >> > >> > connect to database and iterates thru reader object > > >> >> > >> Ben Finney wrote: > >> >> > >>> bansi writes: > > >> >> > >>>> Thanks Chris. Sorry for mis-communicating, the two python > >> >> > >>>> scripts are > >> >> > >>>> dependant in a way that namelookupWrapper.py needs to pass csv > >> >> > >>>> record > >> >> > >>>> object to another python script > > >> >> > >>> Why have you structured them that way, though? What constraint is > >> >> > >>> keeping you from doing the work in a single process, where the CSV > >> >> > >>> reader object can be shared? > > >> >> > >>>> If thats not possible then please let me know how to do the > >> >> > >>>> workaround > >> >> > >>>> i didnt understood the import thing and not sure if it helps in > >> >> > >>>> my > >> >> > >>>> case > > >> >> > >>> The problem as you've described it so far is best solved by > >> >> > >>> having a > >> >> > >>> single process accessing the CSV reader object in memory. If that > >> >> > >>> doesn't suit your use case, you'll need to explain why not. > > >> >> > >> In other words, why can't you use Python 2.7 to accept input and > >> >> > >> generate a csv.reader? > > >> >> > >> ~Ethan~- Hide quoted text - > > >> >> > >> - Show quoted text - > > >> >> > > Ethan, > >> >> > > The python script takes the input from Splunk > >> >> > > (http://www.splunk.com/ > >> >> > > base/Documentation/) which supports only Python 2.6 > >> >> > > So the real constraint is Splunk supports only Python 2.6 . > > >> >> > > As you know Python 2.6 doesnt support or doesnt have pyodbc install > >> >> > > for Windows 64 bit OS > >> >> > > So i installed Python 2.7 and thereafter pyodbc install for Windows > >> >> > > 64 > >> >> > > bit OS for Python 2.7 > > >> >> > Have you actually tried Splunk with Python 2.7? It might not work with > >> >> > versions which are earlier than Python 2.6, but that doesn't > >> >> > necessarily mean that it won't work with versions of Python 2 which > >> >> > are > >> >> > later than Python 2.6 (unless the documentation says that it must be > >> >> > Python 2.6).- Hide quoted text - > > >> >> > - Show quoted text - > > >> >> Splunk's latest version 4.1.6 doesn't support Python 2.7 > >> >> I tried the import trick but it didnt work because the real script > >> >> which runs under Python 2.7 has import pyodbc so it results in > >> >> following error > > >> >> c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py > >> >> memberId memberName < memberInput.csv > >> >> Traceback (most recent call last): > >> >> File "namelookupWrapper.py", line 3, in > >> >&g
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Fri, Jan 28, 2011 at 3:42 PM, bansi wrote: > On Jan 28, 1:52 pm, Benjamin Kaplan wrote: >> On Fri, Jan 28, 2011 at 1:33 PM, bansi wrote: >> > On Jan 28, 9:46 am, bansi wrote: >> >> On Jan 26, 8:31 pm, MRAB wrote: >> >> >> > On 27/01/2011 00:57, bansi wrote: >> >> >> > > On Jan 26, 6:25 pm, Ethan Furman wrote: >> >> > >> bansi wrote: >> >> >> > >> > First namelookupWrapper.py running under Python 2.6 accept >> >> > >> arguments >> >> > >> > from stdin and uses csv reader object to read it i.e. >> >> > >> > r=csv.reader(sys.stdin) >> >> >> > >> > And then it has to pass csv reader object to another python >> >> > >> script >> >> > >> > namelookup.py running under Python 2.7 because it uses pyodbc to >> >> > >> > connect to database and iterates thru reader object >> >> >> > >> Ben Finney wrote: >> >> > >>> bansi writes: >> >> >> > >>>> Thanks Chris. Sorry for mis-communicating, the two python scripts >> >> > >>>> are >> >> > >>>> dependant in a way that namelookupWrapper.py needs to pass csv >> >> > >>>> record >> >> > >>>> object to another python script >> >> >> > >>> Why have you structured them that way, though? What constraint is >> >> > >>> keeping you from doing the work in a single process, where the CSV >> >> > >>> reader object can be shared? >> >> >> > >>>> If thats not possible then please let me know how to do the >> >> > >>>> workaround >> >> > >>>> i didnt understood the import thing and not sure if it helps in my >> >> > >>>> case >> >> >> > >>> The problem as you've described it so far is best solved by having a >> >> > >>> single process accessing the CSV reader object in memory. If that >> >> > >>> doesn't suit your use case, you'll need to explain why not. >> >> >> > >> In other words, why can't you use Python 2.7 to accept input and >> >> > >> generate a csv.reader? >> >> >> > >> ~Ethan~- Hide quoted text - >> >> >> > >> - Show quoted text - >> >> >> > > Ethan, >> >> > > The python script takes the input from Splunk (http://www.splunk.com/ >> >> > > base/Documentation/) which supports only Python 2.6 >> >> > > So the real constraint is Splunk supports only Python 2.6 . >> >> >> > > As you know Python 2.6 doesnt support or doesnt have pyodbc install >> >> > > for Windows 64 bit OS >> >> > > So i installed Python 2.7 and thereafter pyodbc install for Windows 64 >> >> > > bit OS for Python 2.7 >> >> >> > Have you actually tried Splunk with Python 2.7? It might not work with >> >> > versions which are earlier than Python 2.6, but that doesn't >> >> > necessarily mean that it won't work with versions of Python 2 which are >> >> > later than Python 2.6 (unless the documentation says that it must be >> >> > Python 2.6).- Hide quoted text - >> >> >> > - Show quoted text - >> >> >> Splunk's latest version 4.1.6 doesn't support Python 2.7 >> >> I tried the import trick but it didnt work because the real script >> >> which runs under Python 2.7 has import pyodbc so it results in >> >> following error >> >> >> c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py >> >> memberId memberName < memberInput.csv >> >> Traceback (most recent call last): >> >> File "namelookupWrapper.py", line 3, in >> >> import namelookup >> >> File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in >> >> >> >> import pyodbc >> >> ImportError: DLL load failed: The specified module could not be found. >> >> >> Please let me know if i am missing something on import. If so please >> >> provide me with an example- Hide quoted text - >> >> >> - Show quoted text - >> >> >
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 28, 1:52 pm, Benjamin Kaplan wrote: > On Fri, Jan 28, 2011 at 1:33 PM, bansi wrote: > > On Jan 28, 9:46 am, bansi wrote: > >> On Jan 26, 8:31 pm, MRAB wrote: > > >> > On 27/01/2011 00:57, bansi wrote: > > >> > > On Jan 26, 6:25 pm, Ethan Furman wrote: > >> > >> bansi wrote: > > >> > >> > First namelookupWrapper.py running under Python 2.6 accept > >> > >> arguments > >> > >> > from stdin and uses csv reader object to read it i.e. > >> > >> > r=csv.reader(sys.stdin) > > >> > >> > And then it has to pass csv reader object to another python > >> > >> script > >> > >> > namelookup.py running under Python 2.7 because it uses pyodbc to > >> > >> > connect to database and iterates thru reader object > > >> > >> Ben Finney wrote: > >> > >>> bansi writes: > > >> > >>>> Thanks Chris. Sorry for mis-communicating, the two python scripts > >> > >>>> are > >> > >>>> dependant in a way that namelookupWrapper.py needs to pass csv > >> > >>>> record > >> > >>>> object to another python script > > >> > >>> Why have you structured them that way, though? What constraint is > >> > >>> keeping you from doing the work in a single process, where the CSV > >> > >>> reader object can be shared? > > >> > >>>> If thats not possible then please let me know how to do the > >> > >>>> workaround > >> > >>>> i didnt understood the import thing and not sure if it helps in my > >> > >>>> case > > >> > >>> The problem as you've described it so far is best solved by having a > >> > >>> single process accessing the CSV reader object in memory. If that > >> > >>> doesn't suit your use case, you'll need to explain why not. > > >> > >> In other words, why can't you use Python 2.7 to accept input and > >> > >> generate a csv.reader? > > >> > >> ~Ethan~- Hide quoted text - > > >> > >> - Show quoted text - > > >> > > Ethan, > >> > > The python script takes the input from Splunk (http://www.splunk.com/ > >> > > base/Documentation/) which supports only Python 2.6 > >> > > So the real constraint is Splunk supports only Python 2.6 . > > >> > > As you know Python 2.6 doesnt support or doesnt have pyodbc install > >> > > for Windows 64 bit OS > >> > > So i installed Python 2.7 and thereafter pyodbc install for Windows 64 > >> > > bit OS for Python 2.7 > > >> > Have you actually tried Splunk with Python 2.7? It might not work with > >> > versions which are earlier than Python 2.6, but that doesn't > >> > necessarily mean that it won't work with versions of Python 2 which are > >> > later than Python 2.6 (unless the documentation says that it must be > >> > Python 2.6).- Hide quoted text - > > >> > - Show quoted text - > > >> Splunk's latest version 4.1.6 doesn't support Python 2.7 > >> I tried the import trick but it didnt work because the real script > >> which runs under Python 2.7 has import pyodbc so it results in > >> following error > > >> c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py > >> memberId memberName < memberInput.csv > >> Traceback (most recent call last): > >> File "namelookupWrapper.py", line 3, in > >> import namelookup > >> File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in > >> > >> import pyodbc > >> ImportError: DLL load failed: The specified module could not be found. > > >> Please let me know if i am missing something on import. If so please > >> provide me with an example- Hide quoted text - > > >> - Show quoted text - > > > Here are some more details from my earlier posting. Please click the > > below link > > >http://answers.splunk.com/questions/11145/its-getting-mysterious-to-m... > > -- > >http://mail.python.org/mailman/listinfo/python-list > > Have you tried downloading the source for PyODBC and compiling it > yourself? All you need to do is python setup.py install. My guess > would be that it works just fine on 64-bit Python 2.6, they just never > released a re-compiled version of it for that platform.- Hide quoted text - > > - Show quoted text - Thanks Benjamin. Please point me to the website from where i can download pyodbc for Windows 64 bit OS under Python 2.6 and installation instructions -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Fri, Jan 28, 2011 at 1:33 PM, bansi wrote: > On Jan 28, 9:46 am, bansi wrote: >> On Jan 26, 8:31 pm, MRAB wrote: >> >> >> >> >> >> > On 27/01/2011 00:57, bansi wrote: >> >> > > On Jan 26, 6:25 pm, Ethan Furman wrote: >> > >> bansi wrote: >> >> > >> > First namelookupWrapper.py running under Python 2.6 accept >> > >> arguments >> > >> > from stdin and uses csv reader object to read it i.e. >> > >> > r=csv.reader(sys.stdin) >> >> > >> > And then it has to pass csv reader object to another python script >> > >> > namelookup.py running under Python 2.7 because it uses pyodbc to >> > >> > connect to database and iterates thru reader object >> >> > >> Ben Finney wrote: >> > >>> bansi writes: >> >> > >>>> Thanks Chris. Sorry for mis-communicating, the two python scripts are >> > >>>> dependant in a way that namelookupWrapper.py needs to pass csv record >> > >>>> object to another python script >> >> > >>> Why have you structured them that way, though? What constraint is >> > >>> keeping you from doing the work in a single process, where the CSV >> > >>> reader object can be shared? >> >> > >>>> If thats not possible then please let me know how to do the workaround >> > >>>> i didnt understood the import thing and not sure if it helps in my >> > >>>> case >> >> > >>> The problem as you've described it so far is best solved by having a >> > >>> single process accessing the CSV reader object in memory. If that >> > >>> doesn't suit your use case, you'll need to explain why not. >> >> > >> In other words, why can't you use Python 2.7 to accept input and >> > >> generate a csv.reader? >> >> > >> ~Ethan~- Hide quoted text - >> >> > >> - Show quoted text - >> >> > > Ethan, >> > > The python script takes the input from Splunk (http://www.splunk.com/ >> > > base/Documentation/) which supports only Python 2.6 >> > > So the real constraint is Splunk supports only Python 2.6 . >> >> > > As you know Python 2.6 doesnt support or doesnt have pyodbc install >> > > for Windows 64 bit OS >> > > So i installed Python 2.7 and thereafter pyodbc install for Windows 64 >> > > bit OS for Python 2.7 >> >> > Have you actually tried Splunk with Python 2.7? It might not work with >> > versions which are earlier than Python 2.6, but that doesn't >> > necessarily mean that it won't work with versions of Python 2 which are >> > later than Python 2.6 (unless the documentation says that it must be >> > Python 2.6).- Hide quoted text - >> >> > - Show quoted text - >> >> Splunk's latest version 4.1.6 doesn't support Python 2.7 >> I tried the import trick but it didnt work because the real script >> which runs under Python 2.7 has import pyodbc so it results in >> following error >> >> c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py >> memberId memberName < memberInput.csv >> Traceback (most recent call last): >> File "namelookupWrapper.py", line 3, in >> import namelookup >> File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in >> >> import pyodbc >> ImportError: DLL load failed: The specified module could not be found. >> >> Please let me know if i am missing something on import. If so please >> provide me with an example- Hide quoted text - >> >> - Show quoted text - > > Here are some more details from my earlier posting. Please click the > below link > > http://answers.splunk.com/questions/11145/its-getting-mysterious-to-make-the-lookup-script-work > -- > http://mail.python.org/mailman/listinfo/python-list > Have you tried downloading the source for PyODBC and compiling it yourself? All you need to do is python setup.py install. My guess would be that it works just fine on 64-bit Python 2.6, they just never released a re-compiled version of it for that platform. -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 28, 9:46 am, bansi wrote: > On Jan 26, 8:31 pm, MRAB wrote: > > > > > > > On 27/01/2011 00:57, bansi wrote: > > > > On Jan 26, 6:25 pm, Ethan Furman wrote: > > >> bansi wrote: > > > >> > First namelookupWrapper.py running under Python 2.6 accept arguments > > >> > from stdin and uses csv reader object to read it i.e. > > >> > r=csv.reader(sys.stdin) > > > >> > And then it has to pass csv reader object to another python script > > >> > namelookup.py running under Python 2.7 because it uses pyodbc to > > >> > connect to database and iterates thru reader object > > > >> Ben Finney wrote: > > >>> bansi writes: > > > >>>> Thanks Chris. Sorry for mis-communicating, the two python scripts are > > >>>> dependant in a way that namelookupWrapper.py needs to pass csv record > > >>>> object to another python script > > > >>> Why have you structured them that way, though? What constraint is > > >>> keeping you from doing the work in a single process, where the CSV > > >>> reader object can be shared? > > > >>>> If thats not possible then please let me know how to do the workaround > > >>>> i didnt understood the import thing and not sure if it helps in my > > >>>> case > > > >>> The problem as you've described it so far is best solved by having a > > >>> single process accessing the CSV reader object in memory. If that > > >>> doesn't suit your use case, you'll need to explain why not. > > > >> In other words, why can't you use Python 2.7 to accept input and > > >> generate a csv.reader? > > > >> ~Ethan~- Hide quoted text - > > > >> - Show quoted text - > > > > Ethan, > > > The python script takes the input from Splunk (http://www.splunk.com/ > > > base/Documentation/) which supports only Python 2.6 > > > So the real constraint is Splunk supports only Python 2.6 . > > > > As you know Python 2.6 doesnt support or doesnt have pyodbc install > > > for Windows 64 bit OS > > > So i installed Python 2.7 and thereafter pyodbc install for Windows 64 > > > bit OS for Python 2.7 > > > Have you actually tried Splunk with Python 2.7? It might not work with > > versions which are earlier than Python 2.6, but that doesn't > > necessarily mean that it won't work with versions of Python 2 which are > > later than Python 2.6 (unless the documentation says that it must be > > Python 2.6).- Hide quoted text - > > > - Show quoted text - > > Splunk's latest version 4.1.6 doesn't support Python 2.7 > I tried the import trick but it didnt work because the real script > which runs under Python 2.7 has import pyodbc so it results in > following error > > c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py > memberId memberName < memberInput.csv > Traceback (most recent call last): > File "namelookupWrapper.py", line 3, in > import namelookup > File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in > > import pyodbc > ImportError: DLL load failed: The specified module could not be found. > > Please let me know if i am missing something on import. If so please > provide me with an example- Hide quoted text - > > - Show quoted text - Here are some more details from my earlier posting. Please click the below link http://answers.splunk.com/questions/11145/its-getting-mysterious-to-make-the-lookup-script-work -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 26, 8:31 pm, MRAB wrote: > On 27/01/2011 00:57, bansi wrote: > > > > > > > On Jan 26, 6:25 pm, Ethan Furman wrote: > >> bansi wrote: > > >> > First namelookupWrapper.py running under Python 2.6 accept arguments > >> > from stdin and uses csv reader object to read it i.e. > >> > r=csv.reader(sys.stdin) > > >> > And then it has to pass csv reader object to another python script > >> > namelookup.py running under Python 2.7 because it uses pyodbc to > >> > connect to database and iterates thru reader object > > >> Ben Finney wrote: > >>> bansi writes: > > >>>> Thanks Chris. Sorry for mis-communicating, the two python scripts are > >>>> dependant in a way that namelookupWrapper.py needs to pass csv record > >>>> object to another python script > > >>> Why have you structured them that way, though? What constraint is > >>> keeping you from doing the work in a single process, where the CSV > >>> reader object can be shared? > > >>>> If thats not possible then please let me know how to do the workaround > >>>> i didnt understood the import thing and not sure if it helps in my > >>>> case > > >>> The problem as you've described it so far is best solved by having a > >>> single process accessing the CSV reader object in memory. If that > >>> doesn't suit your use case, you'll need to explain why not. > > >> In other words, why can't you use Python 2.7 to accept input and > >> generate a csv.reader? > > >> ~Ethan~- Hide quoted text - > > >> - Show quoted text - > > > Ethan, > > The python script takes the input from Splunk (http://www.splunk.com/ > > base/Documentation/) which supports only Python 2.6 > > So the real constraint is Splunk supports only Python 2.6 . > > > As you know Python 2.6 doesnt support or doesnt have pyodbc install > > for Windows 64 bit OS > > So i installed Python 2.7 and thereafter pyodbc install for Windows 64 > > bit OS for Python 2.7 > > Have you actually tried Splunk with Python 2.7? It might not work with > versions which are earlier than Python 2.6, but that doesn't > necessarily mean that it won't work with versions of Python 2 which are > later than Python 2.6 (unless the documentation says that it must be > Python 2.6).- Hide quoted text - > > - Show quoted text - Splunk's latest version 4.1.6 doesn't support Python 2.7 I tried the import trick but it didnt work because the real script which runs under Python 2.7 has import pyodbc so it results in following error c:\Splunk\etc\apps\search\bin>splunk cmd python namelookupWrapper.py memberId memberName < memberInput.csv Traceback (most recent call last): File "namelookupWrapper.py", line 3, in import namelookup File "c:\Splunk\etc\apps\search\bin\namelookup.py", line 7, in import pyodbc ImportError: DLL load failed: The specified module could not be found. Please let me know if i am missing something on import. If so please provide me with an example -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On 27/01/2011 00:57, bansi wrote: On Jan 26, 6:25 pm, Ethan Furman wrote: bansi wrote: > First namelookupWrapper.py running under Python 2.6 accept arguments > from stdin and uses csv reader object to read it i.e. > r=csv.reader(sys.stdin) > > And then it has to pass csv reader object to another python script > namelookup.py running under Python 2.7 because it uses pyodbc to > connect to database and iterates thru reader object Ben Finney wrote: bansi writes: Thanks Chris. Sorry for mis-communicating, the two python scripts are dependant in a way that namelookupWrapper.py needs to pass csv record object to another python script Why have you structured them that way, though? What constraint is keeping you from doing the work in a single process, where the CSV reader object can be shared? If thats not possible then please let me know how to do the workaround i didnt understood the import thing and not sure if it helps in my case The problem as you've described it so far is best solved by having a single process accessing the CSV reader object in memory. If that doesn't suit your use case, you'll need to explain why not. In other words, why can't you use Python 2.7 to accept input and generate a csv.reader? ~Ethan~- Hide quoted text - - Show quoted text - Ethan, The python script takes the input from Splunk (http://www.splunk.com/ base/Documentation/) which supports only Python 2.6 So the real constraint is Splunk supports only Python 2.6 . As you know Python 2.6 doesnt support or doesnt have pyodbc install for Windows 64 bit OS So i installed Python 2.7 and thereafter pyodbc install for Windows 64 bit OS for Python 2.7 Have you actually tried Splunk with Python 2.7? It might not work with versions which are earlier than Python 2.6, but that doesn't necessarily mean that it won't work with versions of Python 2 which are later than Python 2.6 (unless the documentation says that it must be Python 2.6). -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 26, 6:25 pm, Ethan Furman wrote: > bansi wrote: > > > First namelookupWrapper.py running under Python 2.6 accept arguments > > from stdin and uses csv reader object to read it i.e. > > r=csv.reader(sys.stdin) > > > > And then it has to pass csv reader object to another python script > > namelookup.py running under Python 2.7 because it uses pyodbc to > > connect to database and iterates thru reader object > > > > > > Ben Finney wrote: > > bansi writes: > > >> Thanks Chris. Sorry for mis-communicating, the two python scripts are > >> dependant in a way that namelookupWrapper.py needs to pass csv record > >> object to another python script > > > Why have you structured them that way, though? What constraint is > > keeping you from doing the work in a single process, where the CSV > > reader object can be shared? > > >> If thats not possible then please let me know how to do the workaround > >> i didnt understood the import thing and not sure if it helps in my > >> case > > > The problem as you've described it so far is best solved by having a > > single process accessing the CSV reader object in memory. If that > > doesn't suit your use case, you'll need to explain why not. > > In other words, why can't you use Python 2.7 to accept input and > generate a csv.reader? > > ~Ethan~- Hide quoted text - > > - Show quoted text - Ethan, The python script takes the input from Splunk (http://www.splunk.com/ base/Documentation/) which supports only Python 2.6 So the real constraint is Splunk supports only Python 2.6 . As you know Python 2.6 doesnt support or doesnt have pyodbc install for Windows 64 bit OS So i installed Python 2.7 and thereafter pyodbc install for Windows 64 bit OS for Python 2.7 -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
bansi wrote: > First namelookupWrapper.py running under Python 2.6 accept arguments > from stdin and uses csv reader object to read it i.e. > r=csv.reader(sys.stdin) > > And then it has to pass csv reader object to another python script > namelookup.py running under Python 2.7 because it uses pyodbc to > connect to database and iterates thru reader object Ben Finney wrote: bansi writes: Thanks Chris. Sorry for mis-communicating, the two python scripts are dependant in a way that namelookupWrapper.py needs to pass csv record object to another python script Why have you structured them that way, though? What constraint is keeping you from doing the work in a single process, where the CSV reader object can be shared? If thats not possible then please let me know how to do the workaround i didnt understood the import thing and not sure if it helps in my case The problem as you've described it so far is best solved by having a single process accessing the CSV reader object in memory. If that doesn't suit your use case, you'll need to explain why not. In other words, why can't you use Python 2.7 to accept input and generate a csv.reader? ~Ethan~ -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 26, 4:36 pm, Ben Finney wrote: > bansi writes: > > Thanks Chris. Sorry for mis-communicating, the two python scripts are > > dependant in a way that namelookupWrapper.py needs to pass csv record > > object to another python script > > Why have you structured them that way, though? What constraint is > keeping you from doing the work in a single process, where the CSV > reader object can be shared? > > > If thats not possible then please let me know how to do the workaround > > i didnt understood the import thing and not sure if it helps in my > > case > > The problem as you've described it so far is best solved by having a > single process accessing the CSV reader object in memory. If that > doesn't suit your use case, you'll need to explain why not. > > -- > \ “To have the choice between proprietary software packages, is | > `\ being able to choose your master. Freedom means not having a | > _o__) master.” —Richard M. Stallman, 2007-05-16 | > Ben Finney Thanks Ben for quick response. The constraint is in using third party tool called Splunk which has in-built Python 2.6 which doesnt support pyodbc on Windows 64 bit OS. Hence i have to install Python 2.7 for pyodbc. That means namelookupwrapper.py acts as a wrapper running under Splunk environment taking input from Splunk via stdin and storing it in csv reader object and then needs to call actual script namelookup.py under Python 2.7 for making connection to database Hope i have clarified a bit -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
bansi writes: > Thanks Chris. Sorry for mis-communicating, the two python scripts are > dependant in a way that namelookupWrapper.py needs to pass csv record > object to another python script Why have you structured them that way, though? What constraint is keeping you from doing the work in a single process, where the CSV reader object can be shared? > If thats not possible then please let me know how to do the workaround > i didnt understood the import thing and not sure if it helps in my > case The problem as you've described it so far is best solved by having a single process accessing the CSV reader object in memory. If that doesn't suit your use case, you'll need to explain why not. -- \ “To have the choice between proprietary software packages, is | `\ being able to choose your master. Freedom means not having a | _o__)master.” —Richard M. Stallman, 2007-05-16 | Ben Finney -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 26, 1:31 pm, Chris Rebert wrote: > On Wed, Jan 26, 2011 at 7:51 AM, bansi wrote: > > I have following two python scripts > > -namelookupWrapper.py > > -namelookup.py > > > The namelookupWrapper.py takes input of "memberId", "memberName" from > > CLI and has following code snippet > > > idf = sys.argv[1] > > namef = sys.argv[2] > > real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" > > r = csv.reader(sys.stdin) > > os.execv(python_executable, [ python_executable, real_script ] + > > sys.argv[1:] ) > > > Wondering how would i pass csv reader object "r" as an argument using > > os.execv() to another python script i.e. namelookup.py > > It's not possible to pass Python objects between processes in such a > manner. Given that "independent" scripts can't directly take objects > as input anyway, I doubt the two scripts are truly independent from > each other. I would therefore concur with van Sebille that you should > just rewrite them so that one script imports from the other rather > than spawning the other. It should not be too hard to port the Python > 2.6 script to Python 2.7 (or vice-versa if necessary). > > Cheers, > Chris > --http://blog.rebertia.com- Hide quoted text - > > - Show quoted text - Thanks Chris. Sorry for mis-communicating, the two python scripts are dependant in a way that namelookupWrapper.py needs to pass csv record object to another python script If thats not possible then please let me know how to do the workaround i didnt understood the import thing and not sure if it helps in my case Here are the details namelookupwrapper.py - takes input from stdin. Using csv reader object i iterate thru the input which looks like as shown below [MemberId, MemberName] [123, ] [456, ] [989, ] Now i have another script i.e. namelookup.py running under Python 2.7 using pyodbc to retrieve Member Names from database for a given Member Id in namelooupWrapper.py So please let me know how to accomplish this -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Wed, Jan 26, 2011 at 7:51 AM, bansi wrote: > I have following two python scripts > -namelookupWrapper.py > -namelookup.py > > > The namelookupWrapper.py takes input of "memberId", "memberName" from > CLI and has following code snippet > > idf = sys.argv[1] > namef = sys.argv[2] > real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" > r = csv.reader(sys.stdin) > os.execv(python_executable, [ python_executable, real_script ] + > sys.argv[1:] ) > > Wondering how would i pass csv reader object "r" as an argument using > os.execv() to another python script i.e. namelookup.py It's not possible to pass Python objects between processes in such a manner. Given that "independent" scripts can't directly take objects as input anyway, I doubt the two scripts are truly independent from each other. I would therefore concur with van Sebille that you should just rewrite them so that one script imports from the other rather than spawning the other. It should not be too hard to port the Python 2.6 script to Python 2.7 (or vice-versa if necessary). Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On Jan 26, 11:30 am, Emile van Sebille wrote: > On 1/26/2011 7:51 AM bansi said... > > > > > > > I have following two python scripts > > -namelookupWrapper.py > > -namelookup.py > > > The namelookupWrapper.py takes input of "memberId", "memberName" from > > CLI and has following code snippet > > > idf = sys.argv[1] > > namef = sys.argv[2] > > real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" > > r = csv.reader(sys.stdin) > > os.execv(python_executable, [ python_executable, real_script ] + > > sys.argv[1:] ) > > > Wondering how would i pass csv reader object "r" as an argument using > > os.execv() to another python script i.e. namelookup.py > > I suspect you're on the wrong path. You probably want to import > namelookup within namelooupWrapper to use the functions it defines. > > Consider: > > [root@fcfw2 src]# cat > test1.py > > def say(what): print what > > [root@fcfw2 src]# cat > test2.py > > #!/usr/local/bin/python > import sys > from test1 import say > say(sys.argv[1]) > > [root@fcfw2 src]# chmod a+x test2.py > > [root@fcfw2 src]# ./test2.py hello > hello > > HTH, > > Emile- Hide quoted text - > > - Show quoted text - Emile, Thanks for quick response. I am not sure if "import namelookup within namelooupWrapper" helps because they are two independent scripts which has to be executed in sequence. First namelookupWrapper.py running under Python 2.6 accept arguments from stdin and uses csv reader object to read it i.e. r=csv.reader(sys.stdin) And then it has to pass csv reader object to another python script namelookup.py running under Python 2.7 because it uses pyodbc to connect to database and iterates thru reader object Any better ideas/suggestions will be greatly appreciated -- http://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to pass CSV Reader Object As Argument to another Python File ???
On 1/26/2011 7:51 AM bansi said... I have following two python scripts -namelookupWrapper.py -namelookup.py The namelookupWrapper.py takes input of "memberId", "memberName" from CLI and has following code snippet idf = sys.argv[1] namef = sys.argv[2] real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" r = csv.reader(sys.stdin) os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:] ) Wondering how would i pass csv reader object "r" as an argument using os.execv() to another python script i.e. namelookup.py I suspect you're on the wrong path. You probably want to import namelookup within namelooupWrapper to use the functions it defines. Consider: [root@fcfw2 src]# cat > test1.py def say(what): print what [root@fcfw2 src]# cat > test2.py #!/usr/local/bin/python import sys from test1 import say say(sys.argv[1]) [root@fcfw2 src]# chmod a+x test2.py [root@fcfw2 src]# ./test2.py hello hello HTH, Emile -- http://mail.python.org/mailman/listinfo/python-list
Is it possible to pass CSV Reader Object As Argument to another Python File ??? Options
I have following two python scripts -namelookupWrapper.py -namelookup.py The namelookupWrapper.py takes input of "memberId", "memberName" from CLI and has following code snippet idf = sys.argv[1] namef = sys.argv[2] real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" r = csv.reader(sys.stdin) os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:] ) Wondering how would i pass csv reader object "r" as an argument using os.execv() to another python script i.e. namelookup.py Regards, Bansi - Bansilal Haudakari SUN Certified Enterprise JAVA Architect / Programmer http://bansihaudakari.wordpress.com/ L:703-445-2894 -- http://mail.python.org/mailman/listinfo/python-list
Is it possible to pass CSV Reader Object As Argument to another Python File ???
I have following two python scripts -namelookupWrapper.py -namelookup.py The namelookupWrapper.py takes input of "memberId", "memberName" from CLI and has following code snippet idf = sys.argv[1] namef = sys.argv[2] real_script = "C:\\Splunk\\etc\\apps\\search\\bin\\namelookup.py" r = csv.reader(sys.stdin) os.execv(python_executable, [ python_executable, real_script ] + sys.argv[1:] ) Wondering how would i pass csv reader object "r" as an argument using os.execv() to another python script i.e. namelookup.py -- http://mail.python.org/mailman/listinfo/python-list
Re: csv reader
As csv.reader does not suport utf-8 encoded files, I'm using:
fp = codecs.open(arquivoCSV, "r", "utf-8")
self.tab=[]
for l in fp:
l=l.replace('\"','').strip()
self.tab.append(l.split(','))
It works much better except that when I do self.sel.type("q", ustring)
where ustring is a unicode string obtained from the file using the
code showed above.
Remaining problem is that I obtain insted of a regular space...
--
http://mail.python.org/mailman/listinfo/python-list
Re: csv reader
En Tue, 15 Dec 2009 19:12:01 -0300, Emmanuel escribió:
Then my problem is diferent!
In fact I'm reading a csv file saved from openoffice oocalc using
UTF-8 encoding. I get a list of list (let's cal it tab) with the csv
data.
If I do:
print tab[2][4]
In ipython, I get:
equação de Toricelli. Tarefa exercícios PVR 1 e 2 ; PVP 1
If I only do:
tab[2][4]
In ipython, I get:
'equa\xc3\xa7\xc3\xa3o de Toricelli. Tarefa exerc\xc3\xadcios PVR 1 e
2 ; PVP 1'
Does that mean that my problem is not the one I'm thinking?
Yes. You have a real problem, but not this one. When you say `print
something`, you get a nice view of `something`, basically the result of
doing `str(something)`. When you say `something` alone in the interpreter,
you get a more formal representation, the result of calling
`repr(something)`:
py> x = "ecuação"
py> print x
ecuação
py> x
'ecua\x87\xc6o'
py> print repr(x)
'ecua\x87\xc6o'
Those '' around the text and the \xNN notation allow for an unambiguous
representation. Two strings may "look like" the same but be different, and
repr shows that.
('ecua\x87\xc6o' is encoded in windows-1252; you should see
'equa\xc3\xa7\xc3\xa3o' in utf-8)
My real problem is when I use that that kind of UTF-8 encoded (?) with
selenium here.
If I just switch the folowing line:
self.sel.type("q", "equação")
by:
self.sel.type("q", u"equação")
It works fine!
Yes: you should work with unicode most of the time. The "recipe" for
having as little unicode problems as possible says:
- convert the input data (read from external sources, like a file) from
bytes to unicode, using the (known) encoding of those bytes
- handle unicode internally everywhere in your program
- and convert from unicode to bytes as late as possible, when writing
output (to screen, other files, etc) using the encoding expected by those
external files.
See the Unicode How To: http://docs.python.org/howto/unicode.html
The problem is that the csv.reader does give a "equação" and not a
u"equação"
The csv module cannot handle unicode text directly, but see the last
example in the csv documentation for a simple workaround:
http://docs.python.org/library/csv.html
--
Gabriel Genellina
--
http://mail.python.org/mailman/listinfo/python-list
Re: csv reader
Then my problem is diferent!
In fact I'm reading a csv file saved from openoffice oocalc using
UTF-8 encoding. I get a list of list (let's cal it tab) with the csv
data.
If I do:
print tab[2][4]
In ipython, I get:
equação de Toricelli. Tarefa exercícios PVR 1 e 2 ; PVP 1
If I only do:
tab[2][4]
In ipython, I get:
'equa\xc3\xa7\xc3\xa3o de Toricelli. Tarefa exerc\xc3\xadcios PVR 1 e
2 ; PVP 1'
Does that mean that my problem is not the one I'm thinking?
My real problem is when I use that that kind of UTF-8 encoded (?) with
selenium here.
Here is an small code example of a not-working case giving the same
error that on my bigger program:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from selenium import selenium
import sys,os,csv,re
class test:
'''classe para interagir com o sistema acadêmico'''
def __init__(self):
self.webpage=''
self.arquivo=''
self.script=[]
self.sel = selenium('localhost', , '*firefox', 'http://
www.google.com.br')
self.sel.start()
self.sel.open('/')
self.sel.wait_for_page_to_load(3)
self.sel.type("q", "equação")
#self.sel.type("q", u"equacao")
self.sel.click("btnG")
self.sel.wait_for_page_to_load("3")
def main():
teste=test()
if __name__ == "__main__":
main()
If I just switch the folowing line:
self.sel.type("q", "equação")
by:
self.sel.type("q", u"equação")
It works fine!
The problem is that the csv.reader does give a "equação" and not a
u"equação"
Here is the error given with bad code (with "equação"):
ERROR: An unexpected error occurred while tokenizing input
The following traceback may be corrupted or invalid
The error message is: ('EOF in multi-line statement', (1202, 0))
---
UnicodeDecodeErrorTraceback (most recent call
last)
/home/manu/Labo/Cefetes_Colatina/Scripts/
20091215_test_acentuated_caracters.py in ()
27
28 if __name__ == "__main__":
---> 29 main()
30
31
/home/manu/Labo/Cefetes_Colatina/Scripts/
20091215_test_acentuated_caracters.py in main()
23
24 def main():
---> 25 teste=test()
26
27
/home/manu/Labo/Cefetes_Colatina/Scripts/
20091215_test_acentuated_caracters.py in __init__(self)
16 self.sel.open('/')
17 self.sel.wait_for_page_to_load(3)
---> 18 self.sel.type("q", "equação")
19 #self.sel.type("q", u"equacao")
20 self.sel.click("btnG")
/home/manu/Labo/Cefetes_Colatina/Scripts/selenium.pyc in type(self,
locator, value)
588 'value' is the value to type
589 """
--> 590 self.do_command("type", [locator,value,])
591
592
/home/manu/Labo/Cefetes_Colatina/Scripts/selenium.pyc in do_command
(self, verb, args)
201 body = u'cmd=' + urllib.quote_plus(unicode(verb).encode
('utf-8'))
202 for i in range(len(args)):
--> 203 body += '&' + unicode(i+1) + '=' +
urllib.quote_plus(unicode(args[i]).encode('utf-8'))
204 if (None != self.sessionId):
205 body += "&sessionId=" + unicode(self.sessionId)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
4: ordinal not in range(128)
WARNING: Failure executing file:
<20091215_test_acentuated_caracters.py>
Python 2.6.4 (r264:75706, Oct 27 2009, 06:16:59)
--
http://mail.python.org/mailman/listinfo/python-list
Re: csv reader
On Tue, Dec 15, 2009 at 4:24 PM, Emmanuel wrote:
> I have a problem with csv.reader from the library csv. I'm not able to
> import accentuated caracters. For example, I'm trying to import a
> simple file containing a single word "equação" using the following
> code:
>
> import csv
> arquivoCSV='test'
> a=csv.reader(open(arquivoCSV),delimiter=',')
> tab=[]
> for row in a:
> tab.append(row)
> print tab
>
> As a result, I get:
>
> [['equa\xe7\xe3o']]
>
> How can I solve this problem?
I don't think it is a problem. \xe7 is the character ç encoded in
Windows-1252, which is probably the encoding of your csv file. If you
want to convert that to a unicode string, do something like the
following.
s = 'equa\xe7\xe3o'
uni_s = s.decode('Windows-1252')
print uni_s
--
Jerry
--
http://mail.python.org/mailman/listinfo/python-list
Re: csv reader
On Tue, Dec 15, 2009 at 1:24 PM, Emmanuel wrote: > I have a problem with csv.reader from the library csv. I'm not able to > import accentuated caracters. For example, I'm trying to import a > simple file containing a single word "equação" using the following > code: > > import csv > arquivoCSV='test' > a=csv.reader(open(arquivoCSV),delimiter=',') > tab=[] > for row in a: > tab.append(row) > print tab > > As a result, I get: > > [['equa\xe7\xe3o']] > > How can I solve this problem? From http://docs.python.org/library/csv.html : """ Note: This version of the csv module doesn’t support Unicode input. Also, there are currently some issues regarding ASCII NUL characters. Accordingly, all input should be UTF-8 or printable ASCII to be safe; see the examples in section Examples. These restrictions will be removed in the future. """ Thus, you'll have to decode the results into Unicode manually; this will require knowing what encoding your file is using. Files in some encodings may not parse correctly due to the aforementioned NUL problem. Cheers, Chris -- http://blog.rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
csv reader
I have a problem with csv.reader from the library csv. I'm not able to import accentuated caracters. For example, I'm trying to import a simple file containing a single word "equação" using the following code: import csv arquivoCSV='test' a=csv.reader(open(arquivoCSV),delimiter=',') tab=[] for row in a: tab.append(row) print tab As a result, I get: [['equa\xe7\xe3o']] How can I solve this problem? -- http://mail.python.org/mailman/listinfo/python-list
Re: intermediate python csv reader/writer question from a beginner
Learning Python wrote:
> anything related to csv, I usually use VB within excel to manipulate
> the data, nonetheless, i finally got the courage to take a dive into
> python. i have viewed a lot of googled csv tutorials, but none of
> them address everything i need. Nonetheless, I was wondering if
> someone can help me manipulate the sample csv (sample.csv) I have
> generated:
>
> ,,
> someinfo,,,
> somotherinfo,,,
> SEQ,Names,Test1,Test2,Date,Time,,
> 1,Adam,1,2,Monday,1:00 PM,,
> 2,Bob,3,4,Monday,1:00 PM,,
> 3,Charlie,5,6,Monday,1:00 PM,,
> 4,Adam,7,8,Monday,2:00 PM,,
> 5,Bob,9,10,Monday,2:00 PM,,
> 6,Charlie,11,12,Monday,2:00 PM,,
> 7,Adam,13,14,Tuesday,1:00 PM,,
> 8,Bob,15,16,Tuesday,1:00 PM,,
> 9,Charlie,17,18,Tuesday,1:00 PM,,
>
> into (newfile.csv):
>
> Adam-Test1,Adam-Test2,Bob-Test1,Bob-Test2,Charlie-Test1,Charlie-
> Test2,Date,Time
> 1,2,3,4,5,6,Monday,1:00 PM
> 7,8,9,10,11,12,Monday,2:00 PM
> 13,14,15,16,17,18,Tuesday,1:00 PM
>
> note:
> 1. the true header doesn't start line 4 (if this is the case would i
> have to use "split"?)
> 2. if there were SEQ#10-12, or 13-15, it would still be Adam, Bob,
> Charlie, but with different Test1/Test2/Date/Time
I'm not really sure what you are trying to calculate, but this should
give you some ideas...
import csv
from collections import defaultdict
reader = csv.reader(open("sample.csv"))
result = defaultdict(list)
for row in reader:
# ignore unless first row is numeric
if not row or not row[0].isdigit():
continue
n, name, a, b, day, time = row[:6]
print "n=%r, name=%r, a=%r, b=%r, day=%r, time=%r" % (n, name, a,
b, day, time)
result[(day, time)].append(n)
writer = csv.writer(open("newfile.csv", "w"))
for key, values in result.iteritems():
day, time = key
values = values + [day, time]
writer.writerow(values)
This prints
n='1', name='Adam', a='1', b='2', day='Monday', time='1:00 PM'
n='2', name='Bob', a='3', b='4', day='Monday', time='1:00 PM'
n='3', name='Charlie', a='5', b='6', day='Monday', time='1:00 PM'
n='4', name='Adam', a='7', b='8', day='Monday', time='2:00 PM'
n='5', name='Bob', a='9', b='10', day='Monday', time='2:00 PM'
n='6', name='Charlie', a='11', b='12', day='Monday', time='2:00 PM'
n='7', name='Adam', a='13', b='14', day='Tuesday', time='1:00 PM'
n='8', name='Bob', a='15', b='16', day='Tuesday', time='1:00 PM'
n='9', name='Charlie', a='17', b='18', day='Tuesday', time='1:00 PM'
And leaves newfile.csv with the contents
1,2,3,Monday,1:00 PM
7,8,9,Tuesday,1:00 PM
4,5,6,Monday,2:00 PM
--
Nick Craig-Wood -- http://www.craig-wood.com/nick
--
http://mail.python.org/mailman/listinfo/python-list
Re: intermediate python csv reader/writer question from a beginner
On Mon, Feb 23, 2009 at 1:33 PM, Learning Python wrote: > anything related to csv, I usually use VB within excel to manipulate > the data, nonetheless, i finally got the courage to take a dive into > python. i have viewed a lot of googled csv tutorials, but none of > them address everything i need. Nonetheless, I was wondering if > someone can help me manipulate the sample csv (sample.csv) I have > generated: For reading and writing CSV files, see the `csv` module in the standard library (http://docs.python.org/library/csv.html). Cheers, Chris -- Follow the path of the Iguana... http://rebertia.com -- http://mail.python.org/mailman/listinfo/python-list
intermediate python csv reader/writer question from a beginner
anything related to csv, I usually use VB within excel to manipulate the data, nonetheless, i finally got the courage to take a dive into python. i have viewed a lot of googled csv tutorials, but none of them address everything i need. Nonetheless, I was wondering if someone can help me manipulate the sample csv (sample.csv) I have generated: ,, someinfo,,, somotherinfo,,, SEQ,Names,Test1,Test2,Date,Time,, 1,Adam,1,2,Monday,1:00 PM,, 2,Bob,3,4,Monday,1:00 PM,, 3,Charlie,5,6,Monday,1:00 PM,, 4,Adam,7,8,Monday,2:00 PM,, 5,Bob,9,10,Monday,2:00 PM,, 6,Charlie,11,12,Monday,2:00 PM,, 7,Adam,13,14,Tuesday,1:00 PM,, 8,Bob,15,16,Tuesday,1:00 PM,, 9,Charlie,17,18,Tuesday,1:00 PM,, into (newfile.csv): Adam-Test1,Adam-Test2,Bob-Test1,Bob-Test2,Charlie-Test1,Charlie- Test2,Date,Time 1,2,3,4,5,6,Monday,1:00 PM 7,8,9,10,11,12,Monday,2:00 PM 13,14,15,16,17,18,Tuesday,1:00 PM note: 1. the true header doesn't start line 4 (if this is the case would i have to use "split"?) 2. if there were SEQ#10-12, or 13-15, it would still be Adam, Bob, Charlie, but with different Test1/Test2/Date/Time -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV reader and unique ids
Mike P wrote:
I'm trying to use the CSV module to read in some data and then use a
hashable method (as there are millions of records) to find unique ids
and push these out to another file,
You could either zip with a counter or use the uuid module,
depending on just how unique you want your ids to be.
import os, sys
import csv
import itertools
import uuid
stuff = "the quick brown fox jumps over the lazy dog".split ()
f = open ("output.csv", "wb")
writer = csv.writer (f)
#
# Style 1 - numeric counter
#
writer.writerows (zip (itertools.count (), stuff))
#
# Style 2 - uuid
#
writer.writerows ((uuid.uuid1 (), s) for s in stuff)
f.close ()
os.startfile ("output.csv")
TJG
--
http://mail.python.org/mailman/listinfo/python-list
CSV reader and unique ids
Hi All,
I'm trying to use the CSV module to read in some data and then use a
hashable method (as there are millions of records) to find unique ids
and push these out to another file,
can anyone advise? Below is the code so far
fin = open(CSV_INPUT, "rb")
fout = open(CSV_OUTPUT, "wb")
reader = csv.reader(fin, delimiter=chr(254))
writer = csv.writer(fout)
headerList = reader.next()
UID = {}
#For help
#print headerList
# ['Time', 'User-ID', 'IP']
try:
for row in reader[1]:
UID[row] = 1
else:
List= UID.keys()
writer.writerows(List)
fin.close()
fout.close()
Mike
--
http://mail.python.org/mailman/listinfo/python-list
RE: CSV Reader
> -Original Message-
> From: [EMAIL PROTECTED] [mailto:python-
> [EMAIL PROTECTED] On Behalf Of Mike P
> Sent: Tuesday, February 12, 2008 5:37 AM
> To: python-list@python.org
> Subject: Re: CSV Reader
>
> just saw i needed to change record.startswith to row.startswith
> but i get hte following traceback error
>
> Traceback (most recent call last):
> File "C:\Python25\Lib\site-packages\pythonwin\pywin\framework
> \scriptutils.py", line 310, in RunScript
> exec codeObject in __main__.__dict__
> File "Y:\technical\Research\E2C\Template_CSV\import CSV test.py",
> line 10, in
> if not start_line and row.startswith('Transaction ID'):
> AttributeError: 'list' object has no attribute 'startswith'
> --
> http://mail.python.org/mailman/listinfo/python-list
Algorithms + Data Structures = Programs
You need to understand what kind of data structure a "list" is. You're
foundering a bit because you don't have a good understanding of your
tools (the list data structure in this case.) Try going through the
O'Reilly Learning Python book. Even better would be to take/audit a
college/university class on data structures and algorithms.
*
The information transmitted is intended only for the person or entity to which
it is addressed and may contain confidential, proprietary, and/or privileged
material. Any review, retransmission, dissemination or other use of, or taking
of any action in reliance upon this information by persons or entities other
than the intended recipient is prohibited. If you received this in error,
please contact the sender and delete the material from all computers. GA623
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
En Tue, 12 Feb 2008 08:37:13 -0200, Mike P
<[EMAIL PROTECTED]> escribi�:
> just saw i needed to change record.startswith to row.startswith
> but i get hte following traceback error
>
> Traceback (most recent call last):
> File "C:\Python25\Lib\site-packages\pythonwin\pywin\framework
> \scriptutils.py", line 310, in RunScript
> exec codeObject in __main__.__dict__
> File "Y:\technical\Research\E2C\Template_CSV\import CSV test.py",
> line 10, in
> if not start_line and row.startswith('Transaction ID'):
> AttributeError: 'list' object has no attribute 'startswith'
The csv reader doesn't return complete lines, but a list of fields for
each line. If you want to check the first field, use:
if not start_line and row[0].startswith('Transaction ID'):
--
Gabriel Genellina
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
I was just trying to do it with the CSV module -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
Hi Chris that's exactley what i wanted to do, Many thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
On Feb 12, 12:21 pm, Mike P <[EMAIL PROTECTED]> wrote:
> I did just try to post, but it doesn't look like it has appeared?
>
> I've used your advice Andrew and tried to use the CSV module, but now
> it doesn't seem to pick up the startswith command?
> Is this because of the way the CSV module is reading the data in?
> I've looked into the module description but i can't find anything that
> i hould be using?
>
> Can anyone offer an advice?
>
> Cheers again
>
> Mike
>
> working_CSV = "//filer/common/technical/Research/E2C/Template_CSV/
> DFAExposureToConversionQueryTool.csv"
>
> save_file = "//filer/common/technical/Research/E2C/Template_CSV/
> CSV_Data2.csv"
>
> start_line=False
> import csv
> reader = csv.reader(open(working_CSV, "rb"))
> writer = csv.writer(open(save_file, "wb"))
> for row in reader:
> if not start_line and record.startswith("'Transaction ID'"):
> start_line=True
> if start_line:
> print row
> writer.writerows(rows)
> #writer.close()
record won't have an attribute 'startswith' because record is a list
and startswith is a function of a string.
Also, your code isn't exactly clear on what you want to do, if it is
just "Find the first occurence of Transaction ID and pump the file
from then onwards into a new file" why not
output = open('output_file.csv','wb')
start_line = False
for each_line in open('input_file.csv','rb'):
if not start_line and each_line.startswith("'Transaction ID'"):
start_line = True
if start_line:
output.write( each_line )
output.close()
also, if you need a line number for any purposes, take a look at
enumerate() and with that it will return a counter and your data, for
eg. 'for (line_num, each_line) in enumerate(input_file):'. Counting
starts @ zero though so you would need to add 1.
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
just saw i needed to change record.startswith to row.startswith
but i get hte following traceback error
Traceback (most recent call last):
File "C:\Python25\Lib\site-packages\pythonwin\pywin\framework
\scriptutils.py", line 310, in RunScript
exec codeObject in __main__.__dict__
File "Y:\technical\Research\E2C\Template_CSV\import CSV test.py",
line 10, in
if not start_line and row.startswith('Transaction ID'):
AttributeError: 'list' object has no attribute 'startswith'
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
I did just try to post, but it doesn't look like it has appeared?
I've used your advice Andrew and tried to use the CSV module, but now
it doesn't seem to pick up the startswith command?
Is this because of the way the CSV module is reading the data in?
I've looked into the module description but i can't find anything that
i hould be using?
Can anyone offer an advice?
Cheers again
Mike
working_CSV = "//filer/common/technical/Research/E2C/Template_CSV/
DFAExposureToConversionQueryTool.csv"
save_file = "//filer/common/technical/Research/E2C/Template_CSV/
CSV_Data2.csv"
start_line=False
import csv
reader = csv.reader(open(working_CSV, "rb"))
writer = csv.writer(open(save_file, "wb"))
for row in reader:
if not start_line and record.startswith("'Transaction ID'"):
start_line=True
if start_line:
print row
writer.writerows(rows)
#writer.close()
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
En Mon, 11 Feb 2008 14:41:54 -0200, Mike P
<[EMAIL PROTECTED]> escribi�:
> CSV_Data = open(working_CSV)
> data = CSV_Data.readlines()
> flag=False
> for record in data:
> if record.startswith('"Transaction ID"'):
> [...]
Files are already iterable by lines. There is no need to use readlines(),
and you can avoid the already menctioned potential slowdown problem. Just
remove the data=CSV_data.readlines() line, and change that for statement
to be:
for record in CSV_Data:
Reading the style guide may be beneficial:
http://www.python.org/dev/peps/pep-0008/
--
Gabriel Genellina
--
http://mail.python.org/mailman/listinfo/python-list
RE: CSV Reader
> -Original Message-
> From: [EMAIL PROTECTED] [mailto:python-
> [EMAIL PROTECTED] On Behalf Of Mike P
> Sent: Monday, February 11, 2008 11:42 AM
> To: python-list@python.org
> Subject: Re: CSV Reader
>
> Cheers for the help, the second way looked to be the best in the end,
> and thanks for the boolean idea
>
> Mike
>
>
>
> working_CSV = "//filer/common/technical/Research/E2C/Template_CSV/
> DFAExposureToConversionQueryTool.csv"
>
> save_file = open("//filer/common/technical/Research/E2C/Template_CSV/
> CSV_Data2.csv","w")
>
> CSV_Data = open(working_CSV)
> data = CSV_Data.readlines()
> flag=False
> for record in data:
> if record.startswith('"Transaction ID"'):
> flag=True
> if flag:
> save_file.write(record)
> save_file.close()
Don't be a pansy.
Use the csv module, or add a check for
record.startswith('TransactionID'). There's no guarantee that csv
columns will be double-quoted. (Leading whitespace may or may not be
acceptable, too.) Look at the first piece of sample code in the
documentation for the csv module. (Section 9.1.5 in python 2.5) You're
99% of the way to using csv.reader() properly.
Nitpick: If the boolean check is expensive, then
if not flag and record.startswith(...):
flag = true
Nitpick: flag is a weak name. Use something like bPrint, okay2print,
or print_now or anything that's more descriptive. In larger and/or more
complex programs, meaningful variable names are a must.
*
The information transmitted is intended only for the person or entity to which
it is addressed and may contain confidential, proprietary, and/or privileged
material. Any review, retransmission, dissemination or other use of, or taking
of any action in reliance upon this information by persons or entities other
than the intended recipient is prohibited. If you received this in error,
please contact the sender and delete the material from all computers. GA622
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
Cheers for the help, the second way looked to be the best in the end,
and thanks for the boolean idea
Mike
working_CSV = "//filer/common/technical/Research/E2C/Template_CSV/
DFAExposureToConversionQueryTool.csv"
save_file = open("//filer/common/technical/Research/E2C/Template_CSV/
CSV_Data2.csv","w")
CSV_Data = open(working_CSV)
data = CSV_Data.readlines()
flag=False
for record in data:
if record.startswith('"Transaction ID"'):
flag=True
if flag:
save_file.write(record)
save_file.close()
--
http://mail.python.org/mailman/listinfo/python-list
RE: CSV Reader
> -Original Message- > From: [EMAIL PROTECTED] [mailto:python- > [EMAIL PROTECTED] On Behalf Of Mike P > Sent: Monday, February 11, 2008 11:10 AM > To: python-list@python.org > Subject: Re: CSV Reader > > Hi Larry, > > i'm still getting to grips with python, but rest assured i thinkn it's > better for me to write hte code for learnign purposes > > My basic file is here, it comes up with a syntax error on the > startswith line, is this because it is potentially a list? > My idea was to get the lines number where i can see Transaction ID and > then write out everything from this point into a new datafile. > > >From the docs for reader: "All data read are returned as strings. No automatic data type conversion is performed." Just use print or repr() to see what the row data looks. Then the method to check for 'transaction id' should be abundantly clear. for data in reader: print data print repr(data) > Would a better solution be just to use readlines and search for the > string with a counter and then write out a file from there? Yes you could, but the danger is that you get an insanely large file that blows out your memory or causes the process to swap to disk space (disk is slow.) Just loop through the lines and use a boolean flag to determine when to start printing. * The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential, proprietary, and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from all computers. GA621 -- http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
Hi Larry,
i'm still getting to grips with python, but rest assured i thinkn it's
better for me to write hte code for learnign purposes
My basic file is here, it comes up with a syntax error on the
startswith line, is this because it is potentially a list?
My idea was to get the lines number where i can see Transaction ID and
then write out everything from this point into a new datafile.
Would a better solution be just to use readlines and search for the
string with a counter and then write out a file from there?
Any help is greatly appreciated
Mike
working_CSV = "//filer/common/technical/Research/E2C/Template_CSV/
DFAExposureToConversionQueryTool.csv"
import csv
f = file(working_CSV, 'rb')
reader = csv.reader(f)
CSV_lines = ""
for data in reader:
if lines.startswith("Transaction ID")
append.reader.line_num()
# there will only be 1 instance of this title at the start of the CSV
file
writer(Working_csv.csv[, dialect='excel'][, fmtparam])
--
http://mail.python.org/mailman/listinfo/python-list
Re: CSV Reader
Mike P wrote: > Hi All, > > I want to read in a CSV file, but then write out a new CSV file from a > given line.. > > I'm using the CSV reader and have the the line where i want to start > writing the new file from begins with > "Transaction ID", > > i thought it should be something along the lines of below.. obvioulsy > this doesn't work but any help would be great. > > import csv > f = file(working_CSV, 'rb') > new_data = 0 # a counter to find where the line starts with > "Transaction ID" > reader = csv.reader(f) > for data in reader: > read data file > > write new CSV > > Cheers > > Mike > What part "obviously" doesn't work? Try something, post any tracebacks and we will try to help. Don't ask others to write your code for you without actually trying it yourself. It appears you are on the right track. -Larry -- http://mail.python.org/mailman/listinfo/python-list
CSV Reader
Hi All, I want to read in a CSV file, but then write out a new CSV file from a given line.. I'm using the CSV reader and have the the line where i want to start writing the new file from begins with "Transaction ID", i thought it should be something along the lines of below.. obvioulsy this doesn't work but any help would be great. import csv f = file(working_CSV, 'rb') new_data = 0 # a counter to find where the line starts with "Transaction ID" reader = csv.reader(f) for data in reader: read data file write new CSV Cheers Mike -- http://mail.python.org/mailman/listinfo/python-list
Re: High level csv reader
[EMAIL PROTECTED] wrote: > George> It occured to me that most times I read a csv file, I'm often > George> doing from scratch things like assigning labels to columns, > George> mapping fields to the appropriate type, ignoring some fields, > George> changing their order, etc. Before I go on and reinvent the > George> wheel, is there a generic high level wrapper around csv.reader > George> that does all this ? > > I'm not aware of anything that specifically addresses these ideas. Here are > some thoughts though: > > * If you use the DictReader class you can ignore fields you aren't > interested in more easily since you access the fields of interest by > name and with the fieldnames parameter to the constructor can assign > column names to csv data which lacks it (use similar functionality in > DictWriter to create column labels on output). You can also specify > the restkey parameter to the constructor and thus only specify the > fields of interest in the fieldnames parameter. (I think. I've never > actually used that capability, but that's what the documentation > suggests.) > > * There was a thread earlier this month with this subject: > > paseline(my favorite simple script): does something similar exist? > > Check it out for a number of different solutions to formatting the > fields in a line of text. The idea can easily be extended to a list > or dict of values instead, perhaps in a subclass of DictReader. > > Skip Indeed, they are both relevant; actually Fredrik's suggestion in that thread was my starting point. Here's my my current API for the (most typical) case of fixed-size rows, addressing my most common requirement, field conversions (no named columns for now): http://rafb.net/paste/results/4UgvSD50.html. It looks somewhat involved but it's more along the lines of "making easy things easy and hard things possible". Comments and suggestions are most welcome. George -- http://mail.python.org/mailman/listinfo/python-list
Re: High level csv reader
George> It occured to me that most times I read a csv file, I'm often George> doing from scratch things like assigning labels to columns, George> mapping fields to the appropriate type, ignoring some fields, George> changing their order, etc. Before I go on and reinvent the George> wheel, is there a generic high level wrapper around csv.reader George> that does all this ? I'm not aware of anything that specifically addresses these ideas. Here are some thoughts though: * If you use the DictReader class you can ignore fields you aren't interested in more easily since you access the fields of interest by name and with the fieldnames parameter to the constructor can assign column names to csv data which lacks it (use similar functionality in DictWriter to create column labels on output). You can also specify the restkey parameter to the constructor and thus only specify the fields of interest in the fieldnames parameter. (I think. I've never actually used that capability, but that's what the documentation suggests.) * There was a thread earlier this month with this subject: paseline(my favorite simple script): does something similar exist? Check it out for a number of different solutions to formatting the fields in a line of text. The idea can easily be extended to a list or dict of values instead, perhaps in a subclass of DictReader. Skip -- http://mail.python.org/mailman/listinfo/python-list
Re: High level csv reader
George Sakkis wrote: > It occured to me that most times I read a csv file, I'm often doing > from scratch things like assigning labels to columns, mapping fields to > the appropriate type, ignoring some fields, changing their order, etc. > Before I go on and reinvent the wheel, is there a generic high level > wrapper around csv.reader that does all this ? > Hi George, Firstly one thing to remember: if you are going to reinvent the wheel, don't forget to also reinvent the axle :-) AFAIK there is no such animal. I would need a lot of persuasion that a wrapper or toolbox wasn't just overhead with little benefit. For example, ignoring some fields and/or changing the order of others can be done rather simply: | >>> inrow | ['a', 'b', 'c', 'd', 'e', 'f'] Suppose we want to ignore the "vowels" and reverse the order of the others: | >>> outrow = list(inrow[k] for k in (5, 3, 2, 1)) | >>> outrow | ['f', 'd', 'c', 'b'] I don't see the value in creating (and documenting!) a one-line function def gather_list_items(input_list, item_map): return list(input_list[k] for k in item_map) NB something like this is already in the mx kit somewhere IIRC. Cheers, John -- http://mail.python.org/mailman/listinfo/python-list
Re: High level csv reader
James Stroud wrote: > George Sakkis wrote: > > It occured to me that most times I read a csv file, I'm often doing > > from scratch things like assigning labels to columns, mapping fields to > > the appropriate type, ignoring some fields, changing their order, etc. > > Before I go on and reinvent the wheel, is there a generic high level > > wrapper around csv.reader that does all this ? > > > > Thanks, > > George > > > > There is a csv in the standard library. Though many of us don't mind > answering questions like this, you can get a lot of answers quicker by > (1) looking to see what's in the standard library and (2) using google. > > http://docs.python.org/lib/module-csv.html > > James The OP mentioned "csv.reader". This indicates to me that he *has* read the csv docs (have you?), and is *already* using the csv module. The tasks he says he does often are *not* covered by the standard library. He appears to be asking if there is a higher-level wrapper around the standard library. Please consider reading his question carefully. -- http://mail.python.org/mailman/listinfo/python-list
Re: High level csv reader
James Stroud wrote: >> Before I go on and reinvent the wheel, is there a generic high level >> wrapper around csv.reader that does all this ? > > There is a csv in the standard library. I'm not sure the "csv" module qualifies as a high-level wrapper around itself, though. -- http://mail.python.org/mailman/listinfo/python-list
Re: High level csv reader
George Sakkis wrote: > It occured to me that most times I read a csv file, I'm often doing > from scratch things like assigning labels to columns, mapping fields to > the appropriate type, ignoring some fields, changing their order, etc. > Before I go on and reinvent the wheel, is there a generic high level > wrapper around csv.reader that does all this ? > > Thanks, > George > There is a csv in the standard library. Though many of us don't mind answering questions like this, you can get a lot of answers quicker by (1) looking to see what's in the standard library and (2) using google. http://docs.python.org/lib/module-csv.html James -- http://mail.python.org/mailman/listinfo/python-list
High level csv reader
It occured to me that most times I read a csv file, I'm often doing from scratch things like assigning labels to columns, mapping fields to the appropriate type, ignoring some fields, changing their order, etc. Before I go on and reinvent the wheel, is there a generic high level wrapper around csv.reader that does all this ? Thanks, George -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
Stephan wrote:
> DictReader field names on the fly. Here is a rudimentary example of my
> working code and the data it can parse.
>
> -
> John|Smith
> Beef|Potatos|Dinner Roll|Ice Cream
> Susan|Jones
> Chicken|Peas|Biscuits|Cake
> Roger|Miller
> Pork|Salad|Muffin|Cookies
> -
That sample data would have been valuable information in your original post.
Here's what becomes of your code if you apply the "zip trick" from my first
post (yes, I am sometimes stubborn):
import itertools
import csv
HeaderFields = ["First Name", "Last Name"]
DetailFields = ["Entree", "Side Dish", "Starch", "Desert"]
instream = open("testdata.txt")
heads = csv.DictReader(instream, HeaderFields, delimiter="|")
details = csv.DictReader(instream, DetailFields, delimiter="|")
for header, detail in itertools.izip(heads, details):
print "Header (%d fields): %s" % (len(header), header)
print "Detail (%d fields): %s" % (len(detail), detail)
Peter
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
Andrew McLean wrote:
> You are welcome. One point. I think there have been at least two
> different interpretations of precisely what you task is.
>
> I had assumed that all the different "header" lines contained data for
> the same fields in the same order, and similarly that all the "detail"
> lines contained data for the same fields in the same order.
Indeed, you are correct. Peter's version is interesting in its own
right, but not precisely what I had in mind. However, from his example
I saw what I was missing: I didn't realize that you could reassign the
DictReader field names on the fly. Here is a rudimentary example of my
working code and the data it can parse.
-
John|Smith
Beef|Potatos|Dinner Roll|Ice Cream
Susan|Jones
Chicken|Peas|Biscuits|Cake
Roger|Miller
Pork|Salad|Muffin|Cookies
-
import csv
HeaderFields = ["First Name", "Last Name"]
DetailFields = ["Entree", "Side Dish", "Starch", "Desert"]
reader = csv.DictReader(open("testdata.txt"), [], delimiter="|")
while True:
try:
# Read next "header" line (if there isn't one then exit the
loop)
reader.fieldnames = HeaderFields
header = reader.next()
# Read the next "detail" line
reader.fieldnames = DetailFields
detail = reader.next()
# Print the parsed data
print '-' * 40
print "Header (%d fields): %s" % (len(header), header)
print "Detail (%d fields): %s" % (len(detail), detail)
except StopIteration: break
Regards,
-Stephan
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
In article <[EMAIL PROTECTED]>, Stephan <[EMAIL PROTECTED]> writes >Thank you all for these interesting examples and methods! You are welcome. One point. I think there have been at least two different interpretations of precisely what you task is. I had assumed that all the different "header" lines contained data for the same fields in the same order, and similarly that all the "detail" lines contained data for the same fields in the same order. However, I think Peter has answered on the basis that you have records consisting of pairs of lines, the first line being a header containing field names specific to that record with the second line containing the corresponding data. It would help of you let us know which (if any) was correct. -- Andrew McLean -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
Stephan wrote: > Thank you all for these interesting examples and methods! You're welcome. > Supposing I want to use DictReader to bring in the CSV lines and tie > them to field names, (again, with alternating lines having different > fields), should I use two two DictReaders as in Christopher's example > or is there a better way? For a clean design you would need not just two DictReader instances, but one DictReader for every two lines. However, with the current DictReader implementation, the following works, too: import csv import sys reader = csv.DictReader(sys.stdin) for record in reader: print record reader.fieldnames = None Peter -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
Thank you all for these interesting examples and methods! Supposing I want to use DictReader to bring in the CSV lines and tie them to field names, (again, with alternating lines having different fields), should I use two two DictReaders as in Christopher's example or is there a better way? -- Stephan -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
In article <[EMAIL PROTECTED]>, Stephan <[EMAIL PROTECTED]> writes >I'm fairly new to python and am working on parsing some delimited text >files. I noticed that there's a nice CSV reading/writing module >included in the libraries. > >My data files however, are odd in that they are composed of lines with >alternating formats. (Essentially the rows are a header record and a >corresponding detail record on the next line. Each line type has a >different number of fields.) > >Can the CSV module be coerced to read two line formats at once or am I >better off using read and split? > >Thanks for your insight, >Stephan > The csv module should be suitable. The reader just takes each line, parses it, then returns a list of strings. It doesn't matter if different lines have different numbers of fields. To get an idea of what I mean, try something like the following (untested): import csv reader = csv.reader(open(filename)) while True: # Read next "header" line, if there isn't one then exit the loop header = reader.next() if not header: break # Assume that there is a "detail" line if the preceding # "header" line exists detail = reader.next() # Print the parsed data print '-' * 40 print "Header (%d fields): %s" % (len(header), header) print "Detail (%d fields): %s" % (len(detail), detail) You could wrap this up into a class which returns (header, detail) pairs and does better error handling, but the above code should illustrate the basics. -- Andrew McLean -- http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
Stephan wrote:
> Can the CSV module be coerced to read two line formats at once or am I
> better off using read and split?
Yes, it can:
import csv
import sys
reader = csv.reader(sys.stdin)
while True:
try:
names = reader.next()
values = reader.next()
except StopIteration:
break
print dict(zip(names, values))
Python offers an elegant way to do the same using the zip() or
itertools.izip() function:
import csv
import sys
from itertools import izip
reader = csv.reader(sys.stdin)
for names, values in izip(reader, reader):
print dict(izip(names, values))
Now let's add some minimal error checking, and we are done:
import csv
import sys
from itertools import izip, chain
def check_orphan():
raise Exception("Unexpected end of input")
yield None
reader = csv.reader(sys.stdin)
for names, values in izip(reader, chain(reader, check_orphan())):
if len(names) != len(values):
if len(names) > len(values):
raise Exception("More names than values")
else:
raise Exception("More values than names")
print dict(izip(names, values))
Peter
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python's CSV reader
Stephan wrote: > Can the CSV module be coerced to read two line formats at once or am I > better off using read and split? Well, readlines/split really isn't bad. So long as the file fits comfortably in memory: fi = open(file) lines = fi.readlines() evens = iter(lines[0::2]) odds = iter(lines[1::2]) csv1 = csv.reader(evens) csv2 = csv.reader(odds) The trick is that the "csvfile" in the CSV object doesn't have to be a real file, it just has to be an iterator that returns strings. If the file's too big to fit in memory, you could piece together a pair of iterators that execute read() on the file appropriately. -- http://mail.python.org/mailman/listinfo/python-list
Python's CSV reader
I'm fairly new to python and am working on parsing some delimited text files. I noticed that there's a nice CSV reading/writing module included in the libraries. My data files however, are odd in that they are composed of lines with alternating formats. (Essentially the rows are a header record and a corresponding detail record on the next line. Each line type has a different number of fields.) Can the CSV module be coerced to read two line formats at once or am I better off using read and split? Thanks for your insight, Stephan -- http://mail.python.org/mailman/listinfo/python-list
