Re: How can I debug silent failure - print no output

[cs]

On 28May2016 03:32, Sayth Renshaw  wrote:

On Saturday, 28 May 2016 19:44:53 UTC+10, Sayth Renshaw  wrote:

On Saturday, 28 May 2016 18:02:06 UTC+10, Sayth Renshaw  wrote:
> So how do i get argparse to read the file arguments correctly?
>
> Looking at the namespace it all gets pushed into path and extension remains 
empty.
>
> [sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
> Namespace(extension='', path=['data/', '*.xml'])
> This is the section I am running
>
> parser = argparse.ArgumentParser()
> parser.add_argument("path", nargs="+")
> parser.add_argument('-e', '--extension', default='',
> help='File extension to filter by.')
>
> args = parser.parse_args()
> name_pattern = "*" + args.extension
> print(args)

Ah if only i used argparse properly

python racemeeting.py data/ -e *.xml


There are a couple of things here.

First is that a normal UNIX command puts all the options first, so you should 
be desiging your command line to run like this:


 python racemeeting.py -e *.xml data

or perhaps:

 python racemeeting.py -d data -e *.xml

It is traditional to stop parsing options such as -e when you reach the first 
non-option, because that lets one put whatever is necessary safely _after_ the 
options without fear that one of the arguments will resemble an option. For 
example, suppoing you have a file with the name "-e" and said:


 somecommand -f foo dir *

intending to use all the local filenames after "dir". In your current scheme 
(accepting options after "dir") a "-e" appearing later would be misinterpreted.


The second is to be aware that the shell expands globs _before_ invoking the 
command. This is extremely useful because it means that (a) commands usually 
don't need to do their own glob expansion and (b) all commands end up using the 
same glob syntax because it is common to the shell, not a command-specific 
special syntax. Which makes everything easier to use.


The upshot of that is that you should _either_ be quoting "*.xml" on your 
command line to prevent expansion, _or_ you should not be bothering with he 
asterisk, instead passing the argument ".xml" or even just "xml".


As an experiment, make two dummy XML files in your current directory (where 
your script is):


 >>dummy1.xml
 >>dummy2.xml

and run your script passing *.xml as you currently do, and see what your print 
statements say. This should make the effect obvious.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Sayth Renshaw]

On Saturday, 28 May 2016 19:44:53 UTC+10, Sayth Renshaw  wrote:
> On Saturday, 28 May 2016 18:02:06 UTC+10, Sayth Renshaw  wrote:
> > So how do i get argparse to read the file arguments correctly?
> > 
> > Looking at the namespace it all gets pushed into path and extension remains 
> > empty.
> > 
> > [sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
> > Namespace(extension='', path=['data/', '*.xml'])
> > 
> > This is the section I am running
> > 
> > parser = argparse.ArgumentParser()
> > parser.add_argument("path", nargs="+")
> > parser.add_argument('-e', '--extension', default='',
> > help='File extension to filter by.')
> > 
> > args = parser.parse_args()
> > name_pattern = "*" + args.extension
> > print(args)
> > 
> > Sayth
> 
> Ah if only i used argparse properly
> 
> python racemeeting.py data/ -e *.xml
> 
> Sayth

Which means I can rewrite it like this.

parser = argparse.ArgumentParser()
parser.add_argument("path", type=str, nargs="+")
parser.add_argument('-e', '--extension', default='',
help='File extension to filter by.')

args = parser.parse_args()
name_pattern = "*" + args.extension
my_dir = args.path[0]

for dir_path, subdir_list, file_list in os.walk(my_dir):
for name_pattern in file_list:
full_path = os.path.join(dir_path, name_pattern)

Cheers

Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Sayth Renshaw]

On Saturday, 28 May 2016 18:02:06 UTC+10, Sayth Renshaw  wrote:
> So how do i get argparse to read the file arguments correctly?
> 
> Looking at the namespace it all gets pushed into path and extension remains 
> empty.
> 
> [sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
> Namespace(extension='', path=['data/', '*.xml'])
> 
> This is the section I am running
> 
> parser = argparse.ArgumentParser()
> parser.add_argument("path", nargs="+")
> parser.add_argument('-e', '--extension', default='',
> help='File extension to filter by.')
> 
> args = parser.parse_args()
> name_pattern = "*" + args.extension
> print(args)
> 
> Sayth

Ah if only i used argparse properly

python racemeeting.py data/ -e *.xml

Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Sayth Renshaw]

So how do i get argparse to read the file arguments correctly?

Looking at the namespace it all gets pushed into path and extension remains 
empty.

[sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
Namespace(extension='', path=['data/', '*.xml'])

This is the section I am running

parser = argparse.ArgumentParser()
parser.add_argument("path", nargs="+")
parser.add_argument('-e', '--extension', default='',
help='File extension to filter by.')

args = parser.parse_args()
name_pattern = "*" + args.extension
print(args)

Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Sayth Renshaw]

On Saturday, 28 May 2016 16:35:35 UTC+10, Sayth Renshaw  wrote:
> > >
> > >Ok after printing a few things i have found an error.
> > >
> > >def GetArgs():
> > >'''parse XML from command line'''
> > >parser = argparse.ArgumentParser()
> > >
> > >parser.add_argument("path", nargs="+")
> > >parser.add_argument('-e', '--extension', default='',
> > >help='File extension to filter by.')
> > >args = parser.parse_args()
> > >
> > >files = set()
> > >name_pattern = "*" + args.extension
> > >for path in args.path:
> > >files.update(glob.glob(os.path.join(path, name_pattern)))
> > >
> > >print(files)
> > >return files
> > >
> > >a = GetArgs()
> > >print(a)
> > >
> > >so printing the files or the call to the function returns set() not the 
> > >actual files.
> > 
> > Since you're constructing a set of filenames, this means it is probably 
> > returning the right kind of thing, but it is empty. That points to the glob 
> > not 
> > doing what you want or the for-loop not doing anything.
> > 
> > >[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
> > >set()
> > >set()
> > >set()
> > 
> > So...  Add more prints!
> > 
> > Specificly, print(args) right after it is set, and put a print() _inside_ 
> > the 
> > loop before the call to files.update, probably printing "path", eg 
> > print("path 
> > =", path).
> > 
> > Then see what you learn.
> > 
> > Cheers,
> > Cameron Simpson 
> 
> Having done extra prints 
> 
> name_pattern = "*" + args.extension
> for path in args.path:
> print(args.path)
> print(path)
> files.update(glob.glob(os.path.join(path, name_pattern)))
> 
> it is getting the path and file however I think it is keeping the directory 
> so i am not getting files.
> 
> [sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
> ['data/20160528RAND0.xml']
> data/20160528RAND0.xml
> set()
> set()
> ['data/20160528RAND0.xml']
> data/20160528RAND0.xml
> 
> Sayth
Actually think I have found the cause and its really small but on way its 
called.
I was calling
python3 racemeeting.py data/*.xml
which gives the directory and file as the path 
['data/20160528RAND0.xml']

But with arguments separated by a space I actually receive what i thought I 
would get a path and extension such as

sayth@localhost pyXML]$ python3 racemeeting.py data/ *.xml
Namespace(extension='', path=['data/', '*.xml'])
Traceback (most recent call last):
  File "racemeeting.py", line 35, in 

Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Sayth Renshaw]

> >
> >Ok after printing a few things i have found an error.
> >
> >def GetArgs():
> >'''parse XML from command line'''
> >parser = argparse.ArgumentParser()
> >
> >parser.add_argument("path", nargs="+")
> >parser.add_argument('-e', '--extension', default='',
> >help='File extension to filter by.')
> >args = parser.parse_args()
> >
> >files = set()
> >name_pattern = "*" + args.extension
> >for path in args.path:
> >files.update(glob.glob(os.path.join(path, name_pattern)))
> >
> >print(files)
> >return files
> >
> >a = GetArgs()
> >print(a)
> >
> >so printing the files or the call to the function returns set() not the 
> >actual files.
> 
> Since you're constructing a set of filenames, this means it is probably 
> returning the right kind of thing, but it is empty. That points to the glob 
> not 
> doing what you want or the for-loop not doing anything.
> 
> >[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
> >set()
> >set()
> >set()
> 
> So...  Add more prints!
> 
> Specificly, print(args) right after it is set, and put a print() _inside_ the 
> loop before the call to files.update, probably printing "path", eg 
> print("path 
> =", path).
> 
> Then see what you learn.
> 
> Cheers,
> Cameron Simpson 

Having done extra prints 

name_pattern = "*" + args.extension
for path in args.path:
print(args.path)
print(path)
files.update(glob.glob(os.path.join(path, name_pattern)))

it is getting the path and file however I think it is keeping the directory so 
i am not getting files.

[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
['data/20160528RAND0.xml']
data/20160528RAND0.xml
set()
set()
['data/20160528RAND0.xml']
data/20160528RAND0.xml

Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[cs]

On 27May2016 21:02, Sayth Renshaw  wrote:

On Saturday, 28 May 2016 13:06:59 UTC+10, Michael Torrie  wrote:

Add more print() calls. Offhand I'd say that pq(filename=filename) is
returning an empty list so that for loop is not doing anything.  Hence
your debugging print() calls never happen.

Add sanity print()'s earlier in your program, and make sure everything
you are iterating over is what you expect.


Ok after printing a few things i have found an error.

def GetArgs():
   '''parse XML from command line'''
   parser = argparse.ArgumentParser()

   parser.add_argument("path", nargs="+")
   parser.add_argument('-e', '--extension', default='',
   help='File extension to filter by.')
   args = parser.parse_args()

   files = set()
   name_pattern = "*" + args.extension
   for path in args.path:
   files.update(glob.glob(os.path.join(path, name_pattern)))

   print(files)
   return files

a = GetArgs()
print(a)

so printing the files or the call to the function returns set() not the actual 
files.


Since you're constructing a set of filenames, this means it is probably 
returning the right kind of thing, but it is empty. That points to the glob not 
doing what you want or the for-loop not doing anything.



[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
set()
set()
set()


So...  Add more prints!

Specificly, print(args) right after it is set, and put a print() _inside_ the 
loop before the call to files.update, probably printing "path", eg print("path 
=", path).


Then see what you learn.

Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Sayth Renshaw]

On Saturday, 28 May 2016 13:06:59 UTC+10, Michael Torrie  wrote:

> Add more print() calls. Offhand I'd say that pq(filename=filename) is
> returning an empty list so that for loop is not doing anything.  Hence
> your debugging print() calls never happen.
> 
> Add sanity print()'s earlier in your program, and make sure everything
> you are iterating over is what you expect.

Ok after printing a few things i have found an error.

def GetArgs():
'''parse XML from command line'''
parser = argparse.ArgumentParser()

parser.add_argument("path", nargs="+")
parser.add_argument('-e', '--extension', default='',
help='File extension to filter by.')
args = parser.parse_args()

files = set()
name_pattern = "*" + args.extension
for path in args.path:
files.update(glob.glob(os.path.join(path, name_pattern)))

print(files)
return files

a = GetArgs()
print(a)

so printing the files or the call to the function returns set() not the actual 
files.

[sayth@localhost pyXML]$ python3 racemeeting.py data/*.xml
set()
set()
set()


Sayth
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Jason Friedman]

>
> def GetArgs():
> '''parse XML from command line'''
> parser = argparse.ArgumentParser()
>
> parser.add_argument("path", nargs="+")
> parser.add_argument('-e', '--extension', default='',
> help='File extension to filter by.')
> args = parser.parse_args()
>
> files = set()
> name_pattern = "*" + args.extension
> for path in args.path:
> files.update(glob.glob(os.path.join(path, name_pattern)))
> return files
>
>
>
> # Now walk the tree and insert data.
> for filename in sorted(GetArgs()):
> for meeting in pq(filename=filename):
> print(filename)
> print(meeting)
> meetdata = [meeting.get(attr) for attr in meetattrs]
> cur.execute("insert into meetings valueme in GetArgs():s (" +
> ",".join(["%s"]*len(meetattrs)) + ")", meetdata)
> for race in meeting.findall("race"):
> race.set("meeting_id", meeting.get("id"))
> racedata = [race.get(attr) for attr in raceattrs]
> cur.execute("insert into races values (" +
> ",".join(["%s"]*len(raceattrs)) + ")",
> racedata)
> for horse in race.findall("nomination"):
> horse.set("race_id", race.get("id"))
> horsedata = [horse.get(attr) for attr in horseattrs]
> cur.execute("insert into horses values (" +
> ",".join(["%s"]*len(horseattrs)) + ")",
> horsedata)
>
> If your actual indentation matches what I see in your post, is your

for filename in sorted(GetArgs())

line within the definition of GetArgs?

If yes, it will not be executed.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: How can I debug silent failure - print no output

[Michael Torrie]

On 05/27/2016 08:41 PM, Sayth Renshaw wrote:
> This is my terminal and directory structure.

Add more print() calls. Offhand I'd say that pq(filename=filename) is
returning an empty list so that for loop is not doing anything.  Hence
your debugging print() calls never happen.

Add sanity print()'s earlier in your program, and make sure everything
you are iterating over is what you expect.


-- 
https://mail.python.org/mailman/listinfo/python-list