Re: Hypothesis 1.0: A production quality property-based testing library for Python

[David MacIver]

On 31 March 2015 at 02:46, Terry Reedy  wrote:

> On 3/30/2015 4:46 PM, David MacIver wrote:
>
>> On 30 March 2015 at 22:37, Terry Reedy > > wrote:
>>
>
>  https://www.python.org/dev/__peps/pep-0484/
>> 
>> proposes the addition of a 'typing' module for defining types beyond
>> the builtins and ABCs, such as List(int), Union(tuple, list).  If
>> and when it is accepted and added (maybe 3.5, maybe later), you
>> should consider having Hypothesis accept the notations that it can
>> work with.
>>
>> Yes, definitely. Having something like that as standard would be great
>> for Hypothesis and I intend to support it once it becomes available.
>>
>
> I just posted "PEP 484: Generating test inputs from type hints" to
> python-ideas list.


Thanks. I'll pop over there and join in the discussion.


>
>
>  (It will force me to finally figure out how to do staged APIs, with some
>> versions of the API only supported on some versions of python, but I
>> need to do that anyway. This might involve just shipping a compatibility
>> layer for previous versions of Python in with Hypothesis)
>>
>
> I believe Guido intends that typing.py should be available on PyPI for use
> with current versions.
>
>
Ah, that would be useful. Does that include Python 2.7? The examples
suggest yes.

(I really hate having to ask that. I don't want to support Python 2.7 at
all, but...)


>  Full documentation is available at
>> http://hypothesis.readthedocs.__org/en/latest/
>> you can
>> skip straight to the quick start guide:
>> http://hypothesis.readthedocs.__org/en/latest/quickstart.html
>>
>
> --
> Terry Jan Reedy
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Hypothesis 1.0: A production quality property-based testing library for Python

[David MacIver]

On 31 March 2015 at 02:51, Paul Rubin  wrote:

> David MacIver  writes:
> > Hypothesis is based on Quickcheck
> > (https://wiki.haskell.org/Introduction_to_QuickCheck2)
>
> This is great.  Have you looked at the Erlang version of Quickcheck?  It
> may have aspects more directly applicable to Python, since Erlang is
> dynamically typed like Python is.
>

I actually haven't looked at it much. I've read some of the associated
papers, but due to a mix of my not really knowing erlang well and it being
proprietary it's not as useful for idea mining as I'd like it to be. I have
some friends who are more extensive users of it who I've talked to a bit
about it for feature comparisons though.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A simple single line, triple-quoted comment is giving syntax error. Why?

[Ian Kelly]

On Sun, Mar 29, 2015 at 4:41 AM, Thomas 'PointedEars' Lahn
 wrote:
> Ian Kelly wrote:
>
>> […] Thomas 'PointedEars' Lahn […] wrote:
>>> Ian Kelly wrote:
>> Why should the burden of proof be on me in the first place?
>
> Because *you* made the claim that “STRING+” could be part of an AST in this
> way.

Okay, I didn't understand before that this was the part that you were
objecting to. I had written "There is only one expr node, and it
contains both STRING tokens", to which you replied "Prove it." I did
not intend to imply anything at all by the inclusion of STRING+ in the
tree. That pseudo-tree was hastily sketched out, and as both you and
Gregory have pointed out, STRING+ is not a symbol of the grammar and
should not have been included as a node. None of this has any bearing
on what I was trying to demonstrate with that tree.

>> Within a grammar, the question of "is an X a Y" is nonsensical in
>> isolation. It can only be answered in relation to a parse tree.
>> Consider the simple grammar:
>>
>> S -> A | B
>> A -> x
>> B -> x
>>
>> Is x an A? It depends.
>
> No, by the definition 2 below, that we all accepted implicitly up to this
> point, x is *definitely* an A.

What gives you the impression that I ever accepted it?

>> If the tree that generates the x produces it from an A node, then yes.
>> Otherwise, no.
>
> Fallacy.
>
> First, two definitions, so that we are clear what we are talking about:
>
> (1) Let a *production chain* be the repeated application of the production
> rules of a formal grammar such that
>
>   C ⇒ D ⇒ x
>
> is a production chain if there are production rules
>
>   C → D
>   D → x.
>
> [Note the difference between “⇒” and “→”.]

Sure.

> (2) Let the statement “x is an A” be true if x can be produced in a
> production chain starting with or including the non-terminal A
> left-hand side –
>
>   x ∈ A ↔ ∃A (… ⇒ A ⇒ … ⇒ x).

Sorry, but this definition just seems entirely arbitrary to me.
Mathematically, it looks nonsensical; A is a symbol, not a set. This
question of whether "x is an A" is informal and not a topic of formal
language theory so far as I'm aware. Can you cite some source for it?

> Now, according to these definitions, in the offered grammar x is *both* an A
> and a B.  Because what matters is _not_ the practical result of production
> chains (the actual parse tree), but the certainty of the theoretical
> possibility of it.

This strikes me as being a lot like arguing, "some kites are toys, and
some kites are birds; therefore, all kites are both toys and birds."

>> So when I write that the "foo" in "foo" "bar" is not an expression, I
>> am only speaking in relation to the parse tree that generates "foo"
>> "bar".
>
> But it has been indicated by others that the parse tree that you presented
> is wrong, based on a misconception about the syntax of the formal grammar,
> and you have not yet substantiated your claim that it is correct.

As noted above, the inaccuracy that Gregory pointed out has no bearing
on my argument.

You're really going to make me spell it out, aren't you? Fine, here you go.

single_input -> simple_stmt -> expr_stmt -> testlist_star_expr -> test
-> or_test -> and_test -> not_test -> comparison -> expr -> xor_expr
-> and_expr -> shift_expr -> arith_expr -> term -> factor -> power ->
atom -> STRING STRING

Note: the derivation contains exactly one expr node, which indirectly
produces both STRINGs. Neither STRING in this derivation is
individually produced from the expr.

>> I don't know what you mean by a "backdoor".
>
> Appending a statement that is contradictory to what was stated just before,
> or at least ambiguous, so that the possibility arises for one to say “I did
> not mean that” when that/a contradiction to the former statement is pointed
> out later.

The purpose was not to make the possibility arise to say that. The
purpose was to point out the discontinuity *immediately* so that it
would be already addressed and not *need* to be pointed out later
(although we can see how well that turned out).

I have to say that I find it highly surprising that you find this
construction objectionable. Would you still consider it
"contradictory" if I had phrased it equivalently as a logical "or"
rather than using the word "except"?

>> The purpose of that parenthetical was to explicitly acknowledge that the
>> single STRING could be viewed as an expression when taken out of context,
>
> Yes, your fallacy is mainly based on ignoring the context.  Context is
> important; you cannot just ignore it and still make correct arguments.

Funny, I thought that was my line. This whole argument came about
because you seem to be stubbornly insistent on ignoring the context
that I set for my statement, calling it "contradictory" instead.

>> and to disclaim that I was excluding that case from the preceding
>> statement. I still stand by that statement; The "foo" in "foo" "bar" is
>> not an expression in the same sense that the "42" is

Re: Using csv DictWriter - add a extra field

[Peter Otten]

Victor Hooi wrote:

> Hi,
> 
> I have a dict named "connections", with items like the following:
> 
> In [18]: connections
> Out[18]:
> {'3424234': {'end_timestamp': datetime.datetime(2015, 3, 25, 5, 31, 30,
> {406000, tzinfo=datetime.timezone(datetime.timedelta(-1, 61200))),
>   'ip_address': '10.168.8.36:52440',
>   'open_timestamp': datetime.datetime(2015, 3, 25, 5, 31, 0, 383000,
>   tzinfo=datetime.timezone(datetime.timedelta(-1, 61200))), 'time_open':
>   datetime.timedelta(0, 30, 23000)}}
> 
> In this case, the key is a connection id (e.g. "3424234"), and the value
> is a another dict, which contains things like 'end_timestamp',
> 'ip_address" etc.
> 
> I'm writing the output of "connections" to a CSV file using DictWriter:
> 
> fieldnames = ['connection_id', 'ip_address', 'open_timestamp',
> 'end_timestamp', 'time_open'] with open('output.csv', 'w') as csvfile:
> writer = DictWriter(csvfile, fieldnames)
> writer.writeheader()
> for connection, values in sorted(connections.items()):
> if 'time_open' in values:
> writer.writerow(values, {'connection_id': connection})
> else:
> pass
> # DO SOME STUFF
> 
> The only problem is, I'd also like output the connection_id field as part
> of each CSV record.
> 
> However, connection_id in this case is the key for the parent dict.
> 
> Is there a clean way to add a extra field to DictWriter writerows, or it
> is the contents of the dict and that's it?

The latter. The obvious solution is to add the extra field to the dict:

values['connection_id'] = connection
writer.writerow(values)

Or you use a collections.ChainMap (Python 3 only I think, but 
 should work as well here)

writer.writerow(
collections.ChainMap(values, {"connection_id": connection}))


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using csv DictWriter - add a extra field

[Victor Hooi]

Hi,

Aha, yeah, I can add the connection_id as another field in the inner dict - the 
only drawback is that the data is duplicated twice. However, I suppose even if 
it's not elegant, it does work.

However, that ChainMap does look interesting =). And yes, I am actually using 
Python 3.x (mainly because of http://bugs.python.org/issue6641).

So if I understand correctly, I can just use ChainMap to join any arbitrary 
number of dicts together - it seems like the right solution here.

Are there any drawbacks to using ChainMap here? (Aside from needing Python 3.x).

Cheers,
Victor
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using csv DictWriter - add a extra field

[Peter Otten]

Victor Hooi wrote:

> Aha, yeah, I can add the connection_id as another field in the inner dict
> - the only drawback is that the data is duplicated twice. However, I
> suppose even if it's not elegant, it does work.

The elegance lies in its simplicity, so it's still my personal favourite.

> However, that ChainMap does look interesting =). And yes, I am actually
> using Python 3.x (mainly because of http://bugs.python.org/issue6641).
> 
> So if I understand correctly, I can just use ChainMap to join any
> arbitrary number of dicts together - it seems like the right solution
> here.
> 
> Are there any drawbacks to using ChainMap here? (Aside from needing Python
> 3.x).

As it introduces another level of indirection it costs a little time, but 
compared to the actual I/O that should be negligible. 

So no, if there is a drawback I don't see it.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Strategy/ Advice for How to Best Attack this Problem?

[Saran A]

On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote:
> On 03/30/2015 12:45 PM, Saran A wrote:
> > On Sunday, March 29, 2015 at 10:04:45 PM UTC-4, Chris Angelico wrote:
> >> On Mon, Mar 30, 2015 at 12:08 PM, Paul Rubin  
> >> wrote:
> >>> Saran Ahluwalia  writes:
>  cross-platform...
>  * Monitors a folder for files that are dropped throughout the day
> >>>
> >>> I don't see a cross-platform way to do that other than by waking up and
> >>> scanning the folder every so often (once a minute, say).  The Linux way
> >>> is with inotify and there's a Python module for it (search terms: python
> >>> inotify).  There might be comparable but non-identical interfaces for
> >>> other platforms.
> >>
> >> All too often, "cross-platform" means probing for one option, then
> >> another, then another, and using whichever one you can. On Windows,
> >> there's FindFirstChangeNotification and ReadDirectoryChanges, which
> >> Tim Golden wrote about, and which I coded up into a teleporter for
> >> getting files out of a VM automatically:
> >>
> >> http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
> >> https://github.com/Rosuav/shed/blob/master/senddir.py
> >>
> >> ChrisA
> >
> > @Dave, Chris, Paul and Dennis: Thank you for resources and the notes 
> > regarding what I should keep in mind. I have an initial commit: 
> > https://github.com/ahlusar1989/IntroToPython/blob/master/Project1WG_with_assumptions_and_comments.py
> >
> > I welcome your thoughts on this
> >
> 
> It's missing a number of your requirements.  But it's a start.
> 
> If it were my file, I'd have a TODO comment at the bottom stating known 
> changes that are needed.  In it, I'd mention:
> 
> 1) your present code is assuming all filenames come directly from the 
> commandline.  No searching of a directory.
> 
> 2) your present code does not move any files to success or failure 
> directories
> 
> 3) your present code doesn't calculate or write to a text file any 
> statistics.
> 
> 4) your present code runs once through the names, and terminates.  It 
> doesn't "monitor" anything.
> 
> 5) your present code doesn't check for zero-length files
> 
> I'd also wonder why you bother checking whether the 
> os.path.getsize(file) function returns the same value as the os.SEEK_END 
> and ftell() code does.  Is it that you don't trust the library?  Or that 
> you have to run on Windows, where the line-ending logic can change the 
> apparent file size?
> 
> I notice you're not specifying a file mode on the open.  So in Python 3, 
> your sizes are going to be specified in unicode characters after 
> decoding.  Is that what the spec says?  It's probably safer to 
> explicitly specify the mode (and the file encoding if you're in text).
> 
> I see you call strip() before comparing the length.  Could there ever be 
> leading or trailing whitespace that's significant?  Is that the actual 
> specification of line size?
> 
> -- 
> DaveA

@ Dave A

On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote:
> On 03/30/2015 12:45 PM, Saran A wrote:
> > On Sunday, March 29, 2015 at 10:04:45 PM UTC-4, Chris Angelico wrote:
> >> On Mon, Mar 30, 2015 at 12:08 PM, Paul Rubin  
> >> wrote:
> >>> Saran Ahluwalia  writes:
>  cross-platform...
>  * Monitors a folder for files that are dropped throughout the day
> >>>
> >>> I don't see a cross-platform way to do that other than by waking up and
> >>> scanning the folder every so often (once a minute, say).  The Linux way
> >>> is with inotify and there's a Python module for it (search terms: python
> >>> inotify).  There might be comparable but non-identical interfaces for
> >>> other platforms.
> >>
> >> All too often, "cross-platform" means probing for one option, then
> >> another, then another, and using whichever one you can. On Windows,
> >> there's FindFirstChangeNotification and ReadDirectoryChanges, which
> >> Tim Golden wrote about, and which I coded up into a teleporter for
> >> getting files out of a VM automatically:
> >>
> >> http://timgolden.me.uk/python/win32_how_do_i/watch_directory_for_changes.html
> >> https://github.com/Rosuav/shed/blob/master/senddir.py
> >>
> >> ChrisA
> >
> > @Dave, Chris, Paul and Dennis: Thank you for resources and the notes 
> > regarding what I should keep in mind. I have an initial commit: 
> > https://github.com/ahlusar1989/IntroToPython/blob/master/Project1WG_with_assumptions_and_comments.py
> >
> > I welcome your thoughts on this
> >
> 
> It's missing a number of your requirements.  But it's a start.
> 
> If it were my file, I'd have a TODO comment at the bottom stating known 
> changes that are needed.  In it, I'd mention:
> 
> 1) your present code is assuming all filenames come directly from the 
> commandline.  No searching of a directory.
> 
> 2) your present code does not move any files to success or failure 
> directories
> 
> 3) your present code doesn't calculate or write to a text file any 
> statistics.
> 
> 4) your present c

Project, how to debug

[Robert Clove]

Hi All,

I am facing a problem.
I have been given a project written in python and asked to debug it.
I have not been given the flow they said understand and debug.

Can someone suggest me how to debug it in Wings IDE.
Project have approx 10 files.

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A simple single line, triple-quoted comment is giving syntax error. Why?

[Jean-Michel Pichavant]

- Original Message -
> From: "Terry Reedy" 
> To: python-list@python.org
> Sent: Wednesday, 18 March, 2015 10:47:40 PM
> Subject: Re: A simple single line, triple-quoted comment is giving syntax 
> error. Why?
> 
> On 3/18/2015 3:53 PM, Thomas 'PointedEars' Lahn wrote:
> 
> > I must strongly object and recommend against getting accustomed to
> > the
> > suggested use of multi-line string literals.
> 
> I agree.

I remember loosing a lot of time finding a bug that can be summarized like this 
(immagine a much larger list where the temptation of commenting a block is 
high):

a =  [  '1',
'''shorten the list to speedup tests
'2',
'3',
'''
'4',
]


print a
>>> ['1', "shorten the list to speedup tests\n\t\t'2',\n\t\t'3',\n4"]

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Project, how to debug

[Peter Otten]

Robert Clove wrote:

> Hi All,
> 
> I am facing a problem.
> I have been given a project written in python and asked to debug it.
> I have not been given the flow they said understand and debug.
> 
> Can someone suggest me how to debug it in Wings IDE.

Are those specific bugs that you are supposed to fix or are you to both find 
and fix bugs?

> Project have approx 10 files.

Generally speaking speaking this is not so much a matter of tools; if there 
is any documentation you can trust reading that is usually the best start;
reading the code is important, too.

Then you can consider your next steps depending on the "messiness" of the 
code base and the degree of coverage with unit tests.


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Project, how to debug

[Chris Angelico]

On Tue, Mar 31, 2015 at 10:23 PM, Robert Clove  wrote:
>
> I am facing a problem.
> I have been given a project written in python and asked to debug it.
> I have not been given the flow they said understand and debug.
>
> Can someone suggest me how to debug it in Wings IDE.
> Project have approx 10 files.

Hi, sounds like this is a homework assignment.

I don't know the IDE, so I'll just give some basic debugging advice.
Most of this isn't even specific to Python - it's simply how you go
about figuring out where a problem is in a piece of code.

The first step is *always* to find out what the code ought to be
doing. Should it be firing up a server of some sort? Running,
performing some task, and terminating? Producing output on the console
(stdout/stderr)? Bringing up a graphical window? If you don't already
know, you might be able to find out from hints in the code itself -
comments at the top, function names, string literals, that kind of
thing. Otherwise, ask someone. You can't assess whether the code's
doing the right thing or not without knowing what "the right thing"
means.

(Parenthesis: Some people think that a program crash is "the wrong
thing" and the program not crashing is "the right thing". This is
often enough true that it's very tempting, but if you believe your job
is to stop the program crashing, then all you have to do is remove
code until the failing part is gone. The result might not achieve
anything, but hey! It's not crashing, right? You need to start by
knowing what actual goal to not-remove. Also, sometimes it's
absolutely right for a program to throw an exception; a test script
for PEP 479 is simply probing to see which of two exceptions it gets,
and "success" is defined by getting a RuntimeError. End of
parenthesis.)

Once you know what the code ought to do, check to see whether you
trust the code enough to run it. In your case, you can probably trust
the code-provider to not have anything openly malicious in the code,
but otherwise, you'll need to look through the code, or run it in some
specialized environment like a virtual machine, or something like
that, to protect yourself. Remember, you're running code that someone
else gave you, and it's buggy. If the program's purpose is "delete
unnecessary files from your hard disk", then you want to be VERY
careful with it - one small bug, and it might wipe out a whole lot of
actively-used files. Just ask any sysadmin about the time he typed "rm
* .pyc" or equivalent.

So, you know what the program should do, and you either trust the
code, or have taken measures to protect yourself. Great! Now comes the
simple part: Run the program, see if it's doing what it should, and if
not, change something. Repeat until success.

'Course, I never said it was easy. Just simple. :)

Chances are you don't have enough information to know WHY it isn't
doing what it should, though, and that's where most of what people
talk of "debugging techniques" comes in. Your IDE might let you
single-step through the code, place breakpoints, and all sorts of
other fancy things; if so, use as many of those features as you're
comfortable with, and ignore the rest. Your basic debugging technique
will usually come down to one thing: Find out what the program's doing
at . And the simplest and most basic form of that
is what I call IIDPIO debugging: If In Doubt, Print It Out. Just add
print() calls to key parts of the code, reporting on the contents of
variables, or just announcing that control flow reached a certain
point. It's amazing how easy a debugging job can become when you just
get a little more info out of it.

Give that a try, and see how you go. If symptoms persist, see your teacher.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Project, how to debug

[Robert Clove]

On Tue, Mar 31, 2015 at 5:19 PM, Peter Otten <__pete...@web.de> wrote:

> Robert Clove wrote:
>
> > Hi All,
> >
> > I am facing a problem.
> > I have been given a project written in python and asked to debug it.
> > I have not been given the flow they said understand and debug.
> >
> > Can someone suggest me how to debug it in Wings IDE.
>
> Are those specific bugs that you are supposed to fix or are you to both
> find
> and fix bugs?
>
> > Project have approx 10 files.
>
> Generally speaking speaking this is not so much a matter of tools; if there
> is any documentation you can trust reading that is usually the best start;
> reading the code is important, too.
>
> Then you can consider your next steps depending on the "messiness" of the
> code base and the degree of coverage with unit tests.
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>


There is no documentation provided i have to read the code understand the
flow first.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: generator/coroutine terminology

[Albert van der Horst]

In article <83d579c1-ab61-4a3d-a834-e65d28eac...@googlegroups.com>,
Rustom Mody   wrote:
>On Saturday, March 14, 2015 at 8:59:22 PM UTC+5:30, Rustom Mody wrote:
>> On Saturday, March 14, 2015 at 11:34:27 AM UTC+5:30, Steven D'Aprano wrote:
>> >
>> > A generator (function) may be a function which returns an iterator,...
>>
>> I find "generator-function" misleading in the same way that "pineapple"
>> misleadingly suggests "apple that grows on pines"
>>
>> A builtin function is a function in the builtin (or builtins -- can
>never remember) module
>> A pure function is function that does not assign or mutate non-locals
>> A Steven-function is a function that presumably Steven wrote
>>
>> However a "generator function" is a weird sort of function (at best).
>> Not regarding it as a function is IMO more reasonable.
>
>Another analogy somewhat closer home than pineapples.
>In Pascal there are procedures and functions.
>Even in the venerable Fortran there are subroutine-subprogram and
>(sub)function-subprogram.

The Algol 68 designers considered it a defect in the design.
They created a situation like in Python, where a "def"-thingy need
not return a value.

>
>C took the stupid approach of just throwing out one of these.
>A decade of troubles was enough to convince people that it was needed and the
>mysterious 'void-returning' function was introduced to simulate procedures

The mistake this was intended to fix, was the rule that by default a
function returns int in C.

Groetjes Albert
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

-- 
https://mail.python.org/mailman/listinfo/python-list

Python regex exercise

[Robert Clove]

Hi All,

Do anyone have good links to python regex or other python problems for
beginners but with solution.

Please mail me.

Regards
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Strategy/ Advice for How to Best Attack this Problem?

[Dave Angel]

On 03/31/2015 07:00 AM, Saran A wrote:

> @DaveA: This is a homework assignment.  Is it possible that you 
could provide me with some snippets or guidance on where to place your 
suggestions (for your TO DOs 2,3,4,5)?

>



On Monday, March 30, 2015 at 2:36:02 PM UTC-4, Dave Angel wrote:




It's missing a number of your requirements.  But it's a start.

If it were my file, I'd have a TODO comment at the bottom stating known
changes that are needed.  In it, I'd mention:

1) your present code is assuming all filenames come directly from the
commandline.  No searching of a directory.

2) your present code does not move any files to success or failure
directories



In function validate_files()
Just after the line
print('success with %s on %d reco...
you could move the file, using shutil.  Likewise after the failure print.


3) your present code doesn't calculate or write to a text file any
statistics.


You successfully print to sys.stderr.  So you could print to some other 
file in the exact same way.




4) your present code runs once through the names, and terminates.  It
doesn't "monitor" anything.


Make a new function, perhaps called main(), with a loop that calls 
validate_files(), with a sleep after each pass.  Of course, unless you 
fix TODO#1, that'll keep looking for the same files.  No harm in that if 
that's the spec, since you moved the earlier versions of the files.


But if you want to "monitor" the directory, let the directory name be 
the argument to main, and let main do a dirlist each time through the 
loop, and pass the corresponding list to validate_files.




5) your present code doesn't check for zero-length files



In validate_and_process_data(), instead of checking filesize against 
ftell, check it against zero.



I'd also wonder why you bother checking whether the
os.path.getsize(file) function returns the same value as the os.SEEK_END
and ftell() code does.  Is it that you don't trust the library?  Or that
you have to run on Windows, where the line-ending logic can change the
apparent file size?

I notice you're not specifying a file mode on the open.  So in Python 3,
your sizes are going to be specified in unicode characters after
decoding.  Is that what the spec says?  It's probably safer to
explicitly specify the mode (and the file encoding if you're in text).

I see you call strip() before comparing the length.  Could there ever be
leading or trailing whitespace that's significant?  Is that the actual
specification of line size?

--
DaveA






I ask this because I have been searching fruitlessly through for some time and 
there are so many permutations that I am bamboozled by which is considered best 
practice.

Moreover, as to the other comments, those are too specific. The scope of the 
assignment is very limited, but I am learning what I need to look out or ask 
questions regarding specs - in the future.




--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: generator/coroutine terminology

[Albert van der Horst]

In article <55062bda$0$12998$c3e8da3$54964...@news.astraweb.com>,
Steven D'Aprano   wrote:
>Marko Rauhamaa wrote:
>
>> Chris Angelico :
>>
>>> On Sun, Mar 15, 2015 at 9:15 AM, Marko Rauhamaa  wrote:
 Is it necessary/useful for a Python application programmer to be
 conscious of the different types of iterator? What mistaken usage
 could arise if the application just treated all iterators as, well,
 iterators?
>>>
>>> If you treat them all as iterators, then you're safe, because that's
>>> the lowest common denominator. But there are a number of other
>>> iterators that have more features, including files, generators, etc.
>>
>> What features do generator iterators provide on top of generic
>> iterators?
>
>The biggest difference is syntactic. Here's an iterator which returns a
>never-ending sequence of squared numbers 1, 4, 9, 16, ...
>
>class Squares:
>def __init__(self):
>self.i = 0
>def __next__(self):
>self.i += 1
>return self.i**2
>def __iter__(self):
>return self

You should give an example of usage. As a newby I'm not up to
figuring out the specification from source for
something built of the mysterious __ internal
thingies.
(I did experiment with Squares interactively. But I didn't get
further than creating a Squares object.)


>
>
>Here's the same thing written as a generator:
>
>def squares():
>i = 1
>while True:
>yield i**2
>i += 1
>
>
>Four lines, versus eight. The iterator version has a lot of boilerplate
>(although some of it, the two-line __iter__ method, could be eliminated if
>there was a standard Iterator builtin to inherit from).
>
>Here's a more complicated example:
>
>class Randoms:
>def __init__(self):
>self.finished = False
>def __next__(self):
>x = random.random()
>if x > 0.5:
>self.finished = True
>if self.finished:
>raise StopIteration
>else:
>return x
>def __iter__(self):
>return self
>
>
>def randoms():
>x = random.random()
>while x < 0.5:
>yield x
>x = random.random()
>
>
>Generators, as a rule, are significantly easier to write, understand, and
>debug. There's nothing they can do that can't be done with an iterator
>class, but the fiddly, unexciting bits related to halting and resuming and
>saving state are all handled for you, allowing you to concentrate on the
>behaviour you want, not the boilerplate.

This is illuminating. Thanks.

>
>--
>Steven

Groetjes Albert
>
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

-- 
https://mail.python.org/mailman/listinfo/python-list

How to handle file in Whoosh?

[subhabrata . banerji]

I am trying to build a search engine, I started with Whoosh. The tutorial and 
web based materials are fine. Web has sizable question and answers. The initial 
experiments seem going fine. But I want to handle files located in various 
parts of my machine. I found "from whoosh.filedb.filestore import FileStorage", 
but I am looking for a simple example to start with. I tried something as 
below, it may be giving some result but is it going okay? If any one may please 
see and correct if required?

>>> txt_file1=open("/python27/whooshtext1.txt","r").read()
>>> txt_file2=open("/python27/whooshtext3.txt","r").read()
>>> writer.add_document(title=u"First document", path= 
>>> unicode("indexdir"+os.sep+"a"),content=u"txt_file1")
>>> writer.add_document(title=u"Second document", path= 
>>> unicode("indexdir"+os.sep+"b"),content=u"txt_file2")
>>> writer.commit()
>>> with ix.searcher() as searcher:
query = QueryParser("content", ix.schema).parse("flood")
results = searcher.search(query)
print results
for result in results:
print result



>>> 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: generator/coroutine terminology

[Dave Angel]

On 03/31/2015 09:18 AM, Albert van der Horst wrote:

In article <55062bda$0$12998$c3e8da3$54964...@news.astraweb.com>,
Steven D'Aprano   wrote:




The biggest difference is syntactic. Here's an iterator which returns a
never-ending sequence of squared numbers 1, 4, 9, 16, ...

class Squares:
def __init__(self):
self.i = 0
def __next__(self):
self.i += 1
return self.i**2
def __iter__(self):
return self


You should give an example of usage. As a newby I'm not up to
figuring out the specification from source for
something built of the mysterious __ internal
thingies.
(I did experiment with Squares interactively. But I didn't get
further than creating a Squares object.)



He did say it was an iterator.  So for a first try, write a for loop:

class Squares:
   def __init__(self):
   self.i = 0
   def __next__(self):
   self.i += 1
   return self.i**2
   def __iter__(self):
   return self

for i in Squares():
print(i)
if i > 50:
break

print("done")







Here's the same thing written as a generator:

def squares():
i = 1
while True:
yield i**2
i += 1


Four lines, versus eight. The iterator version has a lot of boilerplate
(although some of it, the two-line __iter__ method, could be eliminated if
there was a standard Iterator builtin to inherit from).


--
DaveA
--
https://mail.python.org/mailman/listinfo/python-list

Re: How to handle file in Whoosh?

[Peter Otten]

subhabrata.bane...@gmail.com wrote:

> I am trying to build a search engine, I started with Whoosh. The tutorial 
and web based materials are fine. Web has sizable question and answers. The 
initial experiments seem going fine. But I want to handle files located in 
various parts of my machine. I found "from whoosh.filedb.filestore import 
FileStorage", but I am looking for a simple example to start with. I tried 
something as below, it may be giving some result but is it going okay? If 
any one may please see and correct if required?
> 
> >>> txt_file1=open("/python27/whooshtext1.txt","r").read()
> >>> txt_file2=open("/python27/whooshtext3.txt","r").read()
> >>> writer.add_document(title=u"First document", path= 
unicode("indexdir"+os.sep+"a"),content=u"txt_file1")

I'm not a whoosh user myself, but if I were to guess:

The content argument should contain the contents of the file, not the 
variable name, e. g.:
import io
...
ENCODING = "ascii" # replace with actual encoding of the file
txt_file1 = io.open("/python27/whooshtext1.txt", encoding=ENCODING).read()
...
writer.add_document(
title=u"First document", 
path=os.path.join(u"indexdir", u"a"),
content=txt_file1)


> >>> writer.add_document(title=u"Second document", path= 
unicode("indexdir"+os.sep+"b"),content=u"txt_file2")
> >>> writer.commit()
> >>> with ix.searcher() as searcher:
> query = QueryParser("content", ix.schema).parse("flood")
> results = searcher.search(query)
> print results
> for result in results:
> print result
> 
> 
> 

u"flood" is indeed not in the string u"txt_file2" nor the string 
u"txt_file1", so that's the expected number of matches.


-- 
https://mail.python.org/mailman/listinfo/python-list

Lockfile hanling

[Ervin Hegedüs]

Hello,

there is an app, written in Python, which stores few bytes of
datas in a single file. The application uses threads. Every
thread can modify the file, but only one at a time. I'm using a
lock file to prevent the multiple access.

Here is the lock method:

  while True:
  try:
  fl = os.open(self.lockfile, os.O_CREAT | os.O_EXCL | os.O_RDWR)
  except OSError, e:
  if e.errno != errno.EEXIST:
  raise
  time.sleep(0.2)
  continue
  except:
  syslog.syslog(syslog.LOG_DEBUG, "Sync error: " + 
str(sys.exc_info()[1]))
  else:
  break

This works as well for me - about 3-4 weeks. After some weeks, I
got this error:

OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'


Today the app had been restarted, about 3-4 hours ago. Now I see
under the proc/PID/fd:

lrwx-- 1 root root 64 márc 31 16.45 5 -> 
/var/spool/myapp/queue.lock (deleted)

there are about 50 deleted FD's. After few weeks the process
reaches the number if max fd's.

How can I prevent or avoid this issue? What's the correct way to
handle the lockfile in Python?


Thanks,


Ervin

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Skip Montanaro]

On Tue, Mar 31, 2015 at 9:50 AM, Ervin Hegedüs  wrote:
> After few weeks the process
> reaches the number if max fd's.
>
> How can I prevent or avoid this issue? What's the correct way to
> handle the lockfile in Python?

Ervin,

You need to close the files you open. I don't see that happening in
your code snippet. You might consider the pylockfile module, available
from PyPI:

https://pypi.python.org/pypi/lockfile

(I'm the original author, though others have taken it over.)

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: generator/coroutine terminology

[Albert van der Horst]

In article ,
Dave Angel   wrote:
>On 03/31/2015 09:18 AM, Albert van der Horst wrote:
>> In article <55062bda$0$12998$c3e8da3$54964...@news.astraweb.com>,
>> Steven D'Aprano   wrote:
>
>>>
>>> The biggest difference is syntactic. Here's an iterator which returns a
>>> never-ending sequence of squared numbers 1, 4, 9, 16, ...
>>>
>>> class Squares:
>>> def __init__(self):
>>> self.i = 0
>>> def __next__(self):
>>> self.i += 1
>>> return self.i**2
>>> def __iter__(self):
>>> return self
>>
>> You should give an example of usage. As a newby I'm not up to
>> figuring out the specification from source for
>> something built of the mysterious __ internal
>> thingies.
>> (I did experiment with Squares interactively. But I didn't get
>> further than creating a Squares object.)
>>
>
>He did say it was an iterator.  So for a first try, write a for loop:
>
>class Squares:
>def __init__(self):
>self.i = 0
>def __next__(self):
>self.i += 1
>return self.i**2
>def __iter__(self):
>return self
>
>for i in Squares():
> print(i)
> if i > 50:
> break
>

This is what I get:
/ --
albert@cherry:/tmp$ more aap.py
class Squares:
def __init__(self):
self.i = 0
def __next__(self):
self.i += 1
return self.i**2
def __iter__(self):
return self

albert@cherry:/tmp$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from aap import *
>>> for i in Squares():
... print i
... if i>50: break
...
Traceback (most recent call last):
  File "", line 1, in 
TypeError: instance has no next() method
>>>

/ --

Probably not what is intended.

Last minute note:
   renaming __next__() into next() did the job.

>--
>DaveA

Groetjes Albert
-- 
Albert van der Horst, UTRECHT,THE NETHERLANDS
Economic growth -- being exponential -- ultimately falters.
albert@spe&ar&c.xs4all.nl &=n http://home.hccnet.nl/a.w.m.van.der.horst

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Ervin Hegedüs]

Hi Skip,

thanks for the reply,

On Tue, Mar 31, 2015 at 09:55:57AM -0500, Skip Montanaro wrote:
> On Tue, Mar 31, 2015 at 9:50 AM, Ervin Hegedüs  wrote:
> > After few weeks the process
> > reaches the number if max fd's.
> >
> > How can I prevent or avoid this issue? What's the correct way to
> > handle the lockfile in Python?
> 
> Ervin,
> 
> You need to close the files you open. I don't see that happening in
> your code snippet.

sorry - at the end of the function there is a close() method to a
file, after the thread passes the modifications:

try:
os.remove(self.lockfile)
except:
syslog.syslog(syslog.LOG_DEBUG, "Sync error: " + str(sys.exc_info()[1]))

And I think, the "(deleted)" info near the filename in proc/PID/fd
means the lockfile had been deleted.

> You might consider the pylockfile module, available
> from PyPI:
> 
> https://pypi.python.org/pypi/lockfile
> 
> (I'm the original author, though others have taken it over.)

sounds good, thanks - what's the minimal version of Python? I've
only 2.5 on that server...


Regards,


a.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Skip Montanaro]

> sorry - at the end of the function there is a close() method to a
> file, after the thread passes the modifications:
>
> try:
> os.remove(self.lockfile)
> except:
> syslog.syslog(syslog.LOG_DEBUG, "Sync error: " + 
> str(sys.exc_info()[1]))

Hmmm... Still not seeing os.close(fl)...

> And I think, the "(deleted)" info near the filename in proc/PID/fd
> means the lockfile had been deleted.

os.remove() will remove the lockfile name from the directory. It
doesn't automatically close the file.

>> You might consider the pylockfile module...

> sounds good, thanks - what's the minimal version of Python? I've
> only 2.5 on that server...

When I was still maintaining it, it worked with 2.5, and I distributed
a patch you could apply to get it to work with 2.4. I'm not sure if
the current maintainers have changed that. You'd have to look. If
worse comes to worst, I believe 0.9.1 (the last version I released)
should still work with 2.5.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Logging Custom Levels?

[Didymus]

Hi,

I've create a Python file called "log.py" and placed in the custom levels:

# Performance Debug...
logging.addLevelName(PDEBUG_NUM, "PDEBUG")

def pdebug(self, message, *args, **kws):
""" Performance Debug Message Level """
self.log(PDEBUG_NUM, message, *args, **kws) 

logging.Logger.pdebug = pdebug


This works except that the %(module) and %(lineno) does not print properly, it 
instead prints out as the "log.py" and the line that this is on. I think I 
figured out a way to get the module and line from the calling custom level:

import inspect
frame = inspect.currentframe()
filename = 
os.path.splitext(os.path.basename(frame.f_back.f_code.co_filename))[0]
linenumber = frame.f_back.f_lineno


My question is how do I pass this into the "self.log" call properly? I've tried 
a few different things without any luck. Any ideas how I can pass this into the 
custom logging level and get it to work?

Thanks in Advance for any help!
Tom
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: generator/coroutine terminology

[Chris Angelico]

On Wed, Apr 1, 2015 at 2:03 AM, Albert van der Horst
 wrote:
> class Squares:
> def __init__(self):
> self.i = 0
> def __next__(self):
> self.i += 1
> return self.i**2
> def __iter__(self):
> return self
>
> albert@cherry:/tmp$ python
> Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
> [GCC 4.4.5] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
 from aap import *
 for i in Squares():
> ... print i
> ... if i>50: break
> ...
> Traceback (most recent call last):
>   File "", line 1, in 
> TypeError: instance has no next() method

>
> / --
>
> Probably not what is intended.
>
> Last minute note:
>renaming __next__() into next() did the job.

That class was written for Python 3, not Python 2. In Py2, you need to
rename __next__ to next, as you noted, and you probably also want to
explicitly subclass object. Or just run it under Python 3. :)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Ervin Hegedüs]

Hi Skip,

On Tue, Mar 31, 2015 at 10:19:27AM -0500, Skip Montanaro wrote:
> > sorry - at the end of the function there is a close() method to a
> > file, after the thread passes the modifications:
> >
> > try:
> > os.remove(self.lockfile)
> > except:
> > syslog.syslog(syslog.LOG_DEBUG, "Sync error: " + 
> > str(sys.exc_info()[1]))
> 
> Hmmm... Still not seeing os.close(fl)...
> 
> > And I think, the "(deleted)" info near the filename in proc/PID/fd
> > means the lockfile had been deleted.
> 
> os.remove() will remove the lockfile name from the directory. It
> doesn't automatically close the file.

ah, well, many-many-many thanks - that was what I misses...

> >> You might consider the pylockfile module...
> 
> > sounds good, thanks - what's the minimal version of Python? I've
> > only 2.5 on that server...
> 
> When I was still maintaining it, it worked with 2.5, and I distributed
> a patch you could apply to get it to work with 2.4. I'm not sure if
> the current maintainers have changed that. You'd have to look. If
> worse comes to worst, I believe 0.9.1 (the last version I released)
> should still work with 2.5.

thanks for the info - now it works as very well.


a.
 
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Matthew Ruffalo]

On 2015-03-31 10:50, Ervin Hegedüs wrote:
> there is an app, written in Python, which stores few bytes of
> datas in a single file. The application uses threads. Every
> thread can modify the file, but only one at a time. I'm using a
> lock file to prevent the multiple access.
>
> ...
>
> How can I prevent or avoid this issue? What's the correct way to
> handle the lockfile in Python?

Hi Ervin-

If only one instance of the app is running at a given time, and you only
need to ensure mutual exclusion between its threads, you should probably
not use a lock *file* for this. I would strongly recommend that you use
a threading.Lock as per
https://docs.python.org/2.5/lib/module-threading.html instead of a lock
file. This will also allow you to avoid a 0.2-second polling loop; a
call to threading.Lock.acquire() will block until it is released by
another thread.

MMR...
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Ervin Hegedüs]

Hi Matthew,

On Tue, Mar 31, 2015 at 11:50:06AM -0400, Matthew Ruffalo wrote:
> On 2015-03-31 10:50, Ervin Hegedüs wrote:
> > there is an app, written in Python, which stores few bytes of
> > datas in a single file. The application uses threads. Every
> > thread can modify the file, but only one at a time. I'm using a
> > lock file to prevent the multiple access.
> >
> > ...
> >
> > How can I prevent or avoid this issue? What's the correct way to
> > handle the lockfile in Python?
> 
> Hi Ervin-
> 
> If only one instance of the app is running at a given time, and you only
> need to ensure mutual exclusion between its threads, you should probably
> not use a lock *file* for this. I would strongly recommend that you use
> a threading.Lock as per
> https://docs.python.org/2.5/lib/module-threading.html instead of a lock
> file. This will also allow you to avoid a 0.2-second polling loop; a
> call to threading.Lock.acquire() will block until it is released by
> another thread.
> 
> MMR...

thanks, I hope in the next version I can try this :)


a.

-- 
https://mail.python.org/mailman/listinfo/python-list

Quick question, if you please

[John Kelly]

Pythonites,
I received Python with another install and my update software keeps signaling I 
need to install a newer version, and once I do, the older version is still 
there, so I keep getting told I need to update. Should I be able to uninstall 
the old version each time?
Thanks for your kind attention,John Kelly
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quick question, if you please

[Russell Owen]

On 3/31/15 10:09 AM, John Kelly wrote:

Pythonites,

I received Python with another install and my update software keeps
signaling I need to install a newer version, and once I do, the older
version is still there, so I keep getting told I need to update. Should
I be able to uninstall the old version each time?

Thanks for your kind attention,
John Kelly


I would need more information to help. What operating system are you on? 
How and where are you installing Python (and what do you mean by 
"received Python with another install"?).


-- Russell

--
https://mail.python.org/mailman/listinfo/python-list

Re: Logging Custom Levels?

[Jean-Michel Pichavant]

- Original Message -
> From: "Didymus" 
> To: python-list@python.org
> Sent: Tuesday, 31 March, 2015 5:20:52 PM
> Subject: Logging Custom Levels?
> 
> Hi,
> 
> I've create a Python file called "log.py" and placed in the custom
> levels:
> 
> # Performance Debug...
> logging.addLevelName(PDEBUG_NUM, "PDEBUG")
> 
> def pdebug(self, message, *args, **kws):
> """ Performance Debug Message Level """
> self.log(PDEBUG_NUM, message, *args, **kws)
> 
> logging.Logger.pdebug = pdebug
> 
> 
> This works except that the %(module) and %(lineno) does not print
> properly, it instead prints out as the "log.py" and the line that
> this is on. I think I figured out a way to get the module and line
> from the calling custom level:
> 
> import inspect
> frame = inspect.currentframe()
> filename =
> os.path.splitext(os.path.basename(frame.f_back.f_code.co_filename))[0]
> linenumber = frame.f_back.f_lineno
> 
> 
> My question is how do I pass this into the "self.log" call properly?
> I've tried a few different things without any luck. Any ideas how I
> can pass this into the custom logging level and get it to work?
> 
> Thanks in Advance for any help!
> Tom

A solution is pretty simple, do not use an intermediate log function pdebug.

import logging
PDEBUG_NUM=20
logging.addLevelName(PDEBUG_NUM, "PDEBUG")
 
logger = logging.getLogger('foo')
logging.basicConfig(level=logging.DEBUG, format='%(message)s %(lineno)d')

logger.log(PDEBUG_NUM, 'This will work :')


If you *really* want to go for the hackish way, forget about the inspect 
module, the following pdebug function should do the trick:

def pdebug(self, message, *args, **kws):
if self.isEnabledFor(PDEBUG_NUM):
self._log(PDEBUG_NUM, message, args, **kws)

Cheers,

JM


-- IMPORTANT NOTICE: 

The contents of this email and any attachments are confidential and may also be 
privileged. If you are not the intended recipient, please notify the sender 
immediately and do not disclose the contents to any other person, use it for 
any purpose, or store or copy the information in any medium. Thank you.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Christian Gollwitzer]

Am 31.03.15 um 16:50 schrieb Ervin Hegedüs:

there is an app, written in Python, which stores few bytes of
datas in a single file. The application uses threads. Every
thread can modify the file, but only one at a time. I'm using a
lock file to prevent the multiple access.

Here is the lock method:

[...]
This works as well for me - about 3-4 weeks. After some weeks, I
got this error:

OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'



Can you try if fcntl.flock() does what you want? Should be better than 
inventing your own locking mechanism.


Christian
--
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Ian Kelly]

On Tue, Mar 31, 2015 at 12:58 PM, Christian Gollwitzer  wrote:
> Am 31.03.15 um 16:50 schrieb Ervin Hegedüs:
>>
>> there is an app, written in Python, which stores few bytes of
>> datas in a single file. The application uses threads. Every
>> thread can modify the file, but only one at a time. I'm using a
>> lock file to prevent the multiple access.
>>
>> Here is the lock method:
>>
>> [...]
>> This works as well for me - about 3-4 weeks. After some weeks, I
>> got this error:
>>
>> OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'
>>
>
> Can you try if fcntl.flock() does what you want? Should be better than
> inventing your own locking mechanism.

flock locks are per-process; they won't help for synchronizing access
between competing threads in the same process.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Error in processing JSON files in Python

[Denis McMahon]

On Mon, 30 Mar 2015 14:27:14 -0700, Karthik Sharma wrote:

> I have the following python program to read a set of JSON files do some
> processing on it and dump them back to the same folder. However When I
> run the below program and then try to see the output of the JSON file
> using
> 
> `cat file.json | python -m json.tool`
> 
> I get the following error
> 
> `extra data: line 1 column 307 - line 1 column 852 (char 306 - 851)`
> 
> What is wrong with my program?
>  
> #Process 'new' events to extract more info from 'Messages'
> rootDir = '/home/s_parts'
> for dirName, subdirList, fileList in os.walk(rootDir):
> print('Found directory: %s' % dirName)
> for fname in fileList:
> fname='s_parts/'+fname with open(fname, 'r+') as f:
> json_data = json.load(f)
> # do stuff to the data
> json.dump(json_data,f)

You're writing back to the same file as you loaded the data from having 
opened the file in append mode.

This probably leads to a file containing two json objects, the original 
one and the one which you have processed.

That's a bad json file.

Note the caution in the python documentation:

"Note - Unlike pickle and marshal, JSON is not a framed protocol, so 
trying to serialize multiple objects with repeated calls to dump() using 
the same fp will result in an invalid JSON file."

Writing back to the same file as you read with json.load is the same 
thing. If you want to use the same file, you need to close it after the 
json.load(), and then open it again in write mode before the json.dump(). 

Write is "w", *NOT* "w+".

-- 
Denis McMahon, denismfmcma...@gmail.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Christian Gollwitzer]

Am 31.03.15 um 21:17 schrieb Ian Kelly:

On Tue, Mar 31, 2015 at 12:58 PM, Christian Gollwitzer  wrote:

Am 31.03.15 um 16:50 schrieb Ervin Hegedüs:


there is an app, written in Python, which stores few bytes of
datas in a single file. The application uses threads. Every
thread can modify the file, but only one at a time. I'm using a
lock file to prevent the multiple access.

Here is the lock method:

[...]
This works as well for me - about 3-4 weeks. After some weeks, I
got this error:

OSError: [Errno 24] Too many open files: '/var/spool/myapp/queue.lock'



Can you try if fcntl.flock() does what you want? Should be better than
inventing your own locking mechanism.


flock locks are per-process; they won't help for synchronizing access
between competing threads in the same process.



Ok. But if it is really all in one process, then a classical mutex would 
do, wouldn't it? I'm not experienced with threading in Python, but it 
seems from the documentation, that threading.Lock() works like a mutex.


Christian
--
https://mail.python.org/mailman/listinfo/python-list

Re: Lockfile hanling

[Ian Kelly]

On Tue, Mar 31, 2015 at 1:59 PM, Christian Gollwitzer  wrote:
> Am 31.03.15 um 21:17 schrieb Ian Kelly:
>> flock locks are per-process; they won't help for synchronizing access
>> between competing threads in the same process.
>>
>
> Ok. But if it is really all in one process, then a classical mutex would do,
> wouldn't it? I'm not experienced with threading in Python, but it seems from
> the documentation, that threading.Lock() works like a mutex.

As long as there's not *also* some other external process that needs
to access the file occasionally. :-)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Project, how to debug

[Andrew Farrell]

points of general advice:
- As noted by Chris, you first need to figure out what the code is
*supposed* to do.
- Have a notebook and pencil next to you as you are working through this so
that you can make notes and draw out the structure of things.
- Write down the general structure of the code on your notebook so you
always keep yourself oriented. If you find yourself lost, don't just keep
reading code; stop and write out what you do understand and try to turn
your sense of confusion into concrete questions.
- Use The Silver Searcher . If you find yourself
thinking "I wish I could just see all the places where the variable
'herring' is used", then `ag -C 5 herring` is going to be your friend.
- Use pdb , the python debugger
that comes in the standard library. If you find yourself wanting to jump
into a section of the code and print out variables and look at stack
frames, sticking `import pdb;pdb.set_trace()` in there is going to be your
friend.
- Actually try using ipdb
,
which has a slightly better user interface than pdb. But don't worry if you
can't install it. pdb is fine.
- Much like you shouldn't just blindly read through code, don't just
blindly step through code. If you lose sight of where you are, step back
and re-orient yourself.
- Find a colleague, friend, or rubber duck. Explain to them what you think
the code is doing. This will force you to think about things more clearly.
- Try writing unit tests for the code with with py.test
. It will help you verify "okay, this piece
right here is working correctly" and thereby mean that you don't have to
check that piece again.
- Use assert statements to check that things are as they should be.
- Try sticking `assert False` into the middle of a function. If the
function raises an error other than AssertionError, then the problem is
before the assert statement.
- In your spare time, read 9 Indespensible Rules for Debugging Software and
Hardware Problems
.
It is both useful and enjoyable.

On Tue, Mar 31, 2015 at 7:05 AM, Robert Clove  wrote:

>
>
> On Tue, Mar 31, 2015 at 5:19 PM, Peter Otten <__pete...@web.de> wrote:
>
>> Robert Clove wrote:
>>
>> > Hi All,
>> >
>> > I am facing a problem.
>> > I have been given a project written in python and asked to debug it.
>> > I have not been given the flow they said understand and debug.
>> >
>> > Can someone suggest me how to debug it in Wings IDE.
>>
>> Are those specific bugs that you are supposed to fix or are you to both
>> find
>> and fix bugs?
>>
>> > Project have approx 10 files.
>>
>> Generally speaking speaking this is not so much a matter of tools; if
>> there
>> is any documentation you can trust reading that is usually the best start;
>> reading the code is important, too.
>>
>> Then you can consider your next steps depending on the "messiness" of the
>> code base and the degree of coverage with unit tests.
>>
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
>
> There is no documentation provided i have to read the code understand the
> flow first.
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Quick question, if you please

[John Gordon]

In  John Kelly 
 writes:

> I received Python with another install and my update software keeps
> signaling I need to install a newer version, and once I do, the older
> version is still there, so I keep getting told I need to update. Should
> I be able to uninstall the old version each time?

Yes, assuming you have no other software that depends on the older
Python version.

-- 
John Gordon Imagine what it must be like for a real medical doctor to
gor...@panix.comwatch 'House', or a real serial killer to watch 'Dexter'.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A simple single line, triple-quoted comment is giving syntax error. Why?

[sohcahtoa82]

On Tuesday, March 31, 2015 at 12:24:53 AM UTC-7, Ian wrote:
> On Sun, Mar 29, 2015 at 4:41 AM, Thomas 'PointedEars' Lahn
>  wrote:
> > Ian Kelly wrote:
> >
> >> […] Thomas 'PointedEars' Lahn […] wrote:
> >>> Ian Kelly wrote:
> >> Why should the burden of proof be on me in the first place?
> >
> > Because *you* made the claim that “STRING+” could be part of an AST in this
> > way.
> 
> Okay, I didn't understand before that this was the part that you were
> objecting to. I had written "There is only one expr node, and it
> contains both STRING tokens", to which you replied "Prove it." I did
> not intend to imply anything at all by the inclusion of STRING+ in the
> tree. That pseudo-tree was hastily sketched out, and as both you and
> Gregory have pointed out, STRING+ is not a symbol of the grammar and
> should not have been included as a node. None of this has any bearing
> on what I was trying to demonstrate with that tree.
> 
> >> Within a grammar, the question of "is an X a Y" is nonsensical in
> >> isolation. It can only be answered in relation to a parse tree.
> >> Consider the simple grammar:
> >>
> >> S -> A | B
> >> A -> x
> >> B -> x
> >>
> >> Is x an A? It depends.
> >
> > No, by the definition 2 below, that we all accepted implicitly up to this
> > point, x is *definitely* an A.
> 
> What gives you the impression that I ever accepted it?
> 
> >> If the tree that generates the x produces it from an A node, then yes.
> >> Otherwise, no.
> >
> > Fallacy.
> >
> > First, two definitions, so that we are clear what we are talking about:
> >
> > (1) Let a *production chain* be the repeated application of the production
> > rules of a formal grammar such that
> >
> >   C ⇒ D ⇒ x
> >
> > is a production chain if there are production rules
> >
> >   C → D
> >   D → x.
> >
> > [Note the difference between “⇒” and “→”.]
> 
> Sure.
> 
> > (2) Let the statement “x is an A” be true if x can be produced in a
> > production chain starting with or including the non-terminal A
> > left-hand side –
> >
> >   x ∈ A ↔ ∃A (… ⇒ A ⇒ … ⇒ x).
> 
> Sorry, but this definition just seems entirely arbitrary to me.
> Mathematically, it looks nonsensical; A is a symbol, not a set. This
> question of whether "x is an A" is informal and not a topic of formal
> language theory so far as I'm aware. Can you cite some source for it?
> 
> > Now, according to these definitions, in the offered grammar x is *both* an A
> > and a B.  Because what matters is _not_ the practical result of production
> > chains (the actual parse tree), but the certainty of the theoretical
> > possibility of it.
> 
> This strikes me as being a lot like arguing, "some kites are toys, and
> some kites are birds; therefore, all kites are both toys and birds."
> 
> >> So when I write that the "foo" in "foo" "bar" is not an expression, I
> >> am only speaking in relation to the parse tree that generates "foo"
> >> "bar".
> >
> > But it has been indicated by others that the parse tree that you presented
> > is wrong, based on a misconception about the syntax of the formal grammar,
> > and you have not yet substantiated your claim that it is correct.
> 
> As noted above, the inaccuracy that Gregory pointed out has no bearing
> on my argument.
> 
> You're really going to make me spell it out, aren't you? Fine, here you go.
> 
> single_input -> simple_stmt -> expr_stmt -> testlist_star_expr -> test
> -> or_test -> and_test -> not_test -> comparison -> expr -> xor_expr
> -> and_expr -> shift_expr -> arith_expr -> term -> factor -> power ->
> atom -> STRING STRING
> 
> Note: the derivation contains exactly one expr node, which indirectly
> produces both STRINGs. Neither STRING in this derivation is
> individually produced from the expr.
> 
> >> I don't know what you mean by a "backdoor".
> >
> > Appending a statement that is contradictory to what was stated just before,
> > or at least ambiguous, so that the possibility arises for one to say “I did
> > not mean that” when that/a contradiction to the former statement is pointed
> > out later.
> 
> The purpose was not to make the possibility arise to say that. The
> purpose was to point out the discontinuity *immediately* so that it
> would be already addressed and not *need* to be pointed out later
> (although we can see how well that turned out).
> 
> I have to say that I find it highly surprising that you find this
> construction objectionable. Would you still consider it
> "contradictory" if I had phrased it equivalently as a logical "or"
> rather than using the word "except"?
> 
> >> The purpose of that parenthetical was to explicitly acknowledge that the
> >> single STRING could be viewed as an expression when taken out of context,
> >
> > Yes, your fallacy is mainly based on ignoring the context.  Context is
> > important; you cannot just ignore it and still make correct arguments.
> 
> Funny, I thought that was my line. This whole argument came about
> because you seem to be stubbo

Re: A simple single line, triple-quoted comment is giving syntax error. Why?

[Chris Angelico]

On Wed, Apr 1, 2015 at 10:10 AM,   wrote:
> Holy hell, dude, you've been arguing about this for nearly two weeks now.
>
> Let it go.

Not sure you understand the nature of language debates. They can't
hold it back any more!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A simple single line, triple-quoted comment is giving syntax error. Why?

[Ian Kelly]

On Mar 31, 2015 5:16 PM,  wrote:
> Holy hell, dude, you've been arguing about this for nearly two weeks now.
>
> Let it go.

That's rather an exaggeration. I've made seven posts to this thread up
until now, the first of which was six days ago, not two weeks. I don't
think that's excessive.

I suggested dropping it a couple of posts back, but Thomas seemed
interested in continuing to seek clarity, which I have no objection to.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Automation of Windows app?

[alex23]

On 23/03/2015 1:43 PM, Michael Torrie wrote:

As near as I can tell the standard go-to utility for this is a program
called AutoIt.  https://www.autoitscript.com/site/autoit/

Nothing to do with Python, and its scripting language is maybe not that
appealing to many, but it does the job, and does it pretty well.


Actually, it's not *entirely* unrelated, as there's a Python wrapper for it:

https://pypi.python.org/pypi/PyAutoIt/0.3

It's 2.7 only, though, so if 3.x is required, it's also possible to 
create your own wrapper using win32com:


http://stackoverflow.com/questions/151846/get-other-running-processes-window-sizes-in-python/155587#155587

--
https://mail.python.org/mailman/listinfo/python-list

New to Programming - XML Processing

[catperson]

I am new to programming, though not new to computers.  I'm looking to
teach myself Python 3 and am working my way through a tutorial.  At
the point I'm at in the tutorial I am tasked with parsing out an XML
file created with a Garmin Forerunner and am just having a terrible
time getting my head around the concepts.  What I'm looking for is
some suggested reading that might give me some of the theory of
operation behind ElementTree and then how to parse out specific
elements.  Most of what I have been able to find in examples that I
can understand use very simplistic XML files and this Garmin file is
many levels of sub-elements and some of those elements have attributes
assigned, like .

I'm hoping with enough reading I can experiment and work my way
through the problem and end up with a hopefully clear understanding of
the ElementTree module and Dictionairies.  

Thanks for any suggestions in advance.

Jim.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: OT Intuitionism+Platonism+Mysticism => CS (was Deep comparison of dicts...)

[Rustom Mody]

On Saturday, March 21, 2015 at 11:58:43 AM UTC+5:30, Marko Rauhamaa wrote:
> Rustom Mody :
> 
> > However I am talking some historical facts, viz: Because some nuts did
> > the 20th century equivalent of: "Break each others' heads about how
> > many angels can dance on the head of a pin" therefore much of the
> > world around us exists as it does (for better or worse) including my
> > (and I guess your) degree and job.
> 
> The deep philosophical questions around the year 1900 in both
> mathematics and physics were left embarrassingly open. The way out was
> to stop caring.

Ive collected some of the above points and some from earlier threads here:
http://blog.languager.org/2015/03/cs-history-0.html

Anyone un-mentioned there that wants to figure -- or conversely :-) --
just give me the say-so.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New to Programming - XML Processing

[Ben Finney]

catperson  writes:

> I am new to programming, though not new to computers. I'm looking to
> teach myself Python 3 and am working my way through a tutorial.

Congratulations! Python 3 is a fine language to be your first.

Which tutorial are you following? It may be relevant.

> At the point I'm at in the tutorial I am tasked with parsing out an
> XML file created with a Garmin Forerunner and am just having a
> terrible time getting my head around the concepts.

Have you worked with manipulating XML documents in the past?

If I describe XML as a machine-readable hierarchical data serialisation
format, does that sound like noise or are you comfortable that you
understand that description?

> What I'm looking for is some suggested reading that might give me some
> of the theory of operation behind ElementTree and then how to parse
> out specific elements.

You'll need to understand the principles behind XML, then. Warning: it
is not simple! XML is machine-readable, but rarely pleasant for *humans*
to read or understand.

> I'm hoping with enough reading I can experiment and work my way
> through the problem and end up with a hopefully clear understanding of
> the ElementTree module and Dictionairies.

XML is a standard – a large number of standards, really – from the W3
Consortium. They have a good overview of the “core” of XML
http://www.w3.org/standards/xml/core>.

There are countless XML formats though; XML is a markup language to use
as a basis for constructing data serialisation formats. You need to know
what the specific format (called a “schema”) is before you can expect to
make sense of it, and so in order to understand what's in the document
you need a description of the specific schema from whoever wrote the
document.

General XML tutorials do exist; here is one comprehensive one
http://www.xmlobjective.com/category/guide/>. You'll likely want to
learn more about the specifics of whatever XML format is being used,
though, before going much beyond the basic principles.

-- 
 \“Politics is not the art of the possible. It consists in |
  `\   choosing between the disastrous and the unpalatable.” —John |
_o__)Kenneth Galbraith, 1962-03-02 |
Ben Finney

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New to Programming - XML Processing

[Rustom Mody]

On Wednesday, April 1, 2015 at 8:57:15 AM UTC+5:30, catperson wrote:
> I am new to programming, though not new to computers.  I'm looking to
> teach myself Python 3 and am working my way through a tutorial.  At
> the point I'm at in the tutorial I am tasked with parsing out an XML
> file created with a Garmin Forerunner and am just having a terrible
> time getting my head around the concepts.  What I'm looking for is
> some suggested reading that might give me some of the theory of
> operation behind ElementTree and then how to parse out specific
> elements.  Most of what I have been able to find in examples that I
> can understand use very simplistic XML files and this Garmin file is
> many levels of sub-elements and some of those elements have attributes
> assigned, like .
> 
> I'm hoping with enough reading I can experiment and work my way
> through the problem and end up with a hopefully clear understanding of
> the ElementTree module and Dictionairies.  
> 
> Thanks for any suggestions in advance.

Suggestions:
1. Learn to use the interpreter interactively; ie (at the least)¹ ie

a. Start up python (without a program)
b. Play around with trivial expressions
c. Explore introspective features - help(), type() dir()

2. Do you know about triple-quoted strings?
a. Start small (or trivial) sub-parts of your XML as triple-quoted examples in 
the 
interpreter and start throwing them at elementtree
b. If they dont work trivialize further; if they work add complexity
---
¹ At the least because environments like Idle are more conducive to such playing
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New to Programming - XML Processing

[Rustom Mody]

On Wednesday, April 1, 2015 at 8:57:15 AM UTC+5:30, catperson wrote:
> I'm hoping with enough reading I can experiment and work my way
> through the problem and end up with a hopefully clear understanding of
> the ElementTree module and Dictionairies.  

Also:
If you are not familiar with dictionaries, better grok them
(and lists and tuples and some else) from a good tut like

https://docs.python.org/3.4/tutorial/

BEFORE taking on ElementTree
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New to Programming - XML Processing

[Andrew Farrell]

You should follow Rustom's advice before just diving into the blog post I
linked to. Otherwise you risk blindly following things and losing your
bearings when you run into bugs.

On Tue, Mar 31, 2015 at 11:17 PM, Rustom Mody  wrote:

> On Wednesday, April 1, 2015 at 8:57:15 AM UTC+5:30, catperson wrote:
> > I am new to programming, though not new to computers.  I'm looking to
> > teach myself Python 3 and am working my way through a tutorial.  At
> > the point I'm at in the tutorial I am tasked with parsing out an XML
> > file created with a Garmin Forerunner and am just having a terrible
> > time getting my head around the concepts.  What I'm looking for is
> > some suggested reading that might give me some of the theory of
> > operation behind ElementTree and then how to parse out specific
> > elements.  Most of what I have been able to find in examples that I
> > can understand use very simplistic XML files and this Garmin file is
> > many levels of sub-elements and some of those elements have attributes
> > assigned, like .
> >
> > I'm hoping with enough reading I can experiment and work my way
> > through the problem and end up with a hopefully clear understanding of
> > the ElementTree module and Dictionairies.
> >
> > Thanks for any suggestions in advance.
>
> Suggestions:
> 1. Learn to use the interpreter interactively; ie (at the least)¹ ie
>
> a. Start up python (without a program)
> b. Play around with trivial expressions
> c. Explore introspective features - help(), type() dir()
>
> 2. Do you know about triple-quoted strings?
> a. Start small (or trivial) sub-parts of your XML as triple-quoted
> examples in the
> interpreter and start throwing them at elementtree
> b. If they dont work trivialize further; if they work add complexity
> ---
> ¹ At the least because environments like Idle are more conducive to such
> playing
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: New to Programming - XML Processing

[Andrew Farrell]

> I am new to programming, though not new to computers.

One quick tip: when starting a new project, it is sometimes is useful to
see if others have done the same thing and use their approach to guide
yours. In your case, googling "Garmin Forerunner xml python" results in
this blog post which is recent enough that it should be helpful.

http://www.nezirteke.com/2014/02/using-python-to-plot-garmin-forerunner.html

Note that coming up with good phrases for google to help you with a
programming task is a skill that you get better at with time.

Welcome to Python! ^-^

On Tue, Mar 31, 2015 at 11:08 PM, Ben Finney 
wrote:

> catperson  writes:
>
> > I am new to programming, though not new to computers. I'm looking to
> > teach myself Python 3 and am working my way through a tutorial.
>
> Congratulations! Python 3 is a fine language to be your first.
>
> Which tutorial are you following? It may be relevant.
>
> > At the point I'm at in the tutorial I am tasked with parsing out an
> > XML file created with a Garmin Forerunner and am just having a
> > terrible time getting my head around the concepts.
>
> Have you worked with manipulating XML documents in the past?
>
> If I describe XML as a machine-readable hierarchical data serialisation
> format, does that sound like noise or are you comfortable that you
> understand that description?
>
> > What I'm looking for is some suggested reading that might give me some
> > of the theory of operation behind ElementTree and then how to parse
> > out specific elements.
>
> You'll need to understand the principles behind XML, then. Warning: it
> is not simple! XML is machine-readable, but rarely pleasant for *humans*
> to read or understand.
>
> > I'm hoping with enough reading I can experiment and work my way
> > through the problem and end up with a hopefully clear understanding of
> > the ElementTree module and Dictionairies.
>
> XML is a standard – a large number of standards, really – from the W3
> Consortium. They have a good overview of the “core” of XML
> http://www.w3.org/standards/xml/core>.
>
> There are countless XML formats though; XML is a markup language to use
> as a basis for constructing data serialisation formats. You need to know
> what the specific format (called a “schema”) is before you can expect to
> make sense of it, and so in order to understand what's in the document
> you need a description of the specific schema from whoever wrote the
> document.
>
> General XML tutorials do exist; here is one comprehensive one
> http://www.xmlobjective.com/category/guide/>. You'll likely want to
> learn more about the specifics of whatever XML format is being used,
> though, before going much beyond the basic principles.
>
> --
>  \“Politics is not the art of the possible. It consists in |
>   `\   choosing between the disastrous and the unpalatable.” —John |
> _o__)Kenneth Galbraith, 1962-03-02 |
> Ben Finney
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list