Re: XML Parsing
Sepideh Ghanavati schrieb am 06.04.2015 um 04:26: > I know basic of python and I have an xml file created from csv which has > three attributes "category", "definition" and "definition description". > I want to parse through xml file and identify actors, constraints, > principal from the text. However, I am not sure what is the best way to > go. Any suggestion? If it's really generated from a CSV file, you could also parse that instead: https://docs.python.org/3/library/csv.html Admittedly, CSV files are simple, but they also have major problems, especially when it comes to detecting their character encoding and their specific format (tab/comma/semicolon/space/whatever separated, with or without escaping, quoted values, ...). Meaning, you can easily end up reading nonsense from the file instead of the content that was originally put into it. So, if you want to parse from XML instead, use ElementTree: https://docs.python.org/3/library/xml.etree.elementtree.html Stefan -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to deliver different source distributions for different Python versions?
Dave Hein schrieb am 05.04.2015 um 22:38: > I would like to distribute a python package with different code for > Python 2.* than for Python 3.*. (Mostly this is because of different > unicode string handling). > > There is nothing in to setuptools or PyPi that directly supports > this scenario. > > But perhaps there could be some script run at install time that moves > the correct source code to the right location? In other works, if I > included both source code versions in the distribution (in a src2 and > a src3 subdirectory) then a function invoked at install time could > detect the python version and copy the appropriate source code to the > right location. > > Is that at all possible? Is there some install time hook that lets me > supply custom installation code? Sure. You can simply change the directory in which distutils looks for your Python code: https://docs.python.org/2/distutils/setupscript.html#listing-whole-packages However, in general, you shouldn't be doing this. It's usually easier (definitely in the long-term) to keep your sources cross-Py2.x/3.x compatible, maybe with the help of tools like "six" or "python-future", than to try to keep separate source trees in sync. http://python-future.org/ Stefan -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to deliver different source distributions for different Python versions?
On 05/04/2015 21:38, Dave Hein wrote: I would like to distribute a python package with different code for Python 2.* than for Python 3.*. (Mostly this is because of different unicode string handling). There is nothing in to setuptools or PyPi that directly supports this scenario. But perhaps there could be some script run at install time that moves the correct source code to the right location? In other works, if I included both source code versions in the distribution (in a src2 and a src3 subdirectory) then a function invoked at install time could detect the python version and copy the appropriate source code to the right location. Is that at all possible? Is there some install time hook that lets me supply custom installation code? -- Dave Hein I can't help directly but have you looked here https://packaging.python.org/en/latest/ ? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: XML Parsing
Sepideh Ghanavati writes: > I know basic of python and I have an xml file created from csv What XML schema defines the document's format? Without knowing the schema, parsing will be unreliable. What created the document? Why is it relevant that the document was “created from CSV”? > which has three attributes "category", "definition" and "definition > description". What do you mean by “attributes”? In Python, an attribute has a specific meaning. In XML, an attribute has a rather different meaning. Neither of those meanings seems to apply to “the XML document has three attributes”. XML documents don't have attributes; differnt XML elements in a document have different attributes. > I want to parse through xml file and identify actors, constraints, > principal from the text. How are those defined in the document's schema? > However, I am not sure what is the best way to go. Any suggestion? You should: * Learn some more about XML http://www.xmlobjective.com/the-basic-principles-of-xml/>. * Learn exactly what formal document schema defines the document https://en.wikipedia.org/wiki/XML_schema>. If the document isn't accompanied by a specification of exactly what its schema is, you're going to have a difficult time. -- \“If I melt dry ice, can I swim without getting wet?” —Steven | `\Wright | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
XML Parsing
Hi, I know basic of python and I have an xml file created from csv which has three attributes "category", "definition" and "definition description". I want to parse through xml file and identify actors, constraints, principal from the text. However, I am not sure what is the best way to go. Any suggestion? -- https://mail.python.org/mailman/listinfo/python-list
Re: Is it possible to deliver different source distributions for different Python versions?
On Mon, 6 Apr 2015 06:38 am, Dave Hein wrote: > I would like to distribute a python package with different code for > Python 2.* than for Python 3.*. (Mostly this is because of different > unicode string handling). > > There is nothing in to setuptools or PyPi that directly supports > this scenario. > > But perhaps there could be some script run at install time that moves > the correct source code to the right location? In other works, if I > included both source code versions in the distribution (in a src2 and > a src3 subdirectory) then a function invoked at install time could > detect the python version and copy the appropriate source code to the > right location. > > Is that at all possible? Is there some install time hook that lets me > supply custom installation code? I'm not aware of any standard solution to that, but I'm not a setuptools expert. setup.py files are Python code, so you can put any code you like in them. But, as far as I am concerned, having the installer pick and choose what source files to include is not a good solution. Instead, you should pick one of these two alternatives: (1) Supply a separate package for 2.x and 3.x, each with their own installer. The installer confirms that it is running under the correct version of Python, and just installs. (2) Or supply a single package with a single installer that works under both 2.x and 3.x. (This is my preference.) One way of handling the second case is to only support 3.3 or better: that way, you can still use u"..." for Unicode strings. Hardly anyone used 3.1 or 3.2, and 3.0 is no longer supported, so it is quite reasonable to insist people upgrade to 3.3. Another way to handle the second case is to use conditional imports: # mymodule2 mystring = u"äπЖ☃" # Also works in 3.3 or better. # mymodule3 mystring = "äπЖ☃" # main application from __future__ import print_function from future_builtins import * if sys.version < '3': import mymodule2 as mymodule else: import mymodule3 as mymodule print(mymodule.mystring) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Help with pipes, buffering and pseudoterminals
On 05Apr2015 12:20, Daniel Ellis wrote: I have a small little tool I'd like to make. It essentially takes piped input, modifies the text in some way, and immediately prints the output. The problem I'm having is that any output I pipe to the program seems to be buffered, removing the desired effect. That depends on the upstream program; it does the buffering. The pipe itself presents received data downstream immediately. However, as you've seen, almost every program buffers its standard output if the output is not a tty; this is automatic in the stdio C library and results in more fficient use of I/O. From what I understand, I need to somehow have the input be retrieved via a pseudoterminal. This is a rather gross hack, though sometimes all you can do. While some programs have an option to force unbuffered output, most do not. Attaching their output to a pty is one way to encourage them to at least line buffer their output. However, you should bear in mind that the reason that programs line buffer to a terminal is that they presume they are in an interactive situation with a person watching. The program _may_ act differently in other ways as well, such as asking question it might not otherwise ask in "batch" mode (where it might cautiously not ask and presume "no"). Also, output sent through a pty is subject to the line discipline in the terminal; temrinals are funny things with much historical behaviour. At the least you pobably want your pty in "raw" mode to avoid all sorts of stuff that can be done to your data. The problem that I'm having is that most examples on the internet seem to assume I would like to launch a program in a forked pty process, which doesn't really fit my use case. Indeed not, but not to worry. You don't need to fork. I've tried a number of things, but I seem to be unable to get even a basic understanding of how to use the pty module. Have you every used a pty from C? Do you know how ptys work? (master side, slave side, etc). Here's a piece of code I whipped up just to try to get a feel for what is going on when I use pty.fork, but it doesn't seem to do what I think it should: import pty import os import sys pid, fd = pty.fork() print pid, fd sys.stdout.flush() os.read(fd, 1024) This only seems to print from the parent process. The documentation for pty.fork says: Return value is (pid, fd). Note that the child gets pid 0, and the fd is invalid. So the child cannot used "fd". It further says that the child has its stdin and stdout attached to the pty, and that the pty is the child's controlling terminal (this means it is affected by things like "typing" ^C at the pty, etc). I read that I need to do the os.read call for the fork to happen. I've also tried printing *after* the os.read call. Don't try to adapt fork-based tutorials to your needs. Understand ptys directly first. I realize this does very little to solve my overall goal, but I figure understanding what is going on is probably a worthwhile first step. What you probably want to use is pty.openpty() instead. No fork. You will get back file descriptors for the master and slave sides of the pty. Then you can use these with the subprocess module to connect your input program. Or, guessing from your opening sentence, you can write a wrapper script whose whole purpose is to run a program on a pty. Regarding terminology: a pseudoterminal (pty) is a device that looks like a traditional serial terminal. All terminal emulators like xterm use one, and so do other programs presenting a terminal session such as the sshd process handling an interactive remote login. When you call pty.openpty() you are handed two file descriptors: one for the master side of the pty and one for the slave side. The slave side is the side that looks like a terminal, and is what a typical use would connect a child process to. The master side is the other side of the pty. When a program writes to the "slave" side, the output is available for read on the master side, much like a pipe. When a program writes to the master side, the output is available for read on the slave side, _as_ _if_ _typed_ at the terminal. A pty is not necessarily going to solve your problem unless you can get your input via the pty. From the sounds of it you're in this situation: command-generating-output | your-program such that your input is attached to a pipe, and because "command-generating-output" is attached to a pipe it is block buffering its output, hence your problem. You can't undo that situation after the fact. To solve your problem via a pty you need to contrive to set up "command-generating-output" already attached to a pty. One way to do that is for "your-program" to open a pty and itself invoke "command-generating-output" with its output via the pty, which is why so many tutorials suppose a "fork" situation. One typical away to do that is to pass
Re: Help with pipes, buffering and pseudoterminals
On Sun, 05 Apr 2015 12:20:48 -0700, Daniel Ellis wrote:
> This only seems to print from the parent process. I read that I need to
> do the os.read call for the fork to happen. I've also tried printing
> *after* the os.read call.
The child process has its std{in,out,err} attached to the newly-created
pty, so that's where the output from the child's "print" goes.
You'll see that output if the parent prints the string returned from the
os.read() call.
--
https://mail.python.org/mailman/listinfo/python-list
ANN: polynice 0.7 - a nice(1) like utility for throttling processes
polynice is a nice(1)-like command line utility for unix systems to throttle long running processes beyond what can be achieved by nice(1), by repeatedly suspending and resuming the process. It is written for python3, though there is some python2.7 compatibility. Author: Radovan Garabík URL: http://kassiopeia.juls.savba.sk/~garabik/software/polynice.html License: GPL (v2) Notable changes: * This is the first public release under the name `polynice' - the utility has been renamed from `verynice' due to name clash with an existing software. * polynice can now emulate timeout(1) - terminate process after given time has passed * MacOS X improvements * better python2 compatiblity -- --- | Radovan Garabík http://kassiopeia.juls.savba.sk/~garabik/ | | __..--^^^--..__garabik @ kassiopeia.juls.savba.sk | --- Antivirus alert: file .signature infected by signature virus. Hi! I'm a signature virus! Copy me into your signature file to help me spread! -- https://mail.python.org/mailman/listinfo/python-list
Is it possible to deliver different source distributions for different Python versions?
I would like to distribute a python package with different code for Python 2.* than for Python 3.*. (Mostly this is because of different unicode string handling). There is nothing in to setuptools or PyPi that directly supports this scenario. But perhaps there could be some script run at install time that moves the correct source code to the right location? In other works, if I included both source code versions in the distribution (in a src2 and a src3 subdirectory) then a function invoked at install time could detect the python version and copy the appropriate source code to the right location. Is that at all possible? Is there some install time hook that lets me supply custom installation code? -- Dave Hein -- https://mail.python.org/mailman/listinfo/python-list
Re: Permission denied when opening a file that was created concurrently by os.rename (Windows)
On 4/5/2015 1:45 PM, Alexey Izbyshev wrote: Hello! I've hit a strange problem that I reduced to the following test case: * Run several python processes in parallel that spin in the following loop: while True: if os.path.isfile(fname): with open(fname, 'rb') as f: f.read() break Besides anything else, I would consider adding a minimal sleep in the loop. * Then, run another process that creates a temporary file and then renames it to the name than other processes are expecting * Now, some of the reading processes occasionally fail with "Permission denied" OSError -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Help with pipes, buffering and pseudoterminals
I have a small little tool I'd like to make. It essentially takes piped input, modifies the text in some way, and immediately prints the output. The problem I'm having is that any output I pipe to the program seems to be buffered, removing the desired effect. >From what I understand, I need to somehow have the input be retrieved via a >pseudoterminal. The problem that I'm having is that most examples on the >internet seem to assume I would like to launch a program in a forked pty >process, which doesn't really fit my use case. I've tried a number of things, but I seem to be unable to get even a basic understanding of how to use the pty module. Here's a piece of code I whipped up just to try to get a feel for what is going on when I use pty.fork, but it doesn't seem to do what I think it should: import pty import os import sys pid, fd = pty.fork() print pid, fd sys.stdout.flush() os.read(fd, 1024) This only seems to print from the parent process. I read that I need to do the os.read call for the fork to happen. I've also tried printing *after* the os.read call. I realize this does very little to solve my overall goal, but I figure understanding what is going on is probably a worthwhile first step. -- https://mail.python.org/mailman/listinfo/python-list
Re: Permission denied when opening a file that was created concurrently by os.rename (Windows)
On 04/05/2015 01:45 PM, Alexey Izbyshev wrote: Hello! I've hit a strange problem that I reduced to the following test case: * Run several python processes in parallel that spin in the following loop: while True: if os.path.isfile(fname): with open(fname, 'rb') as f: f.read() break * Then, run another process that creates a temporary file and then renames it to the name than other processes are expecting * Now, some of the reading processes occasionally fail with "Permission denied" OSError I was able to reproduce it on two Windows 7 64-bit machines. It seems when the file appears on the filesystem it is still unavailable to reading, but I have no idea how it can happen. Both source and destination files are in the same directory, and the destination doesn't exist before calling os.rename. Everything I could find indicates that os.rename should be atomic under this conditions even on Windows, so nobody should be able to observe the destination in unaccessible state. I know that I can workaround this problem by removing useless os.path.isfile() check and wrapping open() with try-except, but I'd like to know the root cause of the problem. Please share you thoughts. The test case is attached, the main file is test.bat. Python is expected to be in PATH. Stderr of readers is redirected to *.log. You may need to run several times to hit the issue. Alexey Izbyshev, research assistant, ISP RAS The attachment is missing; please just include it inline, after reducing it to a reasonably minimal sample. My guess is that the process that does the os.rename is not closing the original file before renaming it. So even though the rename is atomic, the file is still locked by the first process. -- DaveA -- https://mail.python.org/mailman/listinfo/python-list
Re: Permission denied when opening a file that was created concurrently by os.rename (Windows)
On Mon, Apr 6, 2015 at 3:45 AM, Alexey Izbyshev wrote: > The test case is attached, the main file is test.bat. Python is expected to > be in PATH. Stderr of readers is redirected to *.log. You may need to run > several times to hit the issue. You have an interesting-looking problem, but the attachment didn't arrive. Is it short enough to include in-line as text in your email? ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Permission denied when opening a file that was created concurrently by os.rename (Windows)
Hello! I've hit a strange problem that I reduced to the following test case: * Run several python processes in parallel that spin in the following loop: while True: if os.path.isfile(fname): with open(fname, 'rb') as f: f.read() break * Then, run another process that creates a temporary file and then renames it to the name than other processes are expecting * Now, some of the reading processes occasionally fail with "Permission denied" OSError I was able to reproduce it on two Windows 7 64-bit machines. It seems when the file appears on the filesystem it is still unavailable to reading, but I have no idea how it can happen. Both source and destination files are in the same directory, and the destination doesn't exist before calling os.rename. Everything I could find indicates that os.rename should be atomic under this conditions even on Windows, so nobody should be able to observe the destination in unaccessible state. I know that I can workaround this problem by removing useless os.path.isfile() check and wrapping open() with try-except, but I'd like to know the root cause of the problem. Please share you thoughts. The test case is attached, the main file is test.bat. Python is expected to be in PATH. Stderr of readers is redirected to *.log. You may need to run several times to hit the issue. Alexey Izbyshev, research assistant, ISP RAS -- https://mail.python.org/mailman/listinfo/python-list
