Re: improving performance of writing into a pipe
On 20 February 2013 17:54, wrote:
> On Tuesday, February 19, 2013 5:47:16 PM UTC, Michael Torrie wrote:
>> On 02/19/2013 02:24 AM, mikp...@gmail.com wrote:
>>
>> > Or rather: what would you try to catch in this particular case?
>>
>>
>> As Peter said, nothing for now. But you seem very resistant to telling
>>
>> us what exception was raised.
>
>
> Michael believe me:
> I am not resistant or try to hide anything!
> As written before, I don't know what exception to search for, so I wrote the
> (wrong) code:
> except:
> print "error"
> Let's why I don't have a clue about it.
> But someone already explained me that I should not do this.
You don't need to look for errors. If you remove the try/except then
they show up automatically. For example (in the interpreter):
$ python
Python 2.7.3 (default, Sep 26 2012, 21:51:14)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> open('Desktop')
Traceback (most recent call last):
File "", line 1, in
IOError: [Errno 21] Is a directory: 'Desktop'
The last three lines above are the error message that people were
expecting you to show here. They contains lots of useful information:
1) The type of the error
2) A message "Is a directory" and in this case a cryptic code 21 (that
some might find useful).
3) The line of code that caused the error (this is more useful when
running code saved in a file).
What you are doing, however, is this:
>>> try:
... open('Desktop')
... except:
... print('An error occurred...')
...
An error occurred...
Which gives a much less useful error message. So just remove the try/except.
Oscar
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
In mikp...@gmail.com writes: > As written before, I don't know what exception to search for, so I wrote > the (wrong) code: > except: > print "error" > Let's why I don't have a clue about it. > But someone already explained me that I should not do this. If you don't know what exception is being raised, temporarily remove the try/except statements and run the code directly. You'll get the exception, and then you'll know which one it is. -- John Gordon A is for Amy, who fell down the stairs gor...@panix.com B is for Basil, assaulted by bears -- Edward Gorey, "The Gashlycrumb Tinies" -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On Tuesday, February 19, 2013 5:47:16 PM UTC, Michael Torrie wrote: > On 02/19/2013 02:24 AM, mikp...@gmail.com wrote: > > > Or rather: what would you try to catch in this particular case? > > > As Peter said, nothing for now. But you seem very resistant to telling > > us what exception was raised. Michael believe me: I am not resistant or try to hide anything! As written before, I don't know what exception to search for, so I wrote the (wrong) code: except: print "error" Let's why I don't have a clue about it. But someone already explained me that I should not do this. > > Though looking at your code more closely I can see that likely the error > > is related to the fact that /tmp/mypipe is not an executable program. Yes it is and has rwx permissions. Unfortunately I don't have access to the code in the pipe. > > popen (which is deprecated and replaced by the subprocess module) is for > > running programs and communicating with them over pipes created by the > > popen function. So your code is not likely to ever work as it is > > presently given. > > > > Here's the bash equivalent of your code: > > > > $ mkfifo /tmp/path > > $ cat > $ echo hello, world | /tmp/path > > > > Bash will say, "bash: /tmp/path: Permission denied" > > > > The correct bash line is: > > $ echo hello, world > /tmp/path > > > > popen() (and subprocess) is the equivalent of the first bash command. > > open() is the equivalent of the second line. > > Do you understand the difference? I think I do now, thanks. mik -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On 02/19/2013 02:24 AM, mikp...@gmail.com wrote: > Or rather: what would you try to catch in this particular case? As Peter said, nothing for now. But you seem very resistant to telling us what exception was raised. Though looking at your code more closely I can see that likely the error is related to the fact that /tmp/mypipe is not an executable program. popen (which is deprecated and replaced by the subprocess module) is for running programs and communicating with them over pipes created by the popen function. So your code is not likely to ever work as it is presently given. Here's the bash equivalent of your code: $ mkfifo /tmp/path $ cat /tmp/path popen() (and subprocess) is the equivalent of the first bash command. open() is the equivalent of the second line. Do you understand the difference? -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
> > Thanks a lot Serhiy to you and to everyone else. > > > Do you mind telling us what fix you applied? Oh, apologies Peter, I thought it was clear as I posted it after the lines written by Serhiy. So it was what Serhiy suggest in addition to some (?minor?) modification to the pipe itself, which I cannot comment about as I don't have access to it. But apparently the problem of not being able to open it was due to it. (It does not help much I am afraid... but that's all I know). Mik -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
mikp...@gmail.com wrote:
>> def write_to_pipe(line):
>> hexbytes = ''.join('\\x%02x' % ord(c) for c in line)
I thought this was only needed to have 'echo' except your data.
>> with open('/tmp/mypipe', 'w') as f:
>> f.write(hexbytes)
> Update:
> with a fix in the pipe THIS was the right way to do it, and it now works.
> Thanks a lot Serhiy to you and to everyone else.
Do you mind telling us what fix you applied?
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
>
> def write_to_pipe(line):
>
> hexbytes = ''.join('\\x%02x' % ord(c) for c in line)
>
> with open('/tmp/mypipe', 'w') as f:
>
> f.write(hexbytes)
Update:
with a fix in the pipe THIS was the right way to do it, and it now works.
Thanks a lot Serhiy to you and to everyone else.
Mik
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On Monday, February 18, 2013 7:29:09 PM UTC, Serhiy Storchaka wrote:
> On 18.02.13 17:12, mikp...@gmail.com wrote:
>
> > on an embedded linux system (BeagleBoard) I am writing data coming from
> > bluetooth dongle into a pipe.
>
> > The function is the following one:
>
> >
>
> >
>
> > def write_to_pipe(line):
>
> >
>
> > # next line ensures that bytes like '0x09' are not translated into
> > '\t' for
>
> > #example, and they are sent as such
>
> > hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])
>
> > wrap = ["echo -en '", "' > /tmp/mypipe"]
>
> > msg = hexbytes.join(wrap)
>
> > print "DBG: sending: ", msg
>
> >
>
> > try:
>
> > os.popen( msg )
>
> > except:
>
> > print "Error: write_to_pipe has failed!"
>
> >
>
> >
>
> > Now I typically receive 4 bytes from the bluetooth dongle and that is fine.
>
> > However when I receive many more than that it seems that the writing into
> > the pipe is too slow.
>
> >
>
> > Is there any clever/obvious way to improve the code above?
>
> > (I am quite sure there is to be honest).
>
>
>
> def write_to_pipe(line):
>
> hexbytes = ''.join('\\x%02x' % ord(c) for c in line)
>
> with open('/tmp/mypipe', 'w') as f:
>
> f.write(hexbytes)
I'll take your hexbytes = '' line (which is surely more efficient than mine).
However whit this approach open + write it seems the pipe doesn't get the
data...
I am not sure what is going on.
At this point I suspect it could be a problem on the pipe itself (which I
inherited).
It is just weird that the pipe accept this correctly:
wrap = ["echo -en '", "' > /tmp/midi"]
msg = hexbytes.join(wrap)
os.popen( msg )
but seems to be careless of approach open + write.
I need to investigate there.
Thanks a lot, to you and to everyone else.
Mik
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On 19 February 2013 10:27, wrote: >> can work. As a few people already told you the built-in open() > > > Few people? > I thought Oscar was a singular person, not a group of people :-) Serhiy also suggested it. > Seriously, I am convinced by that approach (thanks) and I wish to go that > way, but the problem I am getting now is that the open fails and then I can't > go on. Perhaps no-one has been explicit enough about what you should do here: 1) Remove all try/except from your code. 2) Run the code 3) Look at the *unadulterated* error message that Python prints out 4) Either fix the error if you know how or 5) Reply here posting the exact error message. Also, in future: 6) Don't use bare try/except and don't catch errors while you're debugging. Allow the errors to be printed as they are so that you can read the message and see the line that triggers the error. 7) Don't post to a mailing list saying "I get an error", "I can see errors" or "it doesn't work". If you have errors paste the exact error message (all of it!). If you don't get errors but it doesn't do what you want explain exactly what happened and also what you wanted to happen. If you had followed this procedure at the start of this thread, then you would already have a solution to (or at least an explanation of) your problem by now. > Also, I am now looking at the subprocess as os.popen seems deprecated. > Any opinion on that? I'm not convinced that either is appropriate for your problem. Oscar -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
> > Once you get your script working you can try to provoke errors, and for > > those errors you can recover from you can write error handlers. For IOError > > and Python < 3.3 that may involve inspecting the errno attribute and > > conditionally reraising. Ok. > By the way, I don't think > > > >> > PIPEPATH = ["/tmp/mypipe"] > > >> > self.process = os.popen( self.PIPEPATH, 'w') > > > > can work. As a few people already told you the built-in open() Few people? I thought Oscar was a singular person, not a group of people :-) Seriously, I am convinced by that approach (thanks) and I wish to go that way, but the problem I am getting now is that the open fails and then I can't go on. Also, I am now looking at the subprocess as os.popen seems deprecated. Any opinion on that? Thanks for your suggestion. Mik -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
mikp...@gmail.com wrote: > On Monday, February 18, 2013 6:12:01 PM UTC, Michael Torrie wrote: >> On 02/18/2013 10:00 AM, mikp...@gmail.com wrote: >> >> > [..] >> >> >> >> >> >> I don't see an exception in your answer. Where did you put it for us? >> >> >> >> >> > >> >> > well I just did print a message: >> >> > >> >> > PIPEPATH = ["/tmp/mypipe"] >> >> > >> >> > [..] >> >> > try: >> >> > self.process = os.popen( self.PIPEPATH, 'w') >> >> > except: >> >> > print "Error while trying opening the pipe!" >> >> > print "check: ", self.PIPEPATH >> >> > exit() >> >> > >> >> > I see the error messages. >> >> >> >> Unfortunately your attempt to catch this exception is hiding the true >> >> cause. You need to give us the actual exception. Otherwise it could be >> >> anything from self.PIPEPATH not existing to who knows what. >> >> >> >> Almost never do you want to catch all exceptions like you're doing. You >> >> should only catch the specific exceptions you know how to deal with in >> >> your code. >> >> >> >> For testing purposes, if your code really is as you put it, then >> catching exceptions is kind of silly since you're just re-raising the >> exception (sort of) but without any contextual information that would >> make the error meaningful. > > > Ok, I get your point. > But on the other hand how do I know what to catch if I have no clue what > is causing the error? There must be a way to catch all the possible errors > and then investigate what is the problem, right? (which is not what I have > done so far). > > Or rather: what would you try to catch in this particular case? Nothing. Once you get your script working you can try to provoke errors, and for those errors you can recover from you can write error handlers. For IOError and Python < 3.3 that may involve inspecting the errno attribute and conditionally reraising. By the way, I don't think >> > PIPEPATH = ["/tmp/mypipe"] >> > self.process = os.popen( self.PIPEPATH, 'w') can work. As a few people already told you the built-in open() with open(PIPEPATH, "w") as f: f.write(...) is the way to go. -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On Monday, February 18, 2013 6:12:01 PM UTC, Michael Torrie wrote: > On 02/18/2013 10:00 AM, mikp...@gmail.com wrote: > > > [..] > > >> > > >> I don't see an exception in your answer. Where did you put it for us? > > >> > > > > > > well I just did print a message: > > > > > > PIPEPATH = ["/tmp/mypipe"] > > > > > > [..] > > > try: > > > self.process = os.popen( self.PIPEPATH, 'w') > > > except: > > > print "Error while trying opening the pipe!" > > > print "check: ", self.PIPEPATH > > > exit() > > > > > > I see the error messages. > > > > Unfortunately your attempt to catch this exception is hiding the true > > cause. You need to give us the actual exception. Otherwise it could be > > anything from self.PIPEPATH not existing to who knows what. > > > > Almost never do you want to catch all exceptions like you're doing. You > > should only catch the specific exceptions you know how to deal with in > > your code. > > > > For testing purposes, if your code really is as you put it, then > catching exceptions is kind of silly since you're just re-raising the > exception (sort of) but without any contextual information that would > make the error meaningful. Ok, I get your point. But on the other hand how do I know what to catch if I have no clue what is causing the error? There must be a way to catch all the possible errors and then investigate what is the problem, right? (which is not what I have done so far). Or rather: what would you try to catch in this particular case? Thanks -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On 18.02.13 17:12, mikp...@gmail.com wrote:
on an embedded linux system (BeagleBoard) I am writing data coming from
bluetooth dongle into a pipe.
The function is the following one:
def write_to_pipe(line):
# next line ensures that bytes like '0x09' are not translated into '\t' for
#example, and they are sent as such
hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])
wrap = ["echo -en '", "' > /tmp/mypipe"]
msg = hexbytes.join(wrap)
print "DBG: sending: ", msg
try:
os.popen( msg )
except:
print "Error: write_to_pipe has failed!"
Now I typically receive 4 bytes from the bluetooth dongle and that is fine.
However when I receive many more than that it seems that the writing into the
pipe is too slow.
Is there any clever/obvious way to improve the code above?
(I am quite sure there is to be honest).
def write_to_pipe(line):
hexbytes = ''.join('\\x%02x' % ord(c) for c in line)
with open('/tmp/mypipe', 'w') as f:
f.write(hexbytes)
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On 02/18/2013 10:00 AM, mikp...@gmail.com wrote: > [..] >> >> I don't see an exception in your answer. Where did you put it for us? >> > > well I just did print a message: > > PIPEPATH = ["/tmp/mypipe"] > > [..] > try: > self.process = os.popen( self.PIPEPATH, 'w') > except: > print "Error while trying opening the pipe!" > print "check: ", self.PIPEPATH > exit() > > I see the error messages. Unfortunately your attempt to catch this exception is hiding the true cause. You need to give us the actual exception. Otherwise it could be anything from self.PIPEPATH not existing to who knows what. Almost never do you want to catch all exceptions like you're doing. You should only catch the specific exceptions you know how to deal with in your code. For testing purposes, if your code really is as you put it, then catching exceptions is kind of silly since you're just re-raising the exception (sort of) but without any contextual information that would make the error meaningful. -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
[..] > > I don't see an exception in your answer. Where did you put it for us? > well I just did print a message: PIPEPATH = ["/tmp/mypipe"] [..] try: self.process = os.popen( self.PIPEPATH, 'w') except: print "Error while trying opening the pipe!" print "check: ", self.PIPEPATH exit() I see the error messages. It's quite frustrating as I think I am doing something really stupid in here. Mik -- http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
Am 18.02.2013 17:31 schrieb mikp...@gmail.com:
However I get an exception while trying to open the queue:
fout = open('/tmp/mypipe', 'w')
I don't see an exception in your answer. Where did you put it for us?
I have tried it in a command line and the call doesn't return until in another
terminal I open the same queue for reading (???)
That's normal. An open call to a pipe blocks until someone reads from it.
But it's the same with your popen() version.
Thomas
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On Monday, February 18, 2013 3:21:53 PM UTC, Oscar Benjamin wrote:
[..]
>
> Can you not open the pipe file directly in Python code? e.g.
>
>
>
> fout = open('/tmp/mypipe', 'w')
>
> fout.write(data)
>
>
>
> I guess that this would be more efficient than using os.popen to run echo.
>
>
that's an idea,
thanks Oscar.
However I get an exception while trying to open the queue:
fout = open('/tmp/mypipe', 'w')
I have tried it in a command line and the call doesn't return until in another
terminal I open the same queue for reading (???)
I have created the queue with:
mkfifo /tmp/mypipe
any clue?
mik
--
http://mail.python.org/mailman/listinfo/python-list
Re: improving performance of writing into a pipe
On 18 February 2013 15:12, wrote:
> Hi guys,
>
> on an embedded linux system (BeagleBoard) I am writing data coming from
> bluetooth dongle into a pipe.
> The function is the following one:
>
>
> def write_to_pipe(line):
>
> # next line ensures that bytes like '0x09' are not translated into '\t'
> for
> #example, and they are sent as such
> hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line])
> wrap = ["echo -en '", "' > /tmp/mypipe"]
> msg = hexbytes.join(wrap)
> print "DBG: sending: ", msg
>
> try:
> os.popen( msg )
> except:
> print "Error: write_to_pipe has failed!"
>
>
> Now I typically receive 4 bytes from the bluetooth dongle and that is fine.
> However when I receive many more than that it seems that the writing into the
> pipe is too slow.
>
> Is there any clever/obvious way to improve the code above?
> (I am quite sure there is to be honest).
Can you not open the pipe file directly in Python code? e.g.
fout = open('/tmp/mypipe', 'w')
fout.write(data)
I guess that this would be more efficient than using os.popen to run echo.
Oscar
--
http://mail.python.org/mailman/listinfo/python-list
improving performance of writing into a pipe
Hi guys, on an embedded linux system (BeagleBoard) I am writing data coming from bluetooth dongle into a pipe. The function is the following one: def write_to_pipe(line): # next line ensures that bytes like '0x09' are not translated into '\t' for #example, and they are sent as such hexbytes = "\\x" + "\\x".join([hex(ord(c))[2:].zfill(2) for c in line]) wrap = ["echo -en '", "' > /tmp/mypipe"] msg = hexbytes.join(wrap) print "DBG: sending: ", msg try: os.popen( msg ) except: print "Error: write_to_pipe has failed!" Now I typically receive 4 bytes from the bluetooth dongle and that is fine. However when I receive many more than that it seems that the writing into the pipe is too slow. Is there any clever/obvious way to improve the code above? (I am quite sure there is to be honest). Thanks for any suggestion! Mik -- http://mail.python.org/mailman/listinfo/python-list
