subject:"python for loop"

try to make my triple nested loop working. My code would be:
c = 4
y1 = []
m1 = []
std1 = []
while c 24:
c = c + 1
a = []
f.seek(0,0)
for columns in ( raw.strip().split() for raw in f ):
a.append(columns[c])
x = np.array(a, float)
not_nan = np.logical_not(np.isnan(x))
indices = np.arange(len(x))
interp = interp1d(indices[not_nan], x[not_nan], kind = 'nearest')
p = interp(indices)

N = len(p)
dt = 900.0 #Time step (seconds)
fs = 1./dt #Sampling frequency
KA,PSD = oned_Fourierspectrum(p,dt) # Call Song's 1D FS function
time_axis = np.linspace(0.0,N,num = N,endpoint = False)*15/(60*24) 
plot_freq = 24*3600.*KA #Convert to cycles per day 
plot_period = 1.0/plot_freq # convert to days/cycle
fpsd = plot_freq*PSD
d = -1 
while d 335: 
d = d + 1 
y = fpsd[d] 
y1 = y1 + [y]   
   m = np.mean(y1)
m1 = m1 + [m]
print m1


My purpose is make a list of [mean(fpsd[0]), mean(fpsd[1]), mean(fpsd[2]).. 
mean(fpsd[335])]. Each y1 would be the list of fpsd[d].

I check it is working pretty well before second while loop and I can get 
individual mean of fpsd[d]. However, with second whole loop, it produces 
definitely wrong numbers. Would you help me this problem?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Triple nested loop python (While loop insde of for loop inside of while loop)

2013-03-01 Thread Ulrich Eckhardt


Am 01.03.2013 09:59, schrieb Isaac Won:

try to make my triple nested loop working. My code would be:
c = 4

[...]

while c 24:
 c = c + 1


This is bad style and you shouldn't do that in python. The question that 
comes up for me is whether something else is modifying c in that loop, 
but I think the answer is no. For that reason, use Python's way:


  for c in range(5, 25):
  ...

That way it is also clear that the first value in the loop is 5, while 
the initial c = 4 seems to suggest something different. Also, the last 
value is 24, not 23.





 while d 335:
 d = d + 1
 y = fpsd[d]
 y1 = y1 + [y]
m = np.mean(y1)
 m1 = m1 + [m]


Apart from the wrong indention (don't mix tabs and spaces, see PEP 8!) 
and the that d in range(336) is better style, you don't start with an 
empty y1, except on the first iteration of the outer loop.


I'm not really sure if that answers your problem. In any case, please 
drop everything not necessary to demostrate the problem before posting. 
This makes it easier to see what is going wrong both for you and others. 
Also make sure that others can actually run the code.



Greetings from Hamburg!

Uli



--
http://mail.python.org/mailman/listinfo/python-list

Re: Triple nested loop python (While loop insde of for loop inside of while loop)

2013-03-01 Thread Joel Goldstick

On Fri, Mar 1, 2013 at 7:00 AM, Ulrich Eckhardt 
ulrich.eckha...@dominolaser.com wrote:

 Am 01.03.2013 09:59, schrieb Isaac Won:

  try to make my triple nested loop working. My code would be:
 c = 4

 [...]

  while c 24:
  c = c + 1


 This is bad style and you shouldn't do that in python. The question that
 comes up for me is whether something else is modifying c in that loop,
 but I think the answer is no. For that reason, use Python's way:

   for c in range(5, 25):
   ...

 That way it is also clear that the first value in the loop is 5, while the
 initial c = 4 seems to suggest something different. Also, the last value
 is 24, not 23.



 I concur with Uli, and add the following thoughts:  What is going on with
[y]?  Is this really a list?  So what is y1 + y1 + [y] doing?


   while d 335:
  d = d + 1
  y = fpsd[d]
  y1 = y1 + [y]
 m = np.mean(y1)
  m1 = m1 + [m]


 In your outer loop you initialize these values each pass:

dt = 900.0 #Time step (seconds)
fs = 1./dt #Sampling frequency

This should me moved out of the loop since nothing changes with dt or fs



 Apart from the wrong indention (don't mix tabs and spaces, see PEP 8!) and
 the that d in range(336) is better style, you don't start with an empty
 y1, except on the first iteration of the outer loop.

 I'm not really sure if that answers your problem. In any case, please drop
 everything not necessary to demostrate the problem before posting. This
 makes it easier to see what is going wrong both for you and others. Also
 make sure that others can actually run the code.


 Greetings from Hamburg!

 Uli



 --
 http://mail.python.org/**mailman/listinfo/python-listhttp://mail.python.org/mailman/listinfo/python-list




-- 
Joel Goldstick
http://joelgoldstick.com
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Triple nested loop python (While loop insde of for loop inside of while loop)

2013-03-01 Thread Chris Angelico

On Fri, Mar 1, 2013 at 7:59 PM, Isaac Won winef...@gmail.com wrote:
 while c 24:
 for columns in ( raw.strip().split() for raw in f ):
 while d 335:

Note your indentation levels: the code does not agree with your
subject line. The third loop is not actually inside your second.
Should it be?

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Triple nested loop python (While loop insde of for loop inside of while loop)

Thank you, Chris.
I just want to acculate value from y repeatedly.
If y = 1,2,3...10, just have a [1,2,3...10] at onece.
On Friday, March 1, 2013 7:41:05 AM UTC-6, Chris Angelico wrote:
 On Fri, Mar 1, 2013 at 7:59 PM, Isaac Won winef...@gmail.com wrote:
 
  while c 24:
 
  for columns in ( raw.strip().split() for raw in f ):
 
  while d 335:
 
 
 
 Note your indentation levels: the code does not agree with your
 
 subject line. The third loop is not actually inside your second.
 
 Should it be?
 
 
 
 ChrisA

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Triple nested loop python (While loop insde of for loop inside of while loop)

On Friday, March 1, 2013 7:41:05 AM UTC-6, Chris Angelico wrote:
 On Fri, Mar 1, 2013 at 7:59 PM, Isaac Won winef...@gmail.com wrote:
 
  while c 24:
 
  for columns in ( raw.strip().split() for raw in f ):
 
  while d 335:
 
 
 
 Note your indentation levels: the code does not agree with your
 
 subject line. The third loop is not actually inside your second.
 
 Should it be?
 
 
 
 ChrisA

Yes, the thiird lood should be inside of my whole loop.
Thank you,
Isaac
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Triple nested loop python (While loop insde of for loop inside of while loop)

Thank you Ulich for reply,
What I really want to get from this code is m1 as I told. For this purpose, for 
instance, values of fpsd upto second loop and that from third loop should be 
same, but they are not. Actually it is my main question.
Thank you,
Isaac
On Friday, March 1, 2013 6:00:42 AM UTC-6, Ulrich Eckhardt wrote:
 Am 01.03.2013 09:59, schrieb Isaac Won:
 
  try to make my triple nested loop working. My code would be:
 
  c = 4
 
 [...]
 
  while c 24:
 
   c = c + 1
 
 
 
 This is bad style and you shouldn't do that in python. The question that 
 
 comes up for me is whether something else is modifying c in that loop, 
 
 but I think the answer is no. For that reason, use Python's way:
 
 
 
for c in range(5, 25):
 
...
 
 
 
 That way it is also clear that the first value in the loop is 5, while 
 
 the initial c = 4 seems to suggest something different. Also, the last 
 
 value is 24, not 23.
 
 
 
 
 
 
 
   while d 335:
 
   d = d + 1
 
   y = fpsd[d]
 
   y1 = y1 + [y]
 
  m = np.mean(y1)
 
   m1 = m1 + [m]
 
 
 
 Apart from the wrong indention (don't mix tabs and spaces, see PEP 8!) 
 
 and the that d in range(336) is better style, you don't start with an 
 
 empty y1, except on the first iteration of the outer loop.
 
 
 
 I'm not really sure if that answers your problem. In any case, please 
 
 drop everything not necessary to demostrate the problem before posting. 
 
 This makes it easier to see what is going wrong both for you and others. 
 
 Also make sure that others can actually run the code.
 
 
 
 
 
 Greetings from Hamburg!
 
 
 
 Uli

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

Hi Roy,

On Feb 11, 2013, at 10:24 AM, Roy Smith r...@panix.com wrote:
 
 Is this server that you're talking to something that you have control 
 over, i.e. are you stuck with this protocol?  Given a choice, I'd go 
 with something like JSON, for which pre-existing libraries for every 
 language under the sun.
 
I'm running JSON for my application messaging protocol but with JSON and python 
default unordered dict,
there's no guarantee if I put in the length key in the JSON message, it will be 
placed on the first bytes hence
why it was designed for a fixed 4-byte at the start of the message to indicate 
the message length.

Beyond the 4-bytes it is all JSON.

but if you have a better idea, i would certainly welcome it.

 Do you actually *know* what the value of nbuf is?  Is it possible that 
 (somehow) it's 0?  You should print (log, whatever), the value of nbuf, 
 just to make sure.

nbuf is printing the right bytes amount, I removed the print statement before I 
made the first post.

So to clarify, I added a print statement between the first recv and the second.

{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 
1.0, data: {1: igb0, 2: igb1, ifcnt: 2}}
connected to misty:8080
sending data
138 bytes sent: 0x86{msgver: 1.0, msgid: 200, subcode: 100, 
appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 
2}}
receiving data
message length is 188
0 bytes received:

So the subsequent recv() call will be readjusted with 188 bytes buffer size so 
theoretically, recv shouldn't return 0.

The same logic that I used to send to the server from the python client that 
the server will readjust the second recv() call based on the length 
information. On this 2nd recv() call the server is able to obtain the rest of 
the messages.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

Hi Dave,

On Feb 11, 2013, at 9:22 AM, Dave Angel da...@davea.name wrote:

 Exactly how are you sending hexadecimal ?  If that 0xad (which is only one 
 byte, what about the other 3 ?) is intended to be a C description, then it's 
 certainly not hex, it's binary.  And probably little-endian, to boot.  That's 
 a mistake, as network protocols almost always use big-endian.
 
 So what is the range of values for the length, and how are they actually 
 encoded?  Are they uint32 in native binary format?   And are you permitted to 
 change the encoding, to fix it?  (If it were my choice, I'd make it a 
 printable decimal value, first choice, or printable hex, second choice.)

They are ASCII stream, actually JSON with the exception of the first 4 bytes. I 
avoided using struct class for this as it's overkill for my application.

So the idea is that i code the length to a max of 0xff (max length of 256 
bytes), I would only have to assign 4 bytes to it. If i need 1k length, i just 
need to increase it to 6 bytes or 4 if i decide to strip the 0x.

the code for the function is as follows:

def request_get_session(sock, jmsg):
# append message length
plen = hex(len(jmsg))
msg = '{0}{1}'.format(plen, jmsg)

print 'sending data'
n = sock.send(msg)
str = '{0} bytes sent: {1}'.format(n, msg)
print str

# receive message length
print 'receiving data'
mlen = sock.recv(4)
try:
nbuf = int(mlen, 16)
except ValueError as e:
print 'invalid length type'
return -1

print 'message length is {0}'.format(nbuf)

while True:
buf = sock.recv(nbuf)

if not buf:
break

slen = len(buf)
str = {0} bytes received: {1}.format(slen, buf)
print str
return 0
 
 
 I've managed to receive and translate the message length until I reach my 
 second recv which I readjusted the buffer size to include the new message 
 length.
 
 However that failed and recv received 0 bytes. I implemented the same 
 algorithm on the server side using C and it work so appreciate if you can 
 help me on this.
 
 # receive message length
 print 'receiving data'
 mlen = sock.recv(4)
 try:
 nbuf = int(mlen, 16)
 
 That supposes the count is being sent as a printable string of hex digits.  
 That's not what I concluded above.

That is what I meant. it just an ascii string.

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

Hi MRAB,

My code now works thanks to your advice.

{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 
1.0, data: {1: igb0, 2: igb1, ifcnt: 2}}
connected to misty:8080
sending data
138 bytes sent: 0x86{msgver: 1.0, msgid: 200, subcode: 100, 
appver: 1.0, appid: 1.0, data: {1: igb0, 2: igb1, ifcnt: 
2}}
receiving data
message length is 188
188 bytes received: { msgver: 1.00, appver: 1.00, appid: 1000, 
msgid: 10, subcode: 20, data: [ 110.159.183.16, 124.108.16.94, 
2400:3700:50::2, 2400:3700:50::1, 2400:3700:51:: ] }

Many thanks.

On Feb 11, 2013, at 9:55 AM, MRAB pyt...@mrabarnett.plus.com wrote:

 You should keep reading until you get all need or the connection is
 closed:
 
buf = b''
while len(buf)  nbuf:
chunk = sock.recv(nbuf - len(buf))
 
if not chunk:
break
 
buf += chunk
 
 -- 
 http://mail.python.org/mailman/listinfo/python-list

-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-11 Thread MRAB

On 2013-02-11 14:56, Ihsan Junaidi Ibrahim wrote:

Hi Roy,

On Feb 11, 2013, at 10:24 AM, Roy Smith r...@panix.com wrote:

Is this server that you're talking to something that you have control
over, i.e. are you stuck with this protocol? Given a choice, I'd go
with something like JSON, for which pre-existing libraries for every
language under the sun.

I'm running JSON for my application messaging protocol but with JSON and python
default unordered dict,
there's no guarantee if I put in the length key in the JSON message, it will be
placed on the first bytes hence
why it was designed for a fixed 4-byte at the start of the message to indicate
the message length.

Beyond the 4-bytes it is all JSON.

but if you have a better idea, i would certainly welcome it.

I probably wouldn't make it fixed length. I'd have the length in
decimal followed by, say, \n.

Do you actually *know* what the value of nbuf is? Is it possible that
(somehow) it's 0? You should print (log, whatever), the value of nbuf,
just to make sure.

nbuf is printing the right bytes amount, I removed the print statement before I
made the first post.

So to clarify, I added a print statement between the first recv and the second.

{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1: igb0,
2: igb1, ifcnt: 2}}
connected to misty:8080
sending data
138 bytes sent: 0x86{msgver: 1.0, msgid: 200, subcode: 100, appver: 1.0, appid: 1.0, data: {1:
igb0, 2: igb1, ifcnt: 2}}
receiving data
message length is 188
0 bytes received:

So the subsequent recv() call will be readjusted with 188 bytes buffer size so
theoretically, recv shouldn't return 0.

The same logic that I used to send to the server from the python client that
the server will readjust the second recv() call based on the length
information. On this 2nd recv() call the server is able to obtain the rest of
the messages.

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-11 Thread Chris Angelico

On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote:
 I probably wouldn't make it fixed length. I'd have the length in
 decimal followed by, say, \n.

Or even followed by any non-digit. Chances are your JSON data begins
with a non-digit, so you'd just have to insert a space in the event
that you're JSON-encoding a flat integer. (Which might not ever
happen, if you know that your data will always be an object.)

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-11 Thread Dave Angel


On 02/11/2013 10:02 AM, Ihsan Junaidi Ibrahim wrote:


  snip

 print 'message length is {0}'.format(nbuf)

 while True:
 buf = sock.recv(nbuf)

 if not buf:
 break


This loop doesn't terminate till buf is zero length, which it will be 
eventually.  At that point, you've overwritten the real data you may 
have gotten.  So the loop is just plain wrong.


Uwe MRAB's code, since there's no promise that all the data will be 
returned in a single call.  Keep accumulating it as you loop.





 slen = len(buf)
 str = {0} bytes received: {1}.format(slen, buf)
 print str
 return 0





--
DaveA
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop


On Feb 11, 2013, at 11:24 PM, Chris Angelico ros...@gmail.com wrote:

 On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote:
 I probably wouldn't make it fixed length. I'd have the length in
 decimal followed by, say, \n.
 
 Or even followed by any non-digit. Chances are your JSON data begins
 with a non-digit, so you'd just have to insert a space in the event
 that you're JSON-encoding a flat integer. (Which might not ever
 happen, if you know that your data will always be an object.)
 
 ChrisA

So on the first recv() call, I set the buffer at 1 character and I iterate over 
single character until a non-digit character
is encountered?



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-11 Thread Chris Angelico

On Tue, Feb 12, 2013 at 12:41 PM, Ihsan Junaidi Ibrahim ih...@grep.my wrote:

 On Feb 11, 2013, at 11:24 PM, Chris Angelico ros...@gmail.com wrote:

 On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote:
 I probably wouldn't make it fixed length. I'd have the length in
 decimal followed by, say, \n.

 Or even followed by any non-digit. Chances are your JSON data begins
 with a non-digit, so you'd just have to insert a space in the event
 that you're JSON-encoding a flat integer. (Which might not ever
 happen, if you know that your data will always be an object.)

 ChrisA

 So on the first recv() call, I set the buffer at 1 character and I iterate 
 over single character until a non-digit character
 is encountered?

More efficient would be to guess that it'll be, say, 10 bytes, and
then retain any excess for your JSON read loop. But you'd need to sort
that out between the halves of your code.

ChrisA
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-11 Thread Roy Smith

In article mailman.1655.1360594595.2939.python-l...@python.org,
 Ihsan Junaidi Ibrahim ih...@grep.my wrote:

 I'm running JSON for my application messaging protocol but with JSON and 
 python default unordered dict,
 there's no guarantee if I put in the length key in the JSON message, it will 
 be placed on the first bytes hence
 why it was designed for a fixed 4-byte at the start of the message to 
 indicate the message length.
 
 Beyond the 4-bytes it is all JSON.

I'm confused.  It sounds like you're making things way more complicated 
than they have to be.  Can you give us an example of an actual data 
message?
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-11 Thread MRAB


On 2013-02-12 02:20, Chris Angelico wrote:

On Tue, Feb 12, 2013 at 12:41 PM, Ihsan Junaidi Ibrahim ih...@grep.my wrote:


On Feb 11, 2013, at 11:24 PM, Chris Angelico ros...@gmail.com wrote:


On Tue, Feb 12, 2013 at 2:11 AM, MRAB pyt...@mrabarnett.plus.com wrote:

I probably wouldn't make it fixed length. I'd have the length in
decimal followed by, say, \n.


Or even followed by any non-digit. Chances are your JSON data begins
with a non-digit, so you'd just have to insert a space in the event
that you're JSON-encoding a flat integer. (Which might not ever
happen, if you know that your data will always be an object.)

ChrisA


So on the first recv() call, I set the buffer at 1 character and I iterate

 over single character until a non-digit character is encountered?


More efficient would be to guess that it'll be, say, 10 bytes, and
then retain any excess for your JSON read loop. But you'd need to sort
that out between the halves of your code.


If the length is always followed by a space then it's easier to split
it off the input:

buf = sock.recv(10)
space_pos = buf.find(b )
nbuf = int(buf[ : space_pos])
buf = buf[space_pos+ 1 : ]

while len(buf)  nbuf:
chunk = sock.recv(nbuf - len(buf))
if not chunk:
break

buf += chunk

I'm assuming that:

1. The initial recv returns the length followed by a space. It could,
of course, return fewer bytes (space_pos == -1), so you may need to
recv some more bytes, like what's done later on.

2. At least 10 bytes were sent. Imagine what would happen if the sender
sent b2 [] immediately followed by b2 []. The initial recv could
return all of it. In that case you could save the excess until next
time. Alternatively, the sender could guarantee that it would never
send fewer than the 10 bytes, padding with several b  if necessary.

--
http://mail.python.org/mailman/listinfo/python-list

Python recv loop

2013-02-10 Thread Ihsan Junaidi Ibrahim

Hi,

I'm implementing a python client connecting to a C-backend server and am 
currently stuck to as to how to proceed with receiving variable-length byte 
stream coming in from the server.

I have coded the first 4 bytes (in hexadecimal) of message coming in from the 
server to specify the length of the message payload i.e. 0xad{...}

I've managed to receive and translate the message length until I reach my 
second recv which I readjusted the buffer size to include the new message 
length.

However that failed and recv received 0 bytes. I implemented the same algorithm 
on the server side using C and it work so appreciate if you can help me on this.

# receive message length
print 'receiving data'
mlen = sock.recv(4)
try:
nbuf = int(mlen, 16)
except ValueError as e:
print 'invalid length type'
return -1

while True:
buf = sock.recv(nbuf)

if not buf:
break

slen = len(buf)
str = {0} bytes received: {1}.format(slen, buf)
print str
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-10 Thread Dave Angel


On 02/10/2013 07:48 PM, Ihsan Junaidi Ibrahim wrote:

Hi,

I'm implementing a python client connecting to a C-backend server and am 
currently stuck to as to how to proceed with receiving variable-length byte 
stream coming in from the server.

I have coded the first 4 bytes (in hexadecimal) of message coming in from the 
server to specify the length of the message payload i.e. 0xad{...}


Exactly how are you sending hexadecimal ?  If that 0xad (which is only 
one byte, what about the other 3 ?) is intended to be a C description, 
then it's certainly not hex, it's binary.  And probably little-endian, 
to boot.  That's a mistake, as network protocols almost always use 
big-endian.


So what is the range of values for the length, and how are they actually 
encoded?  Are they uint32 in native binary format?   And are you 
permitted to change the encoding, to fix it?  (If it were my choice, I'd 
make it a printable decimal value, first choice, or printable hex, 
second choice.)




I've managed to receive and translate the message length until I reach my 
second recv which I readjusted the buffer size to include the new message 
length.

However that failed and recv received 0 bytes. I implemented the same algorithm 
on the server side using C and it work so appreciate if you can help me on this.

# receive message length
 print 'receiving data'
 mlen = sock.recv(4)
 try:
 nbuf = int(mlen, 16)


That supposes the count is being sent as a printable string of hex 
digits.  That's not what I concluded above.



 except ValueError as e:


If the count starts as 0xad, I can't imagine why this exception wasn't 
thrown.



 print 'invalid length type'
 return -1

 while True:
 buf = sock.recv(nbuf)

 if not buf:
 break

 slen = len(buf)
 str = {0} bytes received: {1}.format(slen, buf)
 print str



You might need to play with hexlify, if you persist in sending the count 
in binary.


--
DaveA
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-10 Thread MRAB


On 2013-02-11 00:48, Ihsan Junaidi Ibrahim wrote:

Hi,

I'm implementing a python client connecting to a C-backend server and am 
currently stuck to as to how to proceed with receiving variable-length byte 
stream coming in from the server.

I have coded the first 4 bytes (in hexadecimal) of message coming in from the 
server to specify the length of the message payload i.e. 0xad{...}

I've managed to receive and translate the message length until I reach my 
second recv which I readjusted the buffer size to include the new message 
length.

However that failed and recv received 0 bytes. I implemented the same algorithm 
on the server side using C and it work so appreciate if you can help me on this.

# receive message length
 print 'receiving data'
 mlen = sock.recv(4)
 try:
 nbuf = int(mlen, 16)
 except ValueError as e:
 print 'invalid length type'
 return -1

 while True:
 buf = sock.recv(nbuf)

 if not buf:
 break

 slen = len(buf)
 str = {0} bytes received: {1}.format(slen, buf)
 print str


You should keep reading until you get all need or the connection is
closed:

buf = b''
while len(buf)  nbuf:
chunk = sock.recv(nbuf - len(buf))

if not chunk:
break

buf += chunk

--
http://mail.python.org/mailman/listinfo/python-list

Re: Python recv loop

2013-02-10 Thread Roy Smith

In article mailman.1612.1360544258.2939.python-l...@python.org,
 Ihsan Junaidi Ibrahim ih...@grep.my wrote:

 I'm implementing a python client connecting to a C-backend server and am 
 currently stuck to as to how to proceed with receiving variable-length byte 
 stream coming in from the server.
 
 I have coded the first 4 bytes (in hexadecimal) of message coming in from the 
 server to specify the length of the message payload i.e. 0xad{...}

Is this server that you're talking to something that you have control 
over, i.e. are you stuck with this protocol?  Given a choice, I'd go 
with something like JSON, for which pre-existing libraries for every 
language under the sun.

But, let's assume for the moment that you're stuck with this 
length-value encoding.  OK, but it's going to be more complicated than 
you think.  [I assume we're talking TCP here?]

Carefully read the documentation for socket.recv():

 socket.recv(bufsize[, flags]) [...] The maximum amount of data to be received 
 at once is specified by bufsize. 

Linger on the word maximum, and try to grok the fullness of of how 
annoying that can be.  What it means is that if the other side sent 120 
bytes (octets), recv() might return all 120 at once, or it might return 
them one at a time, or anything in between.

So, what you need to do is call recv() repeatedly in a loop, each time 
passing it a value for bufsize which represents the amount left in the 
message (i.e. the original message length parsed earlier minus all the 
little bits and pieces that have been read so far).

Keep in mind, you also need to do this when you recv() the first 4 
octets, which make up the length field.  What you've got, recv(4), will 
work MOST of the time, but it's perfectly legal for recv() to return a 
short read.  You can't predict how fragmentation and retry timeouts and 
all sorts of low-level crud will cause your message boundaries to get 
scrambled.

 # receive message length
 print 'receiving data'
 mlen = sock.recv(4)
 try:
 nbuf = int(mlen, 16)
 except ValueError as e:
 print 'invalid length type'
 return -1
 
 while True:
 buf = sock.recv(nbuf)
 
 if not buf:
 break
 
 slen = len(buf)
 str = {0} bytes received: {1}.format(slen, buf)
 print str

Do you actually *know* what the value of nbuf is?  Is it possible that 
(somehow) it's 0?  You should print (log, whatever), the value of nbuf, 
just to make sure.

And, once you'e got all this working, tear it all out and convert to 
using something sane like JSON.  Let somebody else worry about all the 
horrible details.
-- 
http://mail.python.org/mailman/listinfo/python-list

Question of Python second loop break and indexes

2012-05-09 Thread lilin Yi

//final_1 is a list of Identifier which I need to find corresponding
files(four lines) in x(x is the  file) and write following four lines
in a new file.

//because the order of the identifier is the same, so after I find the
same identifier in x , the next time I want to start from next index
in x,which will save time. That is to say , when the if command
satisfied ,it can automatically jump out out the second while loop and
come to the next identifier of final_1 ,meanwhile the j should start
not from the beginning but the position previous.

//when I run the code it takes too much time more than one hour and
give the wrong resultso could you help me make some improvement of
the code?

i=0

offset_1=0


while i len(final_1):
j = offset_1
while j len(x1):
if final_1[i] == x1[j]:
new_file.write(x1[j])
new_file.write(x1[j+1])
new_file.write(x1[j+2])
new_file.write(x1[j+3])
offset_1 = j+4
quit_loop=True
if quit_loop == True:break
else: j=j +1
i=i+1
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Question of Python second loop break and indexes

2012-05-09 Thread MRAB


On 09/05/2012 09:36, lilin Yi wrote:

//final_1 is a list of Identifier which I need to find corresponding
files(four lines) in x(x is the  file) and write following four lines
in a new file.

//because the order of the identifier is the same, so after I find the
same identifier in x , the next time I want to start from next index
in x,which will save time. That is to say , when the if command
satisfied ,it can automatically jump out out the second while loop and
come to the next identifier of final_1 ,meanwhile the j should start
not from the beginning but the position previous.

//when I run the code it takes too much time more than one hour and
give the wrong resultso could you help me make some improvement of
the code?

i=0

offset_1=0


while ilen(final_1):
j = offset_1
while jlen(x1):
if final_1[i] == x1[j]:
new_file.write(x1[j])
new_file.write(x1[j+1])
new_file.write(x1[j+2])
new_file.write(x1[j+3])
offset_1 = j+4
quit_loop=True
if quit_loop == True:break
else: j=j +1
i=i+1


This is roughly equivalent:

j = 0

for f in final_1:
try:
# Look for the identifier starting at index 'j'.
j = x1.index(f, j)
except ValueError:
# Failed to find the identifier.
pass
else:
# Found the identifier at index 'j'.
new_file.write(x1[j])
new_file.write(x1[j + 1])
new_file.write(x1[j + 2])
new_file.write(x1[j + 3])
# Skip over the 4 lines.
j += 4
--
http://mail.python.org/mailman/listinfo/python-list

Re: Question of Python second loop break and indexes

2012-05-09 Thread Ulrich Eckhardt

Am 09.05.2012 10:36, schrieb lilin Yi:
 //final_1 is a list of Identifier which I need to find corresponding
 files(four lines) in x(x is the  file) and write following four lines
 in a new file.
 
 //because the order of the identifier is the same, so after I find the
 same identifier in x , the next time I want to start from next index
 in x,which will save time. That is to say , when the if command
 satisfied ,it can automatically jump out out the second while loop and
 come to the next identifier of final_1 ,meanwhile the j should start
 not from the beginning but the position previous.

 //when I run the code it takes too much time more than one hour and
 give the wrong resultso could you help me make some improvement of
 the code?

If the code takes too much time and gives the wrong results, you must
fix and improve it. In order to do that, the first thing you should do
is get familiar with test-driven development and Python's unittest
library. You can start by fixing the code, but chances are that you will
break it again trying to make it fast then. Having tests that tell you
after each step that the code still works correctly is invaluable.

Some more comments below...

 i=0
 
 offset_1=0
 
 
 while i len(final_1):
   j = offset_1
   while j len(x1):
   if final_1[i] == x1[j]:
   new_file.write(x1[j])
   new_file.write(x1[j+1])
   new_file.write(x1[j+2])
   new_file.write(x1[j+3])
   offset_1 = j+4
   quit_loop=True
   if quit_loop == True:break
   else: j=j +1
   i=i+1

Just looking at the code, there are a few things to note:
1. You are iterating i from zero to len(final_1)-1. The pythonic way
to code this is using for i in range(len(final_1)): However, since
you only use the index i to look up an element inside the final_1
sequence, the proper way is for f in final_1:...
2. Instead of writing four lines separately, you could write them in a
loop: for x in x1[j:j+4]: new_file.write(x).
3. x1 is a list, right? In that case, there is a member function
index() that searches for an element and accepts an optional start
position.
4. The quit_loop is useless, and you probably are getting wrong
results because you don't reset this value. If you use break at the
place where you assign True to it, you will probably get what you
want. Also, Python has real boolean variables with the two values True
and False, you don't have to use strings.


Concerning the speed, you can probably improve it by not storing the
lines of the input file in x1, but rather creating a dictionary
mapping between the input value and the according four lines:

content = open(...).readlines()
d = {}
for i in range(0, len(content), 4):
d[content[i]] = tuple(content[i, i+4])

Then, drop the offset_1 (at least do that until you have the code
working correctly), as it doesn't work with a dictionary and the
dictionary will probably be faster anyway.

The whole loop above then becomes:

for idf in final_1:
for l in d.get(idf):
new_file.write(l)

;)

I hope I gave you a few ideas, good luck!


Uli
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-18 Thread exarkun


On 03:56 am, tjre...@udel.edu wrote:

exar...@twistedmatrix.com wrote:
There's a lot of things in Python that I don't strictly *need*.  That 
doesn't mean that they wouldn't be welcome if I could have them. 
Getting rid of the range/xrange dichotomy would improve things.


The developers agreed a couple of years ago. Starting using 3.1 if you 
want this.


And there was much rejoicing, et cetera.
Since 'range' could refer to a user-defined object, rather than the 
builtin function, there is no way the interpreter should substitute 
'xrange'.


See the earlier parts of this thread for the reasons this isn't true. :)

Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient


On 02:12 am, pavlovevide...@gmail.com wrote:

On Aug 16, 3:35ï¿½pm, sturlamolden sturlamol...@yahoo.no wrote:

On 16 Aug, 14:57, Dennis Lee Bieber wlfr...@ix.netcom.com wrote:

         Well, the alternative would be to have two keywords for 
looping: one
 for your simple incrementing integer loop, and another for a loop 
that

 operates over the elements of some collection type.

A compiler could easily recognise a statement like

ï¿½ ï¿½for i in range(n):

as a simple integer loop.


It would be a simple to do if you were writing it for a different
langauge was a lot less dynamic than Python is.  It'd be quite a
complex hack to add it to CPython's compiler while maintaing the
current highly dynamic runtime semantics and backwards compatibility,
which is a design constraint of Python whether you like it or not.


In your other message, you said this wasn't a legitimate CPython 
complaint.  Here, you say that it would be a complex hack to implement 
this in CPython.  complex hack has negative connotations in my mind. 
This seems contradictory to me.


And all this complaining about an issue that can be worked around by
xrange instead of range.  Sheesh.


Sure.  The specific issue of range vs xrange is quite a small one. 
There is a slightly larger one regarding the flexibility (or relative 
lack thereof) of the CPython runtime, though.


Jean-Paul
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient


On 01:53 am, pavlovevide...@gmail.com wrote:

On Aug 16, 6:28�pm, exar...@twistedmatrix.com wrote:

On 01:23 am, benjamin.kap...@case.edu wrote:

On Sun, Aug 16, 2009 at 6:35 PM, sturlamolden sturlamol...@yahoo.no
wrote:

A compiler could easily recognise a statement like

� for i in range(n):

as a simple integer loop. In fact, Cython is able to do this.

but special cases aren't special enough to break the rules.

Although I think PyPy also recognizes this case and makes it as
efficient as using xrange, and does so without breaking any rules.


PyPy uses a JIT compiler (which is still slower than CPython,
suggesting that perhaps they should have spent more time optimizing
the general case than optimizing for an easily avoided special case).


It's true that PyPy has a JIT compiler (which is still slower than 
CPython).  However, this isn't where the range speedup comes from.  The 
range optimization is handled specifically in the implementation of 
range (or possibly of list, or a combination of the two, I can't 
remember exactly).  It's effective even when the JIT compiler is 
disabled.



There *are* _some_ legitimate complaints to be made about the CPython
runtime. :)


This isn't one of them.

xrange has been part of Python for 10 years.

If there are any complaints to be made about this situation it's that
there are any 2.x learning materials anythere that continue to use
range() and not xrange() in this context.


Jean-Paul
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient


On 01:44 am, http wrote:

exar...@twistedmatrix.com writes:

Although I think PyPy also recognizes this case and makes it as
efficient as using xrange, and does so without breaking any rules.


How can pypy possibly know that the user hasn't assigned some other
value to range?


It doesn't really need to.  The optimization isn't applied when the 
compiler sees the name range being called.  It's applied after the 
object the default builtin name range is bound to is called.


Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread sturlamolden

On 16 Aug, 19:12, Carl Banks pavlovevide...@gmail.com wrote:

 If you don't care about the dynamic stuff why don't you just use
 Cython?  Or quit complaining and just use xrange.

I think you are the only one complaining here.



-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread MRAB


exar...@twistedmatrix.com wrote:

On 02:12 am, pavlovevide...@gmail.com wrote:

On Aug 16, 3:35�pm, sturlamolden sturlamol...@yahoo.no wrote:

On 16 Aug, 14:57, Dennis Lee Bieber wlfr...@ix.netcom.com wrote:

 � � � � Well, the alternative would be to have two keywords for 
looping: one
 for your simple incrementing integer loop, and another for a loop 
that

 operates over the elements of some collection type.

A compiler could easily recognise a statement like

� �for i in range(n):

as a simple integer loop.


It would be a simple to do if you were writing it for a different
langauge was a lot less dynamic than Python is.  It'd be quite a
complex hack to add it to CPython's compiler while maintaing the
current highly dynamic runtime semantics and backwards compatibility,
which is a design constraint of Python whether you like it or not.


In your other message, you said this wasn't a legitimate CPython 
complaint.  Here, you say that it would be a complex hack to implement 
this in CPython.  complex hack has negative connotations in my mind. 
This seems contradictory to me.


And all this complaining about an issue that can be worked around by
xrange instead of range.  Sheesh.


Sure.  The specific issue of range vs xrange is quite a small one. There 
is a slightly larger one regarding the flexibility (or relative lack 
thereof) of the CPython runtime, though.



A possible solution would be to make 'range' return an instance of, say,
'RangeList', a subclass of list which would behave like 'list'
externally but create the list of values internally only when needed.
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread Stefan Behnel

Carl Banks wrote:
 On Aug 16, 3:35 pm, sturlamolden sturlamol...@yahoo.no wrote:
 On 16 Aug, 14:57, Dennis Lee Bieber wlfr...@ix.netcom.com wrote:

 Well, the alternative would be to have two keywords for looping: one
 for your simple incrementing integer loop, and another for a loop that
 operates over the elements of some collection type.
 A compiler could easily recognise a statement like

for i in range(n):

 as a simple integer loop.
 
 In fact, Cython is able to do this.
 
 Cython can do this easily because it is a different language that is a
 lot less dynamic than Python.

Actually, Cython is able to do this because it knows the global scope of a
module. The only thing that this optimisation makes impossible when you
compile your module using Cython is to inject builtins into the module
namespace *after* the compilation, either by assigning module attributes or
by importing the module into a custom namespace.

Given that both use cases are extremely rare, it was decided that
optimisations like this are more important than the ability to redefine the
most common builtins (such as 'range' or 'enumerate'). So, in a way, Cython
really makes them builtins.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread Steven D'Aprano

On Sun, 16 Aug 2009 15:35:26 -0700, sturlamolden wrote:

 On 16 Aug, 14:57, Dennis Lee Bieber wlfr...@ix.netcom.com wrote:
 
         Well, the alternative would be to have two keywords for
         looping: one
 for your simple incrementing integer loop, and another for a loop
 that operates over the elements of some collection type.
 
 A compiler could easily recognise a statement like
 
for i in range(n):
 
 as a simple integer loop. 

Easily huh? Would you like to put a small wager on that?

Here is an unedited copy-and-paste of the last few lines of an 
interactive Python session:

 for i in range(5):
... print i
...
0
1
2
Surprise!
4
 __builtins__.range is range
True


Can you determine how I did this? How would the compiler avoid this? If 
you can find a method for the compiler to avoid surprises like this, why 
do you think it would be valuable to add all that extra complexity to the 
language? (As opposed to an external tool like Cython.)



-- 
Steven
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread Stefan Behnel

John Machin wrote:
 On Aug 17, 8:35 am, sturlamolden sturlamol...@yahoo.no wrote:
 
 A compiler could easily recognise a statement like
for i in range(n):
 as a simple integer loop. In fact, Cython is able to do this.
 
 Extremely easy, once users relinquish the right to replace built-in
 range with their own concoctions ...

Cython allows you to do that. You just have to do it inside the module. If
Cython determines that range refers the global builtin name, it will
enable the loop optimisation. If you assign anything to that global name
inside your module (even the range function itself), this will disable the
optimisation.

Stefan
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread David Robinow

On Sun, Aug 16, 2009 at 11:10 PM, Nobodynob...@nowhere.com wrote:
 Java also has iterators; it's more a case of people coming from C and BASIC.

 Although, some of those may have come *through* Java without abandoning
 old habits. You see the same thing with people coming from BASIC to C and
 writing:

        #define NUM_DATES 50
        int day[NUM_DATES], month[NUM_DATES], year[NUM_DATES];

 rather than defining a struct.

 Sometimes referred to as I know ten languages and can write in BASIC in
 all of them.

Ha, ha. I learned that pattern in Fortran. I confess to having written
code like that in C. I think I've gotten over it.
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread Ethan Furman


Emmanuel Surleau wrote:

Dr. Phillip M. Feldman wrote:



[snip]



def is_prime(n):
  for j in range(2,n):
 if (n % j) == 0: return False
  return True

It seems as though Python is actually expanding range(2,n) into a list of
numbers, even though this is incredibly wasteful of memory. There should
be a looping mechanism that generates the index variable values
incrementally as they are needed.



[snip]


I will also observe that if you were to stop programming whatever
language you are more familiar with in Python, and start programming
Python in Python, you'll have an easier time of it.



I don't see what's particularly un-Pythonic with this code. Not using xrange() 
is a mistake, certainly, but it remains clear, easily understandable code 
which correctly demonstrates the naive algorithm for detecting whether n is a 
prime. It doesn't call for condescension



[snip]


Cheers,

Emm


My comment about programming Python in Python was geared more towards 
the subject line than the actual code, and the biases evident in his 
comments in both this thread and earlier ones.


~Ethan~
--
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient

2009-08-17 Thread Carl Banks

On Aug 17, 4:40 am, exar...@twistedmatrix.com wrote:
 On 02:12 am, pavlovevide...@gmail.com wrote:



 On Aug 16, 3:35 pm, sturlamolden sturlamol...@yahoo.no wrote:
 On 16 Aug, 14:57, Dennis Lee Bieber wlfr...@ix.netcom.com wrote:

           Well, the alternative would be to have two keywords for
 looping: one
   for your simple incrementing integer loop, and another for a loop
 that
   operates over the elements of some collection type.

 A compiler could easily recognise a statement like

    for i in range(n):

 as a simple integer loop.

 It would be a simple to do if you were writing it for a different
 langauge was a lot less dynamic than Python is.  It'd be quite a
 complex hack to add it to CPython's compiler while maintaing the
 current highly dynamic runtime semantics and backwards compatibility,
 which is a design constraint of Python whether you like it or not.

 In your other message, you said this wasn't a legitimate CPython
 complaint.  Here, you say that it would be a complex hack to implement
 this in CPython.  complex hack has negative connotations in my mind.
 This seems contradictory to me.

Well, you missed the point, chief.

It's not a legitimate complaint because you can use xrange, you don't
need compiler magic to recognize and optimize range.


Carl Banks
-- 
http://mail.python.org/mailman/listinfo/python-list

Re: Python 'for' loop is memory inefficient