Re: Tuples and immutability
On Thu, Feb 27, 2014 at 10:33 AM, Chris Angelico wrote: > On Fri, Feb 28, 2014 at 3:27 AM, Eric Jacoboni > wrote: > > But, imho, it's far from being a intuitive result, to say the least. > > It's unintuitive, but it's a consequence of the way += is defined. If > you don't want assignment, don't use assignment :) > > ChrisA > Where is `.__iadd__()` called outside of `list += X`? If the only difference from `.extend()` is that it returns `self`, but the list was already modified anyway, why bother with reassignment? -- https://mail.python.org/mailman/listinfo/python-list
Re: intersection, union, difference, symmetric difference for dictionaries
On Tue, Feb 25, 2014 at 2:32 PM, mauro wrote: > > So I wonder why operations such us intersection, union, difference, > symmetric difference that are available for sets and are not available > for dictionaries without going via key dictviews. How would the set operations apply to the dictionary values? -- https://mail.python.org/mailman/listinfo/python-list
Re: Generator using item[n-1] + item[n] memory
OK, now the trick; adding `data = None` inside the generator works, but in my actual code I wrap my generator inside of `enumerate()`, which seems to obviate the "fix". Can I get it to play nice or am I forced to count manually. Is that a feature? On Fri, Feb 14, 2014 at 9:21 PM, Roy Smith wrote: > In article , > Nick Timkovich wrote: > > > Ah, I think I was equating `yield` too closely with `return` in my head. > > Whereas `return` results in the destruction of the function's locals, > > `yield` I should have known keeps them around, a la C's `static` > functions. > > Many thanks! > > It's not quite like C's static. With C's static, the static variables > are per-function. In Python, yield creates a context per invocation. > Thus, I can do > > def f(): > for i in range(1): > yield i > > g1 = f() > g2 = f() > print g1.next() > print g1.next() > print g1.next() > print g2.next() > print g1.next() > > > which prints 0, 1, 2, 0, 3. There's two contexts active at the same > time, with a distinct instance of "i" in each one. > -- > https://mail.python.org/mailman/listinfo/python-list > -- https://mail.python.org/mailman/listinfo/python-list
Re: Generator using item[n-1] + item[n] memory
Ah, I think I was equating `yield` too closely with `return` in my head. Whereas `return` results in the destruction of the function's locals, `yield` I should have known keeps them around, a la C's `static` functions. Many thanks! -- https://mail.python.org/mailman/listinfo/python-list
Generator using item[n-1] + item[n] memory
I have a Python 3.x program that processes several large text files that
contain sizeable arrays of data that can occasionally brush up against the
memory limit of my puny workstation. From some basic memory profiling, it
seems like when using the generator, the memory usage of my script balloons
to hold consecutive elements, using up to twice the memory I expect.
I made a simple, stand alone example to test the generator and I get
similar results in Python 2.7, 3.3, and 3.4. My test code follows,
`memory_usage()` is a modifed version of [this function from an SO
question](http://stackoverflow.com/a/898406/194586) which uses
`/proc/self/status` and agrees with `top` as I watch it. `resource` is
probably a more cross-platform method:
###
import sys, resource, gc, time
def biggen():
sizes = 1, 1, 10, 1, 1, 10, 10, 1, 1, 10, 10, 20, 1, 1, 20, 20, 1, 1
for size in sizes:
data = [1] * int(size * 1e6)
#time.sleep(1)
yield data
def consumer():
for data in biggen():
rusage = resource.getrusage(resource.RUSAGE_SELF)
peak_mb = rusage.ru_maxrss/1024.0
print('Peak: {0:6.1f} MB, Data Len: {1:6.1f} M'.format(
peak_mb, len(data)/1e6))
#print(memory_usage())
data = None # go
del data # away
gc.collect() # please.
# def memory_usage():
# """Memory usage of the current process, requires /proc/self/status"""
# # http://stackoverflow.com/a/898406/194586
# result = {'peak': 0, 'rss': 0}
# for line in open('/proc/self/status'):
# parts = line.split()
# key = parts[0][2:-1].lower()
# if key in result:
# result[key] = int(parts[1])/1024.0
# return 'Peak: {peak:6.1f} MB, Current: {rss:6.1f} MB'.format(**result)
print(sys.version)
consumer()
###
In practice I'll process data coming from such a generator loop, saving
just what I need, then discard it.
When I run the above script, and two large elements come in series (the
data size can be highly variable), it seems like Python computes the next
before freeing the previous, leading to up to double the memory usage.
$ python genmem.py
2.7.3 (default, Sep 26 2013, 20:08:41)
[GCC 4.6.3]
Peak:7.9 MB, Data Len:1.0 M
Peak: 11.5 MB, Data Len:1.0 M
Peak: 45.8 MB, Data Len: 10.0 M
Peak: 45.9 MB, Data Len:1.0 M
Peak: 45.9 MB, Data Len:1.0 M
Peak: 45.9 MB, Data Len: 10.0 M
#^^ not much different versus previous 10M-list
Peak: 80.2 MB, Data Len: 10.0 M
#^^ same list size, but new memory peak at roughly twice the
usage
Peak: 80.2 MB, Data Len:1.0 M
Peak: 80.2 MB, Data Len:1.0 M
Peak: 80.2 MB, Data Len: 10.0 M
Peak: 80.2 MB, Data Len: 10.0 M
Peak: 118.3 MB, Data Len: 20.0 M
#^^ and again... (20+10)*c
Peak: 118.3 MB, Data Len:1.0 M
Peak: 118.3 MB, Data Len:1.0 M
Peak: 118.3 MB, Data Len: 20.0 M
Peak: 156.5 MB, Data Len: 20.0 M
#^^ and again. (20+20)*c
Peak: 156.5 MB, Data Len:1.0 M
Peak: 156.5 MB, Data Len:1.0 M
The crazy belt-and-suspenders-and-duct-tape approach `data = None`, `del
data`, and `gc.collect()` does nothing.
I'm pretty sure the generator itself is not doubling up on memory because
otherwise a single large value it yields would increase the peak usage, and
in the *same iteration* a large object appeared; it's only large
consecutive objects.
How can I save my memory?
Cheers,
Nick
cc: StackOverflow http://stackoverflow.com/q/21787099/194586
--
https://mail.python.org/mailman/listinfo/python-list
Re: Async serial communication/threads sharing data
On Mar 21, 9:19 pm, Jean-Paul Calderone wrote: > On Sat, 21 Mar 2009 13:52:21 -0700 (PDT), Nick Timkovich > wrote: > >I've been working on a program that will talk to an embedded device > >over the serial port, using some basic binary communications with > >messages 4-10 bytes long or so. Most of the nuts and bolts problems > >I've been able to solve, and have learned a little about the threading > >library to avoid blocking all action while waiting for responses > >(which can take 50 ms to 10 s). Ultimately, this program will test > >the device on the COM port by sending it messages and collecting > >responses for 10k-100k cycles; a cycle being: > > 1. tell it to switch a relay, > > 2. get it's response from the event, > > 3. ask it for some measurements, > > 4. get measurements, > > 5. repeat. > >Later I would like to develop a GUI as well, but not a big issue now > >(another reason to use threads? not sure). > > Twisted includes serial port support and will let you integrate with a > GUI toolkit. Since Twisted encourages you to write programs which deal > with things asynchronously in a single thread, if you use it, your > concerns about data exchange, locking, and timing should be addressed > as a simple consequence of your overall program structure. > > Jean-Paul I've looked at Twisted a little bit because of some searches on serial port comm turning up advice for it. However, there seems to be no/ minimal documentation for the serial portions, like they are some old relic that nobody uses from this seemingly massive package. Do you have any examples or somewhere in particular you could point me? Thanks for the responses, Nick -- http://mail.python.org/mailman/listinfo/python-list
Async serial communication/threads sharing data
I've been working on a program that will talk to an embedded device over the serial port, using some basic binary communications with messages 4-10 bytes long or so. Most of the nuts and bolts problems I've been able to solve, and have learned a little about the threading library to avoid blocking all action while waiting for responses (which can take 50 ms to 10 s). Ultimately, this program will test the device on the COM port by sending it messages and collecting responses for 10k-100k cycles; a cycle being: 1. tell it to switch a relay, 2. get it's response from the event, 3. ask it for some measurements, 4. get measurements, 5. repeat. Later I would like to develop a GUI as well, but not a big issue now (another reason to use threads? not sure). The overall structure of what I should do is not very apparent to me on how to efficiently deal with the communications. Have the main loop handle the overall timing of how often to run the test cycle, then have a thread deal with all the communications, and within that another thread that just receives data? My main issue is with how to exchange data between different threads; can I just do something like have a global list of messages, appending, modifying, and removing as needed? Does the threading.Lock object just prevent every other thread from running, or is it bound somehow to a specific object (like my list)? -- http://mail.python.org/mailman/listinfo/python-list
