Re: Idiom for partial failures
On 2/21/20 9:18 AM, David Wihl wrote: Yes, the API has to support partial failures across all six supported languages. Not all operations need to be atomic and there is considerable efficiency in having multiple operations sent in a single request. Thanks, -David On Thu, Feb 20, 2020 at 6:07 PM Rob Gaddi wrote: On 2/20/20 9:30 AM, David Wihl wrote: (first post) I'm working on the Python client library [0]for the Google Ads API [1]. In some cases, we can start a request with a partial failure [2] flag = True. This means that the request may contain say 1000 operations. If any of the operations fail, the request will return with a success status without an exception. Then the developer has to iterate through the list of operation return statuses to determine which specific ones failed (example [3]). I believe that it would be more idiomatic in Python (and other languages like Ruby) to throw an exception when one of these partial errors occur. That way there would be the same control flow if a major or minor error occurred. The team is asking me for other examples or justification of this being idiomatic of Python. Can you recommend any examples or related best practices? [0] https://github.com/googleads/google-ads-python [1] https://developers.google.com/google-ads/api/docs/start [2] https://developers.google.com/google-ads/api/docs/best-practices/partial-failures [3] https://github.com/googleads/google-ads-python/blob/master/examples/error_handling/handle_partial_failure.py Thanks! -David Potentially stupid question, does your library to the API have to support partial failures? Sure Google Ads does, but does it really add value to do so? And then you've got to decide what to DO with those partially failed results. You could, as a library, just say "No, this library never sets the partial_failure field. We only support atomic transactions that fully succeed or fully fail and raise an exception." Makes your life a lot easier; so unless it makes your customer's lives a lot harder... -- https://mail.python.org/mailman/listinfo/python-list In that case, I think Richard Damon's answer smells rightest. A partial failure is a partial success and should not trigger a mandatory flow-control change, which is what an exception is. If there is any information in the successes, then I'd expect an operation like this to return me a list corresponding 1-to-1 with the list of operations such that I could: results = do_the_thing() for op, result in zip(operations, results): if isinstance(result, GoogleApiError): deal_with_error(op, result) If successes have nothing to add to the party, then I'd expect the operation to return a list containing only the errors, and an empty list means no errors. -- https://mail.python.org/mailman/listinfo/python-list
Re: Idiom for partial failures
On 2/20/20 9:30 AM, David Wihl wrote: (first post) I'm working on the Python client library [0]for the Google Ads API [1]. In some cases, we can start a request with a partial failure [2] flag = True. This means that the request may contain say 1000 operations. If any of the operations fail, the request will return with a success status without an exception. Then the developer has to iterate through the list of operation return statuses to determine which specific ones failed (example [3]). I believe that it would be more idiomatic in Python (and other languages like Ruby) to throw an exception when one of these partial errors occur. That way there would be the same control flow if a major or minor error occurred. The team is asking me for other examples or justification of this being idiomatic of Python. Can you recommend any examples or related best practices? [0] https://github.com/googleads/google-ads-python [1] https://developers.google.com/google-ads/api/docs/start [2] https://developers.google.com/google-ads/api/docs/best-practices/partial-failures [3] https://github.com/googleads/google-ads-python/blob/master/examples/error_handling/handle_partial_failure.py Thanks! -David Potentially stupid question, does your library to the API have to support partial failures? Sure Google Ads does, but does it really add value to do so? And then you've got to decide what to DO with those partially failed results. You could, as a library, just say "No, this library never sets the partial_failure field. We only support atomic transactions that fully succeed or fully fail and raise an exception." Makes your life a lot easier; so unless it makes your customer's lives a lot harder... -- https://mail.python.org/mailman/listinfo/python-list
Re: Damping cofficient from envelope
On 1/13/20 9:45 AM, Shiba Subedi wrote: Dear all, I'm new for python. I'm trying to compute damping coefficient ( for the waveform trace data) but I didn't get how can I do. I can plot the envelope but my interest is to find out the damping coefficient of the trace. How is it possible? If you've already got a method of calculating the envelope then what you want is to minimize the error between that calculated envelope and that of your damping function, which I'll assume is in the form a * exp(-bt). Write a function estimated_envelope(t, a, b) that does that math, and use scipy.optimize.curve_fit to dial in (a,b) against your measured data. This error function doesn't have a lot of exciting topography, so your initial guesses p0 can be pretty far off and you'll still converge. -- https://mail.python.org/mailman/listinfo/python-list
Re: Coding technique: distinguish using type or abc?
On 1/8/20 1:40 PM, DL Neil wrote: Do you prefer to use isinstance() with type() or to refer to collections.abc? This team producing bases statistical analyses for (lesser capable) user-coders to utilise with their own experimental 'control code'; faces situations where a list-parameter is often only one element long. As is so often the way, amongst the 'clients' there are a couple of strong-minded (am not allowed to call them "awkward", or otherwise!) user-coder-analysts, who demand that entry of a single-element not require them to surround it with "unnecessary" square-brackets. Despite complaining, we realise that this is actually quite a good practice, and likely save us (as well as 'them') from interface mistakes. Such single elements appear in both string and numeric formats, but for simplicity (here) let's ignore numerics... The problem rearing its ugly head, is when the string single-element becomes input to a for-loop. If we loop around a list, then each element is handled individually (as desired). Whereas if the element is a string, then each character is treated as if it were a list-element (not)! In Code Review, I noticed that two 'solutions' have been coded. 1 using type()s to distinguish: def format_as_list( parameter:Union[ str, list ] )->list: if isinstance( parameter, str ): parameter_list = [ parameter ] elif isinstance( parameter, list ): parameter_list = parameter else: raise NotImplementedError return parameter_list 2 using abstract base classes from PSL.collections to distinguish: import collections.abc as abc def is_list_not_string( parameter:Union[ str, list ] ) -> bool: return isinstance( parameter, abc.MutableSequence ) def format_as_list( parameter:str )->list: if is_list_not_string( parameter ): return parameter else: return [ parameter, ] (ignoring implicit assumption/error!) NB I've simplified the code and attempted to harmonise the varNMs between snippets. With our preference for EAFP, I went looking for a way to utilise an exception by way of distinguishing between the input-types - but couldn't see how, without major artifice (false/purposeless construct which would confuse the next reader of the code). That said, I'm wondering if using (or appearing to use) tuples and *args might solve the problem - but will have to dig back through the code-base... Meantime, faced with such a challenge, would you recommend utilising one of these ideas over the other, or perhaps some other solution? Are there perhaps circumstances where you would use one solution, and others the other? I try to avoid making assumptions, so I wind up with a lot of if isinstance(parameter, str): plist = [parameter] else: try: plist = list(parameter) except TypeError: plist = [parameter] Any iterable gets listified unless it's a string, which gets treated the same way a non-iterable does. EAFP. -- https://mail.python.org/mailman/listinfo/python-list
Re: Floating point overflow and underflow
On 1/7/20 3:47 PM, Shashank Tiwari wrote:
In Python3 an operation as follows:
10135.1941 * (10**8)
gives the result: 101351941.0001
Similarly, using the pow function also gives the same overflow/underflow
error.
10135.1941 * pow(10,8)
101351941.0001
Like multiplication, division of large or very small floating point numbers
gives similar overflow/underflow errors.
Usage of Decimal doesn't seem to fix it either.
from decimal import *
Decimal(10135.1941 * pow(10,8))
Decimal('101351941.0001220703125')
Decimal(10135.1941)*pow(10,8)
Decimal('101351941.61700120568')
Decimal(10135.1941) * Decimal(pow(10,8))
Decimal('101351941.61700120568')
Decimal(10135.1941 * (10**8))
Decimal('101351941.0001220703125')
How could one do high precision multiplication and division of floating
points numbers (decimals) in Python3 without any overflow/underflow errors?
Thanks, Shanky
You've already polluted your Decimal with floating-point roundoff error before
you even multiply it in any of your examples.
>>> Decimal(10135.1941)
Decimal('10135.19416170012056827545166015625')
Initialize your Decimal from a string instead.
>>> Decimal('10135.1941')
Decimal('10135.1941')
>>> Decimal('10135.1941') * Decimal('1e8')
Decimal('1.01351941E+12')
>>> float(_)
101351941.0
--
https://mail.python.org/mailman/listinfo/python-list
Re: (no subject)
On 1/4/20 2:29 PM, William Johnsson wrote:
Hello! My name is William and im 14 years old and live in sweden. Im pretty
new to programing in python and i need some help with code, (That’s why i’m
here). But i couldn’t really find what i was searching for on the internet. I’m
trying to write code that can check only the first line in a .txt file and
check if it’s the same as the int 1000. I preferably want it to be an if
satement but anything works, i just don’t know how i should write it. I’ve been
browsing the internet for a solution but sadly can’t seem to find a solution to
it.
I don’t really know if this is the right way to get help or even if this email
is for something else. Hopefully i will get some kind of response.
Best regards, William
It won't be; it can't be. A text file contains only text. The line you read
from it might be the string '1000' (or more likely '1000\n', which is 4 digits
followed by a newline character), but it cannot be the int 1000 because data has
types.
str(1000) gives you '1000', because ints can be converted to strings.
int('1000') gives you 1000 because and strings consisting only of digits can be
converted to ints. So the two types are convertible into one another. But
that's a thing you need to do explicitly. If x is a thing you just read from a
text file then
if x == 1000:
can never be true.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Hi!! im searching for a ready app of python for pdf combiner
On 1/6/20 8:13 AM, alon.naj...@gmail.com wrote: Hi!! im searching for a ready app of python for pdf combiner what I need exactly is a pdf combiner for 20 + pdfs to combine them as one, do a merge Why does this need to be in Python? Just download a copy of pdftk (which I think is written in Java?) and move on with your life. -- https://mail.python.org/mailman/listinfo/python-list
Re: python 2 to 3 converter
On 12/9/19 9:55 AM, songbird wrote: jf...@ms4.hinet.net wrote: ... Even string is hard to be handled by the AI:-) Quoted from https://portingguide.readthedocs.io/en/latest/strings.html " ... This means that you need to go through the entire codebase, and decide which value is what type. Unfortunately, this process generally cannot be automated." i don't agree. if the language is already parsed then you have the strings. the contents of the strings will have to be finite if they are fixed strings. so to convert a fixed string you can choose a type for the value and run a test to see if it works. if it does then you've picked the correct type, if it doesn't you pick the next type. there are only a finite number of types. And how exactly do you propose to determine whether the constant I've enclosed in quotation marks in Python2 represents "text" or "binary data"? Not all text is words; think of SQL queries. And some words are binary data, think SCPI commands being sent to a serial port. -- https://mail.python.org/mailman/listinfo/python-list
Re: Make warning an exception?
On 12/6/19 12:58 PM, Israel Brewster wrote: I was running some code and I saw this pop up in the console: 2019-12-06 11:53:54.087 Python[85524:39651849] WARNING: nextEventMatchingMask should only be called from the Main Thread! This will throw an exception in the future. The only problem is, I have no idea what is generating that warning - I never call nextEventMatchingMask directly, so it must be getting called from one of the libraries I’m calling. Is there some way I can force python to throw an exception now, so my debugger can catch it and let me know where in my code the originating call is? I’ve tried stepping through the obvious options, with no luck so far. --- Israel Brewster Software Engineer Alaska Volcano Observatory Geophysical Institute - UAF 2156 Koyukuk Drive Fairbanks AK 99775-7320 Work: 907-474-5172 cell: 907-328-9145 You need to set the warning filter to "error", which you can do either with warnings.simplefilter at the start of your program or by setting the PYTHONWARNINGS environment variable. https://docs.python.org/3/library/warnings.html#the-warnings-filter This the same project you're having PySide/threading problems on? -- https://mail.python.org/mailman/listinfo/python-list
Re: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7
On 12/2/19 5:50 PM, Richard Damon wrote: Perhaps array could be extended so that it took '4' for a 4 byte integer and '8' for an 8 byte integer (maybe 'U4' and 'U8' for unsigned). Might as well also allow 1 and 2 for completeness for char and short (but those are currently consistent). I will note that numpy arrays give exactly this level of control, as do ctypes arrays. The standard array library might just be the wrong tool for the job of reading binary data. -- https://mail.python.org/mailman/listinfo/python-list
Re: array and struct 64-bit Linux change in behavior Python 3.7 and 2.7
On 12/2/19 9:26 AM, Chris Clark wrote:
Test case:
import array
array.array('L', [0])
# x.itemsize == 8 rather than 4
This works fine (returns 4) under Windows Python 3.7.3 64-bit build.
Under Ubuntu; Python 2.7.15rc1, 3.6.5, 3.70b3 64-bit this returns 8.
Documentation at https://docs.python.org/3/library/array.html explicitly states
'L' is for size 4.
It impacts all uses types of array (e.g. reading from byte strings).
The struct module is a little different:
import struct
x = struct.pack('L', 0)
# len(x) ===8 rather than 4
This can be worked around by using '=L' - which is not well documented - so
this maybe a doc issue.
Wanted to post here for comments before opening a bug at
https://bugs.python.org/
Is anyone seeing this under Debian/Ubuntu?
Chris
I'd say not a bug, at least in array. Reading that array documentation you
linked, 4 is explicitly the MINIMUM size in bytes, not the guaranteed size.
The struct situation is, as you said, a bit different. I believe that with the
default native alignment @, you're seeing 4-byte data padded to an 8-byte
alignment, not 8-byte data. That does seem to go against what the struct
documentation says, "Padding is only automatically added between successive
structure members. No padding is added at the beginning or the end of the
encoded struct."
= alignment is documented as having the platform native byte-order, but the size
and alignment is standardized as having no padding, which is exactly the
behavior you seem to want. The documentation is a bit obtuse and scattered, but
no more than any other.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Python Resources related with web security
On 11/26/19 12:41 PM, Grant Edwards wrote: On 2019-11-26, Joel Goldstick wrote: I'm thinking this is a troll or a turing machine experiment? Yea, many of the posts remind me of ELIZA. How do you feel about many of the posts remind you of ELIZA? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to delay until a next increment of time occurs ?
On 11/14/19 12:34 PM, R.Wieser wrote: Dennis, The R-Pi has never, to my knowledge, been billed as suitable for industrial/real-time/process-control... I have not read those specs to be honest. But for some reason I assumed that a 1.4 GHz machine should easily be able to generate a 1.6 KHz signal, even using Python. I was wrong. It can, but just barily and not without jitter and hickups. Lesson learned. Regards, Rudy Wieser The machine itself would be fine. But between Python and Linux (probably the bigger culprit), there are lots of things that aren't interested in your deterministicity. Just run it bare-metal; I'm sure that's straightforward on an rPi and has no gotchas or complications that would lead to days/weeks of frustration. On an actually practical note; ever since they started baking the quasi-realtime stuff into the Linux kernel, you've been able to use the whole sched_setscheduler(2) family of calls (from C) to set a process's real-time priority. That might get things good enough to do the job, though how one gets there from Python I couldn't tell you. Naïvely I'd say to just use ctypes to wrap those calls, but that's some low-level stuff to be getting into from a very high-level language. Might work? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to delay until a next increment of time occurs ?
On 11/13/19 1:26 PM, Dietmar Schwertberger wrote: On 13.11.2019 21:20, R.Wieser wrote: 300us is getting on towards realtime. Not really. Translated to a frequency (toggeling the pin) it would be just 1.6 KHz. Thats rather slow for an ARM machine running on 1.4 Ghz (about a million times as fast). It *is* real-time... Real-time is not about speed, but about guaranteed timing. Yeah, that's always the conversation, but at the end of the day "realtime" is always a question of "by how much". Do you really need to be hard periodic with 0 ns slop in either direction? Can you afford to come in early but not late? By 10%? By 1%? By .01%? If you can afford 1% and you need a 1 second interval, you're only asking for 10 ms and in practice even a giant preemptive non-RT OS like Linux can do it for you. If you can only afford 1% on a 300 us interval then your lack of rigid control over what happens on your processor is a real problem and you're better off bare metal on a dedicated $2 Cortex-M than with random timeslices of the 1.4GHz beast on the rPi. Is what I was shorthanding with "realtime". -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to delay until a next increment of time occurs ?
On 11/13/19 6:02 AM, R.Wieser wrote: Hello all, I'm writing some code to toggle a pin on a Raspberry Pi, and would like to have that happen at (multiples of) 300 uSec increments. I tried time.sleep(), but that one disregards the time after the last one ends and the new one is started. In other words, all the time spend in code (or being interrupted by another thread!) between the end of the previous and the start of the current sleep throws the whole repetitive timing off. So, I'm looking for a method that will allow me to wait for a "last time plus increment". Is there one with the properties of sleep() (not just burning processor cycles way, blocking all threads), but referencing a previous time. Regards, Rudy Wieser 300us is getting on towards realtime. Depending on how rigorous you need to be, Python may not be the right tool for the job; I've never tried it that fast. That said, I'll throw out a different option than the threading based ones people have suggested. Trap SIGALRM with signal.signal and use signal.setitimer to create an interval timer. I can't tell you offhand that signals will give you better realtime performance than threads or vice versa. There's also async in the mix, which I still have no idea how to use. But this way if you strike out on one approach you've got some others to consider. Also, does the rPi have any PWM or counter pins that you can just set and forget, rather than trying to keep it going yourself? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Launching a Script on the Linux Platform
On 11/12/19 10:06 AM, Wildman wrote: What is the best approach for launching a Python GUI program on a Linux platform. The program will be distributed in .deb format. So the .deb will contain a menu file as well as a .desktop file. The post install script will update the system menu. My question is how should the program be executed? Here are two choices for the "command=" entry in the menu file... command="/path/to/program.py" In this case the hash-bang would have to be included in the program script... #!/usr/bin/env python3 The other choice is this... command="python3 /path/to/program.py" (Of course, the Exec command in the .desktop file should match.) Is one method better than the other or does it acutally matter? I will note that without the shebang (and setting the execute bit), the program is only executable from the GUI menu, not the command prompt. I personally start even GUI programs far more often from a prompt. To follow Linux conventions you'd put the shebang, make the file executable, and put the executable somewhere on the PATH. I'd stick to those conventions barring a particular reason not to. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: OOP - how to abort an __init__ when the initialisation code fails ?
On 11/5/19 11:52 AM, Peter J. Holzer wrote:
On 2019-11-04 18:18:39 -0300, Luciano Ramalho wrote:
In addition, as Rob said, it is usually a bad idea to wrap several
lines of code in a single try/except block
I disagree with this. While it is sometimes useful to wrap a single
line, in my experience it rarely is. The scope of the try ... except
should be a unit from a semantic point of view. For example, if you
open a file and write some data to it, you could write:
try:
f = open("a.file", "w")
except OSError as e:
... handle exception
try:
f.write("some")
except OSError as e:
... handle exception
try:
f.write("data")
except OSError as e:
... handle exception
try:
close(f)
except OSError as e:
... handle exception
But not only is this hard to read and ugly as heck, what would be the
point?
Much better to write:
try:
with open("a.file", "w") as f:
f.write("some")
f.write("data")
except OSError as e:
... handle exception
Or maybe don't catch it here at all but just let it bubble up until it
hits a level where dealing with it makes sense from the user's point of
view (who may not care about an individual file, but about the task they
instructed the program to perform).
hp
I mean, you sort of want "the right amount of stuff" in a try block. Lines that
may exhibit a common failure, that you want to route to a common exception
handler go into the same try block (per your example above). Things that are
going to raise unrelated errors for unrelated reasons should be in different try
blocks. Code that should only execute if the try block was error-free, but that
you expect to not be able to raise errors itself, should go into the else block.
Like most of programming, hard-and-fast rules turn out to be impossible.
Reality has squishy edges.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: OOP - how to abort an __init__ when the initialisation code fails ?
On 11/4/19 10:59 AM, R.Wieser wrote: Rob, Returning None will specifically NOT accomplish the thing you want; nothing ever checks the return value of __init__. I thought to have read that when you return a none from it the object itself would return a placeholder singleton. Raise an exception. Yeah, that was what I was thinking too - up until the moment I realized that the than-needed "try" around creating the object would catch /all/ errors in there, and not just the "can't initialize" one ... I might have misunderstood that though. Regards, Rudy Wieser A function with no explicit return returns None; every function returns something. __init__ is not an object creation function (that's __new__), it initializes an already created object that we traditionally call "self". When you exception out of __init__, it causes that initial assignment of the newly created (but only halfway initialized) object to fail, so no one winds up holding a reference to it. Because no one's holding a reference to it, it gets deleted (one day). That's why if you for instance, open a serial port in an __init__ routine, and something fails, you want to catch the exception there in __init__, explicitly close that serial port, and reraise the exception. Otherwise that port still has an object open against it until Python gets around to closing it. The exception you raise should indicate why it failed, such as a TypeError or ValueError if one of the arguments was some flavor of stupid, or a ConnectionRefusedError. Often the exception will already raise itself, and it's simply a matter of not getting in its way. But if the concern is the try..except block around the initialization catching too many things, then either a) your try block is too long and encompasses too many unrelated lines of code, or b) your except condition is too broad. Don't "except Exception". Only catch specific exceptions where you can provide benefit by doing do. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: OOP - how to abort an __init__ when the initialisation code fails ?
On 11/4/19 10:36 AM, Luciano Ramalho wrote: Sorry, I responded only to the OP. My response: """A failed __init__ should raise an appropriate exception. A bare return or returning None is what any __init__ is expected to do in the normal case, so it signals success.""" Actually, the Python interpreter *does* check the return of __init__. If it is anything other than None, Python 3 raises TypeError when you try to create an instance: class X: ...def __init__(self): ...return 1 ... x = X() Traceback (most recent call last): File "", line 1, in TypeError: __init__() should return None, not 'int' Cheers, Luciano On Mon, Nov 4, 2019 at 2:31 PM Rob Gaddi wrote: >> [snip] Raise an exception. Returning None will specifically NOT accomplish the thing you want; nothing ever checks the return value of __init__. I'll be damned; you're right. Learn something new every day. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: OOP - how to abort an __init__ when the initialisation code fails ?
On 11/4/19 7:32 AM, R.Wieser wrote: Hello all, The whole question: How should I handle failed initialisation code inside the __init__ of an object ? I've seen an example doing a plain "return" (of the value "none""), but have no idea if that is all it takes. Also, I wonder what happens to the object itself. Does it still exist and only gets actually destroyed at the next garbage-collection (because of a zero reference count), or is it directly discarded (without having called the __del__ method) ? Regards, Rudy Wieser Raise an exception. Returning None will specifically NOT accomplish the thing you want; nothing ever checks the return value of __init__. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Renaming an import
I'm trying to figure out how to rename an import globally for an entire package. Something like: pkg/__init__.py: import graphing_module_b as graph pkg/foobar.py: from .graph import plot, axis The point being that, if at some point I decide to change from graphing_module_b to graphing_module_a, that decision is made at a single central point in my package rather than scattered through 30 different import statements in a dozen files. Any ideas? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Proper way to pass Queue to process when using multiprocessing.imap()?
On 9/3/19 10:17 AM, Israel Brewster wrote: When using pool.imap to apply a function over a list of values, what is the proper way to pass additional arguments to the function, specifically in my case a Queue that the process can use to communicate back to the main thread (for the purpose of reporting progress)? I have seen suggestions of using starmap, but this doesn’t appear to have a “lazy” variant, which I have found to be very beneficial in my use case. The Queue is the same one for all processes, if that makes a difference. I could just make the Queue global, but I have always been told not too. Perhaps this is an exception? --- Israel Brewster Software Engineer Alaska Volcano Observatory Geophysical Institute - UAF 2156 Koyukuk Drive Fairbanks AK 99775-7320 Work: 907-474-5172 cell: 907-328-9145 The first rule is to never use global variables. The second is to never put too much stock in sweeping generalizations. So long as you can keep that Queue's usage pattern fairly well constrained, go ahead and make it global. One thing to think about that might make this all easier though; have you looked at the concurrent.futures module? I find it does a fantastic job of handling this sort of parallelization in a straightforward way. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: itertools cycle() docs question
On 8/21/19 11:27 AM, Tobiah wrote:
In the docs for itertools.cycle() there is
a bit of equivalent code given:
def cycle(iterable):
# cycle('ABCD') --> A B C D A B C D A B C D ...
saved = []
for element in iterable:
yield element
saved.append(element)
while saved:
for element in saved:
yield element
Is that really how it works? Why make
the copy of the elements? This seems
to be equivalent:
def cycle(iterable):
while iterable:
for thing in iterable:
yield thing
You assume that the initial iterable is reusable. If its not, the only way you
can go back to the beginning is to have kept track of it yourself.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: My pseudocode to Python?
On 8/19/19 10:43 AM, Stefan Ram wrote: Can someone kindly translate my pseudocode to terse Python: list = \ [ 'afaueghauihaiuhgaiuhgaiuhgaeihui', 'erghariughauieghaiughahgaihgaiuhgaiuh', 'rejganregairghaiurghaiuhgauihgauighaei', if x: 'argaeruighaiurhauirguiahuiahgiauhgaeuihi', 'reiugaheirughauierhgauiaeihauiehgiuaehuih' 'ejiaeigaiuegreaiugheiugheaiughauighaiughaiu' 'egairughauirghauiruiegihgruiehgiuaehgiaue' ] ? I want the list to have the seven strings shown as entries if bool(x) is True, but otherwise the list should only have six entries - without the entry directly behind "if x: ". mylist = ['a', 'b', 'c', 'd', 'e', 'f', 'g'] if not x: del mylist[3] -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Boolean comparison & PEP8
On 7/29/19 10:44 AM, Michael F. Stemper wrote: On 28/07/2019 19.04, Chris Angelico wrote: On Mon, Jul 29, 2019 at 9:48 AM Michael Torrie wrote: On 7/28/19 5:55 AM, Jonathan Moules wrote: But this appears to be explicitly called out as being "Worse" in PEP8: """ Don't compare boolean values to True or False using ==. Yes: if greeting: No:if greeting == True: Worse: if greeting is True: """ Yet the recommended solution to the problem of wanting a default argument of an empty list is something like this: def foo(bar=False); if bar is False: bar = [] Clearly in this case the expression "not bar" would be incorrect. This is a fairly unusual case, though. More commonly, the default would be None, not False, and "if bar is None:" is extremely well known and idiomatic. That's certainly how I would have done it until I read your post. But reading it immediately raised the question of why not: def foo( bar=[] ): if len(bar)==0: print( "Pretty short" ) else: print( bar ) return Seems to work just fine. Works find right up until you do anything that modifies bar, and find that bar always points not to a new empty list each time but to the same empty list on each call. >>> def foo(bar=[]): ... bar.append(5) ... return bar ... >>> foo() [5] >>> foo() [5, 5] >>> foo() [5, 5, 5] As far as ways to shoot one's own foot in Python, this is one of the most common. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Nesting Custom Errors in Classes
On 7/22/19 7:54 PM, DL Neil wrote: Do you use nested classes? [following-on from the earlier, "Namespaces: memory vs 'pollution'" discussion thread, wherein a certain 'someone' remembered to from ... import ... as ... an 'action' class but forgot to also import the related custom error class! The original quest was for a wild-card import device. This discussion may obviate continuing the quest and/or needing to remember...] Python manages nested classes. I've NEVER seen such 'in the wild'. (but perhaps I lead a sheltered life?) What are proposed as use-cases for such a technique? - other languages may not offer decent "inheritance", and this is an alternative method/hack - the class is a 'class factory', generating/returning an object ? any others Why not use it as an "encapsulation" device? (please be gentle - reminder: I am too old to have been an OO-native!) * stub definitions >>> class PythonEnvironment(): ... class PythonVersionError( EnvironmentError ): ... pass ... def iscompatible( self ): ''' Ensure that the Python in-use will support all of the facilities employed by the application. ''' ... # stub to simulate failure ... raise self.PythonVersionError ... (code would require an import or from ... import ...) >>> pe = PythonEnvironment() * its basic application becomes:- >>> pe.iscompatible() Traceback (most recent call last): File "", line 1, in File "", line 6, in compatibility __main__.PythonVersionError * somewhat more realistic use:- >>> try: ... pe.iscompatible() ... except PythonEnvironment.PythonVersionError: ... print( "Trapped! -> informative errmsg" ) ... Trapped! -> informative errmsg With this construct, one only has to import the 'outer' class, rather than both the class AND its ancillary error class! Why haven't I seen it before? Can you see anything 'wrong' with this picture? I've used them sometimes for basic encapsulation principles without really gaining anything. I use inheritance of nested classes in my https://pypi.org/project/indexedproperty/ project, where subclasses of IndexedProperty have a _Trampoline inner class subclassed from Trampoline that can be redefined as needed, but that's a bit of an obscure use case. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Pythonic custom multi-line parsers
On 7/10/19 10:50 AM, Johannes Bauer wrote: Hi list, I'm looking for ideas as to a pretty, Pythonic solution for a specific problem that I am solving over and over but where I'm never happy about the solution in the end. It always works, but never is pretty. So see this as an open-ended brainstorming question. Here's the task: There's a custom file format. Each line can be parsed individually and, given the current context, the meaning of each individual line is always clearly distinguishable. I'll give an easy example to demonstrate: moo = koo bar = foo foo := abc def baz = abc Let's say the root context knows only two regexes and give them names: keyvalue: \w+ = \w+ start-multiblock: \w+ := The keyvalue is contained in itself, when the line is successfully parsed all the information is present. The start-multiblock however gives us only part of the puzzle, namely the name of the following block. In the multiblock context, there's different regexes that can happen (actually only one): multiblock-item: \s\w+ Now obviously whe the block is finished, there's no delimiter. It's implicit by the multiblock-item regex not matching and therefore we backtrack to the previous parser (root parser) and can successfully parse the last line baz = abc. Especially consider that even though this is a simple example, generally you'll have multiple contexts, many more regexes and especially nesting inside these contexts. Without having to use a parser generator (for those the examples I deal with are usually too much overhead) what I usually end up doing is building a state machine by hand. I.e., I memorize the context, match those and upon no match manually delegate the input data to backtracked matchers. This results in AWFULLY ugly code. I'm wondering what your ideas are to solve this neatly in a Pythonic fashion without having to rely on third-party dependencies. Cheers, Joe That's pretty much what I do. I generally make the parser a class and each state a method. Every line the parser takes out of the file it passes to self.statefn, which processes the line in the current context and updates self.statefn to a different method if necessary. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Use global, or not
On 6/28/19 6:44 AM, Cecil Westerhof wrote:
I have written a GUI program where I have quit a few global variables.
I did not like this, so I now use one global dict. Something like:
[snip]
global_dict = {
'messages': messages,
'progress': progress,
'window': window,
}
And in the program do things like:
global_dict['progress']['value'] += 1
global_dict['window'].update_idletasks()
global_dict['window'].update_idletasks()
and:
messagebox.showwarning(global_dict['titles']['warning'],
global_dict['messages']['nofiles'])
Is that an acceptable way to do this?
It works. I guess it makes your use of items in the global namespace inherently
intentional rather than allowing accidental globals. I'm not sure how much
value it adds though.
That said, I've got a whole bunch of programs that all use a dict called
Registry that they all import from a shared package; making them program-wide
globals. I use it for a few pieces of static information ('application',
'version') as well as to pass device handles around, since an open connection to
a given piece of physical hardware is an inherently global thing.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: How control a GUI for an unrelated application from a Python script?
On 6/14/19 11:14 AM, Christian Seberino wrote: Out of curiosity, what hardware? Texas Instruments ADS1675REF card Condolences. TI is a world-leader in giving every eval board its own complicated, proprietary digital interface, then not documenting it because "You can just use the provided software" that hasn't been updated since 2001 and doesn't actually let you test the thing you need to. The underlying FPGA board that it's built on has its own page at https://opalkelly.com/products/xem3010/ with an SDK. That may turn out to be your best way in. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How control a GUI for an unrelated application from a Python script?
On 6/14/19 8:49 AM, Christian Seberino wrote: Thanks for all the help. I'll definitely try to bypass the GUI first if possible. This is on Windows 7 so maybe AutoIt will do the trick if can't avoid the GUI. Thanks again everyone. Out of curiosity, what hardware? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Why I am getting IndexError: tuple index out of range when converting a float value to a string?
On 6/11/19 9:52 AM, madhavanbom...@gmail.com wrote:
I wrote the following lines of code:
xm = np.nanmean(x[indx],axis=None,dtype=float)
xsd = np.nanstd(x[indx],axis=None,dtype=float)
type(xm)# np.float64
type(xsd# np.float64
print(xm) # 0.5414720812182742
print(xsd) # 0.15748041033663002
print(str("{6.5f}".format(xm)))
I get the following error:
IndexError: tuple index out of range
Can someone suggest me why I am getting this error and how to overcome this?
Thanks in advance
print(str("{:6.5f}".format(xm)))
You want to apply the format 6.5f to the autoindexed element. You could
also explicitly index the 1st element (#0) as {0:6.5f}.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: More CPUs doen't equal more speed
On 5/23/19 6:32 PM, Cameron Simpson wrote: On 23May2019 17:04, bvdp wrote: Anyway, yes the problem is that I was naively using command.getoutput() which blocks until the command is finished. So, of course, only one process was being run at one time! Bad me! I guess I should be looking at subprocess.Popen(). Now, a more relevant question ... if I do it this way I then need to poll though a list of saved process IDs to see which have finished? Right? My initial thought is to batch them up in small groups (say CPU_COUNT-1) and wait for that batch to finish, etc. Would it be foolish to send send a large number (1200 in this case since this is the number of files) and let the OS worry about scheduling and have my program poll 1200 IDs? Someone mentioned the GIL. If I launch separate processes then I don't encounter this issue? Right? Yes, but it becomes more painful to manage. If you're issues distinct separate commands anyway, dispatch many or all and then wait for them as a distinct step. If the commands start thrashing the rest of the OS resources (such as the disc) then you may want to do some capacity limitation, such as a counter or semaphore to limit how many go at once. Now, waiting for a subcommand can be done in a few ways. If you're then parent of all the processes you can keep a set() of the issued process ids and then call os.wait() repeatedly, which returns the pid of a completed child process. Check it against your set. If you need to act on the specific process, use a dict to map pids to some record of the subprocess. Alternatively, you can spawn a Python Thread for each subcommand, have the Thread dispatch the subcommand _and_ wait for it (i.e. keep your command.getoutput() method, but in a Thread). Main programme waits for the Threads by join()ing them. I'll just note, because no one else has brought it up yet, that rather than manually creating threads and/or process pools for all these things, this is exactly what the standard concurrent.futures module is for. It's a fairly brilliant wrapper around all this stuff, and I feel like it often doesn't get enough love. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Instance vs Class variable oddity
On 5/15/19 10:52 AM, Irv Kalb wrote:
I just saw some code that confused me. The confusion has to do with class
variables and instance variables. In order to understand it better, I built
this very small example:
class Test:
x = 0
def __init__(self, id):
self.id = id
self.show()
def show(self):
print('ID:', self.id, 'self.x is:', self.x)
def increment(self):
self.x = self.x + 1
# Create two instances of the Test object, each shows itself
t1 = Test('first')
t2 = Test('second')
# Ask t1 to increment itself twice
t1.increment()
t1.increment()
# Ask each to show themselves
t1.show()
t2.show()
# Let's see what the class has
print('Test.x is:', Test.x)
When I instantiate two objects (t1 and t2), the __init__ method calls the show
method, which prints a value of self.x. I'm not understanding why this is
legal. I would expect that I would get an error message saying that self.x
does not exist, since no instance variable named self.x has been defined.
However, this code runs fine, and gives the following output:
ID: first self.x is: 0
ID: second self.x is: 0
ID: first self.x is: 2
ID: second self.x is: 0
Test.x is: 0
My guess is that there is some scoping rule that says that if there is no
instance variable by a given name, then see if there is one in the class. If
that is the case, then this line in the increment method seems odd:
self.x = self.x + 1
If the self.x on the right hand side refers to the class variable, and creates
an instance variable called self.x on the left hand side, then how does the
second call work using the value of the instance variable on the right hand
side? Can someone explain what's going on here?
Pretty much exactly like that. The first time you call t1.increment,
you fail to find x in the instance dictionary, fall back and find it
equal to 0, add 1 makes 1, and store x=1 into the instance dictionary.
The second time, you find x=1 in the instance dictionary, add 1 makes 2,
and overwrite x=2 into the instance dictionary.
You never call increment on t2, so it never gets an x in its instance
dictionary, so t2.x still refers to Test.x.
Disclaimer: I would never write code like this, but I saw this in someone
else's code and didn't understand how it was working.
Sometimes when I'm feeling lazy it's a convenient way to set defaults on
instance variables. I don't love it, but I also can't explain what I
find wrong with it.
Irv
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Using a Variable Inside the String
On 5/7/19 7:55 AM, sveem...@gmail.com wrote:
On Tuesday, May 7, 2019 at 1:20:49 AM UTC+5:30, MRAB wrote:
On 2019-05-06 19:44, sveem...@gmail.com wrote:
Hi all,
I am new learner of python programming and I am into my basics.
I got struck with the code in a sample program and refer the same with IDE
and googling which did not help me for more than a day.
What is wrong with my last 2 lines?
what I am missing?
##
my_name = 'Veera'
my_age = 40 # Actually 42
my_height = 155 # Inches
my_weight = "80" # checking the double quotes
my_eyes = 'brown'
my_teeth = 'white'
my_hair = 'black'
print(f"Let's talk about {my_name}.")
print(f"He's {my_height} inches tall.")
print(f"He's {my_weight} weight heavy.")
print("Actually thats not too Heavy.")
print(f"He's got {my_eyes} eyes and {my_hair} hair.")
print(f"His teeth are actually {my_teeth} depending on the coffee.")
### this line are where I Struck #
'my_age' and 'my_height' are numbers, but 'my_weight' is a string
(why?). You can't add a number and a string together.
total = my_age + my_height + my_weight
print(f"If I add {my_age}, {my_height}, and {my_weight} I get {total} .")
##
Thanks . It worked .
Will have to read more on strings to get the exact total value..
The comment can't be ignored because it's sitting there flagging your
problem. Strings (in single or double quote) are strings; a sequence of
characters as used to write words. Numbers are numbers. You can't add
them together.
This over here is my friend Bob. What's 31 + 18 + Bob? The question
makes no sense, because the operation of addition isn't defined in such
a way that allows you to add numbers to human beings. You can't add
them to strings either.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Function to determine list max without itertools
On 4/19/19 5:34 PM, MRAB wrote: [snip] How would you avoid comparing the initial max with list[0]? By slicing the list? By using 'iter' and 'next'? Do you expect that doing either of those to avoid a single comparison would make it faster? Is a comparison very expensive? You could, of course, do some benchmarks to see if it makes a difference, but, personally, I'd just leave it. Personally yes, but because it allows it to execute on non-sequence iterables rather than because it saves a single near-instantaneous comparison. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Function to determine list max without itertools
On 4/19/19 12:23 AM, Sayth Renshaw wrote: On Friday, 19 April 2019 17:01:33 UTC+10, Sayth Renshaw wrote: Set the first item in the list as the current largest. Compare each subsequent integer to the first. if this element is larger, set integer. def maxitwo(listarg): myMax = listarg[0] for item in listarg: if item > myMax: myMax = item return myMax Sayth When you understand what it is you intend to write (barring DL Neil's comments), and THEN write it, you write the correct thing. Thus endith the lesson. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Function to determine list max without itertools
On 4/18/19 4:35 PM, Sayth Renshaw wrote: It's still overly complicated. This is where I have ended up. Without itertools and max its what I got currently. def maximum(listarg): myMax = listarg[0] for item in listarg: for i in listarg[listarg.index(item)+1:len(listarg)]: if myMax < i: myMax = i return myMax How would you simplify it? In English rather than Python, how do you find the maximum element in a list? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: What does "TypeError: unsupported operand type(s) for +: 'function' and 'int'" mean?
On 3/25/19 12:30 PM, CrazyVideoGamez wrote: I have no idea what "TypeError: unsupported operand type(s) for +: 'function' and 'int'" means and I don't know how to fix it. Help! It means you can't add (i.e. apply the + operator) a function to an int. Which is only a problem because somewhere in your code, you're asking it to do that. In all likelihood, if you follow the line number provided in the traceback, you'll see that somewhere that you planned to use the result of calling a function you left out the parentheses and are instead using the function itself. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Why float('Nan') == float('Nan') is False
On 2/13/19 12:32 PM, Marko Rauhamaa wrote:
"Avi Gross" :
A NaN is a bit like a black hole. Anything thrown in disappears and
that is about all we know about it. No two black holes are the same
even if they seem to have the same mass, spin and charge. All they
share is that we don't know what is in them.
Then, how do you explain:
>>> float("nan") != float("nan")
True
Why's that not False?
Marko
Because IEEE-754 decided that it was non-optional that (x != y) was
equal to not (x == y). Which is not the case for the ordering
operators, since ordering is inherently undefined.
In part, these decisions were made to make it possible to detect a NaN
in C in the absence of an isnan() function. If (x != x), then x must be
a NaN.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: How can I find the indices of an array with float values in python?
On 1/10/19 9:25 AM, Peter Otten wrote: Madhavan Bomidi wrote: I have an array (numpy.ndarray) with shape (1500L,) as below: x = array([ 3.e+01, 6.e+01, 9.e+01, ..., 4.4940e+04, 4.4970e+04, 4.5000e+04]) Now, I wanted to determine the indices of the x values between 0.0 and 15.0. While this is simple in MATLAB or IDL by using find or where functions, I was unable to find a best way to find the indices of all elements between 0.0 and 15.0 in the x array. Can you please suggest me a solution for the same? Like this? a = numpy.array([-1, 10, 100, 5, 1000.]) numpy.where((a < 15) & (a > 0))[0] array([1, 3]) I was going to suggest numpy.nonzero, which gives the same result as the simple case of numpy.where as above. In either case, the result is a tuple of indices per axis. >>> a = [[2, 7, 6], [9, 5, 1], [4, 3, 8]] >>> a = np.array([[2, 7, 6], [9, 5, 1], [4, 3, 8]]) >>> b = np.array([1, 5, 2, 6, 3, 7, 4, 8, 5, 9]) >>> (a > 5).nonzero() (array([0, 0, 1, 2]), array([1, 2, 0, 2])) >>> np.nonzero(b % 2) (array([0, 1, 4, 5, 8, 9]),) -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: polar coordinates?
On 12/17/18 5:20 PM, Brian Christiansen wrote: I don't know if this follow up to my original message will even be seen because of all the spam advertising e-books that at least from the titles appear to have nothing to do with python. I apologize for being overly wordy in my description, but I suppose I was worried that if I did not go into great detail about exactly what I was trying to do, someone might suggest a package that solves part of my problem, but not all of it. I downloaded and installed matplotlib. At first I was a bit worried because I have seem to have 3 libraries that contain python "include" files: /usr/lib/python/python27, .../python3, and .../python35, and it at least appeared to install it into the python3 directory, but my IDE uses 3.5. Well I did some of the tutorials that are on YouTube, and it seems to work fine. Matplotlib makes the very simple plots that I wish to make in just a few statements, (scatter plot, etc.), but to make them into representations of PI, each dot on the scatter plot or whatever has to be a different color: point 1 is color(3), point 2 is color(1), point 3 is color(4), etc. I am not sure if matplotlib is able to make each dot in the scatterplot or whatever a differnt color in the single staement that makes to plot, or if a loop has to be set up to draw each of the dots individually in a different color, but I think it will work and with relatively simple code. Other statements in the thread I will take under advisement, I suppose. If I recall correctly you can make the color argument a list with the same number of elements as you have points. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Python, the stack, and the heap
I've been writing Python for good long while now without thinking too hard about this, but I just had a realization this weekend. Back when the earth's crust was still cooling and we all rode dinosaurs to our jobs, local variables got allocated onto the stack, and dynamic memory from malloc or DIM explicitly allocated variables on the heap. Python's objects all have a lifespan dictated by the continued existence of references to them and thus can transcend the lifetime of the current function in ways not known at translation time. So am I right in thinking that all Python objects are out on the heap? And that the references themselves may or may not wind up on the stack depending on what flavor you're running? Answers to these questions have very little bearing on how I actually write Python, mind, but now I'm curious. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: pandas read_csv
On 11/9/18 6:54 AM, Sharan Basappa wrote:
are there any requirements about the format of the CSV file when using read_csv
from pandas?
For example, is it necessary that the csv file has to have same number of
columns in every line etc.
I am trying to load a csv file and I get the following error.
I really don't know what the issue is ...
Any help is appreciated ...
ParserErrorTraceback (most recent call last)
D:\Projects\Initiatives\machine learning\programs\log\log_analysis_1.py in
()
10
11 #data_df = pd.read_csv("BGL_MERGED.log", nrows=100, header=None,
delimiter=r'\s"')
---> 12 data_df = pd.read_csv("d:\Projects\Initiatives\machine
learning\programs\log\BGL_MERGED.log", nrows=100, header=None)
13 logger.debug("data frame %s \n", data_df)
14 print('Cols %d' %(len(data_df.columns)))
D:\Users\sharanb\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\pandas\io\parsers.pyc
in parser_f(filepath_or_buffer, sep, delimiter, header, names, index_col,
usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters,
true_values, false_values, skipinitialspace, skiprows, nrows, na_values,
keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates,
infer_datetime_format, keep_date_col, date_parser, dayfirst, iterator,
chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting,
escapechar, comment, encoding, dialect, tupleize_cols, error_bad_lines,
warn_bad_lines, skipfooter, skip_footer, doublequote, delim_whitespace,
as_recarray, compact_ints, use_unsigned, low_memory, buffer_lines, memory_map,
float_precision)
653 skip_blank_lines=skip_blank_lines)
654
--> 655 return _read(filepath_or_buffer, kwds)
656
657 parser_f.__name__ = name
D:\Users\sharanb\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\pandas\io\parsers.pyc
in _read(filepath_or_buffer, kwds)
409
410 try:
--> 411 data = parser.read(nrows)
412 finally:
413 parser.close()
D:\Users\sharanb\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\pandas\io\parsers.pyc
in read(self, nrows)
1003 raise ValueError('skipfooter not supported for
iteration')
1004
-> 1005 ret = self._engine.read(nrows)
1006
1007 if self.options.get('as_recarray'):
D:\Users\sharanb\AppData\Local\Enthought\Canopy\edm\envs\User\lib\site-packages\pandas\io\parsers.pyc
in read(self, nrows)
1746 def read(self, nrows=None):
1747 try:
-> 1748 data = self._reader.read(nrows)
1749 except StopIteration:
1750 if self._first_chunk:
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader.read
(pandas\_libs\parsers.c:10862)()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_low_memory
(pandas\_libs\parsers.c:11343)()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._read_rows
(pandas\_libs\parsers.c:11884)()
pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._tokenize_rows
(pandas\_libs\parsers.c:11755)()
pandas/_libs/parsers.pyx in pandas._libs.parsers.raise_parser_error
(pandas\_libs\parsers.c:28765)()
ParserError: Error tokenizing data. C error: Expected 1 fields in line 8, saw 3
Offhand, and as a guess based on nothing, I would speculate that
something about line 8 of your CSV file differs from lines 1-7.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Overwhelmed by the Simplicity of Python. Any Recommendation?
On 11/4/18 8:55 AM, Mike C wrote: Same here. Debugging in Python is annoying, I like to step through my code line by line, it's impossible to do it with object-oriented programming language. Also, there's no good REPL IDE. Spyder barely works with some basic features. PyCharm, the most popular, takes too long to start, and you have to setup folders and directories EVERY SINGLE TIME at startup. I've never been a fan of IDEs, but a code editor window on the left and the IPython QtConsole on the right is a pretty efficient way to blaze through code. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: @staticmethod or def function()?
On 10/31/18 8:53 AM, Tobiah wrote: My IDE (pycharm) suggests that I mark my class methods with @staticmethod when they don't use 'self'. Sounds good. I did that then it suggested I had the option to make a regular function instead, at the file level. That's a possibility. I was thinking that I'd leave the method in the class unless it was ever also used by another class. What do you think? It should be a staticmethod if the code is, while not linked to any instance in particular, still fundamentally inherent to the class itself and of little use outside the context of that class. It should be a stand-alone function if it provides stand-alone functionality. The decision is also almost certainly not worth the amount of thought you're giving it. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Important Language Choice Considerations
https://boingboing.net/2018/10/15/python-falls-from-ceiling-in-b.html Say what you want about performance and linguistic elegance, but Julia almost never falls in through the ceiling. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Overwhelmed by the Simplicity of Python. Any Recommendation?
On 10/11/2018 11:29 PM, Kaan Taze wrote: Hi everyone, Since this is my first post to mail-list I'm kind of hesitant to ask this question here but as many of you spend years working with Python maybe some of you can guide me. What I trouble with is not a logical error that exist on a program I wrote. It's the Python itself. Well, I'm 22 years old CS student -from Turkey- and what they showed us at university was C Language and Java but I mainly use C in school projects etc. So it's been few months that I started to use Python for my personal side-projects. There are lots of resources to learn language. I do what I need to do with Python too but I was kinda shocked when I solve Python questions at Hackerrank. Even with list comprehensions you can implement in very smart way to get things done and easy. Iterations, string operations. The codes I see on the Internet using basics in a very clever way which I couldn't come up with the same solution if I tried to for some time. I do understand this ways but coming from ANSI C makes it hard to see this flexibility. I probably do things in a both inefficient and hard way in my projects. How do I get used to this? Is this just another "practice, practice, practice" situation? Anything you can recommend? All the best. Kaan. A) Yes, it's practice practice practice. B) Don't get hung up on finding the clever solution. Comprehensions and generators and lots of other things are great under some circumstances for making the code clearer and easier to read, but they too can become the hammer that makes everything look like a nail. The most important thing is that your code is logical, clean, and easy to understand. If it doesn't take full advantage of the language features, or if the performance isn't optimized to within an inch of its life, well so be it. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Object-oriented philosophy
On 09/06/2018 09:07 AM, Thomas Jollans wrote: On 2018-09-06 17:40, Abdur-Rahmaan Janhangeer wrote: Also, get someone, preferrable a python engineer to review your code. Does anyone here know anyone who would refer to themselves as a "Python engineer" with a straight face? I merely ask... -- Thomas Suddenly I'm filled with visions of pipe, fittings, and a herpetology degree. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Question about floating point
On 08/28/2018 07:11 AM, Frank Millman wrote: Hi all I know about this gotcha - x = 1.1 + 2.2 x 3.3003 According to the docs, the reason is that "numbers like 1.1 and 2.2 do not have exact representations in binary floating point." So when I do this - y = 3.3 y 3.3 what exactly is happening? What is 'y' at this point? Or if I do this - z = (1.1 + 2.2) * 10 / 10 z 3.3 What makes it different from the first example? Thanks Frank Millman https://en.wikipedia.org/wiki/IEEE_754 Python uses what the C folks call doubles, and what IEEE calls a binary64. If you look down at the precision chart, you'll see that for a value V, you have precision limits on the order of (V * 1e-16). If you strip off the 3.3 that you wanted from the result that you got, your error there is 3e-16, and all that's rounded for display, so all you can really talk about is order of magnitude. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: regex pattern to extract repeating groups
On 08/25/2018 04:55 PM, Malcolm wrote:
I am trying to understand why regex is not extracting all of the
characters between two delimiters.
The complete string is the xmp IFD data extracted from a .CR2 image file.
I do have a work around, but it's messy and possibly not future proof.
Any insight greatly appreciated.
Malcolm
My test code is
import re
# environment # Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18)
[MSC v.1900 64 bit (AMD64)] on win32 # extract of real data for test
purposes. This extract is repeated # the delimiters are extract =''' abcdef zxcvb
''' # modify the test data modified_extract_1
=''' abcdef zxcvb
''' # modify test data version 2 this works
modified_extract_2 =''' abcdef zxcvb
''' re_pattern =r'( *re.search(re_pattern, extract, re.DOTALL))
# >>> s1 <_sre.SRE_Match object; span=(1, 89), match=' \n
\n abcd> print('modified_extract_1',
re.search(re_pattern, modified_extract_1, re.DOTALL))
# >>> sre.SRE_Match object; span=(1, 70),
match='\n\nabcdef zxcvb
print('modified_extract_2', re.search(re_pattern, modified_extract_2,
re.DOTALL))
# >>> s <_sre.SRE_Match object; span=(1, 49),
match='\nabcdef zxcvb\n # NOTE the
missing ':' from the
Regexes are generally regarded as a bad way to parse XML data, and I
believe are provably unsuited in the general case (though can be beaten
into sufficiency in specific ones).
Use https://docs.python.org/3.6/library/xml.etree.elementtree.html
instead. Everything will just work. You'll be happier and more
productive, with a brighter smile and glossier coat.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Checking whether type is None
On 07/24/2018 01:07 PM, Chris Angelico wrote:
On Wed, Jul 25, 2018 at 5:33 AM, Tobiah wrote:
Consider:
>>> type({}) is dict
True
>>> type(3) is int
True
>>> type(None) is None
False
Obvious I guess, since the type object is not None.
So what would I compare type(None) to?
>>> type(None)
>>> type(None) is NoneType
Traceback (most recent call last):
File "", line 1, in
NameError: name 'NoneType' is not defined
I know I ask whether:
>>> thing is None
but I wanted a generic test.
I'm trying to get away from things like:
>>> type(thing) is type(None)
because of something I read somewhere preferring
my original test method.
There is nothing more generic in a type test than in simply saying "is
None". There are no other instances of NoneType. Don't try
type-checking None; just check if the object is None.
ChrisA
I suppose one valid usage would be this sort of thing:
fn = {
int: dispatchInt,
str: dispatchStr,
list: dispatchList,
type(None): dispatchNone
}[type(x)]
fn(x)
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Something new which all programmers world wide will appreciate
On 06/27/2018 02:14 PM, skybuck2...@hotmail.com wrote: Now I don't like the French much ! LOL. But this time they have invented something which will fill programmers with tears of joy ! =D http://www.euronews.com/2018/06/27/pizza-making-robot Hopefully this will lead to cheaper and delicious pizzas in the future ! ;) =D Bye, Skybuck. Or, you know, someone didn't bother putting limit checks in and a time out of 20 the thing gets lost and starts putting the sauce directly on the customer. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: configparser v/s file variables
On 06/27/2018 12:15 PM, Jim Lee wrote: On 06/27/18 11:45, Abdur-Rahmaan Janhangeer wrote: and that closes it, thanks !!! Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Importing variables from a file is dangerous because it can execute arbitrary code. It should never be done with files provided by the user. Using configparser is far, far safer. It seems a bit silly to me to worry about arbitrary code execution in an interpreted language like Python whose default runtime execution method is to parse the source code directly. An attacker would be far more likely to simply modify the source to achieve his ends rather than try to inject a payload externally. These days, "execute arbitrary code" implies a deliberate attack. Now, if you used input validation as an argument, I would agree that configparser is, if not safer, easier. -Jim Not at all. Because if you're assuming a malicious user (who wasn't the one to install it), then you're assuming a multi-user environment. In which case the malicious user wouldn't have modify access to the code, unless your program says "Hey, Mal E. Factor, why don't you run your arbitrary code in my environment?" -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python list vs google group
On 06/15/2018 11:44 AM, Larry Martell wrote: On Fri, Jun 15, 2018 at 2:08 PM, Rich Shepard wrote: On Fri, 15 Jun 2018, Larry Martell wrote: Oh, ... MUA == Mail User Agent. I thought it was Made Up Acronym Larry, Could be; depends on the context. My favorite acronym of all time is TWAIN Really? I always thought it didn't scan. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Override built in types... possible? or proposal.
On 05/31/2018 07:49 AM, Dan Strohl wrote:
Is it possible to override the assignment of built in types to the shorthand
representations? And if not, is it a reasonable thought to consider adding?
For example, right now, if I do:
test = "this is a string",
I get back str("this is a string"). What if I want to return this as
my_string("this is a string") (OK, I know I have a recursive issue in my example, but
hopefully you get the point).
Or;
Test = ['item1', 'item2', 'item3'] returns a list, what if I want to add
functionality to all lists in my module? (and yes, I know I could simply not
do [] and always do my_list('item1', 'item2', 'item3']
I am envisioning something in the header like an import statement where I could
do;
override str=my_string
override list=my_list
This would only be scoped to the current module and would not be imported when
that module was imported.
Thoughts?
Dan Strohl
My problem with this idea is that it breaks expectations. If I know one
thing as a Python programmer, it's that 'Bob' is a str. Each time and
every time. If you could override the meaning of basic constant
identifiers to where I have no idea how they behave, that creates an
easy thing to miss that changes the entire meaning of the things you've
written.
What's the use case here? And why is that use case better than, for
instance, simply defining a function in the module that does the things
you want done to strings? Not everything has to be an object method.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pink Floyd: is there anybody in here?
On 05/30/2018 09:34 AM, Paul Rubin wrote: I think Usenet posts are no longer getting forwarded to the mailing list, but now I wonder if this is getting out at all, even to usenet. Does anyone see it? Can't speak for the mailing list, but this came out to Usenet just fine. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: List replication operator
On 05/25/2018 10:13 AM, bartc wrote:
On 25/05/2018 17:58, Rob Gaddi wrote:
So, in the spirit of explicit being better than implicit, please
assume that for actual implementation replicate would be a static
method of actual list, rather than the conveniently executable hackjob
below.
_list = list
_nodefault = object()
class list(_list):
@staticmethod
def replicate(*n, fill=_nodefault, call=list):
That seems to work, but the dimensions are created in reverse order to
what I expected. Which is to have the order of indices corresponding to
the order of dimensions. So:
x=list.replicate(2,3,4)
print (len(x))
print (len(x[0]))
print (len(x[0][0]))
Gives output of 4, 3, 2 rather than 2, 3, 4.
Which means that x[0][0][3] is a bounds error.
A fair point. Something about multidimensional arrays always makes me
semi-dyslexic. And there I was wondering why making it work right had
required I rip dimensions off the list from the end instead.
Corrected version below. list.replicate(3, 2) now means "3 of 2 of
this" as one would expect. This is one of those days when doctest is my
favorite thing about Python.
_list = list
_nodefault = object()
class list(_list):
@staticmethod
def replicate(*n, fill=_nodefault, call=list):
"""Return a list of specified dimensions.
Fill and call can be used to prime the list with initial values, the
default is to create a list of empty lists.
Parameters:
n : List of dimensions
fill: If provided, the fill object is used in all locations.
call: If fill is not provided, the result of call (a function of
no arguments) is used in all locations.
Example:
>>> a = list.replicate(3, 2)
>>> a
[[[], []], [[], []], [[], []]]
>>> a[0][0].append('a')
>>> a
[[['a'], []], [[], []], [[], []]]
>>> b = list.replicate(3, 2, fill=[])
>>> b
[[[], []], [[], []], [[], []]]
>>> b[0][0].append('a')
>>> b
[[['a'], ['a']], [['a'], ['a']], [['a'], ['a']]]
>>> c = list.replicate(3, 2, call=dict)
>>> c
[[{}, {}], [{}, {}], [{}, {}]]
>>> c[0][0]['swallow'] = 'unladen'
>>> c
[[{'swallow': 'unladen'}, {}], [{}, {}], [{}, {}]]
>>> d = list.replicate(3, 2, fill=0)
>>> d
[[0, 0], [0, 0], [0, 0]]
>>> d[0][0] = 5
>>> d
[[5, 0], [0, 0], [0, 0]]
"""
if n:
this = n[0]
future = n[1:]
return [
list.replicate(*future, fill=fill, call=call)
for _ in range(this)
]
elif fill is _nodefault:
return call()
else:
return fill
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: List replication operator
On 05/25/2018 04:53 AM, Steven D'Aprano wrote:
On Fri, 25 May 2018 09:28:01 +0200, Peter Otten wrote:
Yet another arcanum to learn for beginners with little return. If you
cannot refrain from tinkering with the language at least concentrate on
the features with broad application. Thank you.
Broader than multi-dimensional arrays? There are a bazillion uses for
them. How many do you need before it is "broad application"?
> [snip]
This is a frequent, recurring pain point. Experienced programmers forget
how confusing the behaviour of * is because they're so used to the
execution model. They forget that writing a list comp is not even close
to obvious, not only for beginners but even some experienced Python
programmers.
I agree that it's non-obvious, but I disagree with your diagnosis. The
non-obvious bit is that the empty list is a reference to a newly created
mutable, not an immutable constant. Or, more the point, that the word
"mutable" is one that you need to know and think about in the first place.
The thing tripping the young pups is not understanding the underlying
Python concepts of objects and their references. And I sympathize;
Python's handling there is seriously non-trival. I guarantee everyone
on this list got bit by it when they were starting out, either in this
case or as a default argument. I probably still manage to screw it up a
couple times a year due to not thinking about it clearly enough.
So, in the spirit of explicit being better than implicit, please assume
that for actual implementation replicate would be a static method of
actual list, rather than the conveniently executable hackjob below.
_list = list
_nodefault = object()
class list(_list):
@staticmethod
def replicate(*n, fill=_nodefault, call=list):
"""Return a list of specified dimensions.
Fill and call can be used to prime the list with initial values, the
default is to create a list of empty lists.
Parameters:
n : List of dimensions
fill: If provided, the fill object is used in all locations.
call: If fill is not provided, the result of call (a function of
no arguments) is used in all locations.
Example:
>>> a = list.replicate(2, 3)
>>> a
[[[], []], [[], []], [[], []]]
>>> a[0][0].append('a')
>>> a
[[['a'], []], [[], []], [[], []]]
>>> b = list.replicate(2, 3, fill=[])
>>> b
[[[], []], [[], []], [[], []]]
>>> b[0][0].append('a')
>>> b
[[['a'], ['a']], [['a'], ['a']], [['a'], ['a']]]
>>> c = list.replicate(2, 3, call=dict)
>>> c
[[{}, {}], [{}, {}], [{}, {}]]
>>> c[0][0]['swallow'] = 'unladen'
>>> c
[[{'swallow': 'unladen'}, {}], [{}, {}], [{}, {}]]
>>> d = list.replicate(2, 3, fill=0)
>>> d
[[0, 0], [0, 0], [0, 0]]
>>> d[0][0] = 5
>>> d
[[5, 0], [0, 0], [0, 0]]
"""
if n:
this = n[-1]
future = n[:-1]
return [
list.replicate(*future, fill=fill, call=call)
for _ in range(this)
]
elif fill is _nodefault:
return call()
else:
return fill
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: List replication operator
On 05/24/2018 11:17 AM, Steven D'Aprano wrote: Python has a sequence replication operator: py> [1, 2]*3 [1, 2, 1, 2, 1, 2] Unfortunately, it is prone to a common "gotcha": py> x = [[]]*5 # make a multi-dimensional list py> x [[], [], [], [], []] py> x[0].append(1) py> x [[1], [1], [1], [1], [1]] The reason for this behaviour is that * does not copy the original list's items, it simply replicates the references to the items. So we end up with a new list containing five references to the same inner list. This is not a bug and changing the behaviour is not an option. But what do people think about proposing a new list replication with copy operator? [[]]**5 would return a new list consisting of five shallow copies of the inner list. Thoughts? I think the same people making that mistake now would still do so because they didn't know they needed the special operator. [[] for _ in range(5)] works just as well without adding more syntax. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: how to get INDEX count, or last number of Index
On 05/23/2018 02:51 PM, asa32s...@gmail.com wrote: s = "kitti" 0,1,2,3,4 k,i,t,t,i how do i retrieve '4'. i know i can do a len(s)-1, but i assume there is a function that gives me last number of index thanks Not sure I'm following your question; len(s)-1 is much faster than enumerating over the string just to get the last index. If what you want is the current index, though, you can look at the enumerate function s='kitti' for i, c in enumerate(s): print(i, ':', c) -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Numpy array
On 05/18/2018 09:50 PM, Sharan Basappa wrote:
This is regarding numpy array. I am a bit confused how parts of the array are
being accessed in the example below.
1 import scipy as sp
2 data = sp.genfromtxt("web_traffic.tsv", delimiter="\t")
3 print(data[:10])
4 x = data[:,0]
5 y = data[:,1]
Apparently, line 3 prints the first 10 entries in the array
line 4 & 5 is to extract all rows but only 1st and second columns alone for x
and y respectively.
I am confused as to how data[:10] gives the first 10 rows while data[:,0] gives
all rows
Numpy ordering is [rows,columns,...]. Things you don't provide an
explicit answer for are assumed to be "all", which in Python syntax is
":". The reason for THAT goes into how slices are constructed.
So your example 3 could also be written print(data[0:10,:]), which sets
rows to the range starting with 0 and ending before 10, and columns to
everything.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Print Failure or success based on the value of the standalone tool
On 05/10/2018 09:48 AM, Ganesh Pal wrote: I have to test a standalone tool from a python script and I am using os.system() to run the tool . I need to take decision based on the return value of the standalone tool . But since os.system merely throws the output value to STDOUT & returns the exit status (0 => no error) of the shell , how can I make this print Fail or Pass based on the return value of the standalone tool. In the below example ( return_values() function) [snip] By not using os.system, it's been superseded for reasons exactly like yours. https://docs.python.org/3/library/subprocess.html is your friend. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: seeking deeper (language theory) reason behind Python design choice
On 05/10/2018 03:02 AM, bartc wrote: On 10/05/2018 09:09, Marko Rauhamaa wrote: bartc <b...@freeuk.com>: When typing in code (in various languages), I have a habit of typing "..." at places that need to be implemented. For example: if count: ... else: do_something_smart() break the idea being that "..." will surely trigger a syntax error if I forget to address it. I was mildly amused when Python happily executed such code. "..." is a valid expression and thus, a valid statement. I wondered what it meant, so I typed in: print (...) and it displayed: Ellipsis which wasn't very enlightening. No, but if you ever have difficulty remembering how to spell "ellipsis", it's good to know Python comes with a built-in reference. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Looking for advice
On 04/20/2018 10:28 AM, 20/20 Lab wrote: Going to write my first python program that uses a database. Going to store 50-100 rows with 5-10 columns. Which database / module would you advise me to use? It's basically going to be processing order status emails for the sales staff. Producing a webpage (2-3 times daily, as updates arrive) that has the sales staff orders and status on it. I'm thinking just a simple sqlite, but dont want to waste time going down the wrong path. Thank you for your time SQLite, without question. It just works in a way that server-based solutions will give you all manner of initial bootstrap issues. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: RE newbie question
On 04/18/2018 12:37 PM, TUA wrote:
import re
compval = 'A123456_8'
regex = '[a-zA-Z]\w{0,7}'
if re.match(regex, compval):
print('Yes')
else:
print('No')
My intention is to implement a max. length of 8 for an input string. The above
works well in all other respects, but does allow for strings that are too long.
What is the proper way to fix this?
Thanks for any help!
You could put the end marker $ on your regex so that it won't match if
it's not the end.
Or, you know, you could just check len(compval) <= 8 and not get bogged
down in regexes. They tend to be excellent solutions to only a very
specific complexity of problem.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: try-except syntax
On 04/05/2018 02:04 PM, ElChino wrote:
I'm trying to simplify a try-except construct. E.g. how come
this:
try:
_x, pathname, _y = imp.find_module (mod, mod_path)
return ("%s" % pathname)
except ImportError:
pass
except RuntimeError:
pass
return ("")
Cannot be simplified into this:
try:
_x, pathname, _y = imp.find_module (mod, mod_path)
return ("%s" % pathname)
except ImportError:
except RuntimeError:
pass
return ("")
Like a "fall-through" in a C-switch statement.
try:
_x, pathname, _y = imp.find_module (mod, mod_path)
return ("%s" % pathname)
except (ImportError, RuntimeError):
pass
return ("")
That handles the identical case. C-style fall-throughs where you have
one switch have just a bit of code before the fall-through kicks in (the
slightly non-identical case) is often the source of disastrous code errors.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Pip Version Confusion
On 03/24/2018 08:40 AM, Tim Johnson wrote: I'm on Ubuntu 16.04. I'm getting the following message from pip: You are using pip version 8.1.1, however version 9.0.3 is available. You should consider upgrading via the 'pip install --upgrade pip' command. # But then I get this : tim@linus:~/Downloads$ which pip /home/tim/.local/bin/pip # and this: tim@linus:~/Downloads$ pip --version pip 9.0.3 from /home/tim/.local/lib/python2.7/site-packages (python 2.7) Not sure how to resolve - please advise, thanks -- Tim When pip 8.1.1 yells at you about version, are you installing something --user for yourself, or are you using sudo to install it systemwide. Because 'sudo which pip' is probably still pointed to the APT installed one. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Writing a C extension - borrowed references
On 03/20/2018 10:03 AM, Tom Evans wrote: On Tue, Mar 20, 2018 at 4:38 PM, Chris Angelico <ros...@gmail.com> wrote: BTW, have you looked into Cython? It's smart enough to take care of a lot of this sort of thing for you. I did a bit; this work is to replace our old python 2 SAML client, which used python-lasso and python-libxml2, both packages that are built as part of building the C library and thus an utter PITA to package for different versions of python (something our Infra team were extremely reluctant to do). When the latest (PITA to build) version of python-lasso started interfering with python-xmlsec (validating an invalid signature was causing segfaults), I got fed up of fighting it. I actually also maintain a C version of the same code, using the same libraries, so porting those few segments of code to Python/C seemed more expedient than rewriting in Cython. I'm not writing an API to these libraries, just a few functions. Cheers Tom If all you're doing is a thin-wrapper around a C library, have you thought about just using ctypes? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Thank you Python community!
On 03/19/2018 12:40 PM, Tim Daneliuk wrote: On 03/19/2018 02:05 PM, bartc wrote: I've often wondered what the guys who invented C (around 1970) must have been smoking to have come up with some of those ideas. I dunno, but I do know that - if they were smoking something - it was rolled in greenbar paper ... "Look, you can make a filter out of the pinfeed!" -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Context manager on method call from class
On 03/15/2018 11:17 AM, Joseph L. Casale wrote: I have a class which implements a context manager, its __init__ has a signature and the __enter__ returns an instance of the class. Along with several methods which implement functionality on the instance, I have one method which itself must open a context manager against a call on an instance attribute. This context manager does not return an instance of itself, it merely opens a context. I am not thrilled about is the implementation I have used and I am wondering if there is a better way. It's been a long time since I have worked in Python and I am probably overlooking something obvious. Syntactically I achieve what I want in use, but it looks awkward in its implementation. How can I clean up the first class and decorator to implement the functionality on a method in the Foo class? > [snip] from contextlib import contextmanager. Then you just use the @contextmanager decorator on a function, have it set up, yield the context you want, and clean up after. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: I found strange thing while studying through idle
On 03/08/2018 07:57 PM, Steven D'Aprano wrote: [snip] But it is possible that due to differences between platforms, the OP's version of IDLE doesn't display a carriage return as \r but rather as an invisible zero-width space. Just to derail this conversation a bit, does anyone have a use case in the modern (Py3) age for '\r'? I use b'\r' fairly regularly when talking to serial port devices. But the string version? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Unnoticed traceback in a thread (Posting On Python-List Prohibited)
On 03/08/2018 05:06 PM, Lawrence D’Oliveiro wrote: On Friday, March 9, 2018 at 11:43:00 AM UTC+13, Paul Rubin wrote: That you're using threading.Lock for something like this is a borderline code smell. Any use of multithreading in non-CPU-intensive code is a code smell. But CPU intensive code is exactly the circumstance where Python threading lets you down. It really shines when you're I/O-bound. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: RFC: Proposal: Deterministic Object Destruction (Posting On Python-List Prohibited)
On 03/01/2018 02:24 PM, Lawrence D’Oliveiro wrote: On Thursday, March 1, 2018 at 6:44:39 PM UTC+13, Paul Rubin wrote: DOM trees are a classic example (see the various DOM modules in the Python stdlib). Non-leaf nodes have a list of child nodes, child nodes have pointers back upwards to their parent, and each child node has pointers to its left and right siblings if they exist. It's all very circular. This is why you have weak refs. But making sure that the failure to use weakrefs doesn't memory leak the program into the pavement is why you have background garbage collection with intelligent cyclical reference breaking. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there are good DRY fix for this painful design pattern?
On 02/26/2018 07:28 AM, Rhodri James wrote: On 26/02/18 14:41, Steven D'Aprano wrote: I have a class with a large number of parameters (about ten) assigned in `__init__`. The class then has a number of methods which accept *optional* arguments with the same names as the constructor/initialiser parameters. If those arguments are None, the defaults are taken from the instance attributes. An example might be something like this: class Foo: def __init__(self, bashful, doc, dopey, grumpy, happy, sleepy, sneezy): self.bashful = bashful # etc def spam(self, bashful=None, doc=None, dopey=None, grumpy=None, happy=None, sleepy=None, sneezy=None): if bashful is None: bashful = self.bashful if doc is None: doc = self.doc if dopey is None: dopey = self.dopey if grumpy is None: grumpy = self.grumpy if happy is None: happy = self.happy if sleepy is None: sleepy = self.sleepy if sneezy is None: sneezy = self.sneezy # now do the real work... def eggs(self, bashful=None, # etc... ): if bashful is None: bashful = self.bashful # and so on There's a lot of tedious boilerplate repetition in this, and to add insult to injury the class is still under active development with an unstable API, so every time I change one of the parameters, or add a new one, I have to change it in over a dozen places. Is there a good fix for this to reduce the amount of boilerplate? You could use dicts? Untested code: class Foo: def __init__(self, **kwds): self.defaults = kwds def spam(self, **kwds): vars = self.defaults.copy() vars.update(kwds) # Do the work with vars['bashful'] etc That's what I was about to suggest (and equally fail to test). It at least segregates the set of "instance variables I want to have auto-default behavior" from the rest of them. Yeah, **kwargs is clumsy, but it gets the job done. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to work on a package
On 02/07/2018 03:17 PM, Grant Edwards wrote: On 2018-02-07, Rob Gaddi <rgaddi@highlandtechnology.invalid> wrote: When I'm working on a module, the trick is to write a setup.py (using setuptools) from the very get-go. Before I write a single line of code, I've got a setup.py and the directory framework. Then you install the package using pip -e (or in practice --user -e). That's the missing piece. That way you can import your module from the interpreter, because it's now on the path, but its physical location is right there where you left it, complete with your VCS metadata and etc. How do you work on a package that must remain installed and usable the whole time you're working on it? IOW, only specific test apps or apps run in a specific directory should get the "in-progress" foo module when they do an "import foo". Achievable with a virtualenv, but now the process is even that much more complicated. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to work on a package
On 02/07/2018 12:34 PM, Roel Schroeven wrote: dieter schreef op 7/02/2018 8:21: Likely, there are many ways to execute tests for your package. I am using "setuptools" for packaging (an extension of Python's standard "disutils"). Its "setup.py" supports the "test" command. This means, properly set up, I can run tests with "python setup.py test". That can solve the testing issue, I guess, even though it feels weird to me that the most straightforward way doesn't work. But testing is not the only issue. Often I'd like to start the Python interpreter to load one of the modules in the package to try some things out, or write a little script to do the same. These things are very natural to me when writing Python code, so it seems very strange to me that there's no easy way when working on a packages. Don't other people do that? The state of Python packaging is... unfortunate. Improving, due to huge amounts of work by some very dedicated people, but still unfortunate. When I'm working on a module, the trick is to write a setup.py (using setuptools) from the very get-go. Before I write a single line of code, I've got a setup.py and the directory framework. Then you install the package using pip -e (or in practice --user -e). That's the missing piece. That way you can import your module from the interpreter, because it's now on the path, but its physical location is right there where you left it, complete with your VCS metadata and etc. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: pip --user by default
On 01/13/2018 04:54 AM, Thomas Jollans wrote: Hi, I recently discovered the wonders of pip.conf: if I create a file ~/.config/pip/pip.conf* with: [install] user = true then pip will install to the --user site-packages by default, rather than trying to install packages into system directories. The trouble is that this fails when you also use virtualenvs. In a virtualenv, --user doesn't work, so pip fails when trying to install anything in a virtualenv as long as the user pip.conf contains those lines. Short of adding a pip.conf to every single virtualenv, is there any way to work around this, and configure pip to install packages - into the user directories if possible - into the environment when in an environment by default? Thanks, Thomas * the path obviously depends on the OS: https://pip.pypa.io/en/stable/user_guide/#config-file Inside of a virtualenv, what's the difference between a --user install and a system one? -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Native object exposing buffer protocol
On 01/05/2018 04:27 PM, Ben Finney wrote: Rob Gaddi <rgaddi@highlandtechnology.invalid> writes: I'd like to create a native Python object that exposes the buffer protocol. Basically, something with a ._data member which is a bytearray that I can still readinto, make directly into a numpy array, etc. The “etc.” seems pretty important, there. You want the behaviour of ‘bytearray’ without actually inheriting that behaviour from the ‘bytearray’ type. > Well, one specific behavior. Ideally, what I want is, when calls like RawIOBase.readinto and ndarray.frombuffer ask my class "Hey, do you have any raw bytes that I can read and potentially write?" it can answer "Why certainly, here they are." Something like a: class Thingy: def __memoryview__(self): return memoryview(self._data) But it doesn't look as if there's a dunder for that. There's __bytes__, but that specifically requires you give back a bytes() object; even returning a bytearray causes an error. So, it seems your options are: * Enumerate all the things, specifically, that you do want your new type to do. Don't hide anything in “etc.”, so that you know exactly what behaviours need to be implemented. Implement all those behaviours, without benefit of inheriting from ‘bytearray’. * Inherit from ‘bytearray’, but ameliorate the problems you want to avoid. This will require enumerating all those problems, so that you can know whether you have avoided them. Don't hide any of them in an “etc.”. That's ultimately the way I'm going about it, but since this is only going to get used in-house, I'm approaching it the Pythonic way. The problem is "Most methods of bytearray make no sense on a data structure representing a fixed length waveform." The solution is "Well, don't use them." Not the end of the world (the file's less than 2KB), but it seems like something that should be doable easily without having to throw around a lot of extraneous copies. I look forward to your report from having tried it :-) -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Native object exposing buffer protocol
I'd like to create a native Python object that exposes the buffer protocol. Basically, something with a ._data member which is a bytearray that I can still readinto, make directly into a numpy array, etc. I can do it by inheriting the entire thing from bytearray directly, but that gives me a whole lot of methods that are unsuitable for what I'm doing with it, which is reading the contents of a binary file, allowing them to be lightly malleable in memory, and then sending them on to a device. Not the end of the world (the file's less than 2KB), but it seems like something that should be doable easily without having to throw around a lot of extraneous copies. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python goto
On 12/28/2017 04:35 AM, Skip Montanaro wrote: Jorge> I would like to know if there is a goto command or something similar that Jorge> I can use in Python. Ned> Python does not have a goto statement. You have to use structured Ned> statements: for, while, try/except, yield, return, etc. Though it appears some wag has used function decorators to implement goto statements: https://pypi.python.org/pypi/goto-statement/1.1 Rather clever, it seems. Skip If only all that power had been used for good instead of evil... -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: property decorator?
On 12/20/2017 03:56 PM, Irv Kalb wrote: I am trying to work through the concept of the @property decorator with respect to object oriented programming. I believe that I understand how the @property and @.setter work - and that they are used to turn what looks like direct access to instance variables into method calls in an object.It seems like these two decorators are essentially fancy substitutes for traditional getter and setter methods. But, I'm having a hard time understanding why these names were chosen. > [snip] My questions about this are really historical. From my reading, it looks like using an @property decorator is a reference to an older approach using a built in "property" function. But here goes: 1) Why were these decorator names chosen? These two names @property and @.setter don't seem to be very clear to me. At a minimum, they don't match. Wouldn't decorator names like @.getter and @.setter have been better - more explicit? 2) Alternatively, if the getter was going to use the @property decorator, then it would seem complimentary to have the decorator name for the associated setter function to have the word "property" in its also, something like @propertySetter. 3) If I create a method with the @property decorator, is there anything else that is implied about the name of the method other than you can now refer to the . - which calls the appropriate method? My guess/understanding is that in my example above, "salary" is considered the property name, which is how you would refer to it outside of the object. Inside the class, you use the property name as the name of both the setter and the getter methods. Is that the right way to think about it? Finally, it seems very odd to me that when you use the @property decorator and the @.setter, that both of the methods that are decorated need to have the same name (but of course different set of parameters.) As a teacher, this seems like it would be an extremely difficult concept to get across to students, as this does not work the same way as other Python functions (and methods). Without a decorator, the second function of the same name overrides an earlier function of the same name, as in this simple example: > [snip] > Irv It's because of how Python implements decorators. When I write class C: def foobar(self): pass I create a binding in the local scope of class C with the name "foobar" that holds a function object. We call this a "local variable" in keeping with other languages, but it's not really the same as it would be in like C, it's a reference in the local namespace to some other object. When I decorate that foobar instead: class C: @property def foobar(self): pass That variable now, instead of referencing a function object, references an instance of type 'property' that holds a reference to the original foobar method as it's "_getter". When I expand that to class C: @property def foobar(self): return self.__foobar @foobar.setter def foobar(self, value): self.__foobar = value The second decoration calls the 'setter' method of the property instance stored in variable name foobar. That method assigns the new method to the "_setter" of that property instance and returns the updated property, which is assigned redundantly to the variable name foobar. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Learning
On 12/18/2017 01:52 PM, Gene Heskett wrote: On Monday 18 December 2017 16:05:10 Rob Gaddi wrote: On 12/18/2017 08:45 AM, Larry Martell wrote: On Mon, Dec 18, 2017 at 11:33 AM, Marko Rauhamaa <ma...@pacujo.net> wrote: However, one great way to stand out is a portfolio of GitHub projects. Several people have gotten an offer largely based on those (after they aced the technical interviews). For example, we just hired someone who had written a game in sed. That doesn't make him an "interesting person," nor do we look for game or sed developers. But that silly exercise deeply resonated with our team. We expect to have great synergy with him. I have been excluded from even getting an interview because I did not have a portfolio of GitHub projects. I think that is a bad filter. I work 60-70 hours a week for pay, and I have a family and personal interests. When I'm hiring I don't necessarily need a candidate to have an extensive open-source portfolio, but I need to see some kind of portfolio, just as a bar of "This is what I consider my good work to be." The idea that someone is going to have years of experience, but not a single page of code that they can let me look over always strikes me as odd. I've known Larry for years via another list. He has worked on a lot of stuff in the financial arena, wrapped in non-disclosure clauses that prevent him from even saying what color he dots the i's with, so he gets work as much by word of mouth as other more conventional paths. To not hire him because he doesn't have a big portfolio could be a mistake, Rob. Its up to you. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. Cheers, Gene Heskett Not a big portfolio, anything. Shell scripts that you're using to simplify your home login. The garage door monitor you rigged up from an Arduino. The Python script I wrote to fill a USB stick with a randomly chosen selection of MP3s. I as well don't have a huge portfolio of GitHub projects that I spend my life maintaining, but I refuse to believe any of us get away with not writing any code outside out professional lives. Personally, I'd argue I can get a better read of a programmer from the 120 line toy they tossed off one day to solve an immediate need than I can from the 50 kLOC proprietary project they share with two other developers and are NDAed to the hilt on. And as for Larry in particular, he's posted a portfolio's worth of code to this mailing list just in trying to help folks through their issues. For an actual interview I'd want it cleaned up to what he'd consider to be acceptable documentation standards for a project, but "demonstrate competence" is a well-cleared bar. I just don't have any work at the moment that would justify bringing that big a cannon to bear. So I stick to my point. If you have literally nothing that you can show me at an interview, I'm dubious. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Python Learning
On 12/18/2017 08:45 AM, Larry Martell wrote: On Mon, Dec 18, 2017 at 11:33 AM, Marko Rauhamaa <ma...@pacujo.net> wrote: However, one great way to stand out is a portfolio of GitHub projects. Several people have gotten an offer largely based on those (after they aced the technical interviews). For example, we just hired someone who had written a game in sed. That doesn't make him an "interesting person," nor do we look for game or sed developers. But that silly exercise deeply resonated with our team. We expect to have great synergy with him. I have been excluded from even getting an interview because I did not have a portfolio of GitHub projects. I think that is a bad filter. I work 60-70 hours a week for pay, and I have a family and personal interests. When I'm hiring I don't necessarily need a candidate to have an extensive open-source portfolio, but I need to see some kind of portfolio, just as a bar of "This is what I consider my good work to be." The idea that someone is going to have years of experience, but not a single page of code that they can let me look over always strikes me as odd. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Easiest way to access C module in Python
On 11/06/2017 05:59 PM, Grant Edwards wrote: On 2017-11-06, John Pote <johnp...@jptechnical.co.uk> wrote: I have successfully used Python to perform unit and integration tests in the past and I'd like to do the same for some C modules I'm working with at work. There seem to be a number of ways of doing this but being busy at work and home I looking for the approach with the least learning curve. When I want to test C modules (usually destined for an embedded system) using a Python framework, I use ctypes. https://docs.python.org/3/library/ctypes.html I'll second ctypes. It's pretty straightforward and I've made it do some very heavy lifting over the years. Cython is definitely more work. SWIG makes sense if you've got a huge number of interfaces you're trying to map and need to automate the process, but if you've got south of 20 I'd just do it natively with ctypes. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: h5py.File() gives error message
On 10/24/2017 10:58 AM, C W wrote:
Dear list,
The following Python code gives an error message
# Python code starts here:
import numpy as np
import h5py
train_dataset = h5py.File('datasets/train_catvnoncat.h5', "r")
# Python code ends
The error message:
train_dataset = h5py.File('train_catvnoncat.h5', "r")
Traceback (most recent call last):
File "", line 1, in
File "/Users/M/anaconda/lib/python3.6/site-packages/h5py/_hl/files.py",
line 269, in __init__
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/Users/M/anaconda/lib/python3.6/site-packages/h5py/_hl/files.py",
line 99, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 78, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name =
'train_catvnoncat.h5', errno = 2, error message = 'No such file or
directory', flags = 0, o_flags = 0)
My directory is correct, and the dataset folder with file is there.
Why error message? Is it h5py.File() or is it my file? Everything seems
pretty simple, what's going on?
Thank you!
Be 100% sure your directory is correct. Try it again with an absolute
path to the file. Windows makes it far too easy for the working
directory of a program to be other than what you think it is.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: I used list, def. why li += [100,200] , and li = li + [100,200] is different
On 10/23/2017 09:29 AM, 임현준 wrote: I am a Korean student, and I am a beginner in English and Python.;( I can't understand about this def If I want to print [1,2,3,4,5] [1,2,3,4,5,100,200] I will make code like this, and I can understand code. def modify(li): li += [100,200] list = [1,2,3,4,5] print(list) modify(list) print(list) BUT, when I make code like this. I will make code like this, and I can understand code. def modify(li): li = li + [100,200] list = [1,2,3,4,5] print(list) modify(list) print(list) python print [1,2,3,4,5] [1,2,3,4,5] why 'li+= [100,200]'and 'li = li + [100,200]' 's print is different please help me Lists are mutable, they can be changed. Your call to += is equivalent to a call to li.extend([100, 200]), which changes the single existing object pointed to by the "li" reference. The second time, however, you take the existing value that "li" refers to [1,2,3,4,5], create a new object that is ([1,2,3,4,5] + [100,200]), and reassign the local reference "li" to point to that new object. Then your function ends, "li" goes out of scope, nothing points to that newly created object and it gets lost. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Is there a function of ipaddress to get the subnet only from input like 192.168.1.129/25
On 10/17/2017 09:59 AM, Daniel Flick wrote:
I am very new to Python and have been struggling to find some info on
processing IP addresses.
get_network returns 192.168.1.128/25 but I need 192.168.1.128 only. I can do
this with netaddr but I am working with Mako templates and ipaddress is a built
in module so there are less dependencies.
Any ideas?
You mean, other than .split('/')?
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Line terminators in Python?
On 09/29/2017 10:54 AM, Stefan Ram wrote: In some languages, printing »'\n'«, the Unicode code point 10, will have the effect of printing a line terminator, which might mean that the output device actually receives »\r\n«. The line terminator ostensibly depends on the operating system, but actually it depends on the output system. E.g., under one operating system the console might accept another set of line separators than an editor. (Under Windows, »wordpad« accepts »\n«, while »notepad« requires »\r\n«.) What is the recommended way to terminate a line written with Python? Is it »\n« or something else? For example, in Java, in some cases, one should terminate the line with the value of »java.lang.System.lineSeparator()« which might or might not be equal to the value of »"\n"«. Does it possibly depend on the entity being written to, which might be - the Python console, - the IDLE console, - the operating system console or - a text file? As everyone else has said; for general purpose (any of your cases) use you should always just use '\n' and it automagically works. The only time I've ever needed to explicitly worry about '\r' is communicating over sockets or serial ports to devices. And in those cases you need to stuff them with bytes rather than str anyhow, so you're already down in the gory bits. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Real Programmers Write Docs
On 09/27/2017 04:15 PM, Ned Batchelder wrote: On 9/27/17 6:55 PM, Rob Gaddi wrote: Anyone have any good references on using Sphinx to generate a mix of autogenerated API docs and hand-written "Yeah, but this is what you DO with it" docs? Free is good but I'm happy to drop money on books if they're worthwhile. I can offer you an example: the coverage.py docs are in Sphinx, and use both auto-generated and hand-written pages: https://github.com/nedbat/coveragepy/tree/master/doc I'm not sure what information you are looking for, maybe this will help? --Ned. Helps quite a bit, thanks Ned. Basically I'm looking for a good example on structuring this sort of thing to keep it from spiraling out of control. Looks like the big trick is to rely on hand-written structure around autoclass calls rather than letting automodule generate big walls of text. PySerial was my other example, and has gone much the same route. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Real Programmers Write Docs
Anyone have any good references on using Sphinx to generate a mix of autogenerated API docs and hand-written "Yeah, but this is what you DO with it" docs? Free is good but I'm happy to drop money on books if they're worthwhile. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Boolean Expressions
On 09/26/2017 03:23 PM, Cai Gengyang wrote: I'm trying to understand the logic behind AND. I looked up Python logic tables False and False gives False False and True gives False True and False gives False True and True gives True. So does that mean that the way 'and' works in Python is that both terms must be True (1) for the entire expression to be True ? Why is it defined that way, weird ? I was always under the impression that 'and' means that when you have both terms the same, ie either True and True or False and False , then it gives True No, that would actually be an xnor (not xor) operation, a fairly rare usage case. Python doesn't even provide an operator for that, the closest thing would be (bool(x) == bool(y)). "And" means "and". This is true AND that is true. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: How do I check all variables returned buy the functions exists
On 09/15/2017 08:43 AM, Ganesh Pal wrote:
I have a function that return's x variables How do I check if all the the
values returned are not None/False/0/''
Here is the same program to demonstrate this , but I felt this can be
better any suggestions ?
# vi file.py
import random
import string
def return_x_values():
" returns x strings for further processing"
value1 = random.choice(string.ascii_letters)
value2 = random.choice(string.ascii_letters)
value3 = random.choice(string.ascii_letters)
return (value1, value2, value3)
#unpack them
value1, value2 , value3 = return_x_values()
# check if its not none
# I think this can be better
if value1 and value2 and value3 :
print "continue with the program"
else:
print "Exting the program"
# python file.py
continue with the program
I am a Linux user with Python 2.7
Regards,
Ganesh
Don't unpack them yet, you still want them to be aggregated.
vals = return_x_values()
if all(vals):
v1, v2, v3 = vals
print "all values true"
else:
print "at least one false value"
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Exponential Smoothing program
On 08/31/2017 05:35 AM, Ode Idoko wrote:
I am running a master degree programme and very new to programming including python. I
have been given a project to write a python program on exponential smoothing of some
selected stocks. The program should user the user to input alpha, display the graph of
the original data and "smoothed data".
On the same python program, I am to develop a linear regression model to
predict the next period, showing correlation coefficients to indicate the
strength of the model.
I have been able to write a program which returns some values but no graph and
also having difficulties with the linear regression.
This is the program I wrote:
def exponential_smoothing (a,y,f):
ans = (a*y) + (1-a) * f
return ans
print ("Exponential_Smoothing Program")
a = float(input("please enter a: "))
y = float(input("please enter y: "))
f = float(input("please enter f: "))
ans = exponential_smoothing (a,y,f)
print ("the answers are %.2f" %(ans))
Could someone kindly help with tips on how to go about this?
It's time to introduce yourself to Python's data analysis toolkit.
Numpy, Scipy, Matplotlib and, if you're going to be doing stocks, Pandas.
There's a LOT there between them, it's a bit of a drink from the
firehose problem. I like Wes McKinney's "Python for Data Analysis" book
and consider it a good investment if you need to get up to speed on this
stuff.
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Nesting concurrent.futures.ThreadPoolExecutor
On 07/20/2017 12:44 PM, Andrew McLean wrote: I have a program where I am currently using a concurrent.futures.ThreadPoolExecutor to run multiple tasks concurrently. These tasks are typically I/O bound, involving access to local databases and remote REST APIs. However, these tasks could themselves be split into subtasks, which would also benefit from concurrency. What I am hoping is that it is safe to use a concurrent.futures.ThreadPoolExecutor within the tasks. I have coded up a toy example, which seems to work. However, I'd like some confidence that this is intentional. Concurrency is notoriously tricky. I very much hope this is safe, because otherwise it would not be safe to use a ThreadPoolExecutor to execute arbitrary code, in case it also used concurrent.futures to exploit concurrency. Well that last statement is clearly false. It's not safe to use any multiple access mechanism (threading, processes, async stuff) to execute arbitrary code; so it's by definition not safe to use a ThreadPoolExecutor. I'm not being cute and using "safe" in some Turing sense, I'm talking specifically about multiple accesses. Whenever you have multiple access you have interlock issues, which you resolve with mutexes or message queues or however you decide to do so. Those issues don't go away when you use the ThreadPoolExecutor. There is every possibility, especially if you start recursively spawning threads, that A spawns B, A blocks on something that B is supposed to do (such as completing a Future), but due to the thread limit of the pool, the mere existence of A is preventing B from being executed, and you have a deadlock. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: pyserial and end-of-line specification
On 07/18/2017 12:53 PM, FS wrote: Thank you for your response Andre. I had tried some code like that in the document but it did not seem to work. However ever leaving my terminal for a time the code eventually wrote out the records so apparently there is some very deep buffering going on here. A little more searching on the web revealed the following: https://stackoverflow.com/questions/10222788/line-buffered-serial-input It is apparent that pySerial, or at least the documentation is falling short of my needs. It is very unclear what module in the layer is handling the buffering and newlines and so forth. Also unclear is whether the coupled python and OS is reading FIFO or LIFO--something important in quasi realtime scientific applications. This is problematic since the serial port is still so ubiquitous to a lot of scientific instrumentation. I probably will patch up some byte oriented code for this or perhaps write the module in C. Thanks again Fritz Handling it .read(1) at a time is probably your best bet. Append them into a bytearray and pull it all out when you're done. It's a serial port; it's not like there is any degree of inefficiently that you could write the code that will make it slow with respect to the I/O. I write a LOT of serial instrument I/O and I've definitely had to fall back to this plan. Your code gets a little long, you hide it all in a function somewhere and never think on it again. One paradigm I stick for ASCII serial is to have 3 functions: def command(msg: str): """Sends a command, raises CommError if it doesn't get some expected OK sort of thing.""" def query(msg: str): """Sends a commmand, returns a (trimmed) response line.""" def _communicate(msg: str): The ugliest stuff, all the str->bytes->str stuff, the line-ending and protocols, goes into _communicate. Query usually just calls _communicate. Command slaps on whatever checks are needed. It feels a bit heavy, but it leads to highly-usable code and makes it easy to integrate logging, retries, integrating "*OPC?" handshakes, whatever sort of things turn out to be necessary on a given device. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to assert unit test cases with many conditions
On 07/19/2017 03:26 AM, Ganesh Pal wrote: Yes. Just assert each thing as it needs asserting. Asserting each sub test will fail the entire test, I want the to pass the test if any the sub test passes. If the sub test fail try all cases and fail for the last one. Example : def test_this(self): if Sub_test_1(): #passes then PASS the Complete test i.e. test_this() and If sub_test_1() fail then run further subtest!) elif run sub_test_2() : #Then PASS test_this() and don't run next test i.e sub_test_3(),sub_test_4() etc) elif run sub_test_3() if sub_test_3() # Then pass test_this() and don't run next test i.e. sub_test_4() ,sub_test_5(). etc) Regards, Ganesh So you're saying if test 1 passes you don't even bother to run test 2? To be blunt, that sounds convoluted and overcomplicated. How would you ever know that test2 is even doing its job? Why is this superior to writing five tests, all of which always run? Note that "runtime" is not a valid answer unless you're talking about multiple minutes of it. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Best way to assert unit test cases with many conditions
On 07/18/2017 09:56 AM, Ganesh Pal wrote: (1) should I add several asserts per test case, or just warn with the error and fail at the end . In the line 33 – 35 / 37-38 ( sorry this is a dirty pusedo-code) . Yes. Just assert each thing as it needs asserting. (2) Is there a way we can warn the test using assert method and not fail? I was trying to see if I could use assertWarns but the help says that “The test passes if warning is triggered and fails if it isn’t “. I don’t want to fail on warning but just continue which next checks You can, but you're just going to complicate your life. A "test" is a thing that passes (all) or fails (any). If you need it to keep going after a failure, what you have are two tests. There's nothing wrong with having a whole mess of test functions. If there's a lot of common code there you'd have to replicate, that's what setUp() is for. If there are several different flavors of common code you need, you can create a base TestCase subclass and then derive further subclasses from that. Do the things the way the tools want to do them. Unit testing is enough of a pain without trying to drive nails with the butt of a screwdriver. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Re: Users of namedtuple: do you use the _source attribute?
On 07/17/2017 09:57 AM, Steve D'Aprano wrote:
collections.namedtuple generates a new class using exec, and records the source
code for the class as a _source attribute.
Although it has a leading underscore, it is actually a public attribute. The
leading underscore distinguishes it from a named field potentially
called "source", e.g. namedtuple("klass", ['source', 'destination']).
There is some discussion on Python-Dev about:
- changing the way the namedtuple class is generated which may
change the _source attribute
- or even dropping it altogether
in order to speed up namedtuple and reduce Python's startup time.
Is there anyone here who uses the namedtuple _source attribute?
My own tests suggest that changing from the current implementation to one
similar to this recipe here:
https://code.activestate.com/recipes/578918-yet-another-namedtuple/
which only uses exec to generate the __new__ method, not the entire class, has
the potential to speed up namedtuple by a factor of four.
I use namedtuple a lot, and never even HEARD of _source.
That said, it sure feels (as someone who hasn't tried it) like there's a
straightforward namedtuple implementation that calls type() directly
rather than having to exec. I know that exec-gunshyness is overblown,
but is there a simple answer as to why it's necessary here?
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Development testing without reinstalling egg constantly?
On 06/29/2017 08:32 AM, Grant Edwards wrote: This is somebody else's project I'm hacking on, and my understanding of setup tools is pretty much at the "cargo cult" level. I have a sneaking suspicion that's everyone. Setuptools (and distutils before it) has no underlying rhyme or reason; just a collection of stuff and if you do all the stuff then magic. Eggs and wheels and twine make me feel like I'm playing the world's worst Settlers of Catan knockoff every time I try to make a project all pretty, and the XKCD 927 factor is in full and glorious effect. -- Rob Gaddi, Highland Technology -- www.highlandtechnology.com Email address domain is currently out of order. See above to fix. -- https://mail.python.org/mailman/listinfo/python-list
Project Structure
I'm working on a project ( https://github.com/NJDFan/register-maps
http://register-maps.readthedocs.io/en/latest/ ) and running into some
issues with overall project structure and documentation.
The project is a code generator that reads in a set of proprietary
format XML files and uses them to generate output in any number of
languages (C, Python, VHDL) as well as HTML documentation. Input is an
entire directory, output is an entire directory.
registermap ./data/src --format=html --output=../docs/html
One of the things I'd like to do is to write a boilerplate README.rst
into the target directory. I also have Sphinx documentation; I'd like
to incorporate all those READMEs into the HTML docs. I'd also like to
have something like a
registermap --format-help vhdl
Which displays the appropriate README, ideally neatly formatted for the
terminal and running through a pager. You know, civilized.
So, right now I have a project setup which is
./
doc/
...
registermaps/
__init__.py
...
resource/
vhdl/
README.rst
...
html/
README.rst
style.css
...
README.rst
setup.py
...
Output classes have a .outputname, which is both the command-line output
name ('html') and the directory under resource that all the stuff is in.
I have a utility function in the module that allows those classes to
register themselves against that name for the command-line tool.
Resources are accessed through pkg_resource.resource_string internal to
the program, and the READMEs get pulled into the Sphinx docs by having a
doc/source/outputs.rst full of include directives like
.. include:: ../../registermaps/resource/html/README.rst
I still don't have any kind of standard way of pretty-printing and
paging the reST to the terminal. And, more the point, this all feels
VERY tangled. Like there's something that other people just know to do
and I missed the memo. Anyone have any ideas on a cleaner implementation?
--
Rob Gaddi, Highland Technology -- www.highlandtechnology.com
Email address domain is currently out of order. See above to fix.
--
https://mail.python.org/mailman/listinfo/python-list
