Re: parallel subprocess.getoutput
On May 11, 8:04 am, Jaroslav Dobrek wrote: > Hello, > > I wrote the following code for using egrep on many large files: > > MY_DIR = '/my/path/to/dir' > FILES = os.listdir(MY_DIR) > > def grep(regex): > i = 0 > l = len(FILES) > output = [] > while i < l: > command = "egrep " + '"' + regex + '" ' + MY_DIR + '/' + > FILES[i] > result = subprocess.getoutput(command) > if result: > output.append(result) > i += 1 > return output > > Yet, I don't think that the files are searched in parallel. Am I > right? How can I search them in parallel? subprocess.getoutput() blocks until the command writes out all of its output, so no, they're not going to be run in parallel. You really shouldn't use it anyway, as it's very difficult to use it securely. Your code, as it stands, could be exploited if the user can supply the regex or the directory. There are plenty of tools to do parallel execution in a shell, such as: http://code.google.com/p/ppss/. I would use one of those tools first. Nevertheless, if you must do it in Python, then the most portable way to accomplish what you want is to: 0) Create a thread-safe queue object to hold the output. 1) Create each process using a subprocess.Popen object. Do this safely and securely, which means NOT passing shell=True in the constructor, passing stdin=False, and passing stderr=False unless you intend to capture error output. 2) Spawn a new thread for each process. That thread should block reading the Popen.stdout file object. Each time it reads some output, it should then write it to the queue. If you monitor stderr as well, you'll need to spawn two threads per subprocess. When EOF is reached, close the descriptor and call Popen.wait() to terminate the process (this is trickier with two threads and requires additional synchronization). 3) After spawning each process, monitor the queue in the first thread and capture all of the output. 4) Call the join() method on all of the threads to terminate them. The easiest way to do this is to have each thread write a special object (a sentinel) to the queue to indicate that it is done. If you don't mind platform specific code (and it doesn't look like you do), then you can use fcntl.fcntl to make each file-object non- blocking, and then use any of the various asynchronous I/O APIs to avoid the use of threads. You still need to clean up all of the file objects and processes when you are done, though. -- http://mail.python.org/mailman/listinfo/python-list
Re: Alternative to subprocess in order to not wait for calling commands to complete
On May 10, 12:07 pm, ks wrote: > Hi All, > > From within one Python program, I would like to invoke three other > Python programs. Usually I would use the subprocess module to invoke > these sequentially. I now have a use case in which I must invoke the > first one (not wait for it to complete), then invoke the second > (similarly not wait for it to complete) and then go on to the third. > > I am not sure where I should start looking to be able to do this. I am > reading about threads and forking and there are many options out > there. So I was wondering if anyone might have suggestions on where I > can start. > subprocess.Popen objects only block for process termination when you request it, via wait() or communicate(). As such, if you never call those methods, you will never block for process termination. What you want is essentially the default behavior if you're creating Popen objects directly (instead of using the call() function). However, you must call the wait() or poll() methods at least once, after the subprocess has terminated, to release system resources. There are many ways to accomplish this, and the best approach depends on your application. One simple option is to call poll() once each time your application goes through its main loop. Another option is to use a dedicated thread for the purpose of reaping processes. Please note that calling Popen.poll() in a loop over many processes does not scale (each call incurs an expensive system call) and will cause noticeable CPU consumption with even a handful of processes (3 is just fine, though). Unfortunately, the subprocess module lacks any way to wait on multiple processes simultaneously. This is possible with operating-system specific code, however. I would not worry about it for now, but it may be something you need to consider in the future. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: How to get outer class name from an inner class?
On May 8, 4:05 pm, John Gordon wrote: > I'm trying to come up with a scheme for organizing exceptions in > my application. > > Currently, I'm using a base class which knows how to look up the text > of a specific error in a database table, keyed on the error class name. > > The base class looks like this: > > class ApplicationException(Exception): > """Base class for application-specific errors.""" > > def get_message(self): > """Return the error message associated with this class name.""" > > class_name = self.__class__.__name__ > return UserMessage.objects.get(key=class_name).text > > And then I define a bunch of subclasses which all have different names: > > class QuestionTooShortError(NetIDAppsError): > """User entered a security question which is too short.""" > pass > > class QuestionTooLongError(NetIDAppsError): > """User entered a security question which is too long.""" > pass > > This scheme works, but I'd like to make it more streamlined. Specifically, > I'd like to group the classes underneath a parent class, like so: > > class Question(ApplicationException): > > class TooShort(ApplicationException): > pass > > class TooLong(ApplicationException): > pass > > This will make it easier in the future for organizing lots of sub-errors. It's no more or less organized than using a module, so use a module. This is why they exist, after all. That being said, this seems like a bad idea to me: this is a lot of code and types just for a message lookup! Exception types should usually be created based on what you expect users to catch, not based on what you could throw. If all of these exceptions will be handled in the same way, then they shouldn't be distinct types. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: How can I read streaming output of a subprocess
On May 2, 7:46 am, Kiuhnm wrote: > On 5/2/2012 13:08, Damjan Georgievski wrote: > > > I want to read the stream of an external process that I start with > > Python. From what I've seen that's not possible with the subprocess module? > > Try with > cmd = 'your command here' > stdout = Popen(cmd, shell = True, stdout = PIPE, stderr = STDOUT).stdout One should never, ever start a process with shell=True unless it's absolutely necessary. It is a waste of resources and an a security issue waiting to happen. Just say no. Also, stderr shouldn't be combined with stdout unless there is a need to mix the two streams, which usually is not the case. Also, one needs to hold on to the POpen object so the process can be properly reaped later or use a with block. This is where things get tricky for most applications. It gets a little extra tricky for 'ip monitor'. Assuming the intention is to write a command-line application, the right thing to do _most likely_ is: with Popen(['ip', 'monitor', 'neigh'], stdout=PIPE) as proc: try: for line in proc.stdout: # Do your processing here, it need not be line-by-line. finally: process.terminate() Calling process.terminate() explicitly is necessary since "ip monitor" writes output forever. This is not the right thing to do in all situations, and the right thing to do depends on your application and the command being executed as a subprocess. It's not really possible to generalize fully. As such, your error-handling / cleanup code may end up being completely different.Don't use my code if it's not applicable to your situation. In any case, you need to cleanup after yourself, which means waiting until the subprocess terminates. It may also include cleaning up the file objects for the pipes. Using POpen objects in a with block does both, so do that where you can. The official Python documentation for the subprocess module is pretty clear and worth reading. I encourage you to look through it carefully. It does, unfortunately, gloss over which resources need to be cleaned up and when. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 28, 7:26 am, Kiuhnm wrote: > On 4/27/2012 19:15, Adam Skutt wrote: > > On Apr 27, 11:01 am, Kiuhnm wrote: > >> The abstraction is this: > >> - There are primitives and objects. > >> - Primitives are not objects. The converse is also true. > >> - Primitives can become objects (boxing). > >> - Two primitives x and y are equal iff x == y. > >> - Two objects x and y are equal iff x.equals(y). > >> - Two objects are the same object iff x == y. > >> - If x is a primitive, then y = x is a deep copy. > >> - If x is an object, then y = x is a shallow copy. > >> - ... > > > This is not an abstraction at all, but merely a poor explanation of > > how things work in Java. Your last statement is totally incorrect, as > > no copying of the object occurs whatsoever. The reference is merely > > reseated to refer to the new object. If you're going to chide me for > > ignoring the difference between the reference and the referent object, > > then you shouldn't ignore it either, especially in the one case where > > it actually matters! If we try to extend this to other languages, > > then it breaks down completely. > > With shallow copy I meant exactly that. I didn't think that my using the > term with a more general meaning would cause such a reaction. It has a very strict, well-defined meaning in these contexts, especially in languages such as C++. > > So you're saying that I said that "Primitive constructs are references". > Right... No, still wrong. What I said is correct, "References are a form of primitive construct". In C, an int is a primitive but not a reference. An int* is a pointer (reference), and is also (essentially) a primitive. > > > While true, it's still a bad way > > to think about what's going on. It breaks down once we add C++ / > > Pascal reference types to the mix, for example. > > ? Assignment to a C++ reference (T&) effects the underlying object, not the reference itself. A reference can never be reseated once it is bound to an object. Comparing equality on two references directly is the same as comparing two values (it calls operator==). Comparing identity requires doing (&x == &y), like one would do with a value. However, unlike a value, the object is not destroyed when the reference goes out of scope. Most importantly, references to base classes do not slice derived class objects, so virtual calls work correctly through references. As a result, normally the right way to think about a value is as a "temporary name" for an object and not worry about any of the details about how the language makes it work. > > >> Equality or equivalence is a relation which is: > >> - reflexive > >> - symmetric > >> - transitive > >> Everything else... is something else. Call it semi-equality, > >> tricky-equality or whatever, but not equality, please. > > > Sure, but then it's illegal to allow the usage of '==' with floating > > point numbers, which will never have these properties in any usable > > implementation[1]. > > ??? > The operator == is called the equality operator. Floating-point numbers don't really obey those properties in any meaningful fashion. The result is that portions of your view contradict others. Either we must give '==' a different name, meaning what you consider equality is irrelevant, or we must use method names like 'equals', which you find objectionable. > >>> If anything, you have that backwards. Look at Python: all variables > >>> in Python have pointer semantics, not value semantics. > > >> When everything is "white", the word "white" becomes redundant. > >> So the fact that everything in Python have reference semantics means > >> that we can't stop thinking about value and reference semantics. > > > Nope. The behavior of variables is absolutely essential to writing > > correct programs. If I write a program in Python that treats > > variables as if they were values, it will be incorrect. > > You misunderstood what I said. You wouldn't treat variables as if they > were values because you wouldn't even know what that means and that > that's even a possibility. Well, one hopes that is true. I think we have a misunderstanding over language: you said "value and reference semantics" when you really meant "value vs. reference semantics". > I've never heard an old C programmer talk about "value semantics" and > "reference semantics". When everything is a value, your world is pretty > simple.
Re: CPython thread starvation
On Apr 27, 2:54 pm, John Nagle wrote: > I have a multi-threaded CPython program, which has up to four > threads. One thread is simply a wait loop monitoring the other > three and waiting for them to finish, so it can give them more > work to do. When the work threads, which read web pages and > then parse them, are compute-bound, I've had the monitoring thread > starved of CPU time for as long as 120 seconds. How exactly are you determining that this is the case? > I know that the CPython thread dispatcher sucks, but I didn't > realize it sucked that bad. Is there a preference for running > threads at the head of the list (like UNIX, circa 1979) or > something like that? Not in CPython, which is at the mercy of what the operating system does. Under the covers, CPython uses a semaphore on Windows, which do not have FIFO ordering as per http://msdn.microsoft.com/en-us/library/windows/desktop/ms685129(v=vs.85).aspx. As a result, I think your thread is succumbing to the same issues that impact signal delivery as described on 22-24 and 35-41 of http://www.dabeaz.com/python/GIL.pdf. I'm not sure there's any easy or reliable way to "fix" that from your code. I am not a WinAPI programmer though, and I'd suggest finding one to help you out. It doesn't appear possible to change the scheduling policy for semaphore programatically, and I don't know closely they pay any attention to thread priority. That's just a guess though, and finding out for sure would take some low-level debugging. However, it seems to be the most probable situation assuming your code is correct. > > (And yes, I know about "multiprocessing". These threads are already > in one of several service processes. I don't want to launch even more > copies of the Python interpreter. Why? There's little harm in launching more instances. Processes have some additional startup and memory overhead compared to threads, but I can't imagine it woudl be an issue. Given what you're trying to do, I'd expect to run out of other resources long before I ran out of memory because I created too many processes or threads. > The threads are usually I/O bound, > but when they hit unusually long web pages, they go compute-bound > during parsing.) If your concern is being CPU oversubscribed by using lots of processes, I suspect it's probably misplaced. A whole mess of CPU- bound tasks is pretty much the easiest case for a scheduler to handle. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Borg identity [was Re: why () is () and [] is [] work in other way?]
On Apr 27, 2:40 pm, Steven D'Aprano wrote:
> On Fri, 27 Apr 2012 10:33:34 -0700, Adam Skutt wrote:
> >> Why should the caller care whether they are dealing with a singleton
> >> object or an unspecified number of Borg objects all sharing state? A
> >> clever interpreter could make many Borg instances appear to be a
> >> singleton. A really clever one could also make a singleton appear to be
> >> many Borg instances.
>
> > Trivial: to break cyclical references in a deep copy operation.
>
> I asked why the *caller* should care. If the caller has to break cyclical
> references manually, the garbage collector is not doing its job.
It's a necessary requirement to serialize any cyclical structure.
Garbage collection has nothing to do with it. If I have some
structure such that A --> B --> A, I need to be able to determine that
I've seen 'A' before in order to serialize the structure to disk, or I
will never write it out successfully.
There are plenty of situations where we legitimately care whether two
pointers are the same and don't give one whit about the state of
objects they point to. You cannot conflate the two tests, and that's
precisely what your 'give all borg instances the same identity' idea
does.
> I think that if you're talking about per-instance members of a Borg
> class, you're confused as to what Borg means.
I'm not. I'm talking about per-instance members of a subclass of a
Borg class. There's nothing about the Borg pattern that forbids such
behavior, which is one of the reasons it's such a terrible idea in
general. Borg implies promises that it cannot readily keep.
> Since all instances share state, you can't have *per-instance* data.
I most certainly can do so in a subclass. Shared state in a parent
doesn't mandate shared state in a child.
> Not at all. Obviously each Borg subclass will have it's own fake
> identity.
> When I said that Borg instances are indistinguishable except for
> identity, I thought that was obvious that I was talking about instances
> of a single type. Mea culpa.
>
> Clearly if x is an instance of Borg, and y is an instance of
> BorgSubclass, you can distinguish them by looking at the type. The point
> is that you shouldn't be able to distinguish instances of a single type.
No, that's not the least bit obvious nor apparent, and it still
violates LSP. It means every function that takes a Borg as an
argument must know about every subclass in order to distinguish
between them.
The serialization function above would need to do so. Imagine an
object x that holds a Borg object and a BorgSubclass object. If the
serialization function keeps a list of objects it has seen before and
uses that to determine whether to write the object out, it will fail
to write out one or the other if we implemented your harebrained 'All
Borg objects have the same identity' idea.
Your idea means that 'x.borg is x.subborg' must return True. It also
means either x.borg isn't going to be written out, or x.subborg isn't
going to be written out. The program is broken.
If you modify your idea to ignore subtypes, than this function breaks:
def write_many(value, channel1, channel2):
channel1.write(value)
if channel2 is not channel1:
channel2.write(value)
Calling write_many("foo", x.borg, x.subborg) now gives different
behavior than write_many("foo", x.borg, x.borg). That's probably not
what the programmer intended!
Like it or not, whether you have only one object with shared state or
infinite objects with the same shared state is not an implementation
detail. Just because you write code that doesn't care about that fact
does not make it an implementation detail. I can write code that
depends on that fact, and there's not a single thing you can do to
stop me.
This is why the Borg pattern is a bad idea in general, because it
encourages programmers to write code that is subtly wrong. If you
have a Borg class, you can't ignore the fact that you have multiple
objects even if you want to do so. You will eventually end up writing
incorrect code as a result. Yet, many people try to do precisely
that, your idea is attempting to do precisely that!
> Oh please, enough of the religion of LSP.
>
> Barbara Liskov first introduced this idea in 1987, twenty years after
> Simula 67 first appeared and thirty years after MIT researchers came up
> with the concept of object oriented programming. That's hardly
> fundamental to the concept of OOP.
> People have, and still do, violate LSP all the time.
People write code with security flaws all of the time too. This
doesn't even being to approach being an reasonable argument. It's
completely disingenuous. People come up
Re: Direct vs indirect [was Re: why () is () and [] is [] work in other way?]
On Apr 27, 1:06 pm, Steven D'Aprano wrote: > On Thu, 26 Apr 2012 04:42:36 -0700, Adam Skutt wrote: > > On Apr 26, 5:10 am, Steven D'Aprano > +comp.lang.pyt...@pearwood.info> wrote: > >> But I was actually referring to something more fundamental than that. > >> The statement "a is b" is a *direct* statement of identity. "John is my > >> father." "id(a) == id(b)" is *indirect*: "The only child of John's > >> grandfather is the parent of the mother-in-law of my sister-in-law" > >> sort of thing. (Excuse me if I got the relationships mixed up.) > > > Again, the fact that you somehow think this absurd family tree is > > relevant only shows you're fundamentally confused about what object > > oriented identity means. That's rather depressing, seeing as I've given > > you a link to the definition. > > Perhaps you failed to notice that this "absurd" family tree, as you put > it, consists of grandparent+parent+sibling+in-law. What sort of families > are you familiar with that this seems absurd to you? No, I noticed, but who talks like that? It's not remotely comparable to the sort of difference we're talking about. > > I think you have inadvertently demonstrated the point I am clumsily > trying to make. Even when two expressions are logically equivalent, the > form of the expressions make a big difference to the comprehensibility of > the text. And if we were talking about 30, 20, 5, maybe even 2 line function versus it's name, you might have a point. We're not talking about such things though, and it's pretty disingenuous to pretend otherwise. Yet, that's precisely what you did with your absurd family relationship. > Which would you rather read? > > for item in sequence[1:]: ... > > for item in sequence[sum(ord(c) for c in 'avocado') % 183:]: ... > > The two are logically equivalent, so logically you should have no > preference between the two, yes? No, they're not logically equivalent. The first won't even execute, as sequence is undefined. You need two lines in the first case. > A statement is "direct" in the sense I mean if it explicitly states the > thing you intend it to state. And in the case of the two ways to compare identity, both statements state exactly what I intend to state. They're synonyms. > > "a is b" is a direct test of whether a is b. (Duh.) > > "id(a) == id(b)" is an indirect test of whether a is b, since it requires > at least three indirect steps: the knowledge of what the id() function > does, the knowledge of what the == operator does, and the knowledge that > equal IDs imply identity. The problem is that using 'is' correctly requires understanding all of those three things. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Python id() does not return an address [was Re: why () is () and [] is [] work in other way?]
On Apr 27, 1:12 pm, Steven D'Aprano wrote: > On Thu, 26 Apr 2012 04:42:36 -0700, Adam Skutt wrote: > > On Apr 26, 5:10 am, Steven D'Aprano > +comp.lang.pyt...@pearwood.info> wrote: > >> Solution to *what problem*? > > > This confusion that many people have over what 'is' does, including > > yourself. > > I have no confusion over what "is" does. False. If you did, then you would not have suggested the difference in True/False result between "id([1,2]) == id([1, 2])" and "[1, 2] is [1, 2]" matters. You would understand that the result of an identity test with temporary objects is meaningless, since identity is only meaningful while the objects are alive. That's a fundamental mistake. > >> > An address is an identifier: a number that I can use to access a > >> > value[1]. > > >> Then by your own definition, Python's id() does not return an address, > >> since you cannot use it to access a value. > > > The fact Python lacks explicit dereferencing doesn't change the fact > > that id() returns an address. Replace 'can' with 'could' or 'could > > potentially' or the whole phrase with 'represents' if you wish. It's a > > rather pointless thing to quibble over. > > You can't treat id() as an address. Did you miss my post when I > demonstrated that Jython returns IDs generated on demand, starting from > 1? In general, there is *no way even in principle* to go from a Python ID > to the memory location (address) of the object with that ID, because in > general objects *may not even have a fixed address*. Objects in Jython > don't, because the Java virtual machine can move them in memory. Yes, there is a way. You add a function deref() to the language. In CPython, that simply treats the passed value as a memory address and treats it as an object, perhaps with an optional check. In Jython, it'd access a global table of numbers as keys with the corresponding objects as values, and return them. The value of id() is absolutely an address, even in Jython. The fact the values can move about is irrelevant. Again, if this wasn't possible, then you couldn't implement 'is'. Implementing 'is' requires a mechanism for comparing objects that doesn't involve ensuring the contents of the two operands in memory is the same. > > Would you call the result of casting a C pointer to an int an address? > > If so, you must call the result of id() an address as well-- you can't > > dereference either of them. If not, then you need to provide an > > alternate name for the result of casting a C pointer to an int. > > I don't need to do anything of the sort. Yes, you do, because you called such a thing an address when talking about CPython. Even if my definition is wrong (it's not), your definition is wrong too. > (And for the record, in C you can cast an integer into a pointer, > although the results are implementation-specific. There's no equivalent > in Python.) Yes, but the lack of that operation doesn't mean that id() doesn't return an address. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Borg identity [was Re: why () is () and [] is [] work in other way?]
On Apr 27, 12:56 pm, Steven D'Aprano wrote: > On Thu, 26 Apr 2012 04:42:36 -0700, Adam Skutt wrote: > > You're going to have to explain the value of an "ID" that's not 1:1 with > > an object's identity, for at least the object's lifecycle, for a > > programmer. If you can't come up with a useful case, then you haven't > > said anything of merit. > > I gave an example earlier, but you seem to have misunderstood it, so I'll > give more detail. > > In the Borg design pattern, every Borg instance shares state and are > indistinguishable, with only one exception: object identity. We can > distinguish two Borg instances by using "is". > > Since the whole point of the pattern is for Borg instances to be > indistinguishable, the existence of a way to distinguish Borg instances > is a flaw and may be undesirable. At least, it's exposing an > implementation detail which some people argue should not be exposed. > Then people should stop with such idiocy like the Borg pattern. It's a bad example from an even worse idea. > Why should the caller care whether they are dealing with a singleton > object or an unspecified number of Borg objects all sharing state? A > clever interpreter could make many Borg instances appear to be a > singleton. A really clever one could also make a singleton appear to be > many Borg instances. Trivial: to break cyclical references in a deep copy operation. > John's argument is that Python should raise an exception if you compare > "2 is 2", or for that matter "3579 is 3579", which is foolish. > >> identities. The Borg design pattern, for example, would be an excellent > >> candidate for ID:identity being treated as many-to-one. > > > How would inheritance work if I did that? > > You don't inherit from Borg instances, and instances inherit from their > class the same as any other instance. I think you misunderstood me. Define a Borg class where somehow identity is the same for all instances. Inherit from that class and add per-instance members. Now, identity can't be the same for all instances. As a result, you've just violated the Liskov Substituion Principal: code that relies on all Borg class instances having the same identity will fail when passed an instance of the subclass. It's impossible to combine identities and not violate LSP, unless you forbid subclasses. Your idea violates one of the most fundamental tenants of object-oriented programming. This is because object identity is one of the fundamental consequences of object-oriented programming. You can't do away with it, and any attempt to do so really just suggests that you don't understand OOP at all. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 27, 11:01 am, Kiuhnm wrote: > On 4/27/2012 1:57, Adam Skutt wrote: > > On Apr 26, 6:34 pm, Kiuhnm wrote: > >>> If you > >> understand that your 'a' is not really an object but a reference to it, > >> everything becomes clear and you see that '==' always do the same thing. > > > Yes, object identity is implemented almost? everywhere by comparing > > the value of two pointers (references)[1]. I've already said I'm not > > really sure how else one would go about implementing it. > > >> You might tell me that that's just an implementation detail, but when an > >> implementation detail is easier to understand and makes more sense than > >> the whole abstraction which is built upon it, something is seriously wrong. > > > I'm not sure what abstraction is being built here. I think you have > > me confused for someone else, possibly Steven. > > The abstraction is this: > - There are primitives and objects. > - Primitives are not objects. The converse is also true. > - Primitives can become objects (boxing). > - Two primitives x and y are equal iff x == y. > - Two objects x and y are equal iff x.equals(y). > - Two objects are the same object iff x == y. > - If x is a primitive, then y = x is a deep copy. > - If x is an object, then y = x is a shallow copy. > - ... > This is not an abstraction at all, but merely a poor explanation of how things work in Java. Your last statement is totally incorrect, as no copying of the object occurs whatsoever. The reference is merely reseated to refer to the new object. If you're going to chide me for ignoring the difference between the reference and the referent object, then you shouldn't ignore it either, especially in the one case where it actually matters! If we try to extend this to other languages, then it breaks down completely. > The truth: > - Primitives can be references. > - Two primitives are equal iff x == y. > - Operator '.' automatically derefences references. > You have the first statement backwards. References are a primitive construct, not the other way around. While true, it's still a bad way to think about what's going on. It breaks down once we add C++ / Pascal reference types to the mix, for example. It's better to think about variables (names) and just recognize that not all variables have the same semantics. It avoids details that are irrelevant to writing actual programs and remains consistent. > Equality or equivalence is a relation which is: > - reflexive > - symmetric > - transitive > Everything else... is something else. Call it semi-equality, > tricky-equality or whatever, but not equality, please. Sure, but then it's illegal to allow the usage of '==' with floating point numbers, which will never have these properties in any usable implementation[1]. So we're back to what started this tangent, and we end up needing 'equals()' methods on our classes to distinguish between the different forms of equality. That's precisely what you want to avoid. Or we can just accept that '==' doesn't always possess those properties, which is what essentially every programming language does, and call it (value) equality. As long as we don't cross incompatible meanings, it's hard to believe that this isn't the right thing to do. > > > If anything, you have that backwards. Look at Python: all variables > > in Python have pointer semantics, not value semantics. > > When everything is "white", the word "white" becomes redundant. > So the fact that everything in Python have reference semantics means > that we can't stop thinking about value and reference semantics. Nope. The behavior of variables is absolutely essential to writing correct programs. If I write a program in Python that treats variables as if they were values, it will be incorrect. > > > In imperative > > languages, pointers have greater utility over value types because not > > all types can obey the rules for value types. For example, I don't > > know how to give value semantics to something like a I/O object (e.g, > > file, C++ fstream, C FILE), since I don't know how to create > > independent copies. > > By defining a copy constructor. Then write me a working one. I'll wait. To save yourself some time, you can start with std::fstream. > Python is already without pointers (*). > A world where everyone is a lawyer is a world without lawyers (really, > there isn't any other way we can get rid of them :) ). > > (*) By the way, some would argue that references are not pointers. They would be completely and utterly wrong, and probably imbuing pointers with properties they don't actually possess. Unless you're talking about C++ / Pascal references, which really aren't pointers and do possess a different set of semantics (alias might be a better term for them). Adam [1] Not in any fashion that's useful to the programmer, at any rate. -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 27, 8:07 am, Kiuhnm wrote: > Useful... maybe, conceptually sound... no. > Conceptually, NaN is the class of all elements which are not numbers, > therefore NaN = NaN. NaN isn't really the class of all elements which aren't numbers. NaN is the result of a few specific IEEE 754 operations that cannot be computed, like 0/0, and for which there's no other reasonable substitute (e.g., infinity) for practical applications . In the real world, if we were doing the math with pen and paper, we'd stop as soon as we hit such an error. Equality is simply not defined for the operations that can produce NaN, because we don't know to perform those computations. So no, it doesn't conceptually follow that NaN = NaN, what conceptually follows is the operation is undefined because NaN causes a halt. This is what programming languages ought to do if NaN is compared to anything other than a (floating-point) number: disallow the operation in the first place or toss an exception. Any code that tries such an operation has a logic error and must be fixed. However, when comparing NaN against floating point numbers, I don't see why NaN == NaN returning false is any less conceptually correct than any other possible result. NaN's very existence implicitly declares that we're now making up the rules as we go along, so we might as well pick the simplest set of functional rules. Plus, floating point numbers violate our expectations of equality anyway, frequently in surprising ways. 0.1 + 0.1 + 0.1 == 0.3 is true with pen and paper, but likely false on your computer. It's even potentially possible to compare two floating point variables twice and get different results each time[1]! As such, we'd have this problem with defining equality even if NaN didn't exist. We must treat floating-point numbers as a special case in order to write useful working programs. This includes defining equality in a way that's different from what works for nearly every other data type. Adam [1] Due to register spilling causing intermediate rounding. This could happen with the x87 FPU since the registers were 80-bits wide but values were stored in RAM as 64-bits. This behavior is less common now, but hardly impossible. -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 10:56 pm, "OKB (not okblacke)" wrote: > Adam Skutt wrote: > > If I write a function that does a value comparison, then it should > > do value comparison on _every type that can be passed to it_, > > regardless of whether the type is a primitive or an object, whether > > it has value or reference semantics, and regardless of how value > > comparison is actually implemented. If I write some function: > > f(x : T, y : U) => x == y > > where T and U are some unknown types, then I want the function to > > do a value comparison for every type pairing that allows the > > function to compile. Likewise, if I write a function that does > > identity comparison, then it logically wants to do identity > > comparison on _every type that can be passed to it_. > > What you say here makes perfect sense, but also shows that you > really shouldn't be using Python if you want stuff to work this way. In > Python any value of any type can be passed to any function. The claims > you are making about object identity and object equality are reasonable, > but as you show here, to really handle them requires dragging in a huge > amount of type-system baggage. So the check gets deferred to runtime, and the programmer may need to explictly throw 'NotImplemented' or something like that. Which is what happens in Python. Not type-checking arguments simply moves the burden from the language to the programmer, which is a standard consequence of moving from static to dynamic typing. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 7:33 pm, Steven D'Aprano wrote: > On Thu, 26 Apr 2012 12:22:55 -0700, Adam Skutt wrote: > > I often wonder what the world would be like if Python, C#, and Java > > embraced value types more, and had better support for pure functions. > > They would be slower, require more memory, Funny, Haskell frequently beats C in both categories. MATLAB is faster and more memory efficient than naive C matrix code, since it has a very efficient copy-on-write implementation. As the various C++ matrix libraries will show you, efficient COW is much harder when you have to deal with C++ aliasing rules. > harder to use, and far, far less popular. Alas, these two are probably true. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Thu, Apr 26, 2012 at 5:39 PM, Ian Kelly wrote: > On Thu, Apr 26, 2012 at 1:34 PM, Adam Skutt wrote: >> What I think you want is what I said above: ValueError raised when >> either operand is a /temporary/ object. Really, it should probably be >> a parse-time error, since you could (and should) make the >> determination at parse time. > > I'm not sure precisely what you mean by "temporary object", so I am > taking it to mean an object that is referenced only by the VM stack > (or something equivalent for other implementations). > > In that case: no, you can't. Take "f() is g()", where the code > objects of f and g are supplied at runtime. Are the objects returned > by either of those expressions "temporary"? Without being able to do > static analysis of the code of f and g, there is no way to know. A temporary object would be anything that need not be alive longer than the duration of the 'is' operation. I am not a Python language expert so that definition may not be exactly correct or workable for Python. In the example: >>> [1, 2] is [3, 4] [1,2] and [3,4] don't need to exist before the 'is' operation, nor after it, so they are temporaries. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Re: why () is () and [] is [] work in other way?
On Thu, Apr 26, 2012 at 12:05 PM, Evan Driscoll wrote:
> This thread has already beaten a dead horse enough that the horse came back
> as a zombie and was re-killed, but I couldn't help but respond to this part:
>
>
> On 01/-10/-28163 01:59 PM, Adam Skutt wrote:
>>
>> Code that relies on the identity of a temporary object is generally
>> incorrect. This is why C++ explicitly forbids taking the address
>> (identity) of temporaries.
>
>
> Except that C++ *doesn't* really forbid taking the address of a temporary,
> at least indirectly:
>
> #include
>
> int const * address_of(int const & x) {
> return &x;
> }
>
> int main() {
> std::cout << address_of(1+2) << "\n";
> }
>
> That complies without warning with GCC 4.6 '-Wall -Wextra', MSVC 2010 '/W4',
> and Comeau's online front end, and I am pretty confident that the above code
> is perfectly legal in terms of provoking undefined behavior (in the
> technical C++ sense of "your program is now allowed to set your cat on
> fire").
Yes, you can get a const reference to a temporary object, but that's
the only thing you can do. This is intentional, so you can use
temporaries (e.g., std::string("Hello World") ) in the same contexts
where one would use a literal (e.g., 3 or 4.2). Note that it's
impossible to mutate the temporary and impossible for the reference to
outlive the temporary.
What the standard says is: "The result of the unary & operator is a
pointer to its operand. The operand shall be an lvalue or a
qualified-id." The unary & operator is known as the address-of
operator. The C++ standard is actually going further than forbidding
temporaries, it forbids rvalues, which are things one expects to see
on the Right hand side of an assignment, or =.
One of Scott Meyer's Effective C++ books covers all of this in great
detail, including how you can get a temporary that's an lvalue as
opposed to an rvalue.
Adam
--
http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 6:34 pm, Kiuhnm wrote: > On 4/26/2012 20:54, Adam Skutt wrote: > > On Apr 26, 12:02 pm, Kiuhnm wrote: > >> On 4/26/2012 16:00, Adam Skutt wrote: > >>> On Apr 26, 9:37 am, Kiuhnm wrote: > >> The fact that you think that that's "differing behaviour" is what makes > >> it a misfeature. The fact that you think that '==' can take objects as > >> operands confirms that Java *does* confuse programmers. > > > The equality operator can absolutely be used between two objects. Try > > it if you don't believe me. It always does identity comparison when > > given two objects. It can also be given two primitives, and in this > > case, it does value comparison. Despite performing different > > operations with the same symbol, there's little risk of confusion > > because I can trivially figure out if a variable is an object or an > > primitive. > > No, it can't be used between objects but only between primitives and > references (which should be regarded as primitives, by the way). The only way to access an object is through a reference. >> If you > understand that your 'a' is not really an object but a reference to it, > everything becomes clear and you see that '==' always do the same thing. Yes, object identity is implemented almost? everywhere by comparing the value of two pointers (references)[1]. I've already said I'm not really sure how else one would go about implementing it. > You might tell me that that's just an implementation detail, but when an > implementation detail is easier to understand and makes more sense than > the whole abstraction which is built upon it, something is seriously wrong. I'm not sure what abstraction is being built here. I think you have me confused for someone else, possibly Steven. You're missing the big picture. The two comparisons are asking different questions: Value equality asks if the operands 'have the same state' regardless of how they exist in memory. Identity equality asks if the two operands are the same block of memory. The two are distinct because not all types support both operations. If I write a function that does a value comparison, then it should do value comparison on _every type that can be passed to it_, regardless of whether the type is a primitive or an object, whether it has value or reference semantics, and regardless of how value comparison is actually implemented. If I write some function: f(x : T, y : U) => x == y where T and U are some unknown types, then I want the function to do a value comparison for every type pairing that allows the function to compile. Likewise, if I write a function that does identity comparison, then it logically wants to do identity comparison on _every type that can be passed to it_. To accomplish this, I must have a distinct way of asking each question. In Python we have '==' and 'is'[2]; in Java we have 'Object.equals()' and '=='; in C and C++ we distinguish by the types of the variables being compared (T and T*). Java gives '==' a different meaning for primitive types, but that turns out to be OK because I can't write a function that takes both a primitive type and a reference type at the same position. Yes, the reason it does this is due to what I said above, but that doesn't have any bearing on why we pick one operation over the other as programmers. > The distinction between primitives and objects is unfortunate. It is as > if Java tried to get rid of pointers but never completely succeeded in > doing that. > It's the distinction between primitives and objects that should've been > an implementation detail, IMO. > > Python's lack of this misfeature is what I'm really fond of. If anything, you have that backwards. Look at Python: all variables in Python have pointer semantics, not value semantics. In imperative languages, pointers have greater utility over value types because not all types can obey the rules for value types. For example, I don't know how to give value semantics to something like a I/O object (e.g, file, C++ fstream, C FILE), since I don't know how to create independent copies. One can obviously create an imperative language without pointers, but I/O gets rather tricky. Adam [1] Though it need not be (and often isn't) as simple as comparing two integers. [2] Well, I suspect 'is' gets used mostly for comparisons against None, True, and False in Python. -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 2:31 pm, John Nagle wrote: > On 4/26/2012 4:45 AM, Adam Skutt wrote: > > On Apr 26, 1:48 am, John Nagle wrote: > >> On 4/25/2012 5:01 PM, Steven D'Aprano wrote: > > >>> On Wed, 25 Apr 2012 13:49:24 -0700, Adam Skutt wrote: > > >>>> Though, maybe it's better to use a different keyword than 'is' though, > >>>> due to the plain English > >>>> connotations of the term; I like 'sameobj' personally, for whatever > >>>> little it matters. Really, I think taking away the 'is' operator > >>>> altogether is better, so the only way to test identity is: > >>>> id(x) == id(y) > > >>> Four reasons why that's a bad idea: > > >>> 1) The "is" operator is fast, because it can be implemented directly by > >>> the interpreter as a simple pointer comparison (or equivalent). > > >> This assumes that everything is, internally, an object. In CPython, > >> that's the case, because Python is a naive interpreter and everything, > >> including numbers, is "boxed". That's not true of PyPy or Shed Skin. > >> So does "is" have to force the creation of a temporary boxed object? > > > That's what C# does AFAIK. Java defines '==' as value comparison for > > primitives and '==' as identity comparison for objects, but I don't > > exactly know how one would do that in Python. > > I would suggest that "is" raise ValueError for the ambiguous cases. > If both operands are immutable, "is" should raise ValueError. I don't know how you would easily detect user-defined immutable types, nor do I see why such an operation should be an error. I think it would end up violating the principal of least surprise in a lot of cases, especially when talking about things like immutable sets, maps, or other complicated data structures. What I think you want is what I said above: ValueError raised when either operand is a /temporary/ object. Really, it should probably be a parse-time error, since you could (and should) make the determination at parse time. > That's the case where the internal representation of immutables > shows through. You still have this problem with mutable temporary objects, as my little snipped showed. You're still going to get a result that's inconsistent and/or "surprising" sooner or later. The problem is the temporary nature of the object, not mutability. > > If this breaks a program, it was broken anyway. It will > catch bad comparisons like > > if x is 1000 : > ... > > which is implementation dependent. Yes, I agree that a correct fix shouldn't break anything except already broken programs. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 1:34 pm, rusi wrote: > On Apr 26, 7:44 pm, Adam Skutt wrote: > > On Apr 26, 10:18 am, rusi wrote: > > > > On Apr 26, 4:42 pm, Adam Skutt wrote: > > > > > In a mathematical sense, you're saying that given f(x) = x+2, using > > > > f(x) is somehow more "direct" (whatever the hell that even means) than > > > > using 'x+2'. That's just not true. We freely and openly interchange > > > > them all the time doing mathematics. Programming is no different. > > > > If f(x) and x+2 are freely interchangeable then you have referential > > > transparency, a property that only purely functional languages have. > > > In python: > > > I think you misunderstood what I was trying to explain. Steven is > > trying to claim that there's some sort of meaningful difference > > between calling an operation/algorithm/function by some name versus > > handing out its definition. I was merely pointing out that we > > routinely substitute the two when it is appropriate to do so. > > > My apologies if you somehow took that to mean that I was implying > > there was referential transparency here. I couldn't think of a better > > example for what I was trying to say. > > > Adam > > And my apologies... I forgot to state my main point: > Programmer accessible object identity is the principal impediment to > referential transparency. > In a functional language one can bind a name to a value -- period. > There is nothing more essence-ial -- its platonic id -- to the name > than that and so the whole can of worms connected with object identity > remains sealed within the language implementation. Yes, I agree that object identity is a major hold up, but I think side effects are a bigger problem. It's possible in C++ to create types that behave like the primitive types without too much difficulty, hence making object identity unimportant. However, it's considerably more difficult in C++ to write side-effect free code[1]. This is a bit of an apple and orange thing, though. ;) I often wonder what the world would be like if Python, C#, and Java embraced value types more, and had better support for pure functions. Unfortunately, building a language where all types behave like that is rather difficult, as the Haskell guys have shown us ;). Adam [1] Or even just code that only uses side-effects the compiler understands. -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 12:02 pm, Kiuhnm wrote: > On 4/26/2012 16:00, Adam Skutt wrote: > > On Apr 26, 9:37 am, Kiuhnm wrote: > >> On 4/26/2012 13:45, Adam Skutt wrote: > > >>> On Apr 26, 1:48 am, John Naglewrote: > >>>> This assumes that everything is, internally, an object. In > >>>> CPython, > >>>> that's the case, because Python is a naive interpreter and everything, > >>>> including numbers, is "boxed". That's not true of PyPy or Shed Skin. > >>>> So does "is" have to force the creation of a temporary boxed object? > > >>> That's what C# does AFAIK. Java defines '==' as value comparison for > >>> primitives and '==' as identity comparison for objects, but I don't > >>> exactly know how one would do that in Python. > > >> Why should we take from Java one of its worst misfeatures and disfigure > >> Python for life? > > > There are a lot of misfeatures in Java. Lack of operating overloading > > really isn't one of them. I prefer languages that include operator > > overloading, but readily understand and accept the arguments against > > it. Nor is the differing behavior for '==' between primitives and > > objects a misfeature. > > The fact that you think that that's "differing behaviour" is what makes > it a misfeature. The fact that you think that '==' can take objects as > operands confirms that Java *does* confuse programmers. > The equality operator can absolutely be used between two objects. Try it if you don't believe me. It always does identity comparison when given two objects. It can also be given two primitives, and in this case, it does value comparison. Despite performing different operations with the same symbol, there's little risk of confusion because I can trivially figure out if a variable is an object or an primitive. > > C# and Python do have a misfeature: '==' is identity comparison only > > if operator== / __eq__ is not overloaded. Identity comparison and > > value comparison are disjoint operations, so it's entirely > > inappropriate to combine them. > > They're not "disjoint", in fact one almost always implies the other (*). "Almost always" isn't a rebuttal. There's no requirement whatsoever for the results of identity comparison to be related to the results of value comparison, ergo they are disjoint. Changing one doesn't have to influence the other. Please note that I never advocated doing what Java does, I merely noted what it does. > Python's idea is that, by default, any object is equal to itself and > only itself. Which is just wrong-headed. Many types have no meaningful definition for value equality, ergo any code that attempts to perform the operation is incorrect. > (*) nan == nan is false, but, at least conceptually, a 'NotComparable' > exception should be raised instead. That wouldn't be very useful, though. > >> Python's way is much much cleaner. > > > Nope. Automatically substituting identity equality for value equality > > is wrong. While rare, there are legitimate reasons for the former to > > be True while the latter is False. > > There shouldn't be, to be fair. Which is the whole problem. It's nice to keep erroneous conditions out of your domain, but it's just not always possible. I don't know how you implement NaN (which you need) without allowing for this. I don't know how you implement SQL NULL without allowing for this. While lots of problems can avoid this issue, I'm not sure all problems can. Moreover, I don't know how to implement a value comparison for many objects, so the operation should just be undefined. I should point out that I was a little hasty in painting Python with the same brush as C# and excluding Java. Python and Java are equally bad: value equality defaults to identity equality but there are distinct operations for telling them apart. People want identity equality in Python write 'is', not '=='. People who explicitly want value equality in Java write 'equals()'. I apologize, and blame skipping breakfast this morning. C# is arguably worse, since '==' on objects is defined as identity equality unless it has been overridden. This means that that the intent of the operation varies with no easy way to figure it out in context, you simply have to know. C# also provides a way to test only for identity, Object.ReferenceEquals(), but it's underused. Ultimately this is really a problem of documentation: the language shouldn't encourage conflation of intent i
Re: why () is () and [] is [] work in other way?
On Apr 26, 10:18 am, rusi wrote: > On Apr 26, 4:42 pm, Adam Skutt wrote: > > > > > In a mathematical sense, you're saying that given f(x) = x+2, using > > f(x) is somehow more "direct" (whatever the hell that even means) than > > using 'x+2'. That's just not true. We freely and openly interchange > > them all the time doing mathematics. Programming is no different. > > If f(x) and x+2 are freely interchangeable then you have referential > transparency, a property that only purely functional languages have. > In python: I think you misunderstood what I was trying to explain. Steven is trying to claim that there's some sort of meaningful difference between calling an operation/algorithm/function by some name versus handing out its definition. I was merely pointing out that we routinely substitute the two when it is appropriate to do so. My apologies if you somehow took that to mean that I was implying there was referential transparency here. I couldn't think of a better example for what I was trying to say. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 9:37 am, Kiuhnm wrote: > On 4/26/2012 13:45, Adam Skutt wrote: > > > > > > > > > > > On Apr 26, 1:48 am, John Nagle wrote: > >> On 4/25/2012 5:01 PM, Steven D'Aprano wrote: > > >>> On Wed, 25 Apr 2012 13:49:24 -0700, Adam Skutt wrote: > > >>>> Though, maybe it's better to use a different keyword than 'is' though, > >>>> due to the plain English > >>>> connotations of the term; I like 'sameobj' personally, for whatever > >>>> little it matters. Really, I think taking away the 'is' operator > >>>> altogether is better, so the only way to test identity is: > >>>> id(x) == id(y) > > >>> Four reasons why that's a bad idea: > > >>> 1) The "is" operator is fast, because it can be implemented directly by > >>> the interpreter as a simple pointer comparison (or equivalent). > > >> This assumes that everything is, internally, an object. In CPython, > >> that's the case, because Python is a naive interpreter and everything, > >> including numbers, is "boxed". That's not true of PyPy or Shed Skin. > >> So does "is" have to force the creation of a temporary boxed object? > > > That's what C# does AFAIK. Java defines '==' as value comparison for > > primitives and '==' as identity comparison for objects, but I don't > > exactly know how one would do that in Python. > > Why should we take from Java one of its worst misfeatures and disfigure > Python for life? There are a lot of misfeatures in Java. Lack of operating overloading really isn't one of them. I prefer languages that include operator overloading, but readily understand and accept the arguments against it. Nor is the differing behavior for '==' between primitives and objects a misfeature. C# and Python do have a misfeature: '==' is identity comparison only if operator== / __eq__ is not overloaded. Identity comparison and value comparison are disjoint operations, so it's entirely inappropriate to combine them. I don't necessarily mind if the two operations have the same symbol, as long as there's some other way in-context to determine which operation is occurring. This is the case in C and C++, for example. > Python's way is much much cleaner. Nope. Automatically substituting identity equality for value equality is wrong. While rare, there are legitimate reasons for the former to be True while the latter is False. Moreover, it means that class authors must remember to write an __eq__ when appropriate and won't get any sort of error when they forget to do so. That can lead to bugs. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 1:48 am, John Nagle wrote: > On 4/25/2012 5:01 PM, Steven D'Aprano wrote: > > > On Wed, 25 Apr 2012 13:49:24 -0700, Adam Skutt wrote: > > >> Though, maybe it's better to use a different keyword than 'is' though, > >> due to the plain English > >> connotations of the term; I like 'sameobj' personally, for whatever > >> little it matters. Really, I think taking away the 'is' operator > >> altogether is better, so the only way to test identity is: > >> id(x) == id(y) > > > Four reasons why that's a bad idea: > > > 1) The "is" operator is fast, because it can be implemented directly by > > the interpreter as a simple pointer comparison (or equivalent). > > This assumes that everything is, internally, an object. In CPython, > that's the case, because Python is a naive interpreter and everything, > including numbers, is "boxed". That's not true of PyPy or Shed Skin. > So does "is" have to force the creation of a temporary boxed object? That's what C# does AFAIK. Java defines '==' as value comparison for primitives and '==' as identity comparison for objects, but I don't exactly know how one would do that in Python. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: why () is () and [] is [] work in other way?
On Apr 26, 5:10 am, Steven D'Aprano wrote: > On Wed, 25 Apr 2012 20:50:21 -0700, Adam Skutt wrote: > > On Apr 25, 8:01 pm, Steven D'Aprano > +comp.lang.pyt...@pearwood.info> wrote: > >> On Wed, 25 Apr 2012 13:49:24 -0700, Adam Skutt wrote: > >> > Though, maybe it's better to use a different keyword than 'is' > >> > though, due to the plain English > >> > connotations of the term; I like 'sameobj' personally, for whatever > >> > little it matters. Really, I think taking away the 'is' operator > >> > altogether is better, so the only way to test identity is: > >> > id(x) == id(y) > > >> Four reasons why that's a bad idea: > > >> 1) The "is" operator is fast, because it can be implemented directly by > >> the interpreter as a simple pointer comparison (or equivalent). The > >> id() idiom is slow, because it involves two global lookups and an > >> equality comparison. Inside a tight loop, that can make a big > >> difference in speed. > > > The runtime can optimize the two operations to be equivalent, since they > > are logically equivalent operations. If you removed 'is', there's > > little reason to believe it would do otherwise. > > I'm afraid you are mistaken there. *By design*, Python allows shadowing > and monkey-patching of built-ins. (Although not quite to the same degree > as Ruby, and thank goodness!) > Yes, I understand that. You still haven't explained why this behavior is correct in this particular situation. Arguing from the position of, "What Python does must be correct" isn't a valid tactic, I'm afraid. > It's useful for the same reason that shadowing any other builtin is > useful. id() isn't special enough to complicate the simple, and > effective, execution model just to satisfy philosophers. If overriding id() is useful, then overriding 'is' must be useful too. Python is still broken. Unless you can prove the two operations shouldn't be logically equivalent (and you don't), you can't meaningfully argue for different semantics for them. You still end up with a broken language either way. > They are not *logically* equivalent. First you have to define what you > mean by identity, then you have to define what you mean by an ID, and > then you have to decide whether or not to enforce the rule that identity > and IDs are 1:1 or not, and if so, under what circumstances. You're going to have to explain the value of an "ID" that's not 1:1 with an object's identity, for at least the object's lifecycle, for a programmer. If you can't come up with a useful case, then you haven't said anything of merit. Plainly, to show they're not logically equivalent, you need to explain why the guarantee provided by id() is improper. Then, you need to generalize it to all programming languages. Python's concept of identity is not unique nor special. It uses the exact same rules as C+ +, C#, Java, and many other languages. > My library > card ID may, by coincidence, match your drivers licence ID. Doesn't mean > we're the same person. I don't know why you even remotely think this is relevant. All it does is further demonstrate that you don't understand the object- oriented concept of identity at all. Comparing library IDs and drivers license IDs is an improper operation. Languages that can have multiple IDs, possibly overlapping, /do/ disallow such idiocy. > identities. The Borg design pattern, for example, would be an excellent > candidate for ID:identity being treated as many-to-one. How would inheritance work if I did that? > Even if you decide that treating IDs as 1:1 is the only thing that makes > sense in your philosophy, in practice that does not hold for Python. IDs > may be reused by Python. They may be reused in all languages I can think of. They're only unique for the lifetime of the object, because that's all we need as programmers. > There are circumstances where different objects > get the same ID. Hence, comparing IDs is not equivalent to identity > testing. Two objects only get the same ID if one of the objects is dead. The results of such an comparison are obviously meaningless. Some runtimes even try very hard to prevent you from doing such silly things. > > But I was actually referring to something more fundamental than that. The > statement "a is b" is a *direct* statement of identity. "John is my > father." "id(a) == id(b)" is *indirect*: "The only child of John's > grandfather is the parent of the mother-in-law of my sister-in-law" sort > o
Re: why () is () and [] is [] work in other way?
On Apr 25, 8:01 pm, Steven D'Aprano wrote:
> On Wed, 25 Apr 2012 13:49:24 -0700, Adam Skutt wrote:
> > Though, maybe it's better to use a different keyword than 'is' though,
> > due to the plain English
> > connotations of the term; I like 'sameobj' personally, for whatever
> > little it matters. Really, I think taking away the 'is' operator
> > altogether is better, so the only way to test identity is:
> > id(x) == id(y)
>
> Four reasons why that's a bad idea:
>
> 1) The "is" operator is fast, because it can be implemented directly by
> the interpreter as a simple pointer comparison (or equivalent). The id()
> idiom is slow, because it involves two global lookups and an equality
> comparison. Inside a tight loop, that can make a big difference in speed.
The runtime can optimize the two operations to be equivalent, since
they are logically equivalent operations. If you removed 'is',
there's little reason to believe it would do otherwise.
>
> 2) The "is" operator always has the exact same semantics and cannot be
> overridden. The id() function can be monkey-patched.
>
I can't see how that's useful at all. Identity is a fundamental
property of an object; hence retrieval of it must be a language
operation. The fact Python chooses to do otherwise is unfortunate,
but also irrelevant to my position.
> 3) The "is" idiom semantics is direct: "a is b" directly tests the thing
> you want to test, namely whether a is b. The id() idiom is indirect:
> "id(a) == id(b)" only indirectly tests whether a is b.
The two expressions are logically equivalent, so I don't see how this
matters, nor how it is true.
>
> 4) The id() idiom already breaks if you replace names a, b with
> expressions:
>
> >>> id([1,2]) == id([3,4])
>
> True
It's not broken at all. The lifetime of temporary objects is
intentionally undefined, and that's a /good/ thing. What's
unfortunate is that CPython optimizes temporaries differently between
the two logically equivalent expressions.
As long as this holds:
>>> class A(object):
... def __del__(self):
...print "Farewell to: %d" % id(self)
...
>>> A() is A()
Farewell to: 4146953292
Farewell to: 4146953260
False
>>> id(A()) == id(A())
Farewell to: 4146953420
Farewell to: 4146953420
True
then there's nothing "broken" about the behavior of either expression.
I personally think logically equivalent expressions should give the
same results, but since both operations follow the rules of object
identity correctly, it's not the end of the world. It's only
surprising to the programmer if:
1) They don't understand identity.
2) They don't understand what objects are and are not temporaries.
Code that relies on the identity of a temporary object is generally
incorrect. This is why C++ explicitly forbids taking the address
(identity) of temporaries. As such, the language behavior in your
case is inconsequential. Making demons fly out of the programmer's
nose would be equally appropriate.
The other solution is to do what Java and C# do: banish id() entirely
and only provide 'is' (== in Java, Object.ReferenceEquals() in C#).
That seems just as fine, really, Practically, it's also probably the
better solution for CPython, which is fine by me. My preference for
keeping id() and removing 'is' probably comes from my background as a C
++ programmer, and I already said it matters very little.
> But that's absolutely wrong. id(x) returns an ID, not an address.
> It just
> happens that, as an accident of implementation, the CPython interpreter
> uses the object address as an ID, because objects can't move. That's not
> the case for all implementations. In Jython, objects can move and the
> address is not static, and so IDs are assigned on demand starting with 1:
>
> steve@runes:~$ jython
> Jython 2.5.1+ (Release_2_5_1, Aug 4 2010, 07:18:19)
> [OpenJDK Client VM (Sun Microsystems Inc.)] on java1.6.0_18
> Type "help", "copyright", "credits" or "license" for more information.>>>
> id(42)
> 1
> >>> id("Hello World!")
> 2
> >>> id(None)
>
> 3
>
An address is an identifier: a number that I can use to access a
value[1]. I never said that id() must return an address the host CPU
understands (virtual, physical, or otherwise). Most languages use
addresses that the host CPU cannot understand without assistance at
least sometimes, including C on some platforms.
> Other implementations may make other choices. I don't believe that the
> language even defines the id as a number, alth
Re: why () is () and [] is [] work in other way?
On Apr 25, 10:38 am, Nobody wrote: > On Mon, 23 Apr 2012 10:01:24 -0700, Paul Rubin wrote: > >> I can't think of a single case where 'is' is ill-defined. > > > If I can't predict the output of > > > print (20+30 is 30+20) # check whether addition is commutative print > > (20*30 is 30*20) # check whether multiplication is commutative > > > by just reading the language definition and the code, I'd have to say "is" > > is ill-defined. > > If anything is ill-defined, then it's "+" and "*", i.e. it's unspecified > whether the value which they return is a unique object or a reference to > some other object. > Such a definition precludes meaningful operator overloading and is highly problematic for floating-point numbers. There's also no way to enforce it, but I think you know that too. :) Identity and equality are distinct concepts in programming languages. There's nothing that can be done about that, and no particularly good reason to force certain language behaviors because some "programmers" have difficulty with the distinction. Though, maybe it's better to use a different keyword than 'is' though, due to the plain English connotations of the term; I like 'sameobj' personally, for whatever little it matters. Really, I think taking away the 'is' operator altogether is better, so the only way to test identity is: id(x) == id(y) Though I would prefer: addr(x) == addr(y) myself, again, for what little it matters. The right thing to do when confronted with this problem is teach the difference and move on. As an aside, the whole problem with 'is' and literals is perhaps the only really good argument for a 'new' keyword/operator like C++ and Java have. Then it's more explicit to the programmer that they've created two objects (in this case, anyway). > More accurately, the existence of "is", "is not" and "id" cause many other > constructs to have "ill-defined" behaviour. > > >> "a is b" is true iff 'a' and 'b' are the same object. Why should 'is' > >> lie to the user? > > > Whether a and b are the same object is implementation-dependent. > > And what's wrong with that? If you want a language which precisely > specifies all observable behaviour, you're going to end up with a rather > useless language. For a start, it can't have a time() function. For > similar reasons, you can't have networking or any form of preemptive > concurrency (which includes any form of inter-process communication on an > OS which uses preemptive multi-tasking). Fully specified does not mean fully deterministic. What makes a specification of "Any value in the range 0 through N" less 'full' than a specification of "X" or a constant? Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: os.system()
On Apr 20, 6:51 am, Yigit Turgut wrote: > On Apr 19, 11:02 pm, "Steve" wrote: > > > > "Yigit Turgut" wrote in message > > >news:b9a8bb28-3003-4a36-86fb-339ef697b...@i2g2000vbd.googlegroups.com... > > > When I use os.system() function, script waits for termination of the > > > windows that is opened by os.system() to continue thus throwing errors > > and etc. How can i tell Python to let it go and keep on with the next > > execution after os.system() ? > > > Can you make use of subprocess.Popen() ? > > Still waits for the new windows to kill. It should not do so unless you tell it do so, but without more details it's difficult to determine what is happening. The only waiting that should be necessary is to eventually call Popen.poll() or Popen.wait() to reap the subprocess after it terminates, but there are many correct ways to do that depending on your application and the subprocess. A thread, as suggested by Jacob, is one way to accomplish it. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: system call that is killed after n seconds if not finished
On Apr 16, 11:34 am, Alain Ketterlin
wrote:
> Jaroslav Dobrek writes:
> > I would like to execute shell commands, but only if their execution
> > time is not longer than n seconds. Like so:
>
> > monitor(os.system("do_something"), 5)
>
> > I.e. the command do_somthing should be executed by the operating
> > system. If the call has not finished after 5 seconds, the process
> > should be killed.
>
> > How could this be done?
>
> My system (linux) has a timeout command that does just this.
Alas, while timeout(1) is in GNU coreutils it's a rather recent
addition and so you cannot rely on it being on every Linux machine.
Also, there are multiple versions of timeout, from different
applications, that do the same thing with different syntax. However,
if it is available on the systems you care about, I would use it.
Pay close attention to its caveats.
> Otherwise,
> you may use subprocess.Popen to spawn the process, sleep for 5 seconds
> in the parent, then use Popen.poll() to check whether the process has
> finished, and finally Popen.kill() if it has not.
>
That's true subject to two conditions:
1. The subprocess doesn't spawn child processes of its own or you
don't care about killing them (doubtful).
2. The application isn't running on a command-line, where it could be
suspended; or you don't care about timeliness if the application is
suspended.
The first item can be solved by manually performing fork()/exec() and
using setpgid() to create a process group. Terminating the process
group on timeout will terminate the subprocess and all of its
children.
The second one is much more tricky to solve. Basically, you have to
supply enough job control functionality (like a shell) to handle the
terminal signals and ensure your signaling process stays alive. The
GNU coreutils version of timeout basically immunizes itself from the
signals, which works but also creates problems if it is run in the
background. If you need to do this, I'd advise cheating, as I believe
the general solution is pretty ugly and doesn't exist in ready form.
I certainly could not find it.
Adam
--
http://mail.python.org/mailman/listinfo/python-list
Re: Python randomly exits with Linux OS error -9 or -15
On Apr 9, 6:01 am, Janis wrote: > Hello! > > I have this problem with my script exiting randomly with Linux OS > status code -9 (most often) or -15 (also sometimes, but much more > rarely). As far as I understand -9 corresponds to Bad file descriptor > and -15 Block device required. > As Alain already said, you're probably hitting a resource limit of some sort or another and your program is being forcibly terminated as a result. You may just need to increase ulimits or system-wide limits, or you may have a resource leak. There may be output in the kernel logs (dmesg) or some /var/log file if this is the case. > 1) Is there a way how I could find out what exactly causes Python > process to exit? Use a debugger or at least turn on core dumps, so you have something to examine after the crash. Tracking through the output to figure out where you crashed in the Python code is difficult, but possible. > I have a try-catch block in the root of the script which works very > well to catch any kind of exceptions, but in these cases either Python > does not catch the exception or fails to log it into MySQL error log > table. You need to fix your handler to log the exception to stderr or similiar, as it will make fixing your application much easier. It's virtually impossible for anyone to help you if you don't know exactly what's going on. It's also highly likely that your current handler is throwing an exception while running and masking the original error. I would seriously advise taking it out, at least temporarily. This is why catch-all handlers tend to be a poor idea, as they're rarely robust in the cases you didn't consider. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Python randomly exits with Linux OS error -9 or -15
On Apr 9, 6:47 am, Alain Ketterlin wrote: > Janis writes: > > I have this problem with my script exiting randomly with Linux OS > > status code -9 (most often) or -15 (also sometimes, but much more > > rarely). As far as I understand -9 corresponds to Bad file descriptor > > and -15 Block device required. > > How do you get -9 and -15? Exit status is supposed to be between 0 and > 127. 0-255 are perfectly legal in UNIX. Chances are that something is interpreting the unsigned integer as a signed integer accidentally. Of course, without any output, there's no way to know for sure. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrupting a blocking function frolm another thread.
On Apr 8, 5:52 pm, superhac...@gmail.com wrote: > On Sunday, April 8, 2012 3:55:41 PM UTC-5, Adam Skutt wrote: > > On Apr 8, 2:45 pm, "superhac...@gmail.com" > > wrote: > > > I am using the python module nfqueue-bindings which is a nfqueue > > > packet intercepting module. It uses the following snippet of code to > > > start the process: > > > > print "trying to run" > > > try: > > > q.try_run() > > > except KeyboardInterrupt, e: > > > print "interrupted" > > > > The q.try_run() method blocks. I would like to be able to interrupt > > > this the same way you can hit control-c to unblock it, but from > > > another thread. Does anyone have any idea on how to do this? Is > > > there some sort of Exception I can generate from another thread that > > > would do this? > > > The simplest and most reliable way will be to modify the asynchronous > > I/O example > > athttps://www.wzdftpd.net/redmine/projects/nfqueue-bindings/repository/... > > to do what you want. The classical way to accomplish this would be to > > create a second I/O descriptor (via os.pipe or similiar) and wait for > > that descriptor in the same async I/O loop. When you want to stop the > > loop, send a message (could be a single byte) to the second descriptor > > and then respond appropriately in your asynchronous I/O handler. > > > However, simply ignoring the nfqueue socket while doing other > > processing might be acceptable too. I would read the documentation > > and ask questions carefully before taking that approach, as it may > > lead to lost data or other problems. The viability of this approach > > depends heavily on your application. > > > Adam > > Thanks for the reply Adam. I am still confused with the example with the > example you linked to. I have only been working with Python for a couple of > weeks, but I am fluent with C. Given the example you linked to, are you > saying that this is the run loop for asynchronous comm that a end user of the > module could use? Or is this something with the nfqueue-bindings module that > I would I have to modify. I don't recognize the the main loop in the example. asyncore is a standard python module for handling asynchronous I/O operations (i.e., select/poll operations). The main loop in that example is handled by the 'asyncore.loop()' call, which just calls select() in a loop until all channels are closed. You're looking at the standard Python library technique for an asynchronous I/O event loop. asyncore.file_dispatcher enables use of asyncore.loop() with standard UNIX FDs. nfqueue.queue objects provide access to their underlying FDs so you can wait on them. asyncore.loop() is the main loop of the application, performing endless select/poll waits and calling 'handle_read' whenever the netfilter FD has data for reading. You can find more details about the module in the Python library docs. Since your code is Linux specific, you can always call os.select() or os.poll() if it makes you feel better, but I don't think that's going to simplify anything here. Hopefully that helps you. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrupting a blocking function frolm another thread.
On Apr 8, 2:45 pm, "superhac...@gmail.com" wrote: > I am using the python module nfqueue-bindings which is a nfqueue > packet intercepting module. It uses the following snippet of code to > start the process: > > print "trying to run" > try: > q.try_run() > except KeyboardInterrupt, e: > print "interrupted" > > The q.try_run() method blocks. I would like to be able to interrupt > this the same way you can hit control-c to unblock it, but from > another thread. Does anyone have any idea on how to do this? Is > there some sort of Exception I can generate from another thread that > would do this? The simplest and most reliable way will be to modify the asynchronous I/O example at https://www.wzdftpd.net/redmine/projects/nfqueue-bindings/repository/entry/examples/nfq_asyncore.py to do what you want. The classical way to accomplish this would be to create a second I/O descriptor (via os.pipe or similiar) and wait for that descriptor in the same async I/O loop. When you want to stop the loop, send a message (could be a single byte) to the second descriptor and then respond appropriately in your asynchronous I/O handler. However, simply ignoring the nfqueue socket while doing other processing might be acceptable too. I would read the documentation and ask questions carefully before taking that approach, as it may lead to lost data or other problems. The viability of this approach depends heavily on your application. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: multithreading
On Apr 7, 5:06 pm, Kiuhnm wrote:
> On 4/7/2012 22:09, Bryan wrote:>> For instance, let's say I want to make this
> code thread-safe:
>
> >> --->
> >> myDict = {}
>
> >> def f(name, val):
> >> if name not in myDict:
> >> myDict[name] = val
> >> return myDict[name]
> >> <---
>
> > First, don't re-code Python's built-ins. The example is a job for
> > dict.setdefault().
>
> [...]
>
> That was just an example for the sake of the discussion.
> My question is this: can I use 'threading' without interfering with the
> program which will import my module?
'import threading' ought to work everywhere, but that's not enough to
tell you whether whatever you're trying to do will actually work.
However, you shouldn't need to do it unless your application is meant
to /only/ be used in applications that have done 'import threading'
elsewhere. Otherwise, you probably have a pretty serious design
issue.
Global state is bad. TLS state is little better, even if it's common
in a lot of python modules. Non-thread-safe object instances is
usually fine. Object construction needs to be thread-safe, but that's
also the default behavior. You need not worry about it unless you're
doing very unusual things.
Plainly, most of the time you shouldn't need to do anything to support
multiples threads beyond avoiding global state. In fact, you should
stop and give some serious thought to your design if you need to do
anything else.
Adam
--
http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 3, 10:09 am, Jérôme wrote: > Tue, 3 Jan 2012 06:12:59 -0800 (PST) > Adam Skutt a écrit: > > > The conservative approach is to use another IPC mechanism to talk to > > the process, such as a pipe or a socket. Why are you trying to send > > the child process SIGINT in the first place? > > Say, you've got an application that plays a sound for a few seconds, using an > external program for that (beep, sox). If you close the application, you want > the sound to stop as well. If your application wants to play audio and terminate playing audio early, it really should use an audio library and play the sound directly. Especially if you're playing so many sounds in such a fashion you need to terminate them. If you're playing a single beep or short sound, then who really cares if it occurs slightly after your processes ends? GStreamer is one such library for sound processing though it may be more complicated than you need. > Should I try something like this ? > > communicate(input="Ctrl+C") > > (I'd need to figure out how to write Ctrl+C...) > > Would that be cross-platform ? > No, that won't work at all. 'Ctrl-C' is a magic indicator to the terminal driver to send SIGINT to the appropriate processes: it never actually appears as an input and output character to a program unless it's controlling the terminal in raw mode (where it loses its special meaning). If you're really insistent on using the sox(1) command-line tools to play your sounds, then you'll have to send SIGINT or SIGTERM in order to tell it to terminate (which can be done just by calling the terminate() method). Which is fine and will probably work OK subject to three restrictions: 1. Nothing in your application modifies the default handling of SIGCHLD. This means that the child process appears as a zombie (e.g., via 'ps ax') until the parent calls poll() or wait() methods to reap it. 2. You only send signals while the process is alive or a zombie: once the poll() or wait() methods return something other than None, you cannot safely send anymore signals. 3. You trap any errors returned by the send signal call. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 3, 4:38 am, Jérôme wrote: > I have an application that can spawn a subprocess to play a beep. I want it > to kill the subprocess when exiting. Why? You shouldn't need to send a signal to tell the process to terminate: it should terminate when its stdin is closed or when it's done playing the beep in the first place. If it doesn't, I would find a better way to play a beep in the first place. What you're attempting to do sounds superfluous and unnecessary. The only thing you ought to need to do is ensure that the child process is properly reaped if it terminates before your process terminates. > To do so, my close() method must > > a/ Check if any subprocess has actually been launched (I store the Popen in > a variable called _beep_process. If Popen has not been called, the variable > is init to 0 and the call to send_signal will fail.) > > b/ Check if the process is still alive using Popen.poll() and returncode > (otherwise, I might kill a new process) > > c/ Catch the exception in case the process would be dead since the last > check (otherwise, I might get an error from send_signal) > Steps a and c are all that are necessary under standard SIGCHLD handling. However, all of this should be entirely unnecessary in the first place. Plus, I have a hard time imagining why something like 'gtk.gdk.beep()' isn't adequate for your needs anyway. > For instance, Popen.send_signal() should not be able to send a signal to a > subprocess that has already returned, should it ? Is there any good reason > for allowing it to do so ? If not, it would spare me check b/ in this example. > The problem is being able to distinguish "process is a zombie" from "process doesn't exist" which both return ESRCH. This is certainly possible with careful coding but I'm not sure I would bother except in the simplest programs. Too many libraries do too many questionable things with signal handlers so I find it much safer and easier just to avoid the damn things whenever possible. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 2, 11:53 pm, Cameron Simpson wrote: > On 02Jan2012 19:16, Adam Skutt wrote: > | On Jan 2, 8:44 pm, Cameron Simpson wrote: > | > On 02Jan2012 20:31, Devin Jeanpierre wrote: > | > | > I think that catching the exception is probably the most Pythonic way. > | > | > | > | It's the only correct way. > | > > | > Indeed, but be precise - chek that it _is_ error 3, or more portably, > | > errno.ESRCH. POSIX probably mandates that that is a 3, but the symbol > | > should track the local system if it differs. Example: > | > | No. It is possible (however unlikely) for EPERM to be legitimately > | returned in this case. Anything other than EINVAL should be > | interpreted as "The child process is dead". > > Sure. I was more taking the line: catch and accept only the specific > errors you understand. That advice really only applies when we have interfaces that don't fully define their exceptional conditions. That's not the case here. Though I should correct myself and note that EPERM can only occur if SIGCHLD is being handled in a non-default fashion. That being said, I'm not sure any error code should be treated differently from another here. > Of course he can catch EPERM also. But any other > variant should at the least generate a warning to stderr or a log > it is _unexpected_. Are they really unexpected though? Even if they are, what is logging an error going to gain anyone? Even crashing may not be helpful, especially given that there have been bugs in the past with the return codes from kill(2). Certainly, I'd be far more concerned about ensuring my code doesn't accidentally kill the wrong process than worrying about ensuring it handles the return code from kill(2) correctly. > I take your point that reraising the exception may be overkill for > failed signal delivery (if that was your point). But I am arguing for > being very careful about what you silently pass as an ok thing. > Trapping too few conditions is just as erroneous as trapping too many. This is especially true when dealing with UNIX system calls. The right behavior is frequently not as simple as it may appear, unfortunately. > | Hence why you should > | avoid sending the signal in the first place: the situations where you > | don't run the risk of possibly killing an innocent bystander are > | pretty narrow. While unlikely on modern UNiX and Linux, IMO it's best > | to avoid the issue altogether whenever possible. > > Fair enough too. But sometimes you need to nail a rogue child. Even if I believed that were true, I wouldn't use SIGINT to do it. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 3, 3:58 am, Chris Angelico wrote: > On Tue, Jan 3, 2012 at 7:44 PM, Jérôme wrote: > > If so, I don't see how I can protect myself from that. Checking the process > > is alive and then hoping that the time interval for the race condition is so > > small that there are few chances for that to happen (because the OS > > quarantines PID numbers for a while, for instance) ? > > The probability is extremely small. PIDs are generally allocated > sequentially, and obviously one won't be reallocated until the > previous process has terminated. You're looking at a narrow window of > opportunity between a check and an action; you don't really need to > worry about PID reuse within that window, unless there's a particular > reason to fear it (eg your process is very low priority, or there's a > lot of "process spinning" happening). Under normal circumstances, you > won't see a new process start up with the same PID for some time. > Not all operating systems attempt to generate sequential processes IDs. The window can be rather large in certain situations, and if you cannot rely on your child processes becoming zombies and kill being called only when the child is alive or a zombie, then you should not be calling kill(2) at all. Killing an unrelated process will be considered as a bug by users. Hence why I find it easier just to avoid the problem altogether if I can. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 3, 3:44 am, Jérôme wrote: > Mon, 2 Jan 2012 19:16:50 -0800 (PST) > Adam Skutt a écrit: > > > No. It is possible (however unlikely) for EPERM to be legitimately > > returned in this case. Anything other than EINVAL should be > > interpreted as "The child process is dead". Hence why you should > > avoid sending the signal in the first place: the situations where you > > don't run the risk of possibly killing an innocent bystander are > > pretty narrow. While unlikely on modern UNiX and Linux, IMO it's best > > to avoid the issue altogether whenever possible. > > Should I understand that Popen.send_signal blindly sends the signal to the > process of PID Popen.pid, and therefore could kill a new process of the same > PID that would have been launched by the same user in another program ? > Or possibly one launched by another user altogether. By /default/, child processes that terminate become zombies and remain in that state until their parent reaps them via wait(2) or a related syscall. This means that until you make such a call, the signal is delivered to the desired child process. In this case, kill(2) still returns ESRCH since the child process is a zombie and cannot possibly respond to the signal. However, if SIGCHLD has been explicitly ignored or has no SA_NOCLDWAIT set, child processes are reaped automatically. Likewise, it is possible to install a signal handler for SIGCHLD that calls wait and reaps the child processes. Do you know what all of your other Python modules and extensions do? If so, then you can probably rely on the default semantics. If not, I'd strongly suggest a more conservative approach. Regardless, of whether you can rely on the presence of your zombie children or not, you should expect the kill(2) call to fail and be prepared to trap the failure. Obviously, all of this is rather UNIX / Linux specific. > If so, I don't see how I can protect myself from that. Checking the process > is alive and then hoping that the time interval for the race condition is so > small that there are few chances for that to happen (because the OS > quarantines PID numbers for a while, for instance) ? The conservative approach is to use another IPC mechanism to talk to the process, such as a pipe or a socket. Why are you trying to send the child process SIGINT in the first place? Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 3, 7:31 am, Heiko Wundram wrote: > Am 03.01.2012 02:19, schrieb Adam Skutt: > > > On Jan 2, 6:09 pm, Jérôme wrote: > >> What is the clean way to avoid this race condition ? > > > The fundamental race condition cannot be removed nor avoided. Ideally, > > avoid the need to send the subprocess a signal in the first place. If > > it cannot be avoided, then trap the exception. > > Yes, it can be avoided, that's what the default SIGCHLD-handling > (keeping the process as a zombie until it's explicitly collected by a > wait*()) is for, which forces the PID not to be reused by the operating > system until the parent has acknowledged (by actively calling wait*()) > that the child has terminated. No, you still can see ESRCH when sending signals to a zombie process. Code that sends signals to child processes via kill(2) must be prepared for the call to fail at anytime since the process can die at anytime. It can't handle the signal, so it's treated as if it doesn't exist by kill(2) in this case. However, you don't have to worry about sending the signal to the wrong process. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 2, 8:44 pm, Cameron Simpson wrote: > On 02Jan2012 20:31, Devin Jeanpierre wrote: > | > I think that catching the exception is probably the most Pythonic way. > | > | It's the only correct way. > > Indeed, but be precise - chek that it _is_ error 3, or more portably, > errno.ESRCH. POSIX probably mandates that that is a 3, but the symbol > should track the local system if it differs. Example: > No. It is possible (however unlikely) for EPERM to be legitimately returned in this case. Anything other than EINVAL should be interpreted as "The child process is dead". Hence why you should avoid sending the signal in the first place: the situations where you don't run the risk of possibly killing an innocent bystander are pretty narrow. While unlikely on modern UNiX and Linux, IMO it's best to avoid the issue altogether whenever possible. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Avoid race condition with Popen.send_signal
On Jan 2, 6:09 pm, Jérôme wrote: > Hi all. > > When a subprocess is running, it can be sent a signal with the send_signal > method : > > process = Popen( args) > process.send_signal(signal.SIGINT) > > If the SIGINT is sent while the process has already finished, an error is > raised : > > File "/usr/lib/python2.7/subprocess.py", line 1457, in send_signal > os.kill(self.pid, sig) > OSError: [Errno 3] Aucun processus de ce type > > To avoid this, I can check that the process is still alive : > > process = Popen( args) > process.poll() > if (None == process.returncode): > process.send_signal(signal.SIGINT) > > It makes safer, but there is still an issue if the process ends between > poll() and send_signal(). > > What is the clean way to avoid this race condition ? The fundamental race condition cannot be removed nor avoided. Ideally, avoid the need to send the subprocess a signal in the first place. If it cannot be avoided, then trap the exception. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Why doesn't threading.join() return a value?
On Sep 2, 4:14 pm, Chris Torek wrote: > In article > Adam Skutt wrote: > > >No, it can only pass a void*, which isn't much better than passing an > >int. > > It is far better than passing an int, although it leaves you with > an annoying storage-management issue, and sidesteps any reasonable > attempts at type-checking (both of which are of course "par for > the course" in C). And when written out, makes it sound distinctly worse than passing an int :p. And let's not kid ourselves, unless you're a C programmer, it is distinctly worse than passing an int. Heck, your example (snipped) goes out of your way to unnecessarily leverage the functionality provided by pthreads. > Some manual pages are clearer about this than others. Here is one > that I think is not bad: > > The symbolic constant PTHREAD_CANCELED expands to a constant > expression of type (void *), whose value matches no pointer to > an object in memory nor the value NULL. > > So, provided you use pthread_exit() "correctly" (always pass either > NULL or the address of some actual object in memory), the special > reserved value is different from all of "your" values. Unfortunately, I'm not sure all implementations behave that way. Not that cancellation is really worth bothering with anyway, but it's a pretty nasty corner case. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Why doesn't threading.join() return a value?
On Sep 2, 2:23 pm, Alain Ketterlin wrote: > Sorry, you're wrong, at least for POSIX threads: > > void pthread_exit(void *value_ptr); > int pthread_join(pthread_t thread, void **value_ptr); > > pthread_exit can pass anything, and that value will be retrieved with > pthread_join. No, it can only pass a void*, which isn't much better than passing an int. Passing a void* is not equivalent to passing anything, not even in C. Moreover, specific values are still reserved, like PTHREAD_CANCELLED. Yes, it was strictly inappropriate for me to say both return solely integers, but my error doesn't meaningful alter my description of the situation. The interface provided by the underlying APIs is not especially usable for arbitrary data transfer. Doubly so when we're discussing something like Python's threading module. > I'm not sure what you are talking about here. Maybe you confuse threads > with processes? Windows threads have exit codes, just like processes. At least one code is reserved and cannot be used by the programmer. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Why doesn't threading.join() return a value?
On Sep 2, 10:53 am, Roy Smith wrote: > I have a function I want to run in a thread and return a value. It > seems like the most obvious way to do this is to have my target > function return the value, the Thread object stash that someplace, and > return it as the return value for join(). > > Yes, I know there's other ways for a thread to return values (pass the > target a queue, for example), but making the return value of the > target function available would have been the most convenient. I'm > curious why threading wasn't implemented this way. I assume it is because the underlying operating system APIs do not support it. Windows and POSIX threads only support returning an integer when a thread exits, similar to the exit code of a process. More importantly, there's no way to tell whether the exit code of a thread was set by user code or by the system. Even worse, some of those integer values are reserved by some operating systems. If your thread died via an exception, it still has an error code set by the operating system. How would you going to distinguish those codes from your own? Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Python thread
On Sep 1, 5:14 pm, George wrote: > Hi, > Why doesn't python threads show an associated PID? On spawning python > threads using the threading module I can only see the main thread's pid on > using top or ps unix command, no subprocesses are displayed. In otherwords > top or ps in not aware of any subprocesses created using threading module in > python. You probably need to run 'ps axm' or something similar to see to threads associated with a processes on your system. > > Whereas in Java , creating threads will result in separate pid , these > subprocesses can be listed using top or ps. Java threads get mapped to the > cores in the system. No. It depends on your runtime, but if Java uses native threads, it will have the same behavior as Python here. It also doesn't mean they get mapped to cores in the system. All it means is the operating system is responsible for scheduling the threads and managing their lifecycle. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Python thread
On Sep 1, 5:54 pm, Terry Reedy wrote: > > Does it mean that python threads are not mapped to the core in the system. > > They all run on the same core. > No, CPython is a native thread implementation, so they'll be scheduled however the kernel sees fit. Only allowing one thread to run at a time doesn't mean they'll always run on the same core. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Why __slots__ slows down attribute access?
On Aug 23, 5:48 am, Jack wrote:
> People have illusion that it is faster to visit the attribute defined
> by __slots__
> .http://groups.google.com/group/comp.lang.python/msg/c4e413c3d86d80be
>
> That is wrong. The following tests show it is slower.
No, they don't really show anything. The default clocks used by
timeit lack the resolution to measure such things accurately; you're
measuring various noise sources on your system. The range of 100
trials of 1million iterations on my system is 70.3 ms, which is 70ns
when divided by a million, which is about the size of the difference
you show. A 70ns average difference between iterations is trivially
attributable to noise on a modern machine.
Run enough trials or just wait for the moon to move a bit, and I
wouldn't be terribly surprised if you got difference results.
Rebooting your machine might be enough to do it.
Adam
> D:\>d:\python-v3.1.2\python -mtimeit -s "class A(object): __slots__ =
> ('a', 'b', 'c')" -s "inst = A()" "inst.a=5; inst.b=6; inst.c=7"
> 100 loops, best of 3: 0.237 usec per loop
>
> D:\>d:\python-v3.1.2\python -mtimeit -s "class A(object): pass" -s
> "inst = A()" "inst.a=5 inst.b=6; inst.c=7"
> 100 loops, best of 3: 0.214 usec per loop
>
> D:\>d:\python-v2.6.4\python -mtimeit -s "class A(object): __slots__ =
> ('a', 'b', 'c')" -s "inst = A()" "inst.a=5; inst.b=6; inst.c=7"
> 100 loops, best of 3: 0.26 usec per loop
>
> D:\>d:\python-v2.6.4\python -mtimeit -s "class A(object): pass" -s
> "inst = A()" "inst.a=5; inst.b=6; inst.c=7"
> 100 loops, best of 3: 0.217 usec per loop
--
http://mail.python.org/mailman/listinfo/python-list
Re: How to get number of bytes written to nonblocking FIFO when EAGAIN is raised?
On Jul 19, 9:19 pm, Aaron Staley wrote:
> However, if interpreter 1 overfills the FIFO, we get an error (EAGAIN)>>>
> f.write('a'*7)
>
> IOError: [Errno 11] Resource temporarily unavailable
>
> However interpreter 2 still receives data>> len(f.read())
>
> 65536
>
> It looks like interpreter 1 pushed data until the FIFO was full and
> then raised the IOError. Interpreter 2 constantly received some, but
> not all, of what interpreter 2 tried to send.
> Unfortunately, the IOError seems to have no attribute indicating how
> much data was successfully sent. I've looked through the docs and
> can't seem to figure out how; can anyone land some advice?
You need to do as Roy Smith suggested and use the actual OS I/O calls
os.read() and os.write(). os.write() returns the number of bytes
actually written to the underlying descriptor. Python file objects
are akin to FILE structures in C: they perform buffering and other
operations internally that makes them less than suitable for usage
with asynchronous UNIX I/O. In particular, they buffer I/O internally
before writing it to the descriptor, so there's no direct relationship
between calling file.write() and how much data is written to the
stream. In addition, file objects also simply raise the underlying OS
error when it occurs. The UNIX write(2) syscall assumes that you have
been keeping track of how many bytes you've successfully written to
the stream and does not track it for you.
Adam
--
http://mail.python.org/mailman/listinfo/python-list
Re: Best way to gain root privileges
On Feb 18, 9:04 am, Ricardo Aráoz wrote: > I've always asked myself why can't a program be used by users of a > certain group but run with the privileges of some other user, not > necessarily the one that uses it, but one created specifically for the > tasks the program is responsible for. > > AFAIK in Linux a program will only run with the privileges of the user > who runs it. This is precisely what the setuid /setgid bits do: force the program to run as the user and group that owns the program, instead of the user / group of the caller. It is forbidden for scripts due to lots of historical problems and inherent difficultly in writing (shell) scripts that can be elevated securely. That being said, it's typically a terrible, horrible idea. The effort involved in ensuring the program cannot be exploited rarely outweighs the gains involved from switching privileges. > Many a time I have wanted to allow access to certain privileges to a user but > *only* > through a program. As far as security is concerned it would be enough > that only root has permission to give the said program running > privileges (privileges different from those of the user that is actually > running it), that only allowed users may modify the program, and that > *other* users may only run it. This would address the issue of someone > modifying the program to gain access to it's privileges. Now, if someone > is able to gain illegal privileges to modify the program, then there > *is* a security hole and the program is not really the problem. sudo already does this to a limited degree. If you want more granularity than sudo, you're looking at mandatory access controls. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to gain root privileges
On Feb 17, 10:32 am, GSO wrote: > > I'm having a awfully hard time figuring out why a home CCTV > > application might need privilege at all. Are you sure you really need > > privilege? It sounds to me like there may be some larger design > > issues mandating the need for privilege when it's not really > > necessary. > > A user login should only able to view the footage. It's important > that a user login cannot delete any images/video. This much can be > done with ACL - but having said that a user login would still be able > to copy the images/video, so ACL would work but is not ideal - I could > prevent copying with raised privileges. No, you can't. If I can read the data, I can copy the data. How did you think you were going to prevent copying? Remember that copying is just read()/write() in a loop. > If I were to allow a user to > archive footage without using an admin login then that would require > ACL with write access, which is out of the question. > > If a camera loses its connection I think it's OK to let a user restart > the camera without using gksu, but this would require raised > privileges. > > There are other misc. points where I need write access. The directory > where images are stored by the live feed can become 'messy' (for want > of a better way of putting it), write access is needed to tidy it up > before live camera images can be viewed, it's not really appropriate > to use gksu here every time a user wants to view the live images. > Also (I don't know exactly how I'm going to do this yet) but I'm > thinking I might use the system log functions (syslogd) as part of a > scheme to ensure the integrity of saved footage and the archive. > As Terry suggests (and I fully concur), all of these issues are best solved by having a privileged daemon (though it may not need to be root or entirely root). Inter-process communication, if necessary, can be handled by UNIX domain sockets. There are then a few ways to validate the privileges of the users connecting to the sockets: * UNIX sockets support passing credentials across the socket for privilege checks. This is however somewhat non-portable (different platforms do it differently), and I'd imagine a minor PITA in the Python. * Create sockets with different ownership/permissions and then require the users to be in the correct UNIX group. Note that the permissions of the sockets themselves are ignored on some platforms, so each socket needs to be in a directory with the correct permissions and ownership set: /var/run/cctvd/admin/ <--- cctv:cctvadmin 750 /var/run/cctvd/admin/socket <--- Doesn't matter, so 777 cctv:cctvadmin is fine /var/run/cctvd/user/ < cctv:cctv 750 (or cctvuser for the group, if you'd rather) /var/run/cctvd/user/socket <--- Doesn't matter, so 777 cctv:cctv Though these days it may be overkill, you may be able to rely on the socket write permission (the others don't matter) and user/group for protection. You can also do actual authentication, but that's another huge can of worms best left undisturbed if possible. * It also allows you to simply stream the files over the sockets instead of the client accessing the files directly. Your call, though. Done properly, there's a high probability the only thing that needs to run as root is the init script. You may also need startup code that runs as root (not setuid root) that handles daemonization, opens anything that must be open as root, and then drops root privileges. Lots of daemons do this, so you can find examples out there on how to accomplish this. There are also utilities that can handle this for you, which may be adequate in your situation. Privilege separation can be used so that a bug in handling the user socket doesn't accidentally grant someone admin and/or root privileges (basically, have separate processes with separate privileges). There are examples out there of how to achieve this for various purposes as well. As a rule, when you find yourself needing privilege escalation, you should go back and see if there is another way to achieve what you want. There's a good chance the other way will be a better way. It sounds to me like you need to better define all the operations that your software needs to be able to accomplish and what privileges are necessary to accomplish each operation. Once you've defined that, it should become much easier to build a design that properly handles each operation without the need for privilege escalation. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to gain root privileges
On Feb 16, 10:43 pm, GSO wrote: > OK, so I'm heading towards sudo then, aiming to make sure I don't > screw up the configuration. This is a home CCTV application, so I > want things as secure as possible. A setgid wrapper would require the > kind of skilled programming that I couldn't do myself in order to keep > things at a high level of security, but sudo I can handle. I'm having a awfully hard time figuring out why a home CCTV application might need privilege at all. Are you sure you really need privilege? It sounds to me like there may be some larger design issues mandating the need for privilege when it's not really necessary. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to gain root privileges
On Thu, Feb 17, 2011 at 2:12 AM, Dan Stromberg wrote: > > On Wed, Feb 16, 2011 at 6:59 PM, Adam Skutt wrote: >> On Feb 16, 9:00 pm, Dan Stromberg wrote: >>> So yeah, whether you use perl or anything else invoked with #!, you're >>> pretty much better off with sudo, or a tiny C wrapper that's so simple >>> it's hard to get wrong. >> >> UNIX makes this almost impossible > > Please give a source that includes justifying detail. Then you have the > right to ask me if there are well understood ways around each of the issues > identified. I did, the sudo source; kindly read the whole paragraph before replying next time. Do you think they go through all the contortions regarding closing FDs, handling signals, etc. just for fun? Do you think what they go through is unnecessary? No, the correct and portable ways to do many of the necessary actions are hardly obvious. Almost impossible refers to the fact that tasks that ought to be relatively simple, like closing all file descriptors, aren't made trivial by any standard. Heck, even figuring out what file descriptors might need to be closed is non-trivial! There is no one convenient source that plainly documents everything you must do and why, beyond the source code of applications that already escalate privileges safely. In which case, you might as well just run one of them. Very few (any?) programs that run setuid (root) have existed without a security vulnerability in their history. On the continuum from trivial to impossible, writing a program that securely elevates privilege definitely sits much closer to impossible. > Actually, they can be. Here's one, as a very easy-to-find example: > > $ ls -al > cmd started 2011 Wed Feb 16 10:19:21 PM > total 8 > drwxr-xr-x 2 dstromberg dstromberg 4096 2011-02-16 22:19 . > drwxr-xr-x 61 dstromberg dstromberg 4096 2011-02-16 22:18 .. > -rwsr-xr-x 1 root dstromberg 0 2011-02-16 22:19 > correct-and-simple-setuid > benchbox-dstromberg:~/src/correct-and-simple-setuid i686-pc-linux-gnu 12006 > - above cmd done 2011 Wed Feb 16 10:19 PM > > $ ./correct-and-simple-setuid > cmd started 2011 Wed Feb 16 10:19:24 PM > benchbox-dstromberg:~/src/correct-and-simple-setuid i686-pc-linux-gnu 12006 > - above cmd done 2011 Wed Feb 16 10:19 PM > > $ echo $? > cmd started 2011 Wed Feb 16 10:19:26 PM > 0 > benchbox-dstromberg:~/src/correct-and-simple-setuid i686-pc-linux-gnu 12006 > - above cmd done 2011 Wed Feb 16 10:19 PM > > It's a correct, setuid replacement for /bin/true. It's 0 bytes long. Now, > I expect you to exploit it, since I claim that it's both tiny and simple. > IOW, it's a reductio ad absurdum. The only absurd part of this example is why I would do this in the first place. If this is the level of discourse you're going to choose to operate at, then I don't see any reason to waste my time any further with you. Counter-examples that do nothing meaningful are themselves meaningless. > > sudo is an example of something that _isn't_ closely matched with what it's > running (beyond a little text in a config file), not something that _is_. > You're engaging in a category error there. No, I'm not, the entire point is that any wrapper involving user input in any capacity (including on the part of the program it ends up running) must do most, if not all, of the things sudo does wrt actually running something with privilege. Trying to make assumptions on the part of your parent is folly. > A better example is apache suexec - the helper portion of which is 428 > physical SLOC, and it's more complex than most need to be because it keeps > many env vars and prunes out others. See http://labs.idefense.com/intelligence/vulnerabilities/display.php?id=511 (among others) You don't want to be Apache suexec, kids! It's also sounds, from GSO's description, rather useless here. >> Passing things through sudo(1) is really the only sensible route these >> days but even that can be fraught with peril. > Wait - I thought you just said that it was almost impossible unless the > setuid wrapper was well matched with the program in question. > > Oh, you did. The entire point of my statement was that such a thing is an impossibility: the wrapper cannot safely make assumptions about whatever calls it. Nor is it permissible to make assumptions about whatever the wrapper calls except in very trivial circumstances. The inherent contradiction is that we want (legitimately) for our wrapper to assume it will only be executed by our own code, to reduce the problems it has to worry about. Unfortunately, UNIX makes this impossible and makes assuming literally nothing the only safe choice. > Quantifier error: You've given an example of why ta
Re: Best way to gain root privileges
On Feb 16, 9:00 pm, Dan Stromberg wrote: > So yeah, whether you use perl or anything else invoked with #!, you're > pretty much better off with sudo, or a tiny C wrapper that's so simple > it's hard to get wrong. UNIX makes this almost impossible unless your wrapper is cooperative with whatever process invokes it, which is itself a security risk. I advise anyone seriously considering this route to take a long, hard look at just what contortions sudo goes through in order to achieve this safety. A correct suid program is neither tiny nor simple. Passing things through sudo(1) is really the only sensible route these days but even that can be fraught with peril. For something as simple as, 'Write to a normally restricted area' it's probably no more secure than an ACL (and potentially way less if you screw up the sudo configuration). > However, perl's taint feature would be useful > irrespective when writing privileged code; it removes some of the > skill required. I don't really think so. It doesn't help prevent, for example, someone redirecting stdout to /etc/shadow and then running your command. Besides, I'm not even remotely convinced that 'removing skill' is a good idea. It especially doesn't help you very much when the whole point of your script is just a wrapper to elevate privileges (execute another programs) or copy files about. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Best way to gain root privileges
On Feb 16, 8:40 pm, GSO wrote: > Apols for being a nuisance. I'm normally if anything a web programmer. > > It looks like there are set-id functions in the os module. Further I > don't actually need root privileges, just write access to a directory > that a user ordinarily does not have write access to (and preferably > not read). So give them that instead, preferably via ACL. Reliably denying read access may be difficult, however. Chances are pretty good that any solution you create won't be any more secure than this, though. > So a call to os.setegid(egid) with a group created for the > program's use alone would do this then. (Unless this is bad technique > security wise otherwise, as a uid 0 seteuid call would be considered; > but surely what I am thinking of doing is not a security risk.) Except in order to do this you need to be root, of course, or make the users members of that group anyway (in which case, just use the damn ACL). Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Missing SIGCHLD
On Feb 15, 1:28 pm, Dan Stromberg wrote: > *ix signals have historically been rather unreliable and troublesome > when used heavily. > > There are BSD signals, SysV signals, and POSIX signals - they all try > to solve the problems in different ways. No, there are just signals[1]. There are several different APIs for handling signals, depending on the situation, but they're all driving the same functionality underneath the covers. These days, only sigaction(2) is remotely usable (in C) for installing handlers and all the other APIs should normally be ignored. > You might also make sure your SIGCHLD signal handler is not just > waitpid'ing once per invocation, but rather doing a nonblocking > waitpid in a loop until no process is found, in case signals are lost > (especially if/when signals occur during signal handler processing). This is the most likely the issue. Multiple instances of the same pending signals are coalesced together automatically. It would also help to make sure the signal handler just sets a flag, within the application's main loop it should then respond to that flag appropriately. Running anything inside a signal handler is a recipe for disaster. Also, SIGCHLD handlers may not get reinstalled on some operating systems (even in Python), so the application code needs to reinstall it. If not done within the signal handler, this can caused signals to get "lost". That being said, I'd just spawn a thread and wait there and avoid SIGCHLD altogether. It's typically not worth the hassle. > Oh, also, signals in CPython will tend to cause system calls to return > without completing, and giving an EINTR in errno, and not all CPython > modules will understand what to do with that. :( Sadly, many > application programmers tend to ignore the EINTR possibility. This can be disabled by signal.siginterrupt(). Regardless, the signal handling facilities provided by Python are rather poor. Adam [1] Ok, I lied, there's regular signals and realtime signals, which have a few minor differences. -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing & more
On Feb 14, 5:33 am, Andrea Crotti wrote: > Well the other possibility that I had in mind was to spawn the very > long process in an asynchronous way, but then I still have the > problem to notify the rest of the program that the simulation is over. > > Is there a way to have an asynchronous program that also notifies when > it's over easily? > Several, again, what's best depends on what you're doing: * If you can modify the application, you can have it modify that shared Value before it exits. * On UNIX, you can do a non-blocking waitpid() call to determine when the process exited. The catch is that you can only do this from an actual parent process, so you'd have to reconstruct the way you spawn processes. Plus, if you need the stdout/stderr from the process you then must also consume the I/O in an non-blocking fashion, or dedicate a thread solely to consuming the I/O. It's possible to pull similar trickery in Windows, however I don't think the standard Python library makes it convenient (the right Win32 call is GetExitCodeProcess, given a legitimate Win32 process HANDLE). * You have it write something to stdout/stderr and key off of that. Those (or slight variants on them) are the common ways. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: multiprocessing & more
On Feb 13, 12:34 pm, Andrea Crotti wrote: > > First of all, does it make sense to use multiprocessing and a short > value as boolean to check if the simulation is over or not? > Maybe, but without knowing exactly what you're doing it's difficult to say if any other approach would be superior. Plus, most of the other approaches I can think of would require code modifications or platform- specific assumptions. > > As last thing to know when to start to analyze the data I thought about this > > while len(listdir(res_dir)) < len(network): > sleep(0.2) > > which in theory it should be correct, when there are enough files as > the number of nodes in the network everything should be written. BUT > once every 5 times I get an error, telling me one file doens't exists. > > That means that for listdir the file is already there but trying to > access to it gives error, how is that possible? File I/O is inherently a concurrent, unsynchronized activity. A directory listing can become stale at any time, even while the directory listing is being built (e.g., imagine running ls or dir in a directory where rm or del is currently executing). When a directory is being modified while you're listing it, the contents of the listing essentially become "undefined": you may get entries for files that no longer exist, and you may not get entries for that do exist. A directory listing may also return duplicate entries; this is what I expect is happening to you. The right thing to do is actually check to see if all the files you want exist, if you can. If not, you'll have to keep waiting until you've opened all the files you expect to open. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Shared memory python between two separate shell-launched processes
On Feb 12, 3:29 am, John Nagle wrote: > > If you're having trouble debugging a sequential program, > you do not want to add shared memory to the problem. No, you don't want to add additional concurrent threads of execution. But he's not doing that, he's just preloading stuff into RAM. It's not different from using memcached, or sqlite, or any other database for that matter. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Shared memory python between two separate shell-launched processes
On Feb 10, 9:30 am, "Charles Fox (Sheffield)" wrote: > > But when I look at posix_ipc and POSH it looks like you have to fork > the second process from the first one, rather than access the shared > memory though a key ID as in standard C unix shared memory. Am I > missing something? Are there any other ways to do this? > I don't see what would have given you that impression at all, at least with posix_ipc. It's a straight wrapper on the POSIX shared memory functions, which can be used across processes when used correctly. Even if for some reason that implementation lacks the right stuff, there's always SysV IPC. Open the same segment in both processes, use mmap, and go. What will be tricky, of course, is binding to wherever you stuffed the dictionary. I think ctypes.cast() can do what you need, but I've never done it before. Also, just FYI, there is no such thing as "standard C unix shared memory". There are at least three different relatively widely- supported techniques: SysV, (anonymous) mmap, and POSIX Realtime Shared Memory (which normally involves mmap). All three are standardized by the Open Group, and none of the three are implemented with perfect consistency across Unicies. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: WxPython versus Tkinter.
On Jan 23, 3:23 pm, rantingrick wrote: > psst: thats because they are FEATURES and NOT BUGS you idiot! Your application not displaying a directory listing, due to one of these "features", is desirable behavior? I think not. Your application hanging, due to one of these "features" is desirable behavior? I think not, nor do the wxWidgets, Qt, Swing, Cocoa, Gtk, Win32, et al. toolkit developers. > I am not trying to create a working file browser so you can steal my code. I have ethics, morals, and standards, all of which forbid me from stealing anything you write. > I am displaying one use case for a very useful widget. So useful that you've only found one toolkit that provides it OOB! > A ListCtrl is useful for many, many purposes. Just one of them happens > to be a file browser. The specific functionality the wxWidgets ListCtrl has over its alternatives in other toolkits is only really seen in file browsers. Your challenge explicitly relies on making use of that functionality and in no way, shape, or form illustrates the general utility of a wxListCtrl. When you amend the challenge, you can make the claim it's not about wxListCtrl's unique functionality. > > Qt, Swing, > > Gtk, and Win32 common controls[1] don't provide the same exact control > > either, should we view it as deficient? > Yes. Funny, given that the original purpose of wxWidgets was to provide a cross-platform MFC wrapper, I think you're in the minority of thought among those developers, to say nothing of developers in general. > I suppose you gauge the quality of code by how many tracebacks are > printed. And it seems even more perverse that you expect code to spit > exceptions! You're the one who asked for them originally, not I. I merely posited that since you provided code that prints no tracebacks and expected them (i.e., what you expect your code to do and what it does are two different things), that it is highly unlikely you're capable of providing me code that actually prints tracebacks, much less tracebacks that assist you in debugging. Put plainly, you have no clue whatsoever what your code actually does or how it accomplishes it, which is exactly what we'd expect. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: WxPython versus Tkinter.
On Jan 23, 2:07 pm, rantingrick wrote: > Well then post a traceback. None of these issues will supply a traceback; in fact, in some of these cases, you went out of your way to ensure that would not happen. That's an additional bug (or 5 additional bugs depending how mean your tester/QA staff are). > However you still miss the point. You will do anything to distract > from the point. And what IS that point? Well that Tkinter is > lackluster 20 years old rotware The fact it doesn't provide a control found almost exclusively in file browsers and open/save dialogs does not mean that it is "20 years old rotware". > and you need to resort to these BS tactics to discredit me because 1). You > cannot even create a Tkinter > GUI at the basic level, I certainly can, but there's no point in demonstrating my skill when you cannot create a wxWidgets GUI at a basic level. You made the challenge, you intentionally styled it as "rantingrick v. the world", so you need to actually bring something challenge worthy to the table first. That includes picking a challenge that's interesting and not over a relatively minor widget in the grand scheme of things. Qt, Swing, Gtk, and Win32 common controls[1] don't provide the same exact control either, should we view it as deficient? But regardless of the challenge, I don't think you're capable of posting a worthwhile example, which is why you dismiss all the problems found with it instead of actually fixing the code. > Post CODE Adam. Code! Surely you can handle copy/pasting a traceback i > hope! I certainly can. I don't have much hope for you writing code that provides tracebacks, much less actually useful ones. Adam [1] Hell, I'm not sure /any/ toolkit provides the whole of wxListCtrl functionality OOB, besides wxWidgets itself. Like I said, it's specific functionality isn't well suited. -- http://mail.python.org/mailman/listinfo/python-list
Re: WxPython versus Tkinter.
On Jan 23, 12:21 pm, rantingrick wrote: > The code does work. You just lack the skill to run it. I not only possess the skill to run it, but to find fault it in through simple inspection. All of the bugs I found, but one, I found through reading the .py file. Heck, I'm so good that I guessed two of the bugs before I downloaded the code because both are rank amateur mistakes, and I'm not convinced you even rise to the level of amateur. > > What are these "so-called" bugs exactly? 1. There's a bug related to loading of your resources. 2. There's a bug related to when file I/O is performed. 3/4. There's at least two bugs related to handling of a specific mouse event. 5. There's a bug related to reporting errors to the user. All of these bugs, except one[1], show a grave misunderstanding about how GUI toolkits operate and/or how the underlying operating systems behave. >Remember this code is not meant to WOW anyone. That's apparent from simple inspection of the code! Not only does it not wow, it doesn't even illustrate your own example to a degree that could be reasonably considered competent. It demonstrates you didn't even test the functionality you provided in your own code, or that if you did, you're entirely clueless about GUI design fundamentals. > The main point is to create a > ListCtrl that has two view modes, icons, and editable items. You lack > simple reading and comprehension skills Adam. And your code will not do that in a whole host of common situations that a GUI application is reasonably expected to handle. It is not sufficiently robust to be interesting. > Adam you were one of the biggest trolls in the "other" thread. We > don't need you trolling up this one too. Can you write any code? I > have produced code and no one has offered a rebuttal using the Tkinter > module. This is because only a handful of the entire community has the > skills to create something like this with Tkinter. No, it's because your code is complete and utter shite and there's zero point in attempting to replicate it. Your code does not work, even if you think it does. I'm not the only person who's noted this. > HOWEVER YOU ARE NOT IN THIS GROUP ADAM. You are a troll, and that is > all you can do. Prove me wrong if you can... I already have. Confirmation of some of the bugs I've noted exists in this very thread. Thus, I'm quite capable of reading and comprehending Python code. The same remains to be seen with you. Adam [1] Which is only because wxWidgets has a bug in this regard. That being said, a workaround exists and trivial to find online. -- http://mail.python.org/mailman/listinfo/python-list
Re: WxPython versus Tkinter.
On Jan 22, 7:07 pm, rantingrick wrote: > So PUT UP OR SHUT THE HELL UP! You first. Write actual working code first and then you can challenge people. I found 5 bugs in your code before I quit looking[1]. Can you find and fix them all? Also, I'm entirely ignoring the bad styling, bad default sizing, horrible UI, the fact you call rename "Edit", and the fact the context menu doesn't do anything. All of the are legitimate technical errors, i.e., you coded the program wrong. In the spirit of community I open the bug finding to everyone, but humbly ask you don't tell Mr. "rantingrick" Johnson about them. It can be our little secret ;) Adam [1] I think there might even be more, but I got lazy. -- http://mail.python.org/mailman/listinfo/python-list
Re: examples of realistic multiprocessing usage?
On Jan 20, 11:51 pm, Albert van der Horst wrote: > This is what some people want you to believe. Arm twisting by > GPL-ers when you borrow their ideas? That is really unheard of. Doesn't matter, you're still legally liable if your work is found to be derivative and lacking a fair use defense. It's not borrowing "ideas" that's problematic, it's proving that's all you did. For those of us with legal departments, we have no choice: if they don't believe we can prove our case, we're not using the code, period. The risk simply isn't worth it. > GPL-ers are not keen on getting the most monetary award by > setting lawyers on you and go to court only reluctantly to > enforce the license. And? Monetary award is hardly the only issue. > Stealing code means just that, verbatim copies. When you read this > carefully, you can see that reimplementing the stolen code is > an option. Exactly what you say is legally impossible. No, in the United States it means anything that constitutes a derivative work, since derivative works of GPL-licensed works must be released under the GPL. Merely copying ideas does not make one a derivative work, but one also must be prepared to show that's all that happened. As such, it would have to be a substantially different implementation, generally with some sort of added or useful value. Proving that can be difficult and may very well depend on what court you land in. > > So pardon me, but not even looking at code you might learn from > is pretty hysteric. Not at all. Separating ideas from implementation can be difficult, and convincing a judge of that vastly more so. It's a legitimate concern, and people who intend to ship proprietary software should definitely resort to GPL-licensed software last when looking for inspiration. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 20, 3:02 pm, "Octavian Rasnita" wrote: > From: "Adam Skutt" > Yet, for some unfathomable reason, you > keep promoting > I would be glad if you could tell me about a portable solution which is > accessible with JAWS and Window Eyes, the most used screen readers under > Windows (real glad). I did, Qt. I'm not yournanny and I'm not going to go test it for you. There are bugs in the Qt database relating to JAWS functionality, so it others have plainly gotten it working to some degree. But honestly, why should I waste my time replying to you when you're too damn lazy to even use Google? I certainly won't be doing so in the future. "Lead a ignorant, thirsty horse to water, watch it die of thirst" and all that. > Exactly. When you will read that thing on a GUI creator site it means that > that GUI is not accessible because the screen readers don't support it yet. > The GUI libs manufacturers will say that the screen readers are bad because > they don't offer support for their GUI, although that GUI offers > accessibility features. No, that's not what that page says or means. But I'm not going to be able to explain it to you any more clearly, so if you'd prefer to keep living in your own delusional, illydic world, that's your decision. Qt has been pretty clear in painting the blame on who it belongs on, which is not the screen reader vendors. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 20, 1:17 pm, Emile van Sebille wrote: > > The problem with QT is the license. Qt and wxWidgets are both LGPL and equally non-starters due to license, today. There was a hypothetical assumption on my part that the library would be made license compatible somehow, through the entire discussion. I didn't think it worth mentioning because I didn't think it all that relevant to any of the discussions that end ed up occurring here. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 20, 10:44 am, "Octavian Rasnita" wrote: > From: "Adam Skutt" > Actually, JAWS uses MSAA dead last, as I understand it, because the > API is truly that awful. But MSAA is necessary whenever you're not > using a Win32 common control or most of the other stuff developed by > MS. That means essentially every non-MS toolkit that's been > discussed. > > Yes, but WxPython uses wxWIDGETS and wxWIDGETS use the standard Win32 > controls under Windows, so they are accessible. > And they use Gtk under Linux and Gtk is accessible under Linux also. wxGTK is not, though. wxWidgets is only accessible under Windows. Qt is accessible under Windwos, OS X, and anything that supports AT- SPI[1]. Yet, for some unfathomable reason, you keep promoting wxWidgets even though it is plainly the inferior solution. > Yes of course they can do that if they use the wrong tool. They can't do the > right thing if they believe that Tk or Silverlight or Flash is accessible no > matter if they follow the recommendations for accessibility, because the > screen readers don't support those interfaces. No, even using the right tools: http://doc.qt.nokia.com/qq/qq24-accessibility.html#dealingwithquirks (a few years old, but AFAIK still relevant). It's a crap shoot even if I use tools that support accessibility, because the APIs aren't very good. Some of the new APIs improve the problem, but some reading suggests there will still be the need for a lot of special case handling on both sides of the table. > Nope, the cultural problem exists because the programmers use the wrong > interface, not because they just don't make the efforts for making that > interface more accessible. No, they don't think about it, they don't test, they don't make the necessary calls to enable accessible applications. Plenty of applications with poor to non-existent accessibility support are written with toolkits that support accessibility. > To be more clear and not to include in the discussion many interfaces, we are > comparing Tkinter and WxPython. No, we're not comparing just them, because they're hardly the only solutions. > I am not excluding anything. I said that all the Tk-based programs are > inaccessible no matter what the programmer does to make it accessible, > because the screen readers can't work with that interface. > And I have also said which are the accessible interfaces: wxWIDGETS (which > use Win32 GUI under Windows and Gtk under Linux), SWT (which I think it does > exactly the same thing as wxWIDGETS), WindowsForms (DotNet) and SWING (if the > user installs Java Access Bridge). This is not a complete list, or even a correct list, which was my point. > It depends if there is a JAWS script defined. Stop. You've insisted many times that I need to not do anything special for making accessible applications. Yet, JAWS has an API entirely for the purpose of enhancing accessibility for specific applications! So, which is it? If I don't need to do anything special, then why does the API exist? Oh right, it exists so people can compensate for developers shortcomings. It's very hard to take anything you say seriously when you repeatedly get basic facts wrong and when you make claims that aren't consistent with the facts. > If the user doesn't want or don't know how to create that script for > improving the usability, he/she might need to learn how to use the > application by trial and error, but what I want to show is that the user is > *able* to use that application, even if the interface is not very friendly, > but it would be absolutely impossible to use an application created with > Tkinter. He / she might be able to use the application. It's a "maybe", not a "for sure", yet you consistently imply it's the latter. > I am not talking about custom controls. Those things are the worst thing > possible from the perspective of accessibility, because usually nobody cares > about providing accessibility features. Yet I was talking about custom controls, and plenty of applications use custom controls, so you cannot ignore them even if you'd wish to do so. > My point was that Tkinter which uses Tk is bad, and I didn't tried too many > QT-based applications. No, you explicitly stated that Qt has zero accessibility support, which is simply false. You didn't say, "The ones I tried haven't worked", which is certainly possible depending on version and how Qt was compiled (since Qt is normally shipped with the application on Windows). You said, "But the interfaces created with Tk, Gtk and QT are completely inaccessible." > And I don't see what you demonstrate when you wanted to show Tkinter it is > comparing with WxPython... I never said I was going to demonstrate anything between Tkinter and wxPython. Such a demonstrate is in your imagination. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 20, 8:19 am, "Octavian Rasnita" wrote: > The Windows applications are not accessible only if they use MSAA. The > screen readers know how to get the necessary information for making the > interfaces accessible in other ways, but of course, it is harder for the > screen reader manufacturers if the GUIs don't respect a standard. Actually, JAWS uses MSAA dead last, as I understand it, because the API is truly that awful. But MSAA is necessary whenever you're not using a Win32 common control or most of the other stuff developed by MS. That means essentially every non-MS toolkit that's been discussed. > And if MSAA is hated it doesn't mean that the accessibility is so poor as > you might think. No, but it does mean precisely what I said: application developers can do the "right" thing and get the wrong result. > Yes, that's a problem but we divagate. If you believe that is a problem then you cannot say, "Accessibility does not require special effort" without contradicting yourself. If it did not require effort over and above the norm, then the cultural problem would not exist. > Yes of course, Adobe's products are at most very hard accessible or not > accessible at all and I already said that Gtk is not accessible under > Windows, just like Tk. Which means you're excluding a large swath of applications in your analysis. > Well, I haven't used WxPython until now, but only WxPerl, however the WxPerl > programmers don't need to set the tab focus because by default the tab focus > is set in the order of the controls on the form and I thought it should be > the same in WxPython. It's set in the order you create the controls (with some exceptions), which may or may not be the correct tab focus order. Moreover, the same FAQ I linked tells you that you need to set special styles to support proper tab order when tabbed widgets and certain other controls are involved, to say nothing of fall backs for certain other controls that aren't provided automatically. > Yes, that's true, but as I said, some GUIS like WxPython offer the > accessibility, so those controls are visible, and it is less important if > they are in a right tab order or if some of the controls are not labeled, > because the screen reader can solve these things. You've said yourself that it often requires trial and error on the part of the user, so which is it? > A GUI which offers accessible widgets only if they are coded specially for > beeing accessible are not accessible at all, That's the case with every GUI API on the planet: if you write a custom widget you must provide accessibility support for it. This is nothing new. > Have you created that QT application with its default options? Without doing > anything special to make it accessible? Yes, you don't have to do a thing, unless you're custom building Qt yourself. On Linux, you have to set QT_ACCESSIBILITY = 1, but Linux is the exception. > And most important, have you tested it with JAWS screen reader? (Or > WindowEyes at least) because these are the most used screen readers. > NVDA is pretty poor and it can't be used for very complex things (yet). > No, and I have zero interest in doing so. Point was merely to demonstrate your statement was false. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 20, 2:22 am, "Octavian Rasnita" wrote: > Where do you got the conclusion that the accessibility is so poor? By an examination of the facts. MSAA is universally hated and all screen readers actively have to work around it and its many limitations (meaning that as an application programmer, you can do the "right" thing and still get undesired behavior through no fault of your own). I have no clue if it's replacement is better, but I do know few programs use it, so it's little comfort. The situation on Linux is still an outright disaster. Neither Qt4 nor GTK enable accessibility functionality automatically and support from the various distribution vendors is pretty limited. It is improving, however. Accessibility still requires special effort "over and above" the norm by the programmer whether you believe this to be the case or not. Most programmers aren't even aware of the fact they must consider it (a point you've made yourself before!) and hence commit various accessibility sins. This is mostly a cultural problem, not a technical one, but it's still a real problem. Plus, out of the stuff I run daily on Windows: two of the applications are entirely inaccessible: the Adobe AIR one and the Gtk one. Songbird, the XUL based music player, has some accessibility, but none of the navigation controls seem to be enabled, which would make it very difficult to actually use, I imagine. It's also not particularly keyboard friendly: you don't even get a visual focus indicator most of the time, even though the focus is moving. > But You keep telling me that it is hard to create accessible programs, which > is false, but you don't know and don't want to try to see. > Try to create a WxPython app that doesn't do anything but only displays some > menus, buttons, combo boxes, list boxes, list views, tree views, text > fields, check boxes, radio buttons... and you will see that they are very > accessible with that screen reader I told you about, without requiring you > to do anything special that you wouldn't do otherwise. Applications that only display these things aren't very interesting. And no, they still require additional effort. Most developers can't even be assed to get a sane tab focus order set, to say nothing of keyboard shortcuts, accessibility descriptions / proper label associations for buttons and other controls. It gets even more complicated when you get away from controls only meant to display text. How many web pages have you visited where images are missing alt tags? Do you really thing those same people are going to bother trying to do the right thing if asked to create a desktop GUI application? No, they're not, and you're going to get whatever happens automatically. Which isn't nothing (in some toolkits), but it's also not what one really wants, either. > > How do you came to the conclusion that QT is accessible on all operating > system? By actually going out, reading, and even testing? Qt has had functional accessibility support on OS X and Windows since Qt3, and functional support on Linux, OS X, and Windows since Qt4. It even extends to PySide, with a few limitations: you can't write custom accessible widgets in PySide, though I'm pretty confident they'd enable support if someone wrote a patch to enable the bindings. > When you talk about accessibility, try it yourself and don't take the word > of the GUI widgets developers. > I just did, NVDA was able to read my button text and its accessibility description just fine, both in C++ and Python versions of my Qt application. I didn't bother testing Linux or OS X, but I'm sure it would work there too. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 19, 4:04 pm, "Octavian Rasnita" wrote: > Those rules for creating an accessible application are obvious; like the fact > that a button need to contain a text label and not only an image, or that an > image needs to have a tooltip defined, or that a radio button needs to have a > label attached to it, but all those things can be solved by the programmer > and usually the programmer create those text labels. > The fact that /every/ toolkit provides accessibility guidelines over and above whatever other UI guidelines they provide tells me that creating an accessible application is hardly obvious. Plus, if it were really that simple, the accessibility situation wouldn't be so poor. > Yes, those things should be followed for creating a better app, but what I > wanted to say is that no matter if you do those things or not in a Tk, Gtk or > QT GUI, they will be useless, because the screen readers can't understand > those GUIS even they have text labels, and even if you will see a focus > rectangle around buttons. They don't report that those objects have the focus > so the screen readers won't speak anything. Your "something is better than nothing" argument isn't particularly compelling to me personally as a justification for ripping out TkInter. And Qt is the only toolkit with some level of functioning accessibility support on all three major platforms, assuming the library and software are built correctly, so again, your argument is really for Qt, not for wxWidgets. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 19, 11:09 am, "Octavian Rasnita" wrote: > From: "Adam Skutt" > > Accessibility always requires special effort, and I don't see how > > changing toolkits gets away from this. > > This is the most false thing I ever heard and the most dangerous. O RLY? http://www.wxwidgets.org/docs/technote/wxaccesstips.htm sure looks like there's a whole host of things that I, the application programmer, must do manually to enable an accessible application[1]. I can't just plop down controls and have an accessible application. > The programmer doesn't even know that the application will also offer > accessibility features. No, accessibility requires consideration in the design and implementation of the GUIs, in all of those toolkits. It is not transparent, nor can it be transparent. It requires both consideration when laying out the widgets, but also ensuring that the widgets have specific properties set. How many applications have you ever used that had focus order bugs? That's an accessibility issue that requires programmer intervention to get right. Adam [1] That list is not comprehensive by a long shot. -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 19, 10:41 am, "Octavian Rasnita" wrote: > > Not true. WxPython uses wxWIDGETS which uses the default OS widgets which > usually offer the accessibility features. > (At least under Windows, but most users that need accessibility use Windows > anyway). > Right, under Windows, which is a problem. By your line of reasoning, you really should be supporting PySide / PyQt and not wxWidgets, since they at least play at cross-platform support. > > The alternative is to fix the accessibility support issues in the > > underlying library, and Tk is no less amenable to that than > > wxWidgets. If that's what you want, start coding then. You have a > > long climb ahead of you. > > The underlying libraries for Windows offer the necessary accessibility and > WxPython uses it, but Tkinter uses Tk not those libraries. wxWidgets' support is completely inadequate for a true cross-platform system, the developers are aware of this and acknowledge this and believe a rewrite is necessary. Thus, it is currently really in no better state than Tk. > > Or we have cross-platform support as a requirement and no desire to > > develop the GUI interface three times over. Hence the continued > > popularity of Java GUIs. > > Java GUIS are accessible. Maybe that's the reason. No, the reason is as I stated, no more, no less. Accessibility doesn't enter into most designs. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: UTF-8 question from Dive into Python 3
On Jan 19, 9:00 am, Tim Harig wrote: > > So, you can always assume a big-endian and things will work out correctly > while you cannot always make the same assumption as little endian > without potential issues. The same holds true for any byte stream data. You need to spend some serious time programming a serial port or other byte/bit-stream oriented interface, and then you'll realize the folly of your statement. > That is why I say that byte streams are essentially big endian. It is > all a matter of how you look at it. It is nothing of the sort. Some byte streams are in fact, little endian: when the bytes are combined into larger objects, the least- significant byte in the object comes first. A lot of industrial/ embedded stuff has byte streams with LSB leading in the sequence, CAN comes to mind as an example. The only way to know is for the standard describing the stream to tell you what to do. > > I prefer to look at all data as endian even if it doesn't create > endian issues because it forces me to consider any endian issues that > might arise. If none do, I haven't really lost anything. > If you simply assume that any byte sequence cannot have endian issues you > ignore the > possibility that such issues might not arise. No, you must assume nothing unless you're told how to combine the bytes within a sequence into a larger element. Plus, not all byte streams support such operations! Some byte streams really are just a sequence of bytes and the bytes within the stream cannot be meaningfully combined into larger data types. If I give you a series of 8-bit (so 1 byte) samples from an analog-to-digital converter, tell me how to combine them into a 16, 32, or 64-bit integer. You cannot do it without altering the meaning of the samples; it is a completely non-nonsensical operation. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 19, 1:37 am, "Octavian Rasnita" wrote: > > Most of the programmers don't know that and they don't even need to know > that, but if a lib that create accessible GUIS would be promoted, most of > the Python programmers would use that lib and would create good apps by > default, without even knowing this. Have you written an accessible application in any toolkit whatsoever? It is not magic, and it does not happen by default, even when solely using the standard widgets. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 19, 1:37 am, "Octavian Rasnita" wrote: > > The users shouldn't want something better. Everyone should create GUIs which > are accessible for everyone by default, without making any special effort > for doing this. Accessibility always requires special effort, and I don't see how changing toolkits gets away from this. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 3:23 pm, "Octavian Rasnita" wrote: > From: "Alexander Kapps" > > > Tkinter causes damage? Very bad damage? What are you talking about? > > I am talking about the fact that Python promotes Tkinter, and many beginners > will start using it, and they will start creating applications with it, and > they will learn to use it better than WxPython, and they will start believing > that Tkinter is better because it is easier to use than WxPython, so they > will start convincing others that Tkinter is the best, and they will start > finding many reasons that show that Tkinter is better. And after this, they > will say that they don't care about the real problems generated by GUIs like > Tk. > And a very big problem is that the applications that use Tk/GTK are not > accessible for screen readers, so those applications will be just blank for > people with visual impairments which need to use a screen reader. If you want functional accessibility support, you're going to have to write it in Python yourself, and get the accessibility manufacturers to support it. All of the cross-platform toolkits have poor to non- existent accessibility support. Accessibility issues aren't going to be fixed by going to a different GUI toolkit in the standard library. The alternative is to fix the accessibility support issues in the underlying library, and Tk is no less amenable to that than wxWidgets. If that's what you want, start coding then. You have a long climb ahead of you. Accessibility will never be a particular good argument for switching toolkits. All of the cross-platform offerings with Python bindings are far too broken to be even remotely interesting. > Why do we like the portable GUIS and don't really like the native interfaces > that don't work on other platforms? > Because we want our programs to be usable by as many people as possible. > Well, some platforms render the output as sound and Tkinter are not > "portable" on those platforms (screen readers). Or we have cross-platform support as a requirement and no desire to develop the GUI interface three times over. Hence the continued popularity of Java GUIs. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 9:27 pm, Corey Richardson wrote: > > Why would you add in only a part of wxPython, instead of all of it? Is > the work to cut it down really an advantage over the size of the full > toolkit? From what I just checked, the source tarball is 40MB. Can that > much really be added to the Python stdlib? What other alternatives are > there, besides wxPython, that are perhaps a bit smaller. > The source tarball from the wxPython.org website contains a full version of wxWidgets in addition to the actual wxPython functionality. A python distribution would certainly contain solely the latter and require the end user to already have wxWidgets installed in a suitable fashion. The actual full wxPython binding is ~100 MiB uncompressed, ~15 MiB compressed BZIP2, but it also includes a lot of stuff that could possibly be removed and/or reduced, like full documentation, examples, etc. It can be shrunk even further by taking a dependency on swig and regenerating the bindings at compile time (they're shipped prebuilt). At which point, it's pretty damn small. Not as small as all of the Tk functionality, I think, but well under 10MiB compressed. The problem to me isn't the size (though some might find it objectionable), but the system dependencies you have to take: wxWidgets requires GTK+ on UNIX, which requires a whole mess of crap in term, plus swig, plus whatever else I may or may not be missing. I'm also not 100% certain as to whether it's as portable as Tk is today. At any rate, if the size is an issue, culling widgets is a lot of an effort for not much of a gain, especially when you look at the bigger picture of, "Every file I have to download to build python from scratch" Minimizing what's in the Python distribution does not change the size of the dependency set one bit, and that dwarfs the python code in any case. That is what you want to avoid in my opinion. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 8:59 pm, Corey Richardson wrote: > > > I completely agree! And we should expect it to be even better! > > What out there is there that meets those requirements? Nothing, and I somewhat doubt there ever will be. Tk is somewhat of an anomaly at this point. Most of the trend in GUI toolkits is to become more HTML/JS/CSS like in nature, which isn't something I personally agree with. I certainly don't think it makes life any easier for the programmer, especially starting out. It might make certain types of applications (e.g., CRUD) easier, but that's not a laudable goal in the least. > Mobile and web certainly have their place, but it Python the place for > it? Sure, Python can be used as the back-end of web sites, but not up > front like Java or Flash (aside from Jython). Especially mobile. Python > was not intended for a mobile platform not should it be made to fit that > niche. Python has its place, but your cell phone is not it. I don't think that's true per se, but I don't think it's relevant. A single GUI toolkit for traditional computers, for web, and mobile is a difficult task, one that no one has accomplished. MS has gotten closest, and I'd hesitate to call Silverlight a success. Plus, Silverlight is plugin-based rich applications (ala Flash), not HTML/ CSS/Javascript which is what most people mean/want for the web. Adding HTML/CSS/Javascript to the mix takes the problem from bad to awful, in my opinion. I'm sure the various Pyjamas users might take issue with that, but what works best inside a web browser (with its various enforced limitations) isn't what works best inside a client- side application (be it web delivered or not). > As a web interface are you thinking something like Java's Swing or > something like Django? You pretty clearly need both. There are times when web pages are what I want, there are times when they are inadequate and I need more functionality. You certainly don't want either Swing or Django, though. > > Given the above, what do you guys (python-list, not just rantingrick) > think fills the spot the best? > Nothing, but I'm not even convinced the spot needs to be filled. Certainly no one's made an actual case for why, much less how. Regardless, if it's really what you want, you'd have to write it yourself. Personally, while Silverlight has some interesting ideas, I'd recommend not using it as a base, especially if Python is your target language. > Would these items inclusion in the stdlib entail unnecessary cruft added > on to the size of the stdlib, are they completely cross-platform (as far > as Python itself is)? > Depends on exactly what you do, but you'd likely end up no bigger than .NET or Java. I'll leave it to others to decide whether that's a good or bad thing. > Let's try not to get off track like this thing has been since it was > started. That was and rantingrick's goal from the get go and still is his goal. Otherwise, he wouldn't have changed his position three times now and be overdue for a fourth. Otherwise, he would have answered my / your question about why bother putting a minimized version of wxWidgets in the standard library by now as opposed to the whole damn thing. He dodges technical questions because he lacks even the most elementary understanding. He'll do the same to you and only offer absurd platitudes and insults in return, as opposed to actual working technical solutions. Hell, he can't even offer up a consistent problem to solve, and he's honestly over do for changing it altogether.. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 6:36 pm, rantingrick wrote: On Jan 18, 4:57 pm, Adam Skutt wrote: > On Jan 18, 4:45 pm, Arndt Roger Schneider > > > E.g. a button is: > > > A function gets asychnronously performed in response to > > > a finger/mouse click and release inside a certain screen area. > > No, that is not the definition of a 'button', not even when we choose > > to ignore how it is rendered, which you cannot ignore even if you wish > > to pretend you can. Otherwise, I could always treat hyperlinks and > > buttons as equivalent and even interchangeable. Unfortunately, they > > are no such things. > What! YES it is Adam! And you just exposed yourself as an > argumentative moron! > > A hyperlink and a button are EXACTLY the same thing "functionality" > wise. :active, :visited:, :hover, and the concept of "onhover" all would like to have a word with you. They have different presentation which yields to different functionality: if it's important to tell a user "Hey, you've been there before", then a button is entirely unsuitable, while a hyperlink is designed precisely for that situation. Hyperlinks change their presentation to indicate mouse focus (nevermind the mouse cursor itself normally), buttons don't necessarily do either[1]. Hyperlinks can certainly decay to become button-like, but buttons cannot do everything hyperlinks can do. Checkboxes and radio buttons are a much better rebuttal, as they usually present "almost" the same API and expect the same model on part of the application programmer. It's the "almost" that kills us: radio buttons only behave in the desired way when part of a button group, forcing us to be cognizant of the fact we're creating radio buttons (since we must create button groups as well). Checkboxes support tri-state functionality, and sometimes radiobuttons do as well. Pushbuttons do no such thing[2]. It'd be nice to be able to support the abstract notion of a "button" and get what I wanted, but that requires guessing at intent and computers are notoriously lousy at that. > The fact that they look different has nothing to do with the > argument. Actually it does, but they not only look different, they /behave/ differently. Both in terms of how a user interacts with them, and in terms of how we interact with them in code. > More argumentative crap. Adam you are incapable of compromise or > reason... or maybe both. Try more facts next time. I'm not the one suggesting that the only difference between a hyperlink and a button is how they are rendered, FFS man, have you ever even used a modern GUI? Are you posting from tin? Adam [1] Even when they do, it's not intended to be controlled by the application. Native button widgets have no real built-in support for user-controlled styling on mouse focus, you'd have to do the drawing yourself (at which point you might as well write a custom widget). [2] I'm ignoring the UI problems with radio buttons and checkboxes, of course. Point is, even among things where the differences are subtle, the differences have inescapable ramifications on the code I write. -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 4:45 pm, Arndt Roger Schneider wrote: > Adam Skutt schrieb: > > Until we have pixel-perfect touch sensors, toolkits for devices with > > pointer interfaces (e.g., PCs) and toolkits for devices with touch > > interfaces (e.g., phones and tablets) will necessarily be different. > > > You note this yourself: the UI paradigms that work well when you have > > a pixel-perfect pointer do not work at all when you have a touch > > screen, especially on a limited size and resolution display. > > Yes I did and that's how it is. And then you go and advocate a single toolkit! Do you not see the inherent contradiction there? While it's certainly not impossible, the case is hardly obvious for one GUI toolkit for all possible UIs. You certainly have not presented it, and rantingrick never will. > Think about all the programmers earning their butter and bread :-). > Forget toolkits and widgets for awhile. > What remains are specific types of human/computer interactions, > a visual representation on a screen and a predefined behaviour > for said human action. Also known as "toolkits and widgets". Talking about such things are inescapable. > > E.g. a button is: > A function gets asychnronously performed in response to > a finger/mouse click and release inside a certain screen area. > No, that is not the definition of a 'button', not even when we choose to ignore how it is rendered, which you cannot ignore even if you wish to pretend you can. Otherwise, I could always treat hyperlinks and buttons as equivalent and even interchangeable. Unfortunately, they are no such things. > --A widget is essentially a logical abstraction. No, as much as we try we cannot divorce presentation from behavior fully. > > Because it's not cross-platform, I'd imagine. The entire point of > > wxWidgets was to provide a cross-platform "OOP" UI toolkit. It > > closely copies MFC since MFC and XView were the two "backends" it > > supported. > > MacApp is/was cross-platform, Apple pulled the plug > on the non-mac platforms; the industrie > consortium took charge of the other platforms. > MacApp didn't even see the start of cross-platform development until 1996, four years after wxWidgets. It was not cross-platform from the start and only became cross-platform when all of Apple's other cross- platform endeavours failed. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 3:20 pm, Arndt Roger Schneider wrote: > There has been no advancement in GUI-Design. Today it looks and > behaves just the way Bill Atkinson designed it. That doesn't even begin to equate to a lack of advancement. It's also not true in the least. > Technical revolutions are made by disruptive thoughts, > which are never collective. Revolutions are statistical anomalies, so it's best not to depend on them for progress. It's not even apparent what revolution is necessary here. > ...The problem with gui-design:It requires an graphical artist, > a well versed writer, a software architect and a programmer. > The first two job description are the important ones. > > ...No OS-vender is going to allow that, it equals > lost control. > You need to go look at the people Apple, MS, Google, et al. hire; this statement is just patently false too. Plus, after a certainly level of functionality, the OS vendor does become rather irrelevant. Otherwise, the web would have never taken off an application platform: the HTML standard provides the smallest widget set of just about anything discussed, though it's painting / layout capabilities are both simultaneously far advanced and far worse. Yet, it is likely the way of the future for a large portion of us, like it or not. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 2:11 pm, rantingrick wrote: > Adam now you are making sense. Everything you said here is true. > This > is why we must push for the OpenGUI standard. Funny, I write considerable detail about why such a thing is a pipedream and useless even if it came to fruition, and you somehow believe I'm in support of such an absurd idea. If you believe what I said is true, then you cannot seriously support any sort of "OpenGUI" standard, and I advise you to google that term before you use it again. Tell me, are you a political science major? Heck, to be totally honest, I've never been 100% convinced that cross- platform GUI APIs were even such a good idea. I certainly use them, but only in situations that are very simple or where I'm OK with accepting the fact my application will not be truly a first-class application[1][2]. Even minor differences in presentation can have large ramifications on how applications should function and therefore be written. > The entropy in GUIs has > exploded exponentially and rendered them all useless. Only if you have no clue what you're talking about whatsoever. You perceive them as useless because you're apparently incapable of understanding the simplest GUI precepts, nevermind APIs, which is why you've gone from Pure Python GUI to wxWidgets to this OpenGUI bullshit you're now espousing. Desperately clinging to a position doesn't make you look intelligent. Plus, I'm not sure what entropy you're talking about, but I'm not seeing it. MS continues to innovate, Apple continues to innovate, some portions of the Linux community do innovative things. Though most people just want to put something together and call it a day, and the functionality provided by a lot of toolkits is beyond adequate for that. Adam [1] Or solely on Linux where all of the "native" toolkits have cross- platform support. [2] To say nothing about the explosion of web "applications" in the world today... -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 8:09 am, Arndt Roger Schneider wrote: > Back to rantingrick 21st century toolkit/framwork: > Let's have a look at the numbers: > Worlwide pc market are 300 Million pcs per year, > this number includes desktops(2/3) and servers(1/3). > Your gui app is not relevant on servers. You should tell this "fact" to just about every major enterprise software manufacturer out there. They all ship GUI tools intended to be used on the server. Some of them ship only GUI tools or CLI tools that are worthless, making you use the GUI tools. > The desktop pc market is in decline; there is > however a shift toward pc-servers, instead. > It is anybodies guess how far the pc-desktop decline will go. > Every 21st century toolkit or framework must run on > mobile platforms! Until we have pixel-perfect touch sensors, toolkits for devices with pointer interfaces (e.g., PCs) and toolkits for devices with touch interfaces (e.g., phones and tablets) will necessarily be different. You note this yourself: the UI paradigms that work well when you have a pixel-perfect pointer do not work at all when you have a touch screen, especially on a limited size and resolution display. Even if you're provided a "single" toolkit, you still end up with two, maybe three, different applications, each using different widgets targeted for the device they run on. And no one provides a "single" toolkit: while Silverlight can run on the desktop, the web, and now on Windows Phone, you can't use the same widgets everywhere; ditto with Cocoa for OS X and Cocoa Touch for iTouch devices. While some further unification is obviously possible, it's rather doubtful we'll ever have unified widgets that are truly workable on the web, on the "desktop", and on a portable touch screen device. > > wxWidgets was written ~1992, it is a copy of > mfc, which in turn is a copy of MacApp. MacApp > is also OSS, maintained through an industrie consortium. > Why do you not use the original framework? > Because it's not cross-platform, I'd imagine. The entire point of wxWidgets was to provide a cross-platform "OOP" UI toolkit. It closely copies MFC since MFC and XView were the two "backends" it supported. > Screen resolution: > The time of 72ppi CRT monitors is over. A GUI > framework/toolkit must be resolution independent, > including all icons and indicators; > it should use decluttering (newspeak:ZUI). > WPF is the only functional resolution-independent UI toolkit in existence. While I don't disagree with you in principal, practice is pretty heavily divorced from principal here. Principal doesn't help me write GUI applications today. > wxWidgets is not suitable for a modern type > GUI ad thus clearly not the toolkit/framework > of the 21st century. None of the toolkits accessible from CPython are suitable for a 21st century guy by your standard. If we talk about IronPython, Silverlight becomes the closest, but it isn't a panacea by any stretch of the imagination. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 18, 3:49 am, "Octavian Rasnita" wrote: > From: "Adam Skutt" > Subject: Re: Tkinter: The good, the bad, and the ugly! > > On Jan 17, 3:08 pm, "Octavian Rasnita" wrote: > > "Batteries included"? > > > Python doesn't follow this policy at all. We can say that maybe PHP > > follows it, but not Python. > > http://lmgtfy.com/?q=python+%22batteries+included%22&l=1 > > Well, this doesn't mean anything because Perl can also say that includes > batteries, and PHP can say it also. It means that Python intentionally endeavorer to include some sort of rich and useful standard library as part of the language, ergo criticisms of the language may include criticisms of its library, like it or not. I'm not sure what makes you believe Perl or PHP are relevant here. > You are perfectly right. Then why favor Tkinter and not WxPython. Why not > strip the Python package and offer WxPython and Tkinter separately? As of now? There's no reason to remove what's there already as long as carrying it as a dependency isn't creating problems for building and distributing Python. But I already mentioned an important reason earlier: the dependencies of Tk are much, much smaller than the dependencies of WxWidgets (though eventually this may change with the X11 port getting complete). Another is that the scope and scale of WxWidgets is problematic: since it's a cross-platform C++ solution it provides a lot of things that it needs, but that Pyhton doesn't especially need or want. It creates incompatibility issues because you end up with a bunch of libraries that become useless when writing a GUI. This problem is minimized with Tk when compared to WxWidgets, and most other toolkits (certainly all the ones I've ever used). Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 17, 3:08 pm, "Octavian Rasnita" wrote: >> From: "Adam Skutt" >> And we're not discussing those languages, we're discussing Python, >> which has an explicit policy of "batteries included". As such, >> criticism of the standard library is perfectly acceptable under the >> name "Python", whether you like it or not. Besides, it's inevitable >> anyway. > > "Batteries included"? > > Python doesn't follow this policy at all. We can say that maybe PHP follows > it, but not Python. > http://lmgtfy.com/?q=python+%22batteries+included%22&l=1 > And if this is the wanted policy, why should it be "cheap batteries included" > and not "strong batteries included"? You'd have this same argument regardless of GUI toolkit you end up picking, or even if you choose to not include one at all. This would be because none of them are universal solutions, all of them have deficiencies (actual or perceived) that make them unsuitable for certain applications. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 17, 11:01 am, Steven D'Aprano wrote: > > I'm afraid that's precisely what I'm arguing you *can't* do -- there's > nothing reasonable about equating the standard library with the language. > Some languages don't even have a standard library, or for that matter a > standard implementation. And we're not discussing those languages, we're discussing Python, which has an explicit policy of "batteries included". As such, criticism of the standard library is perfectly acceptable under the name "Python", whether you like it or not. Besides, it's inevitable anyway. > Would you argue that Python is unsuitable for parsing real-world (i.e. > broken) HTML because Beautiful Soup is not in the standard library? That > Python is unsuitable for scientific and mathematical processing because > Scipy and Numpy aren't in the standard library? That you can't do natural > language processing with Python because NLTK is not in the standard > library? That you can't do image processing with Python because PIL is a > third-party library? > Out of the box? Absolutely. Again, expecting me to explicitly state that is worthless pedantry, especially when the discussion at hand proposes modifications to the standard library. There's no question, in context, about what my words meant. > There's no doubt that having a good standard library with a rich toolset > is a beneficial feature of Python. Python's philosophy of "batteries > included" has been one of it's strengths. But it would be absurd to claim > that if a tool isn't in the standard library, the language can't do it. > Too bad that isn't what I claimed, nor can what I said be interpreted as such in any way whatsoever, unless you're a pedantic twit. > > and since rick's proposal involves regressing the standard library.. > > If you think I'm supporting Rick's incoherent proposal, you've > misunderstood me. > If you think what I said somehow even implies you support his proposal, then you need to take a few English courses. > Nevertheless, you can do good, useful work > with only a minimal widget set. Back when dinosaurs walked the earth, I > wrote GUI apps using an *extremely* limited widget set, equivalent to > window, button, text field, and simple bit-mapped graphics -- no menus, > no scroll bars, no list boxes, no styled text, and certainly no layout > widgets. By today's standards, that's even more primitive than the > average web UI, and yet some of the apps written in it were useful and > even elegant. (I don't claim credit for the elegant ones, but the ones I > wrote were at least useful to me.) You can get surprisingly far with only > simple widgets and a bit of ingenuity. Or at least by lowering your > expectations. *wink* And when a time machine warps all back to the 1980s, that argument might have some merit. Since you're not Dr. Emmett Brown, I suggest you refrain from making arguments that predicate themselves on time travel. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 17, 8:30 am, Albert van der Horst wrote: > We are not talking about running applications, but about writing > applications. Someone has to write the applications I run... > I count 4000+ .py files in my /usr/share alone on Ubuntu. > Your set is totally unrepresentative for those scripts. Yeah, go back and count how many of those actually use a GUI and were not written by Canonical for the purpose of OS configuration (though it'd be especially odd to find them in /usr/_share_). Such a response is entirely and completely disingenuous and borderline insulting. You and rick both need eye exams, apparently. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: Joking?
On Jan 17, 3:30 am, Steven Howe wrote: > Target your market. Design your software in the Model-View-Controller > format. It becomes easy to configure you frontend, your GUI, your web > page, if your code is written to separate the work from the glitz. > If there were some evidence MVC actually worked, perhaps. Unfortunately, there isn't, so there's little reason to do this. Nevermind that most MVC frameworks get the definitions entirely wrong, despite the plethora of examples from the original paper (e.g., Swing does). > Car companies know this. There is the body design crew, and the > engineers that make it work. No, there really isn't anymore. Integrated teams are the mandatory order of the day, because getting the necessary fuel economy requires paying close attention to body design. Vehicles where the designer runs rampant over the look are getting rarer, and will continue to do so as long as governments continue to tighten fuel economy standards (and to a lesser degree, safety standards). Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: examples of realistic multiprocessing usage?
On Jan 17, 12:44 am, TomF wrote: > Thanks for your reply. I can enqueue all the jobs before waiting for > the results, it's just that I want the parent to process the results as > they come back. I don't want the parent to block until all results are > returned. I was hoping the Pool module had a test for whether all > processes were done, but I guess it isn't hard to keep track of that > myself. > Regardless of whether it does or doesn't, you don't really want to be blocking in two places anyway, so the "FINISHED" event in the queue is the superior solution. It's certainly possible to build a work pool w/ a queue such that you block on both for entries added to the queue and job completion, but I'm pretty sure it's something you'd have to write yourself. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: examples of realistic multiprocessing usage?
On Jan 16, 11:39 pm, TomF wrote: > One difficulty is that there is a queue of work to be done and a queue > of results to be incorporated back into the parent; there is no > one-to-one correspondence between the two. It's not obvious to me how > to coordinate the queues in a natural way to avoid deadlock or > starvation. > Depends on what you are doing. If you can enqueue all the jobs before waiting for your results, then two queues are adequate. The first queue is jobs to be accomplished, the second queue is the results. The items you put on the result queue have both the result and some sort of id so the results can be ordered after the fact. Your parent thread of execution (thread hereafter) then: 1. Adds jobs to the queue 2. Blocks until all the results are returned. Given that you suggested that there isn't a 1:1 correspondence between jobs and results, have the queue support a message saying, 'Job X is done'. You're finished when all jobs send such a message. 3. Sorts the results into the desired ordered. 4. Acts on them. If you cannot enqueue all the jobs before waiting for the results, I suggest turning the problem into a pipeline, such that the thread submitting the jobs and the thread acting on the results are different: submitter -> job processor -> results processor. Again though, the devil is in the details and without more details, it's hard to suggest an explicit approach. The simplest way to avoid contention between two queues is to just remove it entirely (by converting the processing to a single pipeline like I suggested). If that is not possible, then I suggest moving to pipes (or some other form of I/O based IPC) and asynchronous I/O. But I'd only do that if I really couldn't write a pipeline. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: examples of realistic multiprocessing usage?
On Jan 16, 2:05 pm, TomF wrote: > Instead of explaining my problem and asking for design suggestions, > I'll ask: is there a compendium of realistic Python multiprocessing > examples somewhere? Or an open source project to look at? There are tons, but without even a knowledge domain, it's difficult to recommend much of anything. Multiprocessing for I/O (e.g., web serving) tends to look different and be structured differently from multiprocessing for CPU-intensive tasking (e.g., digital signal processing), and both look different from things with specific requirements w.r.t latency (e.g., video game server, hard-real time applications) or other requirements. Even the level at which you parallel process can change things dramatically. Consider the simple case of matrix multiplication. I can make the multiplication itself parallel; or assuming I have multiple sets of matricies I want to multiply (common), I can make execute each multiplication in parallel. Both solutions look different, and a solution that uses both levels of parallelism frequently will look different still. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 16, 2:17 pm, rantingrick wrote: > The IDLE library has had a NoteBook widget for ages. They just choose > to call it TabPages instead. And what is a NoteBook exactly? Well a > Notebook is a compound widget consisting of a "main frame that holds > two sub frames -- which are a "tab" frame and "pages" frame. When any > "tab" is clicked the page it is linked to is either *raised-to-the-top- > of-the-stack* OR *made-visible* depending on the design. Each "tab" > within the "tab frame" is a Radiobutton. You need an eye exam if you actually believe what you just wrote. TabLayouts are not just two frames and radiobuttons inside a bigger frame. To even say that is disingenuous and insulting to every human who's ever used one. > I believe compound widgets have no business in the stdlib. All widgets are compound widgets, ergo you believe no widgets should be in the stdlib. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 16, 6:04 pm, Steven D'Aprano wrote: > > If the situation isn't > > the same on your computer then your application usage is highly unusual > > or you don't understand what widgets are used to construct your > > applications. You've just told me that Python would no longer be > > suitable for constructing the majority of GUI applications on the > > planet. > > No, that does not follow. Unless "he" (I'll assume it is rantingrick) has > proposed hunting down and destroying all third-party GUI tool sets, what > you've been told is that *one specific* tool set is unsuitable for > constructing the majority of GUI apps. > If you're going to expect me to be that pedantic, then pay me the courtesy of taking the time to find the necessary context. Nevertheless, it's not the least bit unreasonable to address deficiencies in the standard library as deficiencies in the language, like it or not; and since rick's proposal involves regressing the standard library.. > > Really, if you believe the case to be otherwise, I truly believe you > > aren't paying attention to your own computer(s), or don't understand how > > the applications you use are constructed. What's out there isn't > > interesting, it's what people use that's interesting, and people tend to > > use GUIs that are moderately to highly complicated. > > Well, true, but people tend to *use* the parts of the GUIs that are > simple and basic. Not only do the big complicated apps get all the press > even when they are actually a niche product (everyone knows about > Photoshop, but more people use MS Paint) but it's a truism that most > people use something like 20% of the functionality of big, complicated > GUI apps. Most people use Microsoft Word or OpenOffice for little more > than text editing with formatting. First, you can't even build MS Paint from Rick's set / the TkInter set alone, so you're already way off mark (even ignoring the ribbon in the latest versions). Second, relevance? If I cannot ship the application with only 20% of the functionality, then your point is meaningless. Plus, look at my list more closely: TweetDeck is really little more than a bunch of listboxes stuck side by side, but we cannot even construct that without replicating what would be considered standard widgets (mostlu layout widgets, but still, that's the hardest stuff to get right and therefore the most important stuff to include). It is not, from a GUI L&F perspective, "complicated". Yet, I still need quite a few widgets in order to assemble it and make it work. And many of those widgets need fairly rich functionality: buttons must support text and images, listboxes must support embedding more than text, text controls must support hyperlinks, the various layout panes must support scrollbars, sizing, and dynamic updates. > I suspect that a variation of Zipf's Law probably holds for GUI > complexity -- if you rank the widgets in order of most to least commonly > used, I expect that you'll see actual use drop away rapidly and at an > accelerated rate. E.g. the widget in second place might be used roughly > half as often as the widget in first place place, the widget in third > place one third as often, the widget in fourth place one quarter as > often, and so forth. Perhaps, but the drop off isn't relevant till we approach well over 30 widgets, at least, quite arguably more (since GUI toolkits include both things that are common, and things that absolutely suck to program, even if they're not used often). Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 16, 11:30 am, rantingrick wrote: > Adam your post is so incoherent that i cannot decide if you are FOR or > AGAINST changing the current Tkinter GUI module into a wxPython GUI > module. And this widget set that you keep referring to is a bit vague > also. If you found my post incoherent then you shouldn't be attempting to respond, you should ask me to fucking clarify! Funny how when I ask you technical questions you refuse to respond, but when you find my post incoherent you find plenty of time to attempt to twist my words to support your own mental masturbation. > If you cannot relize that this is exactly what we have now in Tkinter > then you need to go and read the source for Tkinter. Oh hell, i'd > better do it for you. Watch this... As I already told you, if you think that what's contained only in TkInter is relevant or interesting, then you're hopelessly lost. In fact, you're outright trolling. Python's standard library don't only include TkInter so only talking about TkInter is entirely disingenuous. > Now what seems to be missing from my proposed widget set that hampers > you from re-creating every GUI app on your computer? Go download the applications and run them yourself! If you can't figure it out from a visual examination, you lack the competence to bring forward your proposal. Being able to identify what widgets are used to create an application is a fundamental, rudimentary skill. If you lack that skill, you're tacitly calling all of your credibility (not that you haven't already done a fine job of demolishing any assumed credibility on your part) into question. > Ok, Ok, i let out image support but in my mind PIL > should handle all of that. It fundamentally cannot. That's not how image rendering in native widgets works. > There you have it. The same exact widget set we have now. Sure you > cannot re-create every GUI app on your stinking computer however if > you download the 3rd party wxPython extension module not only will you > be able to re-create every GUI app on your stinking computer it will > wipe your backside too! (psst: "it" refers to wxPython) > Again, I've already responded to this in detail, in other postings, posted serious and detailed technical questions, and asked for your clarifications. So instead of posting the same tired, idiotic tripe, why don't you go find those postings, read them, and respond to them? Your refusal and/or inability to do so harms your position immensely. > i hope you learned something from this exercise Adam. I'm not the one who needs to learn anything. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Tkinter: The good, the bad, and the ugly!
On Jan 14, 5:17 pm, Albert van der Horst wrote: > > I really don't follow that. You need a tremendous set to write gimp. > Obviously you won't write gimp in Python. > You need a tremendous set to write /the majority of the applications on your computer/. On my desktop right now, I have running: * Google Chrome * TweetDeck * PuTTY * Pidgin * Game for Windows Launcher * Free Download Manager * Steam Client * A Windows Control Panel Window None of those applications could be written with his proposed widget set, he's literally 0/7 on running applications. If the situation isn't the same on your computer then your application usage is highly unusual or you don't understand what widgets are used to construct your applications. You've just told me that Python would no longer be suitable for constructing the majority of GUI applications on the planet. > Now you want to jot together three cooperating screens to specify > some properties for say bluetooth. The proposed set is ample for > that, no? Yes, but why would I ever do such a thing? Such things are provided for me by the operating system already, pretty much regardless of platform. His widget set is quite truly only good for OS utilities, but I already have those. I generally don't need nor want more of them. Neither do Python users, because the majority of us aren't writing OS utilities. That leaves the logical equivalent of traditional terrible, awful Visual Basic applications left, and even Microsoft gives them a full widget set (and "real" programming language) now in VB.NET. The effort involved in segmentation doesn't make things any easier and doesn't save them, the library authors, much of anything. > Such things make up a substantial part of the applications > as far as numbers is concerned. They are probably written by > people who don't want to dive very deeply into GUI. > (Maybe they are more bluetooth experts than GUI-experts, what > would you say?) > I could list every application on the my computer and make a table of which ones would be workable given the list above, but I can already tell you that the answer is, "Not very many, and most of the ones that can were written by MS". And I'm entirely ignoring issues like theming as well. Adding a few additional layout controls would expand the list a moderate deal, but that's going to be true pretty much regardless of which 3 or 4 additional widgets you pick. And this ignores the obvious, "You'd still need a full WxWidgets install in order to build the minimized set, so all you've saved is a few .py files" part of the argument, which is just as compelling, if not more so. Really, if you believe the case to be otherwise, I truly believe you aren't paying attention to your own computer(s), or don't understand how the applications you use are constructed. What's out there isn't interesting, it's what people use that's interesting, and people tend to use GUIs that are moderately to highly complicated. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: os.path.realpath() and os.path.abspath()
On Jan 11, 6:35 am, Jurko Gospodnetić wrote: > Hi all. > > os.path.realpath() documentation states that it returns a 'canonical' > path. Does that infer that it returns an absolute path? > A canonical path is supposed to be absolute and at least Python 2.7.1 ensures that is the case. Historically, some versions of the UNIX syscall (Solaris in particular) have not always returned absolute paths, but I believe this is no longer the case and was a very long standing bug (though I may be mistaken). Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrput a thread
On Jan 4, 10:53 pm, John Nagle wrote: > There are systems where there's support designed in for thread > abort. LISP/Scheme systems tend to support it. QNX, the real-time > OS, has well worked out thread-abort semantics at the C level. > (QNX has really good features for "not getting stuck", like the > ability to put a time limit on any system call.) Yes, but "not getting stuck" and ending the thread execution is only one small part of the problem (and arguably the least significant). What we really want is a way to abort without harming other threads of execution, which is the hard part. QNX doesn't ipso facto make that easier. Functionality like time limits on system calls is more about latency guarantees and priority than "getting stuck" in a deadlock sense. > What you'd really like in Python is the ability for one thread > to be able to force an exception in another thread, plus a > mechanism for locking out such exceptions for critical sections. > It's not worth having, though, in a system where you can really only > run one thread at a time. Exceptions and critical sections are rather fundamentally incompatible, hence the absurd amount of gymnastics .NET goes through to attempt to make ThreadAbortException functional (and still fails rather miserably). If you had STM or 'antitry' blocks, then exceptions might be a semi-saneish way to abort a thread. Without either, I'm not entirely convinced of the utility. Only allowing the exception to be thrown from defined cancellation points is much better (ala POSIX threads), but diminishes the utility for things that are mostly grinding away in userspace. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrput a thread
On Jan 4, 10:12 am, Fuzzyman wrote: > > This is a case that .NET (C#) handles better than Python or Java. > Nope, read the documentation for Thread.Abort() carefully. Thread.Abort() can cause termination inside a static constructor, which is very bad. You sure your application can cope with that? Mine can't. Thread.Abort() is only safe for self-abortion, app domain termination and a few other very narrow and obscure scenarios. It is not a safe way to end arbitrary threads doing arbitrary processing. > It's another place where people > sometimes have a genuine need/use case yet people will insist on > telling them they don't *really* want it... > Because there's no safe way to do it. It's fundamentally a racy operation, with the typical horrible consequences when you lose. Removing the race requires support from the application, i.e., you have to write the code yourself. Frequently, it's simply not possible to remove the race. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrput a thread
On Jan 3, 5:24 pm, Jean-Paul Calderone wrote: > Of course. The whole point here is not about threads vs processes. > It's about shared memory concurrency vs non-shared memory > concurrency. You can implement both with threads and both with > processes, but threads are geared towards shared memory and processes > are geared towards non-shared memory. So what most people mean by > "use processes" is "don't use shared memory". > This is entirely my presumption, but I think if the OP were keenly aware of the differences between thread and processes, it's pretty likely he wouldn't have asked his question in the first place. Also, I've written lots and lots of "use processes" code on multiple platforms, and much of it has used some sort of shared memory construct. It's actually pretty common, especially in code bases with a lot of history. Not all the world is Apache. Adam -- http://mail.python.org/mailman/listinfo/python-list
Re: Interrput a thread
On Jan 3, 5:05 pm, gervaz wrote: > Regarding the case pointed out by Adam I think the best way to > deal with it is to create a critical section so that the shared memory > will be updated in an atomic fashion. Ok, so if the OS kills the process between taking the lock and releasing it, what are you going to do? Handled naively, you can end up in a deadlock situation[1]. If a process has locked a semaphore and then terminates, the semaphore retains its current value: all other processes waiting on the semaphore will still be waiting, and any new processes that access the semaphore will wait as well. In short, if a process holding a semaphore dies, you have to manually unlock it. For signals that the process intends to handle, you can always disable them before taking the lock and reenable them after releasing it. Or, you can install signal handlers in each process that ensure everything is properly cleaned up when the signal is delivered. Which solution is right depends on what you're doing. For signals you don't intend to handle (SIGSEGV) or cannot handle (SIGKILL) there's not much you can do. It's potentially dangerous to continue on after a child has received such a signal, so the right solution may be to literally do nothing and let the deadlock occur. If you must do cleanup, it must be done carefully. The key thing to understand is that the problems with killing threads haven't gone away: delivering the "please die" message isn't the hard part; it's safely cleaning up the thread in such a way it doesn't break the rest of the application! This problem still exists if you replace threads with processes (assuming you're using shared memory). As such, the better thing to do, if possible, is to avoid shared memory and use constructs like pipes and sockets for I/O. They have much better defined failure semantics, and do allow you a modicum of fault isolation. At the very least, graceful shutdown is much easier. HTH, Adam [1] For SIGINT from the terminal, the "right thing" /might/ happen. Strong emphasis on the might. -- http://mail.python.org/mailman/listinfo/python-list
