date:20161006

On Fri, Oct 7, 2016 at 3:06 PM, Loren Wilton  wrote:
> If the thread has a current instruction pointer, then it must also know what
> function it is currently in, and therfore have a pointer (or some such) to
> the current function locals:
>
> def func1():
>ham = 0.0;
>ham += 1;// thread 1 here
>
> def func2():
>ham = 0.0;
>ham -= 1;// thread 2 here
>
> In my C++ understanding of threads, those manipulations on ham won't
> interact. I would hope this is also true in Python?
>
> If I'm right that a thread knows what function it is in, and therefore has
> function-level state access to the current function, then I'd think that it
> also has to have module level state. After all, functions occur inside
> modules, or at least they can:
>
> spam.py:
> ham = 0.0;
> def func():
>ham += 1;// thread 1 here
>
> eggs.py:
> ham = 0.0;
> def func():
>ham -= 1;// thread 2 here
>
> Here I would again hope that the simultaneous threads would not cause
> collisions on ham, since (I think?) there should be spam.ham and eggs.ham,
> which would be two separate things.

Correct on both counts. Threads executing different functions cannot
trample on each other's locals. Even if they're executing different
invocations of the same function, they won't trample on locals. With
your module example (assuming you add "global ham" to each function),
they would be fine too; however, if two threads are in two invocations
of the same function, it's possible for them to overwrite each other's
change. (It'd require pinpoint timing, but it could happen.) So if
you're going to use threads, you have to not use mutable global state.
That's honestly not a difficult rule to follow; most programmers
should have a healthy caution around mutable global state anyway,
regardless of threads.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: static, class and instance methods (Reposting On Python-List Prohibited)

2016-10-06 Thread ast

"Gregory Ewing"  a écrit dans le message de 
news:e5mgi9fp1b...@mid.individual.net...

Lawrence D’Oliveiro wrote:

Every function is already a descriptor.

Which you can see with a simple experiment:

>>> def f(self):
...  print("self =", self)
...

I thought yesterday that every thing was clear. But I have
a doubt now with the following line:

>>> g = f.__get__(17, None)

The signature of  __get__ method is  __get__ (self, inst, owner)  so
once again the first parameter is filled automatically.
Is method __get__ itself a descriptor with an attribute __get__ to perform
the operation ? Hum, it would be endless ...

vars(f.__get__)
TypeError: vars() argument must have __dict__ attribute

no, so how does it work here ?

>>> g

>>> g()
self = 17

--
Greg 

--
https://mail.python.org/mailman/listinfo/python-list

[issue18789] XML Vunerability Table Unclear


Mariatta Wijaya added the comment:

Hi, here is the patch. I followed Raymond's suggestion to use 'vulnerable' or 
'safe' instead of the original 'True' or 'False'.

Please check it out.

Thanks :)

--
keywords: +patch
Added file: http://bugs.python.org/file44994/issue18789.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28383] hash documentation recommends naive XOR to combine but this is suboptimal

2016-10-06 Thread Kevin Norris


New submission from Kevin Norris:

The documentation for __hash__ contains this text:

"The only required property is that objects which compare equal have the same 
hash value; it is advised to somehow mix together (e.g. using exclusive or) the 
hash values for the components of the object that also play a part in 
comparison of objects."

The recommendation of "using exclusive or" is likely to result in programmers 
naively doing this:

def __hash__(self):
return hash(self.thing1) ^ hash(self.thing2) ^ hash(self.thing3)

In the event that (say) self.thing1 and self.thing2 have almost or exactly the 
same hash (with "almost" referring to bitwise differences rather than integral 
distance), this wipes out most or all of the entropy from both values and 
greatly increases the likelihood of hash collisions.  Indeed, Python's own 
tuple type does not do this (while it does use XOR, it also does some other 
math to ensure the bits are as mixed up as is practical).[1]

Because the correct algorithm is both nontrivial to implement and already 
exists in the tuple type's __hash__, I propose that the documentation be 
updated to recommend something like the following:

def __hash__(self):
return hash((self.thing1, self.thing2, self.thing3))

One possible wording:

"The only required property is that objects which compare equal have the same 
hash value; it is advised to mix together the hash values of the components of 
the object that also play a part in comparison of objects by packing them into 
a tuple and hashing the tuple: [code example]"

[1]: https://hg.python.org/cpython/file/fca5c4a63251/Objects/tupleobject.c#l348

--
assignee: docs@python
components: Documentation
messages: 278229
nosy: Kevin.Norris, docs@python
priority: normal
severity: normal
status: open
title: __hash__ documentation recommends naive XOR to combine but this is 
suboptimal
type: performance
versions: Python 2.7, Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: A newbie doubt on methods/functions calling


Honestly, the best implementation strategy I can think of is to first
implement a Python interpreter for the actual mainframe environment.
Then invent an RPC layer that can semi-transparently bridge the two for
when you want to call a module that only exists in the Windows
environment (or call _from_ such a module back to an object/module that
only exists in the mainframe environment), whether it's because it
requires a Windows library or because you want the Windows python
interpreter to do the heavy lifting because it's faster.

Do you have C in the mainframe environment?


Essentially the answer is "no". There is a thing that calls itself a C 
compiler, but it can only compile truely trivial programs, and then only 
after a great deal of manual hacking on the source code to change the syntax 
to what this compiler likes. There is no equivalent of "make".


I'd be better off starting with a Python interpreter in JaveScript or the 
like, if I wanted to do a transliteration that would actually work. I'm 
trying to avoid this, it would be months of work.


   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread mr . puneet . goyal

Let me rephrase my question in other way. 

class myClass:
def __init__(self, var):
self.var = var

myObj = myClass(abc)


# I am calling instance with function name and arguments
myObj func1 arg1 arg2 


Can i associate any function like __init__ with instance ? Means if I just use 
instance anywhere as above, it calls that function in the class and take 
everything as argument which I mentioned after instance? Something like below 

class myClass:
  def __call__(self, args):
# Then I parse the args here and call function internally 
self.func1(arg1, arg2)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application


I still don't understand why this has to be in the same process space as
the VM.  Wouldn't it be a lot simpler to create a simple RPC layer (all
localhost of course) to interface between the VM and a Python server
that spins up multiple processes or sessions?  Kind of like how Python
for a web server would work.  If a request comes in-band from the client
to the VM that necessitates handing off to Python, the RPC layer would
do this.


Assuming that all Python usages are single procedure calls, then all of the 
parameters can be marshalled and passed to another process. I suppose it may 
or may not be possible to modify the internals of structured parameters 
(arrays, lists, dictionaries, etc) and return the modified values, depending 
on how marshalling works. And it may or may not be fast.


If the Python code wants to make reverse calls back into the VM environment 
(and it may well want to do this, for instance for database access) then it 
is still possible but starts to get trickier, and again possibly slower.


Depending on use cases, any marshalling slowness might not matter at all. Or 
it might matter a great deal. Unfortunately I don't yet have any clearly 
defined use cases from the people that should be thinking about that. So at 
the moment I'm trying to follow the original concept spec as closely as 
possible, which is "integrate Python in the VM environment."


   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application


To be fair, the sharing of data between threads is no different from
other concerns about global state. The only thing threads change is
that you can have multiple functions doing stuff at once.

state = [1,2,3]

def func():
   state[1] += 5
   # do work
   state[1] -= 5


It occurs to me that threads must have some state of their own. For 
instance, if I have two thread and they are both in that same function 
above, one thread needs to know that it is currently on the bytecode for the 
first statement, and the other needs to know that it is in the bytecode for 
some of the # do work. In other words, each thread has a current instruction 
pointer.


If the thread has a current instruction pointer, then it must also know what 
function it is currently in, and therfore have a pointer (or some such) to 
the current function locals:


def func1():
   ham = 0.0;
   ham += 1;// thread 1 here

def func2():
   ham = 0.0;
   ham -= 1;// thread 2 here

In my C++ understanding of threads, those manipulations on ham won't 
interact. I would hope this is also true in Python?


If I'm right that a thread knows what function it is in, and therefore has 
function-level state access to the current function, then I'd think that it 
also has to have module level state. After all, functions occur inside 
modules, or at least they can:


spam.py:
ham = 0.0;
def func():
   ham += 1;// thread 1 here

eggs.py:
ham = 0.0;
def func():
   ham -= 1;// thread 2 here

Here I would again hope that the simultaneous threads would not cause 
collisions on ham, since (I think?) there should be spam.ham and eggs.ham, 
which would be two separate things.


The worry I can see is if the CPython interpreter has taken some shortcuts 
and has global variables for the pointers to the module globals and the 
funciton locals. In that case, if two threads aren't in the same function I 
can see all hell breaking loose pretty easily.


   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread Random832

On Thu, Oct 6, 2016, at 19:27, Loren Wilton wrote:
> So I don't want to WRITE a Python interpreter for the actual mainframe
> environment. I want to use an interpreter for an existing environment
> (Windows) where there are already a lot of existing libraries. But
> since a lot of the data to be analyzed is on the mainframe
> environment, and the results would be wanted there too, I need to
> extend the Python data access to the mainframe environment.

Honestly, the best implementation strategy I can think of is to first
implement a Python interpreter for the actual mainframe environment.
Then invent an RPC layer that can semi-transparently bridge the two for
when you want to call a module that only exists in the Windows
environment (or call _from_ such a module back to an object/module that
only exists in the mainframe environment), whether it's because it
requires a Windows library or because you want the Windows python
interpreter to do the heavy lifting because it's faster.

Do you have C in the mainframe environment?
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue26171] heap overflow in zipimporter module

2016-10-06 Thread Parvesh jain


Parvesh jain added the comment:

I think patches put up in http://bugs.python.org/msg258736 is at least not 
sufficient enough for Python 2.7. 
POC script(crash.py) provided with the issue calls get_data with data_size = -1.
I am using Python 2.7.8 . I patched the same with the solution provided in 
https://hg.python.org/cpython/rev/985fc64c60d6 . I was still able to reproduce 
the issue and it failed with 

Traceback (most recent call last):
  File "crash.py", line 25, in 
print(importer.get_data(FILE))
IOError: zipimport: can't read data
Segmentation fault (core dumped)


but I couldn't reproduce the same with latest 2.7.12:-

jchang@qasus-ubun12x64-001:~/Downloads/Python-2.7.12$ python2.7 -V
Python 2.7.12
jchang@qasus-ubun12x64-001:~/Downloads/Python-2.7.12$ python2.7 crash.py
Traceback (most recent call last):
  File "crash.py", line 25, in 
print(importer.get_data(FILE))
zipimport.ZipImportError: negative data size

As we can see issue does happen in 2.7.12 because of following extra check :-

if (data_size < 0) {
PyErr_Format(ZipImportError, "negative data size");
return NULL;
}

which was merged in https://hg.python.org/cpython/rev/2edbdb79cd6d. 

I was thinking of backporting the same to Python 2.7.8 as well to completely 
address this issue. Could you guys confirm if my understanding is correct on 
this ? Thanks

--
nosy: +Parvesh jain

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26081] Implement asyncio Future in C to improve performance

2016-10-06 Thread INADA Naoki


INADA Naoki added the comment:

fixed

--
Added file: http://bugs.python.org/file44993/fastfuture2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application


From: "Paul Rubin" 
I don't think Python threads are the answer.  You want a separate
interpreter per user, which is annoying to do with CPython.  Do you have
concrete performance expectations about the sharing of data between
interpreter sessions?  Those old mainframes were very slow compared to
today's machines, even counting Python's interpretation overhead.


Without separate context it sounds like Python threads are moderately 
useless except in special circumstances, and this certainly isn't one of 
those circumstances.


As for performance, especially with shared data, it is an interesting 
question, but one not easily answered at the moment. The VM does a lot of 
interesting JIT type stuff, so gets performance that easily exceeds the most 
recent hardware implementation of the machine, which was only made a few 
years ago. Yes, a raw PC can do things faster (obviously). But people like 
the mainframe environment for the same reason people like Python over C: it 
does what they want, easier. They are willing to pay some performance for 
this -- but only some. They still have network response times to meet and 
need to get the monthend batch processing done before midnight.


In our case, shared data is no slower to access than process-unique data. 
Well, maybe a little slower if you have to acquire and release a lock, 
because that takes a few nanoseconds or so. So I'd prefer that Python shared 
data access was not a lot slower than non-shared access. My impression is 
that all data is in a common heap, so a lock is required to access anything. 
I'd think that makes all data access equally slow or equally fast.


I haven't found much data on multiple separate interpreters within a single 
process space.


What exactly do you get that is unique to each interpreter? Separate heaps? 
Seperate GC locks? Separate process threads? I would guess maybe separate 
builtins.


Does multiple interpreters make it harder to share data between the 
interpreters?


   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

[issue28199] Compact dict resizing is doing too much work

2016-10-06 Thread INADA Naoki


INADA Naoki added the comment:

Since entries array is embedded in PyDictKeysObject, we can't realloc entries.
And while values are split array, dictresize() convert split table into combine 
table.

Split table may have enough size of ma_values at first in typical case.
And in not typical case, split table may not be used.
So I think realloc ma_values is premature optimization, unless profiler says 
make_keys_shared()
is slow.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21443] asyncio logging documentation clarifications


Mariatta Wijaya added the comment:

Hi, I added the paragraph explaining how to change the log level for asyncio.
Please check it out.
Thanks :)

--
keywords: +patch
nosy: +Mariatta
Added file: http://bugs.python.org/file44992/issue21443.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread mr . puneet . goyal

Well I jump from TCL to Python. And found that it was very convenient to use 
Procs there. So I was looking for that luxury in Python. 

I am not trying to reinvent the wheel. I was just curious to know if there is 
any possibility to create a caller function in  my way (TCL) where I can call 
python function/proc inside myProc. 

-- 
https://mail.python.org/mailman/listinfo/python-list

[issue26081] Implement asyncio Future in C to improve performance

2016-10-06 Thread INADA Naoki


INADA Naoki added the comment:

FutureIter_throw is wrong, maybe.
Removing FutureIter_send and FutureIter_throw from FutureIter_methods solves 
the segv and test passed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28199] Compact dict resizing is doing too much work

2016-10-06 Thread Raymond Hettinger


Raymond Hettinger added the comment:

For the simple case with no dummy entries, I was expecting a fast path that 
just realloced the keys/values/hashes arrays and then updated the index table 
with reinsertion logic that only touches the indices.   Use realloc() is nice 
because it makes it possible that the keys/values/hashes don't have to be 
recopied and if they did, it would use a fast memcpy to move them to the newly 
resized array.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application

On 10/06/2016 07:48 PM, Chris Angelico wrote:
> If that had been your original plan, it's dead simple to enhance it to
> use per-user module names. Just do this same work, but substitute a
> different module name right at the beginning! Other
> extremely-high-level interface functions are similar. You should have
> no trouble making this embed work; and then it's just a matter of
> figuring out the bridge code between that and the rest of the VM.

I still don't understand why this has to be in the same process space as
the VM.  Wouldn't it be a lot simpler to create a simple RPC layer (all
localhost of course) to interface between the VM and a Python server
that spins up multiple processes or sessions?  Kind of like how Python
for a web server would work.  If a request comes in-band from the client
to the VM that necessitates handing off to Python, the RPC layer would
do this.

Just trying to think architecturally here.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 12:52 PM, Paul Rubin  wrote:
> "Loren Wilton"  writes:
>> I've read that Python supports 'threads', and I'd assumed (maybe
>> incorrectly) that these were somewhat separate environments that could
>> be operating concurrently (modulo the GC lock). I assume that data can
>> be shared between the threads,
>
> Threads all run in the same address space.  Data "can be" shared is an
> understatement--it IS shared, so threads can interfere with each other
> very easily.  You have to be quite careful to prevent that, and thread
> programming has a reputation for being difficult for that reason.

To be fair, the sharing of data between threads is no different from
other concerns about global state. The only thing threads change is
that you can have multiple functions doing stuff at once.

state = [1,2,3]

def func():
state[1] += 5
# do work
state[1] -= 5

This function would probably work in most single-threaded
environments. (Proper use of try/finally would make it safer.) This
"patch, operate, unpatch" model becomes dangerous in a threaded
system... but it also becomes dangerous in any case of recursion. Or
just calling a function from inside func() that cares about state. Or
anything like that. Mutable globals must be strictly managed, Alice;
unproductive ones should be *eliminated*.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

"Loren Wilton"  writes:
> strength of Python is that there are many existing 3rd party libraries
> that do lots of useful things. Since a lot of them are distributed as
> binaries, they would not work in this mainframe environment.

Python libraries are usually available as source, either in Python or in C.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

"Loren Wilton"  writes:
> I've read that Python supports 'threads', and I'd assumed (maybe
> incorrectly) that these were somewhat separate environments that could
> be operating concurrently (modulo the GC lock). I assume that data can
> be shared between the threads,

Threads all run in the same address space.  Data "can be" shared is an
understatement--it IS shared, so threads can interfere with each other
very easily.  You have to be quite careful to prevent that, and thread
programming has a reputation for being difficult for that reason.

> Would using threads help me do what I want? 

I don't think Python threads are the answer.  You want a separate
interpreter per user, which is annoying to do with CPython.  Do you have
concrete performance expectations about the sharing of data between
interpreter sessions?  Those old mainframes were very slow compared to
today's machines, even counting Python's interpretation overhead.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 12:21 PM, Loren Wilton  wrote:
> The VM is a program running on Windows. The mainframe object code is just
> byte code. The VM has routines to handle every operator and each kind of
> Descriptor that they might validly use. One form of Descriptor is only used
> by the CALL instruction. It allows an "internal call" to a procedure that is
> part of the VM.  Assuming that the VM linked to CPython, I could use one
> form of this to pass a text line to PyRun_SimpleString. Then I cna write a
> trivial program that will read from a terminal file and pass any received
> text to SimpleString, and somehow route any response back to the terminal
> file.

The source code for SimpleString looks like this:

int
PyRun_SimpleStringFlags(const char *command, PyCompilerFlags *flags)
{
PyObject *m, *d, *v;
m = PyImport_AddModule("__main__");
if (m == NULL)
return -1;
d = PyModule_GetDict(m);
v = PyRun_StringFlags(command, Py_file_input, d, d, flags);
if (v == NULL) {
PyErr_Print();
return -1;
}
Py_DECREF(v);
return 0;
}

If that had been your original plan, it's dead simple to enhance it to
use per-user module names. Just do this same work, but substitute a
different module name right at the beginning! Other
extremely-high-level interface functions are similar. You should have
no trouble making this embed work; and then it's just a matter of
figuring out the bridge code between that and the rest of the VM.

(Okay, so it's probably not going to be as easy as that made it sound,
but still. Definitely plausible.)

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 11:22 AM, Loren Wilton  wrote:
>> Ah, that probably means you want separate interpreters, then.
>
>
> I hope not, unless I can get separate simultaneous interpreters out of one
> CPython.dll in one Windows process space. I'm guessing that I can't, but
> since I really don't know what I'm doing with Python yet, maybe I'm wrong
> there.

It is possible to do that, but based on the info you give in this
post, I withdraw the recommendation for separate interpreters.

>> (for instance, the interactive
>> interpreter uses "_" in the builtins to store your last result, and if
>> you have multiple users sharing the builtins, they'd be trampling over
>> each other's underscores).
>
>
> I'm very new to Python and I suspect I don't completely understand what you
> mean by 'builtins' in this instance. Does Python have a non-module-scope
> namespace that contains things like _ ? Could you give me a pointer to any
> docs on this?

In Python, there are three [1] namespaces: function locals, module
globals, and built-ins. Function locals are exactly what you'd expect:
one invocation of a function has one set of locals, and another
invocation (incl recursion) has a completely separate set.
Module-level names are called "globals", but they're not process-wide
like in C; code in a different module has separate globals. Built-ins
are for "standard" names, like range(), str(), len(), and so on. They
are shared among all modules in a process.

[1] There's also the class namespace, used while you're building a
class. Amongst our namespaces are such diverse elements as...

> If it does, do you have an opinion on how hard it might be to wrap it in
> some form of module (or "super module"?) scope?

I wouldn't. It's generally immutable. What I'd advise is having some
way to terminate Python and restart it (which will take care of any
serious messes), and for the rest, just leave it shared.

>> But if you accept that this is a shared
>> environment, with the consequences thereof, you could have easy and
>> convenient sharing - and might even be able to have a tiny hack around
>> the interactive interpreter to make it use a different module name,
>> instead of __main__. So you could do something like this:
>>
>>  User "Fred" 
>>etc.
>>
>> So they would be *visibly* sharing state. This might be acceptable to
>> you and your users. It would be pretty easy to code up (with the
>> possible exception of the interactive mode).
>
>
> What you show in example is just about exactly what I want. Each user by
> default will have his own auto-generated (or maybe overridable by the user)
> namespace, so normally can work independently. But given that he can get the
> namespace handle for some other user (which might not be an interactive
> user, it might be a background process) then they can share data.

That would align very nicely with modules in Python.

> The worry I have is any implicitly global variables, like underscore, and
> whatever else may exist in this global namespace. I'd really want some way
> to wrap those and make them unique for each user. I'm guessing that may mean
> hacking the code for CPython. I'd rather not do that, unless people would
> like the code back so I don't have my own unique branch that would break on
> each new release. But if that is what is needed, it is certainly something I
> could do.

It's probably only the underscore, and that's only for interactive
work. Seems to me it's not going to be too big a deal.

> One concern I have is if two users both "import widget". Are they now
> sharing the widget namespace? I suspect they are, and that is probably
> undesirable. Or does Python have a hierarchical namespace concept, so that
> we would have fred.widget.ham and joe.widget.ham after fred and joe both
> import widget?

Yes, they would. An import would be used in two ways:

1) Standard library modules, or third-party extension modules. It's
99.999% likely that sharing these won't be a problem.
2) Accessing another user's namespace in order to share data. Sharing
these modules is the very mechanic of data sharing.

Your users would have to be aware that they're sharing state. Off the
top of my head, I can think of very few consequences of this, most of
them having workarounds (for instance, they'd share PRNG state, but
you can explicitly construct an RNG and have your own dedicated state;
also, the decimal module has shared contexts, so you'd have to be
careful if you ever want to change its parameters). The only real
requirement would be a Python that doesn't do the usual thing with
__main__, but instead has a special module name. This would be easy to
do for script running, but might require a little more work for
interactive mode. Worst case, you roll your own REPL using exec(). It
wouldn't be too hard.

I think this is doable.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application


No idea as it's still not clear what you want to accomplish exactly.
You are providing a lot of details, just not concrete ones that show how
things are currently done and how you want to change that by adding
Python to the mix.


Hum, this could take a lot of pages of text to describe how the existing 
environment works in detail, and some of it is considered proprietary so I 
couldn't say anyway. But I can try to give you some general concepts. Then 
once I have a description of the existing environment, I can say how I would 
like to extend it to handle Python. Also see my reply a few moments ago to 
Chris, where he siggested how to users might interact if they were sitting 
at terminals.


Viruses are common today on commodity machines and operating systems. 
Mainframes in general avoid this by obscurity. Most people don't believe 
that mainframes still exist (this is what they have been taught in school) 
so they don't think of hacking them. Also, most people have probably never 
heard of a mainframe other than an IBM 370 with AIX, so have no idea what 
the environment looks like, so they can't hack it. Just to make life harder 
for the hackers, mainframes these days usually don't have anything 
resembling interactive shell access that you can physically reach without a 
badge to open a locked door or two. They typically don't access the internet 
directly. They are a backend database machine with a frontend web server and 
a firewall between it and the internet.


The general way that banking and airline networks work is that they are 
private networks. They may be phone lines carrying RS-232 or similar 
traffic. They may be some sort of packet network that may or may not be IP 
based, and quite possibly does not use public network wires at all. When it 
comes to how terminals access the mainframes, there is a routing program on 
the mainframe that knows the name and address of every possible terminal 
that can access the system. A transaction is usually something like a 6 or 8 
character uppercase word followed by positional character data. The routing 
program gets this message, looks up the terminal name, makes sure it is 
allwed to do this trnasaction, looks up th eprogram that will handle the 
transaction, and sends the termianl name and input data to that program. The 
program does whatever it needs to do, and sends the terminal name and a 
screenload of data back to the routing program. The routing program forwards 
the response data to the terminal.


Web sites like Orbitz have web forms that make search or transaction 
requests. The web server takes the information from the web page, transforms 
it into one of these six character (or so) transaction requests, and 
pretends it is a terminal, and sends the 'screen data' to the mainframe. The 
mainframe processes the request from the "terminal" and sends back a 
screenload of data. The reply may be more than one screen, so there may be 
more transactions back and forth to get all the data. Then the CGI program 
running on the web server formats this into the web page response.


Some mainframes may also be web servers. In this case the web server program 
on the mainframe will interact using the same text commands with the backend 
programs that handle the terminals, in many cases.


Mainframes can have interactive terminals, possibly thousands of them. This 
is rare these days except for the programming machines. But the transaction 
termainls could be used as interactive terminals, by running a transaction 
that connects them to a PYTHON application. So there could be many users 
(thousands of them) all doing independent things in Python at once. There 
could be web page transaction servers that are talking to back-end programs 
that invoke Python commands or scripts. In fact the web page handler could 
appear to be a Python interpreter running a different script for each web 
page request. There are also nightly, weekly, monthly, year-end, and ad-hoc 
reports that are run on mainframes, typically to gether from the databases. 
Any or all of these reports could be Python scripts, or could be programs 
that somehow invoked snippits of Python code.


Ok, that is a user-level operational view of a typical mainframe. Now for 
some lower level details of the particular mainframe of interest.


Our mainframe is emulated in a single Windows program. Any given instance 
can have from 1 to 32 processors, and up to about 100 GW of RAM, and tens of 
thousands of disk drives online. It has an operating system of its own that 
can run tens of thousands of independent processoes ("stacks") at the same 
time. The OS does all of the sort of scheduling, IO, and communications 
things that an OS normally does. There are hundreds of system libraries 
available that programs can use to perform common functions.


User programs don't have separate address spaces in the sense that you 
understand. Memory isn't paged. Effectively memory consists of objects that

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 11:03 AM, Loren Wilton  wrote:
> I've read that Python supports 'threads', and I'd assumed (maybe
> incorrectly) that these were somewhat separate environments that could be
> operating concurrently (modulo the GC lock). I assume that data can be
> shared between the threads, and probably will share automatically if the
> name of the item (or the reference) matches. But I'd also assumed that I
> might be able to do something like start a thread and tell it that it is
> working in "module user_42" or some such, so that by default it's data would
> not be global.
>
> Assuming that I'm not completely off-base about using threads, I don't
> really understand how to create a thread or destroy it. I'm also not
> completely clear on how much useful data there is that is thread local, and
> how much in the way of Python objects goes in the gloabl heap.
>

Threads don't really have much to do with namespacing and data
sharing. The two are broadly orthogonal. Basically, threads in Python
exist so that one function call can be running code while another one
is blocked (eg on I/O). Other than that, they're just two "current
stack location" markers, but they use the same namespaces, the same
globals, etc.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling


Oops, apologies for replying to the wrong thread!

   Loren
--
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application


Ah, that probably means you want separate interpreters, then.


I hope not, unless I can get separate simultaneous interpreters out of one 
CPython.dll in one Windows process space. I'm guessing that I can't, but 
since I really don't know what I'm doing with Python yet, maybe I'm wrong 
there.



(for instance, the interactive
interpreter uses "_" in the builtins to store your last result, and if
you have multiple users sharing the builtins, they'd be trampling over
each other's underscores).


I'm very new to Python and I suspect I don't completely understand what you 
mean by 'builtins' in this instance. Does Python have a non-module-scope 
namespace that contains things like _ ? Could you give me a pointer to any 
docs on this?


If it does, do you have an opinion on how hard it might be to wrap it in 
some form of module (or "super module"?) scope?



But if you accept that this is a shared
environment, with the consequences thereof, you could have easy and
convenient sharing - and might even be able to have a tiny hack around
the interactive interpreter to make it use a different module name,
instead of __main__. So you could do something like this:

 User "Fred" 
   etc.

So they would be *visibly* sharing state. This might be acceptable to
you and your users. It would be pretty easy to code up (with the
possible exception of the interactive mode).


What you show in example is just about exactly what I want. Each user by 
default will have his own auto-generated (or maybe overridable by the user) 
namespace, so normally can work independently. But given that he can get the 
namespace handle for some other user (which might not be an interactive 
user, it might be a background process) then they can share data.


The worry I have is any implicitly global variables, like underscore, and 
whatever else may exist in this global namespace. I'd really want some way 
to wrap those and make them unique for each user. I'm guessing that may mean 
hacking the code for CPython. I'd rather not do that, unless people would 
like the code back so I don't have my own unique branch that would break on 
each new release. But if that is what is needed, it is certainly something I 
could do.


One concern I have is if two users both "import widget". Are they now 
sharing the widget namespace? I suspect they are, and that is probably 
undesirable. Or does Python have a hierarchical namespace concept, so that 
we would have fred.widget.ham and joe.widget.ham after fred and joe both 
import widget?


Thanks again!

   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

On 10/06/2016 06:03 PM, Loren Wilton wrote:
>> So I take it that currently users access the software running in the
>> virtual mainframe over telnet or some form of serial link and that they
>> interact with it in a text terminal?  This point is fairly important,
>> because if it's true, then you really don't have any in-band way of
>> clients talking to some Python process(es) running on the machine
>> hosting the virtual mainframe.
> 
> Um... Yes and no.
> 
> Consider a Linux web server environment that happens to have Python 
> installed.
> 
> A programmer in the organization that ownns the server can probably log on 
> with an SSH session and type Python and then have an interactive Python 
> session where does simple computations or database accesses, or where he 
> runs existing Python scripts.
> 
> But that isn't how the machine will normally run.

Please let us know how the machine normally runs and how your clients
interact with it.  We're still very much working in the dark here since
we can't even determine how users or clients interact with the mainframe
and the kinds of general tasks they do.  You mention a web interface.
How do you envision your clients invoking Python?

> Normally the web pages might have queries or other items that run bits of 
> Python code, or Python scripts, some of which might access or even update a 
> database. Normally there might be nightly cron jobs, some of which might be 
> Pyton scripts.

Well that last bit is comparatively easier and doesn't require any
threading or multi-user capability in python itself.  The script runs,
talks to the mainframe over some protocol you've apparently established,
retrieves some data and performs some sort of command.

> 
>> then you really don't have any in-band way of
>> clients talking to some Python process(es) running on the machine
>> hosting the virtual mainframe.
> 
> I'm not clear on what you mean by "in band way". If you mean that there is 
> something on the Windows side of the machine that might wake up and decide 
> it wants to talk to Python, then no, that can't happen.

I mean that if you client is talking to the mainframe over some kind of
protocol already, then you can't really add Python to the mix directly
since it does not run on the mainframe.  But like I said, we don't know
how your clients are using the mainframe and its program, so I can't say
what the "band" actually is precisely.

But in other terms of reference, if you have a web server that lets a
client do something with the mainframe, then using the browser would be
considered "in band" since command and control functions through the
same interface.  If you wanted to add some sort of python console, you'd
have to either add something to the protocol to allow this interaction,
or do it via some other out-of-band way, such as ssh-ing and running a
script.

> But mainframes these days are generally not interactive in the sense of a 
> Telent terminal. The programming machines have something like that, but the 
> production machines usualy don't. There may be terminals, but they are 
> probably automatic teller machines, or airline reservation terminals. These 
> don't interact with a shell in the Unix sense, they interact with a series 
> of predefined transaction commands that control program access, or maybe are 
> accessed by a single dedicated program. Even web pages, if well written, can 
> only access fixed assets.

How do the users interact with these predefined transaction commands?
There must be some protocol you've implemented to allow this over a
remote connection.

>> You'll just have to implement Python support for these frames of
>> reference. Python is not going to be running on the virtual hardware;
>> it's running on Windows.  All you need to do is provide a way for the
>> remote end users to talk to the Python server running on the windows
>> machine would define sessions for the users and manage the interaction
>> with the virtual mainframe.  No matter how you cut it, Python is
>> separate from the users and separate from the mainframe since you can't
>> run Python on the mainframe itself.  The only thing that makes sense is
>> to develop some kind of Python-based server architecture that clients
>> can talk to to do things via Python on behalf of the user.
> 
> If I need to write these sort of hooks, I'm new enough to Python to not have 
> much of a clue where to start, and would appreciate all the help I can get.

Well this is an advanced Python topic for sure.  But it's one you'll
have to delve into eventually it sounds like.  Google for information on
embedding Python.

> I've read that Python supports 'threads', and I'd assumed (maybe 
> incorrectly) that these were somewhat separate environments that could be 
> operating concurrently (modulo the GC lock). I assume that data can be 
> shared between the threads, and probably will share automatically if the 
> name of the item (or the reference) matches. But I'd also assumed

[issue28381] Add a "starcaller" function

2016-10-06 Thread Steven D'Aprano


Steven D'Aprano added the comment:

This was discussed on Python-Ideas back in July:

https://mail.python.org/pipermail/python-ideas/2016-July/041153.html

I don't recall any opposition, although Nick suggested that possibly a better 
idea was to resurrect the `apply` built-in into functools:

https://mail.python.org/pipermail/python-ideas/2016-July/041159.html

--
nosy: +ncoghlan, steven.daprano

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application


So I take it that currently users access the software running in the
virtual mainframe over telnet or some form of serial link and that they
interact with it in a text terminal?  This point is fairly important,
because if it's true, then you really don't have any in-band way of
clients talking to some Python process(es) running on the machine
hosting the virtual mainframe.


Um... Yes and no.

Consider a Linux web server environment that happens to have Python 
installed.


A programmer in the organization that ownns the server can probably log on 
with an SSH session and type Python and then have an interactive Python 
session where does simple computations or database accesses, or where he 
runs existing Python scripts.


But that isn't how the machine will normally run.

Normally the web pages might have queries or other items that run bits of 
Python code, or Python scripts, some of which might access or even update a 
database. Normally there might be nightly cron jobs, some of which might be 
Pyton scripts.



then you really don't have any in-band way of
clients talking to some Python process(es) running on the machine
hosting the virtual mainframe.


I'm not clear on what you mean by "in band way". If you mean that there is 
something on the Windows side of the machine that might wake up and decide 
it wants to talk to Python, then no, that can't happen.


But mainframes these days are generally not interactive in the sense of a 
Telent terminal. The programming machines have something like that, but the 
production machines usualy don't. There may be terminals, but they are 
probably automatic teller machines, or airline reservation terminals. These 
don't interact with a shell in the Unix sense, they interact with a series 
of predefined transaction commands that control program access, or maybe are 
accessed by a single dedicated program. Even web pages, if well written, can 
only access fixed assets.




I don't think my main concern here is being able to call the CPython
interpreter routines, but instead it is to be able to provide separate
sandboxes for the various programs ("stacks", in B6500 terminology) that
might have their own Python sessions or programs.


You'll just have to implement Python support for these frames of
reference. Python is not going to be running on the virtual hardware;
it's running on Windows.  All you need to do is provide a way for the
remote end users to talk to the Python server running on the windows
machine would define sessions for the users and manage the interaction
with the virtual mainframe.  No matter how you cut it, Python is
separate from the users and separate from the mainframe since you can't
run Python on the mainframe itself.  The only thing that makes sense is
to develop some kind of Python-based server architecture that clients
can talk to to do things via Python on behalf of the user.


If I need to write these sort of hooks, I'm new enough to Python to not have 
much of a clue where to start, and would appreciate all the help I can get.


I've read that Python supports 'threads', and I'd assumed (maybe 
incorrectly) that these were somewhat separate environments that could be 
operating concurrently (modulo the GC lock). I assume that data can be 
shared between the threads, and probably will share automatically if the 
name of the item (or the reference) matches. But I'd also assumed that I 
might be able to do something like start a thread and tell it that it is 
working in "module user_42" or some such, so that by default it's data would 
not be global.


Assuming that I'm not completely off-base about using threads, I don't 
really understand how to create a thread or destroy it. I'm also not 
completely clear on how much useful data there is that is thread local, and 
how much in the way of Python objects goes in the gloabl heap.


Would using threads help me do what I want? Or am I completely off base? Or 
is there some other approach that I've missed that might be better?


   Loren

--
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread BartC


On 06/10/2016 18:06, mr.puneet.go...@gmail.com wrote:

Hi

I just started learning python. Is there any way to call functions in different 
way ?

Rather calling obj.function(arg1, arg2) I would like to call like below

"obj function arg1 arg2"


As has been pointed out, it's difficult to tell whether a space should 
mean a "." or "(". Or, as is also possible, "=" or "+" or any other kind 
of operator.


But even sticking only with "." and "(", then something like:

  a b c d e f g

could be interpreted in a myriad different ways. Python requires that 
this is determined at 'compile' time, but information about what each 
name is (class, instance, function, attribute etc), which might help 
resolve the ambiguity, isn't available until run-time.



this function is part of a class.

class myClass:
def function(arg1, arg2):
 # do something

Is it possible to do in python ? May be not directly but using some other 
methods.


It would be hard without extra clues, or extra restrictions on what is 
possible. But even if a translator could figure out what is what, a 
human reader will have trouble.


What's the problem with typing "." and "(" anyway; keyboard problems? 
(If so then keep away from languages such as C and C++ because they have 
a /lot/ more punctuation!) Or used to a language that doesn't require 
(,) around function arguments?


--
Bartc
--
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling


[Cue the decades-old story about the elaborate set of C macros that I
once saw somebody using so he could write a C program that looked like
some flavor of structured BASIC.]


I once wrote a set pf C defines so that I could compile Pascal with a C 
compiler without having to change the Pascal source code.  Fortunately that 
was about 40 years ago, and I've long since lost the source for those 
macros.


The goal here isn't to make one thing look like another that it isn't.

The goal is to "write a Python environment for a mainframe", ideally without 
having to do all of the writing to do it from scratch.


Lots of people seem to have written Python interpeters in various languages. 
I guess I could sit down and write one in Algol or NEWP and have a native 
Python environment. But what good would it do me? This machine isn't 
something at a university where the first-year hackers write their 5-line 
programs with no IO.  Everything I read says the strength of Python is that 
there are many existing 3rd party libraries that do lots of useful things. 
Since a lot of them are distributed as binaries, they would not work in this 
mainframe environment.


So I don't want to WRITE a Python interpreter for the actual mainframe 
environment. I want to use an interpreter for an existing environment 
(Windows) where there are already a lot of existing libraries. But since a 
lot of the data to be analyzed is on the mainframe environment, and the 
results would be wanted there too, I need to extend the Python data access 
to the mainframe environment.


   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

[issue28382] Possible deadlock after many multiprocessing.Process are launch

2016-10-06 Thread Alexis


New submission from Alexis:

I am launching a process inside a pool worker, using the multiprocessing module.
After a while, a deadlock append when I am trying to join the process.

Here is a simple version of the code:

import sys, time, multiprocessing
from multiprocessing.pool import ThreadPool

def main():
# Launch 8 workers
pool = ThreadPool(8)
it = pool.imap(run, range(500))
while True:
try:
it.next()
except StopIteration:
break

def run(value):
# Each worker launch its own Process
process = multiprocessing.Process(target=run_and_might_segfault, 
args=(value,))
process.start()

while process.is_alive():
sys.stdout.write('.')
sys.stdout.flush()
time.sleep(0.1)

# Will never join after a while, because of a mystery deadlock
process.join()

def run_and_might_segfault(value):
print(value)

if __name__ == '__main__':
main()

And here is a possible output:

~ python m.py
..0
1
8
.9
...10
..11
12
13
14
16

As you can see, process.is_alive() is alway true after few iterations, the 
process will never join.

If I CTRL-C the script a get this stacktrace:

Traceback (most recent call last):
  File 
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py",
 line 680, in next
item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "m.py", line 30, in 
main()
  File "m.py", line 9, in main
it.next()
  File 
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/multiprocessing/pool.py", line 684, in next
self._cond.wait(timeout)
  File 
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/threading.py", line 293, in wait
waiter.acquire()
KeyboardInterrupt

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File 
"/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5
/lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt

Using python 3.5.1 on macos, also tried with 3.5.2 with same issue.
Same result on Debian.
I tried using python 2.7, and it is working well. May be a python 3.5 issue 
only?

Here is the link of the stackoverflow question:
http://stackoverflow.com/questions/39884898/large-amount-of-multiprocessing-process-causing-deadlock

--
components: Library (Lib)
messages: 278221
nosy: Hadhoke
priority: normal
severity: normal
status: open
title: Possible deadlock after many multiprocessing.Process are launch
type: behavior
versions: Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 9:47 AM, Loren Wilton  wrote:
> I don't think my main concern here is being able to call the CPython
> interpreter routines, but instead it is to be able to provide separate
> sandboxes for the various programs ("stacks", in B6500 terminology) that
> might have their own Python sessions or programs.
>
> One possible scenario is a B6500 user, sitting at a terminal, and typing
> "run python". That should get him a copyright notice and a >>> prompt, and
> he should be able to carry on just as though he was sitting at a Windows
> command line. The guy at the terminal in the next office should be able to
> be doing the same thing at the same time.

Ah, that probably means you want separate interpreters, then. My
previous idea of just having separate modules wouldn't isolate well
enough to feel properly comfortable (for instance, the interactive
interpreter uses "_" in the builtins to store your last result, and if
you have multiple users sharing the builtins, they'd be trampling over
each other's underscores). But if you accept that this is a shared
environment, with the consequences thereof, you could have easy and
convenient sharing - and might even be able to have a tiny hack around
the interactive interpreter to make it use a different module name,
instead of __main__. So you could do something like this:

 User "Fred" 
run python
Python 3.6, yada yada
Type "help" yada yada
>>> spam = 1234

 User "Joe" 
run python
Python 3.6, yada yada
Type "help" yada yada
>>> spam = 4321
>>> import fred
>>> print(fred.spam)
1234
>>> print(spam)
4321
>>> fred.ham = 2

 User "Fred" 
>>> print(spam)
1234
>>> print(ham)
2

So they would be *visibly* sharing state. This might be acceptable to
you and your users. It would be pretty easy to code up (with the
possible exception of the interactive mode).

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread Grant Edwards

On 2016-10-06, Steve D'Aprano  wrote:

> The only way to do this will be to write your own pre-processor, which will
> parse your source code, and translate it from your language to valid
> Python. That's a lot of work for very little value -- I recommend you just
> learn the Python syntax rather than trying to force it to be something it
> is not.

[Cue the decades-old story about the elaborate set of C macros that I
once saw somebody using so he could write a C program that looked like
some flavor of structured BASIC.]

-- 
Grant Edwards   grant.b.edwardsYow! Don't SANFORIZE me!!
  at   
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

On 10/06/2016 04:47 PM, Loren Wilton wrote:
> The Python code is running as (I hope) a native Windows DLL, so should be 
> able to access any existing Python libraries that exist on the WIndows 
> machine. Obviously this Python code will be using Windows-shaped data 
> objects like integers, floats, and strings.
> 
> The B6500 code also has integers, floats, doubles, and strings, but they all 
> look different from what Windows data looks like. Since Python accesses 
> things as objects and objects have accessor methods, I am reasonably sure I 
> can either subclass existing Python objects to access B6500 data, or simply 
> make new objects to access that data. Thus the Python code, even though it 
> is running in its own interpreter, will be able to access resources within 
> the virtual machine environment.

So I take it that currently users access the software running in the
virtual mainframe over telnet or some form of serial link and that they
interact with it in a text terminal?  This point is fairly important,
because if it's true, then you really don't have any in-band way of
clients talking to some Python process(es) running on the machine
hosting the virtual mainframe.

> 
> Does that help? I can expound more if needed, but I don't want to bloat the 
> newsgroup with irrelevent stuff.

It's a start.  This is fairly interesting. I love the welding of old and
new technologies in innovative ways.  Please continue.

> I don't think my main concern here is being able to call the CPython 
> interpreter routines, but instead it is to be able to provide separate 
> sandboxes for the various programs ("stacks", in B6500 terminology) that 
> might have their own Python sessions or programs.

You'll just have to implement Python support for these frames of
reference. Python is not going to be running on the virtual hardware;
it's running on Windows.  All you need to do is provide a way for the
remote end users to talk to the Python server running on the windows
machine would define sessions for the users and manage the interaction
with the virtual mainframe.  No matter how you cut it, Python is
separate from the users and separate from the mainframe since you can't
run Python on the mainframe itself.  The only thing that makes sense is
to develop some kind of Python-based server architecture that clients
can talk to to do things via Python on behalf of the user.

At least that's the way I see it pending an answer to the question of
how users interact with this mainframe and its programs.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 9:09 AM, Loren Wilton  wrote:
>> Okay. Before you go one micron further, answer this critical question:
>>
>> *Do you trust your users?*
>
>
> The basic answer is yes. This program amounts to a virtual machine
> implementation of a multi-processor and multi-user mainframe computer. It
> runs on a Windows box of its own. It has an IO subsystem of its own with a
> pretty hefty firewall machine in front of it before it gets to internet
> access. The Windows OS doesn't have network access at all. That may sound
> like a joke to you, but some of the biggest companies in the world have this
> machine running in their back rooms.

Good. And no, that's most definitely not a joke; having a VM with no
network access is a very smart way to manage security.

So you don't have to worry about someone deliberately trying to mess
you around. This saves a *TON* of trouble.

> The goal at the moment is to see if I can make it appear that I have Python
> running more or less natively on the virtual machine. This means that Python
> will have direct access to user variables, files, and direct database access
> for the mainframe databases. Since the mainframe data format doesn't match
> that of the commmon Python objects, I'll either have to subclass the Python
> objects or make my own objects similar to the equivalent Python objects for
> things like numbers and strings. That is a project for tomorrow, not today.
> It seems straight-forward, just a lot of work.

That's the inherent work of trying to make one thing talk to another.
Most likely, you can ignore most of the differences, and just make
sure your script writers are aware, for instance, that integer
wrap-around doesn't exist in Python. Unless someone's been
deliberately exploiting those kinds of features, that's unlikely to
cause problems.

>> You can avoid name collisions simply by giving each user a
>> module; one person's "spam" doesn't collide with another user's "spam"
>> any more than math.log collides with logging.log. However, this is
>> *not* protecting one user from another - it just protects against
>> accidents. (I could easily reach into someone else's module by typing
>> "fred.spam = 123".)
>
>
> This particular mainframe architecture is odd, by today's machine standards.
> It is a stack machine. The stack is not a simple linear stack as far as
> addressing is concerned, but more of a tree structure. All user programs
> have stacks that are rooted at D0 in the operating system, at D1 in their
> own code file, and at higher levels in various nested program procedures. A
> new process can be created by forking at any level, and the new process
> shares all data with the original program from that level outward. It is
> possible for one stack to export procedure entry points that can be called
> by other stacks. Variables, arrays, files, etc, can be passed fron one stack
> (process) to another stack (process).
>
>> So the question really becomes: How independent should the Pythons be?
>> Sharing data is far easier if they're less isolated, but then it's
>> harder to multithread.
>
>
> The desirable goal is to make it look like Python is a native language
> running on this machine, but also giving access to existing Python libraries
> running on the PC that contains this virtual machine. Effectively this is
> extending the boundaries of the virtual machine to include processor chips
> of a different architecture within the scope of the VM. This has security
> implications, but at the moment they are (at least partially) understood and
> accepted.

Hmm. Okay, so you have some pretty fundamental architectural
differences to deal with. I think what you're doing is going to be
possible, but either (a) this is a large and ongoing job, or (b)
Python won't feel truly native - it'll be a bridging system, and
people will need to understand both ends. Sounds like it'd be a good
fun project, though.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application


"Loren Wilton"  writes:

I don't think my main concern here is being able to call the CPython
interpreter routines, but instead it is to be able to provide separate
sandboxes for the various programs ("stacks", in B6500 terminology)
that might have their own Python sessions or programs.


This is much easier to do with Lua than CPython.  I haven't looked at
Micropython much yet.


Unfortunately the requirement is to be able to run Python because of its 
popularity and the size of the existing code base of available libraries. 
Other languages would be interesting at some point, but only if there is 
some real business use for the language. Unless banks and clearing houses 
are commonly using Lua, it might be a hard sell. They are using Python.


   Loren 


--
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

"Loren Wilton"  writes:
> I don't think my main concern here is being able to call the CPython
> interpreter routines, but instead it is to be able to provide separate
> sandboxes for the various programs ("stacks", in B6500 terminology)
> that might have their own Python sessions or programs.

This is much easier to do with Lua than CPython.  I haven't looked at
Micropython much yet.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

the multi-user program is a virtual machine
implementation
and the programs running on the machine have resources
like variables, arrays, files, and databases

Python variables? Python arrays (lists? tuples?)? Or some other sort of
"variables" and "arrays" that exist outside of Python?

You _really_ need to describe your execution environment ;)

http://bitsavers.informatik.uni-stuttgart.de/pdf/burroughs/B6500_6700/1035441_B6500_B7500_Stack_Mechanism_1968.pdf

The linked paper By Erv Hauck and Ben Dent describes what our ancestor
machine looked like in 1968, and is about the most recent documentation
commonly available on the web. Its been a few years since then, and the
machine has changed a little over time, but the basics of the word format
and stack mechanism and program structure are basically unchanged.

Since the machine is no longer represented by hardware, but instead by an
emulator running on Windows, we have some additional capabilities. For
instance, instead of writing a Python interpreter that will run inside the
virtual machine structure (effectively being a normal B6500 program, in
terms of the referenced document) I can use a Python interpreter (CPython)
that is an existing DLL running in the WIndows environment, and do a
procedure call from my B6500 environment into the Python environment.

Effectively this is a call from one stack-based procedure using one bytecode
to another stack-based procedure using a different bytecode. The interpreter
for the B6500 bytecode will realize what it is calling, fabricate an
appropriate call into CPython (possibly switching to a different thread if
that is necessary) and then let the Python code do its thing. Eventually the
procedure call will (probably) return, and the B6500 code will resume
execution. Basically there is a fence where we swich between interpreters on
a procedure call or return boundary.

The Python code is running as (I hope) a native Windows DLL, so should be
able to access any existing Python libraries that exist on the WIndows
machine. Obviously this Python code will be using Windows-shaped data
objects like integers, floats, and strings.

The B6500 code also has integers, floats, doubles, and strings, but they all
look different from what Windows data looks like. Since Python accesses
things as objects and objects have accessor methods, I am reasonably sure I
can either subclass existing Python objects to access B6500 data, or simply
make new objects to access that data. Thus the Python code, even though it
is running in its own interpreter, will be able to access resources within
the virtual machine environment.

Does that help? I can expound more if needed, but I don't want to bloat the
newsgroup with irrelevent stuff.

I don't think my main concern here is being able to call the CPython
interpreter routines, but instead it is to be able to provide separate
sandboxes for the various programs ("stacks", in B6500 terminology) that
might have their own Python sessions or programs.

One possible scenario is a B6500 user, sitting at a terminal, and typing
"run python". That should get him a copyright notice and a >>> prompt, and
he should be able to carry on just as though he was sitting at a Windows
command line. The guy at the terminal in the next office should be able to
be doing the same thing at the same time.

Loren

--
https://mail.python.org/mailman/listinfo/python-list

[issue28206] signal.Signals not documented


Changes by Mariatta Wijaya :


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28381] Add a "starcaller" function

2016-10-06 Thread Ned Deily


Changes by Ned Deily :


--
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28381] Add a "starcaller" function


Mariatta Wijaya added the comment:

Hi Josh,

I think python ideas mailing list might have been a better venue for this.
https://mail.python.org/mailman/listinfo/python-ideas

--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application

We need to understand first what the process/threading/per-user model of 
the existing application is.


The program is a virtual machine for an old mainframe architecture. You 
could think of a VM running a Linux implementaiton as a kind of crude mental 
reference model.


The emulated mainframe is a multi-processor machine. We use separate threads 
to represent each physical processor, and those threads are assigned to 
separate real Intel processors underneath the program, in Windows. A given 
user program can have multiple threads. Each thread can be running or 
waiting for something.


If the user thread is running it will be on one of the virtual processors, 
so will be on some one of the processor threads in the application. The user 
thread can be interrupted by an IO completion interrupt or timer interrupt 
or give up control in some way, and be kicked off the virtual processor. 
When next run it might be on some other virtual processor, and thus on some 
other thread in the VM application. Again, this is just like a normal user 
program running in say Linux. So the VM application has many "processor" 
threads, but a user does not correspond (except transiently) to any given 
processor thread.


The VM also has a number of other threads for various purposes. It would be 
possible to make a dedicated thread for each user that started a Python 
program. This could get messy to manage, and there is always the possibility 
of runnnig out of threads, but it might be a simple way to keep things 
separated. I think that might allpw Python to use thread local storage to 
some advantage, if I've understood the docs thatI've read correctly.


   Loren

--
https://mail.python.org/mailman/listinfo/python-list

[issue27972] Confusing error during cyclic yield

2016-10-06 Thread Guido van Rossum


Guido van Rossum added the comment:

Maybe it could be fixed rather than making this a checked failure?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application


Okay. Before you go one micron further, answer this critical question:

*Do you trust your users?*


The basic answer is yes. This program amounts to a virtual machine 
implementation of a multi-processor and multi-user mainframe computer. It 
runs on a Windows box of its own. It has an IO subsystem of its own with a 
pretty hefty firewall machine in front of it before it gets to internet 
access. The Windows OS doesn't have network access at all. That may sound 
like a joke to you, but some of the biggest companies in the world have this 
machine running in their back rooms.



Would you permit your users to have complete access to the computer
that this program is running on? If they're all people in the same
company, running something on the company's own server, you're fine.
But if there's even the slightest chance that a malicious user will be
on this system, you MUST NOT permit arbitrary code. CPython is *not* a
secured environment.


This is understodd and accepted. I should say at the moment this is at the 
feasibility and experimental stage. It would be possible to firewall this 
into a separate application to protect the VM data, but the machine users 
would still have access to the PC side thru CPython. It would be possible to 
go a step farther and put Python in a PC of its own connected thru 
Infiniband. We've done this with Java, but the results are painful.


The goal at the moment is to see if I can make it appear that I have Python 
running more or less natively on the virtual machine. This means that Python 
will have direct access to user variables, files, and direct database access 
for the mainframe databases. Since the mainframe data format doesn't match 
that of the commmon Python objects, I'll either have to subclass the Python 
objects or make my own objects similar to the equivalent Python objects for 
things like numbers and strings. That is a project for tomorrow, not today. 
It seems straight-forward, just a lot of work.



You want them to be able to share data, even at the level of a single
variable.


Yes.


That strongly suggests using the same CPython embed for all
your users.


That was the conclusion I'd initially come to, yes.


You can avoid name collisions simply by giving each user a
module; one person's "spam" doesn't collide with another user's "spam"
any more than math.log collides with logging.log. However, this is
*not* protecting one user from another - it just protects against
accidents. (I could easily reach into someone else's module by typing
"fred.spam = 123".)


This particular mainframe architecture is odd, by today's machine standards. 
It is a stack machine. The stack is not a simple linear stack as far as 
addressing is concerned, but more of a tree structure. All user programs 
have stacks that are rooted at D0 in the operating system, at D1 in their 
own code file, and at higher levels in various nested program procedures. A 
new process can be created by forking at any level, and the new process 
shares all data with the original program from that level outward. It is 
possible for one stack to export procedure entry points that can be called 
by other stacks. Variables, arrays, files, etc, can be passed fron one stack 
(process) to another stack (process).



So the question really becomes: How independent should the Pythons be?
Sharing data is far easier if they're less isolated, but then it's
harder to multithread.


The desirable goal is to make it look like Python is a native language 
running on this machine, but also giving access to existing Python libraries 
running on the PC that contains this virtual machine. Effectively this is 
extending the boundaries of the virtual machine to include processor chips 
of a different architecture within the scope of the VM. This has security 
implications, but at the moment they are (at least partially) understood and 
accepted.



And above all, the security question.


See last sentence above.

Thanks!
   Loren

--
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

2016-10-06 Thread Erik


On 06/10/16 22:40, Loren Wilton wrote:

the multi-user program is a virtual machine
implementation


That's not relevant (unless you mean each user is running in their own 
VM, in which case you _really_ need to describe your execution environment).


BTW, you _really_ need to describe your execution environment ;)



and the programs running on the machine have resources
like variables, arrays, files, and databases


Python variables? Python arrays (lists? tuples?)? Or some other sort of 
"variables" and "arrays" that exist outside of Python?


You _really_ need to describe your execution environment ;)

E ;)
--
https://mail.python.org/mailman/listinfo/python-list

[issue27972] Confusing error during cyclic yield


Yury Selivanov added the comment:

> Is that enough? What if the recursion involves several tasks waiting
for each other in a cycle?

I'm not sure... Maybe it's OK when two tasks await on each other, I think the 
current Task implementation should be able to handle that.  The problem with 
the Task awaiting itself is that the current implementation just crashes with a 
RecursionError.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27972] Confusing error during cyclic yield


Yury Selivanov added the comment:

> It's pretty perverse. But how would you detect this case?

In Task._step, we can check if the future the task is about to await on is 
"self".

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27972] Confusing error during cyclic yield

2016-10-06 Thread Guido van Rossum


Guido van Rossum added the comment:

Is that enough? What if the recursion involves several tasks waiting
for each other in a cycle?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application

"Loren Wilton"  writes:
> While it is certianly possible to marshall every variable access
> through IPC, it isn't real efficient. I would much perfer to avoid
> this if I possibly can.

Maybe you could use Python proxy objects accessing a shared memory
segment.  Though as Chris Angelico mentions, you may have a security
issue depending on what the users are up to.

Can you say what the application is for?  Does the user language have to
be Python?  What you're describing might be easier with Erlang.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue27972] Confusing error during cyclic yield

2016-10-06 Thread Guido van Rossum


Guido van Rossum added the comment:

It's pretty perverse. But how would you detect this case? Does it
require changes to CPython or only to asyncio? Does it require a spec
change anywhere?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Question on multiple Python users in one application

(Apologies for the broken threading on this reply, I'm just getting the list 
access set up.)



Put each Python in a separate process and communicate by IPC.


I thought of that, but the multi-user program is a virtual machine 
implementation, and the programs running on the machine have resources like 
variables, arrays, files, and databases that the users are likely to want to 
access from Python as native Python objects. It's unlikely or impossible 
that they would or could pass all objects as command line parameters to a 
Python program.


While it is certianly possible to marshall every variable access through 
IPC, it isn't real efficient. I would much perfer to avoid this if I 
possibly can.


Thanks,
   Loren

--
https://mail.python.org/mailman/listinfo/python-list

[issue28380] Mock functions with autospec don't support assert_called_once, assert_called, assert_not_called

2016-10-06 Thread Gregory P. Smith


Gregory P. Smith added the comment:

thanks!  I didn't apply the fix to 3.5 (or earlier - those are closed) as it 
could arguably be seen as adding a new API and there are valid workarounds by 
asserting on the list of calls directly.

--
resolution:  -> fixed
stage:  -> commit review
status: open -> closed
versions:  -Python 3.3, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application

On Fri, Oct 7, 2016 at 7:59 AM, Jolly Good Spam
 wrote:
> I have a Windows multi-user program that can be used by an arbitrary number
> of users at once. They can work independently, or they can share data at the
> file or even variable level if they want. I want to give them the ability to
> write Python programs within this environment. So I intend to embed CPython
> access in the program.

Okay. Before you go one micron further, answer this critical question:

*Do you trust your users?*

Would you permit your users to have complete access to the computer
that this program is running on? If they're all people in the same
company, running something on the company's own server, you're fine.
But if there's even the slightest chance that a malicious user will be
on this system, you MUST NOT permit arbitrary code. CPython is *not* a
secured environment.

So, on to the specifics.

> The basic embedding of CPython seems straight forward. But since I have
> multiple users, each needs their own Python sandbox, so if they all compile
> programs with variable 'spam', it doesn't collide. Of course they can all have
> different programs running at the same time too.

You want them to be able to share data, even at the level of a single
variable. That strongly suggests using the same CPython embed for all
your users. You can avoid name collisions simply by giving each user a
module; one person's "spam" doesn't collide with another user's "spam"
any more than math.log collides with logging.log. However, this is
*not* protecting one user from another - it just protects against
accidents. (I could easily reach into someone else's module by typing
"fred.spam = 123".)

So the question really becomes: How independent should the Pythons be?
Sharing data is far easier if they're less isolated, but then it's
harder to multithread.

And above all, the security question.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Question on multiple Python users in one application

2016-10-06 Thread Erik


On 06/10/16 22:11, Paul Rubin wrote:

"Jolly Good Spam"  writes:

Can someone please suggest what I should be looking at and doing to be
able to effectively run multiple independent Pythons in a single
program?


Put each Python in a separate process and communicate by IPC.


Loren says that this is to be CPython embedded into an existing 
application (in which each user can be identified somehow).


We need to understand first what the process/threading/per-user model of 
the existing application is.


E.
--
https://mail.python.org/mailman/listinfo/python-list

[issue28380] Mock functions with autospec don't support assert_called_once, assert_called, assert_not_called

2016-10-06 Thread Roundup Robot


Roundup Robot added the comment:

New changeset 4e39b4e57673 by Gregory P. Smith in branch '3.6':
Fixes issue28380: unittest.mock Mock autospec functions now properly support
https://hg.python.org/cpython/rev/4e39b4e57673

New changeset fca5c4a63251 by Gregory P. Smith in branch 'default':
Issue #28380: unittest.mock Mock autospec functions now properly support
https://hg.python.org/cpython/rev/fca5c4a63251

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28381] Add a "starcaller" function

2016-10-06 Thread Josh Rosenberg


New submission from Josh Rosenberg:

Not sure if this is the right venue to propose this, but I'd like to propose 
adding a starcaller method to the standard library, either functools or 
operator (not sure if the proposal is more like partial or more like 
methodcaller).

Basically, right now, when you want to take an existing function and call it 
repeatedly with preconstructed tuples using functional tools (e.g. sorted or 
groupby's key argument, multiprocessing.Pool's various map-like methods), you 
need to either have a special starmethod (e.g. Pool.map vs. Pool.starmap), or 
if Python doesn't provide one, you need use Python-defined functions (and for 
stuff like Pool, you can't even use lambda due to pickling issues, so you need 
to define the utility somewhere else). This means that you can't reuse builtins 
for stuff like the old "find runs of consecutive numbers" example from the 
itertools docs ( https://docs.python.org/2.6/library/itertools.html#examples ).

In terms of behavior:

starfunc = starcaller(func)

would be behave roughly the same as the following (less efficient) Python 2 
code:

starfunc = functools.partial(apply, func)

(and perhaps starcaller(func, kwonly=True) would behave like 
functools.partial(apply, func, ()) to extend it to dicts).

This would make it possible to write the old consecutive numbers example:

for k, g in groupby(enumerate(data), lambda (i,x):i-x):

(which is no longer legal since Py3 banned sequence unpacking in arguments like 
that) as:

for k, g in groupby(enumerate(data), starcaller(operator.sub)):

It would also mean that instead of constantly needing to write new "star 
methods", you can just reuse existing functions with new batteries. For 
example, multiprocessing.Pool has a map and starmap function, but imap and 
imap_unordered have no starimap or starimap_unordered equivalents, which means 
using them at all requires you to def a function at trivial wrapper at global 
scope (possibly a long way away from the point of use) just to use an existing 
function ( def whydidineedthis(args): return existingfunction(*args) ).

With starcaller, there is no need to add starimap and friends, because the same 
result is easily achieved with:

with multiprocessing.Pool() as pool:
for res in pool.imap(starcaller(existingfunction), ...):

Is this a reasonable proposal? I could get an implementation ready fairly 
quickly, I'd just need to feedback on the name, which module it should be put 
in (functools or operator seem equally plausible) and whether the idea of using 
a keyword argument to the constructor makes more sense (kwonly=True) vs. 
expecting users to do functools.partial(starcaller(existingfunc), ()), vs. a 
separate class like doublestarcaller.

--
components: Library (Lib)
messages: 278212
nosy: josh.r
priority: normal
severity: normal
status: open
title: Add a "starcaller" function
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: Question on multiple Python users in one application

"Jolly Good Spam"  writes:
> Can someone please suggest what I should be looking at and doing to be
> able to effectively run multiple independent Pythons in a single
> program?

Put each Python in a separate process and communicate by IPC.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue28380] Mock functions with autospec don't support assert_called_once, assert_called, assert_not_called

2016-10-06 Thread Gregory P. Smith


Changes by Gregory P. Smith :


--
assignee:  -> gregory.p.smith
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Question on multiple Python users in one application

2016-10-06 Thread Jolly Good Spam


Hello. Please pardon a newbie question.

I have a Windows multi-user program that can be used by an arbitrary number 
of users at once. They can work independently, or they can share data at the 
file or even variable level if they want. I want to give them the ability to 
write Python programs within this environment. So I intend to embed CPython 
access in the program.


The basic embedding of CPython seems straight forward. But since I have 
multiple users, each needs their own Python sandbox, so if they all compile 
programs with variable 'spam', it doesn't collide. Of course they can all 
have different programs running at the same time too.


I think I need to do some behind-the-scenes thread management and possibly 
namespace management to allow this to work correctly. But this is where I'm 
confused, and not sure where to start. I know how to tell my users apart. 
The program is multi-threaded, but there is not a direct correspondence 
between a user and a thread; a single user can switch threads, but I know 
when this happens.


Can someone please suggest what I should be looking at and doing to be able 
to effectively run multiple independent Pythons in a single program?


Thank you for your help!

   Loren

--
https://mail.python.org/mailman/listinfo/python-list

[issue28365] IDLE: don't offer to save text files as .py

2016-10-06 Thread Terry J. Reedy


Terry J. Reedy added the comment:

PS. A.J., you will likely be a happier user if you make a subdirectory under 
c:/Users/angie and put your files there instead of in the installation 
directory under the hidden AppDate

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28365] IDLE: don't offer to save text files as .py

2016-10-06 Thread Terry J. Reedy


Terry J. Reedy added the comment:

I am closing this as a duplicate of existing #21140, to make .txt instead of 
.py the extension for files known not to be code files.  I There is an existing 
simple patch.  I will modify it to not even offer .py as a secondary choice.

The reason Paul had to close IDLE to open the file in an(other) editor window 
is that Python allows only one editor window per file, and Shell is a subclass 
of OutputWindow, which is a subclass of EditorWindow. (The latter inheritance 
is backwards; OutputWindow should be the base TextEditor and 'CodeEditor' a 
subclass thereof.) When Shell is saved, the file name appears on the recent 
files list, and attempts to open it make the exiting editor the active window. 

Angie, as Paul explained, your screenshot shows a text file, mislabeled as a 
.py python file, in the editor.  When you run non-code text as code, you get a 
SyntaxError.  The text file appears to be a saved interactive shell session.  
"Python 3" by itself is a SyntaxError and will continue to be an error no 
matter what follows.

It is possible that your Python installation, or the IDLE part of it, is badly 
broken, and needs to be repaired or re-installed.  But I think it much much  
more likely that you have made a mistake.  If you want to discuss this further, 
please post to python-list (or its news.gmane.org mirror).  Include a link to 
this issue, give a detailed description of what you did (keystrokes and mouse 
clicks), and ask any questions.

--
resolution:  -> duplicate
stage:  -> resolved
status: open -> closed
superseder:  -> Idle: saving Shell or an OutputWindow should default to .txt
title: idle forgets that saved console session is not a python file after 
restart -> IDLE: don't offer to save text files as .py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue21140] Idle: saving Shell or an OutputWindow should default to .txt

2016-10-06 Thread Terry J. Reedy


Terry J. Reedy added the comment:

A more drastic change would be to refuse to save OutputWindow and Shell as 
.py(w,o) files.  #28365 is about a beginner who apparently save a short Shell 
session as .py, quit IDLE, loaded the saved session in an editor, and tried to 
run it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28380] Mock functions with autospec don't support assert_called_once, assert_called, assert_not_called

2016-10-06 Thread Yannick Brehon


New submission from Yannick Brehon:

If one defines a mock for a function, using autospec=True, then the mock will 
correctly support assert_called_once_with(), among others, but not 
assert_called_once, assert_called, and assert_not_called.
The attached file contains a fix for the issue.

--
components: Library (Lib)
files: fix_autospecced_mock_functions.patch
keywords: patch
messages: 278208
nosy: Yannick Brehon
priority: normal
severity: normal
status: open
title: Mock functions with autospec don't support assert_called_once, 
assert_called, assert_not_called
type: behavior
versions: Python 3.3, Python 3.4, Python 3.5, Python 3.6, Python 3.7
Added file: 
http://bugs.python.org/file44991/fix_autospecced_mock_functions.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18789] XML Vunerability Table Unclear


Mariatta Wijaya added the comment:

I'll work on this :)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue18789] XML Vunerability Table Unclear


Changes by Mariatta Wijaya :


--
nosy: +Mariatta

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26081] Implement asyncio Future in C to improve performance


Yury Selivanov added the comment:

INADA, would you be able to take a look?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28379] PyUnicode_CopyCharacters could lead to undefined behaviour

2016-10-06 Thread Serhiy Storchaka


Serhiy Storchaka added the comment:

Added comments on Rietveld. I don't know whether tests for this function are 
needed. It is public, but is not a part of stable API.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue26081] Implement asyncio Future in C to improve performance


Yury Selivanov added the comment:

The most recent patch segfaults... Will try to debug.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27972] Confusing error during cyclic yield


Yury Selivanov added the comment:

This is an interesting mind twister.  The key problem is that 
`self.runner_task` is blocked on *itself*: so Task._fut_waiter is set to the 
Task.

Therefore when the task is being cancelled, `Task.cancel` simply recurses.

One way to solve this is to prohibit tasks to await on themselves.  Right now 
the following code "kind of" works:

  async def f():
loop.call_later(1, lambda: t.set_result(42))
return await t

  loop = asyncio.get_event_loop()
  t = loop.create_task(f())
  print(loop.run_until_complete(t))

albeit it logs errors about invalid Future state.

My proposal is to raise a ValueError if Task is asked to await on itself.

Guido, what do you think?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: segfault using shutil.make_archive

2016-10-06 Thread Tim

On Thursday, October 6, 2016 at 2:04:20 PM UTC-4, Random832 wrote:
> On Thu, Oct 6, 2016, at 13:45, Random832 wrote:
> > On Thu, Oct 6, 2016, at 12:46, Tim wrote:
> > > I need to zip up a directory that's about 400mb.
> > > I'm using shutil.make_archive and I'm getting this response:
> > > 
> > > Segmentation fault: 11 (core dumped)
> > > 
> > > The code is straightforward (and works on other, smaller dirs):
> > 
> > Are you able to make a test case that reproduces it reliably without any
> > specifics about what data you are using (e.g. can you generate a
> > directory full of blank [or /dev/urandom if the compressed size is a
> > factor] files that causes this)? Is it a large number of files or a
> > large total size of files?
> 
> Also consider passing a logging object to make_archive and see if it
> prints out anything interesting before the segfault.
> 
> import shutil
> import logging
> import sys
> logger = logging.getLogger()
> logger.setLevel(logging.DEBUG)
> logger.addHandler(logging.StreamHandler(sys.stderr))
> shutil.make_archive(..., logger=logger)

Interesting. I tried the above in an interactive python session and never saw 
the problem at all.  So then I added the logger to my program code and watched 
it go. Using the program, I still get the segfault. It happens on the same 
file; I removed it and then it happens on the next file in the list. 

It looks like it (python process) has no more space, but the machine has plenty 
of disk space and available memory.

I didn't think about the number of files, I'll look into that.
For now, I'm just using subprocess and the zip command, which is working.

thanks,
--Tim
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: User Interface Suggestions? (newbie)

2016-10-06 Thread mbg1708

On Wednesday, 5 October 2016 14:10:21 UTC+1, Beverly Howard  wrote:
> I'm new to Python, but have three decades of experience with FoxPro and VFP 
> plus I started programming in Basic and still comfortable with that.
> 
> I have spent some time with Python and am now fairly familiar with the syntax 
> and functions, but building a user interface has me stopped.
> 
> A primary question would be, "What are options for building a display that 
> would update displayed values without scrolling?"
> 
> My first goal is to build a program that would interface with a Raspberry Pi 
> to control a heating process.
> 
> Thanks in advance,
> Beverly Howard

I've had great results using glade and python3.

Glade provides a graphical tool for designing GUI windows.  Once the GUI is 
designed, glade writes out an XML file which can be read by either C or Python 
programs.

Reading the XML from python3 requires that you import gi.repository.  The 
python code after that is straightforward.  

(See http://python-gtk-3-tutorial.readthedocs.io/en/latest/builder.html)
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue28379] PyUnicode_CopyCharacters could lead to undefined behaviour

2016-10-06 Thread Xiang Zhang


Changes by Xiang Zhang :


Added file: http://bugs.python.org/file44990/PyUnicode_CopyCharacters.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue28379] PyUnicode_CopyCharacters could lead to undefined behaviour

2016-10-06 Thread Xiang Zhang


Changes by Xiang Zhang :


Removed file: http://bugs.python.org/file44989/PyUnicode_CopyCharacters.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: BeautifulSoup help !!

On 10/06/2016 11:34 AM, Navneet Siddhant wrote:
> I guess I will have to extract data from multiple divs as only
> extracting data from the parent div which has other divs in it with
> the different data is coming up all messed up. Will play around and
> see if I could get through it. Let me clarify once again I dont need
> complete code , a resource where I could find more info about using
> Beautifulsoup will be appreciated.  Also do I need some kind of
> plugin etc to extract data to csv ? or it is built in python and I
> could simply import csv and write other commands needed ??

Writing CSV files from Python is relatively easy.  Though you may need
to understand basic Python data types first, such as lists and dicts.
The module itself is mostly documented here:

https://docs.python.org/3/library/csv.html

And there are numerous examples of its use:
https://pymotw.com/2/csv/
https://www.getdatajoy.com/examples/python-data-analysis/read-and-write-a-csv-file-with-the-csv-module

To name but two of the first of many google search results.

Sounds to me like you need to spend some time learning basic Python data
types and how to iterate through them (lists and dicts mainly).  BS4
uses both lists and dicts for nearly everything.  An hour or two should
be enough to get a handle on this.  The nice thing about Python is you
can build things and run them in an incremental, and mostly interactive
way.  Regularly print out things so you can see what structure the data
has.  BS4 is very good about string representations of all its
structures you can print them out without knowing anything about them.
For example, if you were searching for a tag:

results = soup.find('a', attrs = {'data-search-key': "name" })

you can just do:
print (results)

And easily see how things are nested. Then you can use that to drill
down using list indexing to get just the part you need.

I suspect if they hire you and you work more on Python you'll grow to
really like it.

I suppose that Alister expressed consternation because much of what you
ask can be solved with some good old fashioned searching of google and
is no different from learning any language, including C# which you
already know.  Python is not C# of course, but the basic principles
behind programming are nearly universally-applied to nearly all
programming languages.  For a programmer, it shouldn't be too hard to
move from language to language.  Except for some of the more idiomatic
things about a language, many aspects are syntactically equivalent.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue28379] PyUnicode_CopyCharacters could lead to undefined behaviour

2016-10-06 Thread Xiang Zhang


New submission from Xiang Zhang:

Currently PyUnicode_CopyCharacters doesn't check arguments thoroughly. This 
could lead to undefined behaviour or crash in debug mode. For example, 
from_start > len(from), how_many < 0. Another case is that when how_many > 
len(from), it will choose len(from) but this can still fail since from_start 
can > 0. The doc of it is also not perfect, it does not necessarily return 0 on 
success.

--
components: Interpreter Core
files: PyUnicode_CopyCharacters.patch
keywords: patch
messages: 278202
nosy: haypo, serhiy.storchaka, xiang.zhang
priority: normal
severity: normal
stage: patch review
status: open
title: PyUnicode_CopyCharacters could lead to undefined behaviour
type: behavior
versions: Python 3.5, Python 3.6, Python 3.7
Added file: http://bugs.python.org/file44989/PyUnicode_CopyCharacters.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27392] Add a server_side keyword parameter to create_connection


Yury Selivanov added the comment:

AFAICT this issue was resolved in https://github.com/python/asyncio/pull/378. 
Closing this one. Thanks, Jim!

--
resolution:  -> fixed
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27759] selectors incorrectly retain invalid file descriptors

2016-10-06 Thread Roundup Robot


Roundup Robot added the comment:

New changeset 8cc1fca83fb8 by Yury Selivanov in branch '3.4':
Issue #27759: Fix selectors incorrectly retain invalid file descriptors.
https://hg.python.org/cpython/rev/8cc1fca83fb8

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27386] Asyncio server hang when clients connect and immediately disconnect


Yury Selivanov added the comment:

Alright, I've backported the fix to 3.4. Closing this.

--
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: segfault using shutil.make_archive

2016-10-06 Thread Random832

On Thu, Oct 6, 2016, at 13:45, Random832 wrote:
> On Thu, Oct 6, 2016, at 12:46, Tim wrote:
> > I need to zip up a directory that's about 400mb.
> > I'm using shutil.make_archive and I'm getting this response:
> > 
> > Segmentation fault: 11 (core dumped)
> > 
> > The code is straightforward (and works on other, smaller dirs):
> 
> Are you able to make a test case that reproduces it reliably without any
> specifics about what data you are using (e.g. can you generate a
> directory full of blank [or /dev/urandom if the compressed size is a
> factor] files that causes this)? Is it a large number of files or a
> large total size of files?

Also consider passing a logging object to make_archive and see if it
prints out anything interesting before the segfault.

import shutil
import logging
import sys
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(logging.StreamHandler(sys.stderr))
shutil.make_archive(..., logger=logger)
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: segfault using shutil.make_archive

2016-10-06 Thread Random832

On Thu, Oct 6, 2016, at 12:46, Tim wrote:
> I need to zip up a directory that's about 400mb.
> I'm using shutil.make_archive and I'm getting this response:
> 
> Segmentation fault: 11 (core dumped)
> 
> The code is straightforward (and works on other, smaller dirs):

Are you able to make a test case that reproduces it reliably without any
specifics about what data you are using (e.g. can you generate a
directory full of blank [or /dev/urandom if the compressed size is a
factor] files that causes this)? Is it a large number of files or a
large total size of files?
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!

I guess I will have to extract data from multiple divs as only extracting data 
from the parent div which has other divs in it with the different data is 
coming up all messed up. Will play around and see if I could get through it. 
Let me clarify once again I dont need complete code , a resource where I could 
find more info about using Beautifulsoup will be appreciated.  Also do I need 
some kind of plugin etc to extract data to csv ? or it is built in python and I 
could simply import csv and write other commands needed ??
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread Steve D'Aprano

On Fri, 7 Oct 2016 04:06 am, mr.puneet.go...@gmail.com wrote:

> Hi
> 
> I just started learning python. Is there any way to call functions in
> different way ?
> 
> Rather calling obj.function(arg1, arg2) I would like to call like below
> 
> "obj function arg1 arg2"

No. This will be a syntax error.

> this function is part of a class.
> 
> class myClass:
> def function(arg1, arg2):
>  # do something
> 
> Is it possible to do in python ? May be not directly but using some other
> methods.

The only way to do this will be to write your own pre-processor, which will
parse your source code, and translate it from your language to valid
Python. That's a lot of work for very little value -- I recommend you just
learn the Python syntax rather than trying to force it to be something it
is not.

-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: A newbie doubt on methods/functions calling

2016-10-06 Thread Peter Otten

mr.puneet.go...@gmail.com wrote:

> Hi
> 
> I just started learning python. Is there any way to call functions in
> different way ?
> 
> Rather calling obj.function(arg1, arg2) I would like to call like below
> 
> "obj function arg1 arg2"

How would the machine reading the above know that you didn't mean

obj(function, arg1, arg2)

or

obj.function.arg1(arg2)

?
 
> this function is part of a class.
> 
> class myClass:
> def function(arg1, arg2):
>  # do something
> 
> Is it possible to do in python ? May be not directly but using some other
> methods.

No, that would be a different language, and if someone were to implement it 
this would make you a newbie in two languages -- hardly an improvement ;)

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!

On Fri, Oct 7, 2016 at 4:00 AM, Navneet Siddhant
 wrote:
> I guess I shouldnt have mentioned as this was a recruitment task. If needed I 
> can post a screenshot of the mail I got which says I can take help from 
> anywhere possible as long as the assignment is done. Wont be simply copying 
> pasting the code as question related to the same will be asked in the 
> interview.
> I just need a proper understanding as to what I need to do to get the results.
> Also how to export the result to csv format.

A screenshot isn't necessary - we trust you to not flat-out lie to us.
(And if you did, well, this is a public list, so your deception would
burn you pretty thoroughly once someone finds out.) Stating that you
can take help from anywhere would have been a good clarification.

Anyway.

One of the annoying facts of the real world is that web scraping is
*hard*. We have awesome tools like Beautiful Soup that save us a lot
of hassle, but ultimately, you have to look at the results and figure
out which parts are "interesting" (that is, the parts that have the
data you want, or tell you about its structure, or something like
that). I strongly recommend messing with bs4 at the interactive
prompt; basically, just play around with everything you get hold of.
Eventually, you want to be building up a series of rows, where each
row is a list of column values; you write a row to the CSV file, and
your job's done. Most likely, you're going to have some sort of
primary loop - outside of that loop you have bs4 navigation to get you
the main block of info, and inside it, you parse through some wad of
stuff to find the truly interesting info, and at the bottom of the
loop, you write something to the CSV file.

Hope that's of some help!

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!

On Fri, Oct 7, 2016 at 3:38 AM, Steve D'Aprano
 wrote:
> On Fri, 7 Oct 2016 03:00 am, Chris Angelico wrote:
>
>> You are asking
>> for assistance with something that was assigned to you *as a
>> recruitment task*. Were you told that asking for help was a legitimate
>> solution?
>
> Why should he need to be told that? Asking for help *is* a legitimate
> solution, just as legitimate as Reading The Fine Manual, searching on
> Google, asking your work peers for mentoring, or doing a training course.

Let me clarify a bit.

Is "asking someone else to write your code" acceptable? I would say
no, in the same way that getting someone else to do your job interview
for you is inappropriate.

You're right, though, "asking for help" was too broad a description
for the situation. Apologies to the OP; of course you can ask
questions and get help. You just need to figure out where the
boundaries are between "stuff you know and are being hired for",
"stuff you don't know and are being hired for, and are learning on the
job", and "stuff you don't know and don't need to know, and can get
someone else to do for you". The boundary between the first two is not
overly important, but if there's not much in the first category,
you're going to be very much in over your head. It's the boundary
between the latter two that's more important, and which determines the
kind of help you're asking for.

But please don't accuse us of "trolling around" when you're asked this
kind of thing.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: User Interface Suggestions? (newbie)

2016-10-06 Thread Grant Edwards

On 2016-10-06, Steve D'Aprano  wrote:
> On Thu, 6 Oct 2016 10:03 pm, BartC wrote:
>
>> I'd quite like to know too. However I've just tried a test sequence
>> ("[P1d" to move the cursor to row 1) and it didn't work. If there's
>> reason why something so basic won't work (hence the need for curses etc)
>> then that would be useful to know too. (And how does curses manage it?!)
>
> I believe that curses uses a large database of terminal types and
> associated escape sequences. So when you start it up, it determines
> what terminal type you have (that's black magic itself) and then
> works out what escape sequences you need.

Nothing magic, it just looks at the 'TERM' enviroment varible and
looks up that value in the terminfo database.

-- 
Grant Edwards   grant.b.edwardsYow! I wonder if there's
  at   anything GOOD on tonight?
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!

On Thursday, October 6, 2016 at 8:52:18 PM UTC+5:30, Navneet Siddhant wrote:
> So I've just started up with python and an assignment was given to me by a 
> company as an recruitment task.
> 
> I need to web scrap the coupons of all the websites available on 
> http://www.couponraja.in and export it to csv format. 
> The details which I need to be present in the csv are the coupon title , 
> vendor , validity , description/detail , url to the vendor , image url of the 
> coupon.
> 
> I have gone through many tutorials on beautifulsoup and have a beginners 
> understanding of using it. Wrote a code as well , but the problem Im facing 
> here is when i collect info from the divs which contains all those info , Im 
> getting it in with all the html tags and the info is clustered. 
> 
> Code m using :
> 
> import requests
> from bs4 import BeautifulSoup
>  
> url = "https://www.couponraja.in/amazon;
> r = requests.get(url)
> soup = BeautifulSoup(r.content)
> g_data = soup.find_all("div", {"class": "nw-offrtxt"})
> for item in g_data:
> print item.contents
> 
> also will need help on how to export the info to csv format , I just know I 
> need to import csv then write the information to a csv file.
> But not getting through on how to achieve that.
> 
> Any help will be appreciated.

Thanx for the support Steve . N yes you can call me Sid ..  :) . . .. 


I guess I shouldnt have mentioned as this was a recruitment task. If needed I 
can post a screenshot of the mail I got which says I can take help from 
anywhere possible as long as the assignment is done. Wont be simply copying 
pasting the code as question related to the same will be asked in the interview.
I just need a proper understanding as to what I need to do to get the results.
Also how to export the result to csv format.
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue28365] idle forgets that saved console session is not a python file after restart

2016-10-06 Thread A.J.


A.J. added the comment:

There is no other way to explain it without fail every time no matter what code 
I write there is always syntax error highlighted on the five in the version 
banner.

--
Added file: http://bugs.python.org/file44988/2016-10-06 (4).png

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

A newbie doubt on methods/functions calling

2016-10-06 Thread mr . puneet . goyal

Hi 

I just started learning python. Is there any way to call functions in different 
way ?

Rather calling obj.function(arg1, arg2) I would like to call like below 

"obj function arg1 arg2" 

this function is part of a class.

class myClass:
def function(arg1, arg2):
 # do something

Is it possible to do in python ? May be not directly but using some other 
methods. 

Regards, Puneet
-- 
https://mail.python.org/mailman/listinfo/python-list

[issue28376] rangeiter_new fails when creating a range of step 0

2016-10-06 Thread Serhiy Storchaka


Serhiy Storchaka added the comment:

Good point Naoki. I think we can remove tp_new method from range_iterator in 
3.7. Seems it is never used.

The patch LGTM for 3.5-3.6, but the test should be marked as CPython 
implementation detail (@support.cpython_only).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

Re: User Interface Suggestions? (newbie)

2016-10-06 Thread Grant Edwards

On 2016-10-06, BartC  wrote:

> All this advice seems to be getting out of hand, with suggestions of 
> 'curses' and 'blessings' and using GUI. I've tried 'ncurses' elsewhere 
> and it was over the top for what I wanted to do.
>
> The OP wants to runs on Pi which I think runs Linux.

It can run Linux.  Whether the OP's is or not, I don't know.

> So all they are asking is, is there a way of randomly positioning
> the cursor within the terminal window so that the next output is at
> that position.

Yes.

> Something like an escape sequence. Terminal screens have been around
> a long time, you'd think someone would have had such a requirement
> before!

Yes, they did.  They wrote the curses library.

> I'd quite like to know too. However I've just tried a test sequence
> ("[P1d" to move the cursor to row 1) and it didn't work.

That doesn't look like an ANSI terminal escape sequence to me -- but I
have no idea what terminal type you're using, so I can't tell you
whether it's right or not.

The ANSI seqeunce to move the cursor to row 1, column 1, is
'[1;1m'.  Both positions default to 1, so '[m' does the same
thing.

> If there's reason why something so basic won't work (hence the need
> for curses etc) then that would be useful to know too. (And how does
> curses manage it?!)

Curses knows how to deal with various different terminal types and it
also knows about the intracacies of the Unix tty API.

If you want to, you can just assume your terminal uses ANSI escape
sequences:

  https://en.wikipedia.org/wiki/ANSI_escape_code

That will mostly work on most terminals you run into these days.

If you want to go one step further, you can use the terminfo library
to deal with different terminal types, but I have no idea how to use
it without ncurses.  If people care about their programs working with
different terminal types, they use ncurses.

-- 
Grant Edwards   grant.b.edwardsYow! !  Up ahead!  It's a
  at   DONUT HUT!!
  gmail.com

-- 
https://mail.python.org/mailman/listinfo/python-list

segfault using shutil.make_archive

2016-10-06 Thread Tim

I need to zip up a directory that's about 400mb.
I'm using shutil.make_archive and I'm getting this response:

Segmentation fault: 11 (core dumped)

The code is straightforward (and works on other, smaller dirs):

shutil.make_archive(os.path.join(zip_dir, zname), 'zip', tgt_dir)

I guess I could drop the convenience of make_archive and use zipfile but that 
seems to be exactly what make_archive does.

I'm on FreeBSD, python2.7.

Anyone seen this before?

thanks,
--tim


-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!

2016-10-06 Thread Steve D'Aprano

On Fri, 7 Oct 2016 03:00 am, Chris Angelico wrote:

> You are asking
> for assistance with something that was assigned to you *as a
> recruitment task*. Were you told that asking for help was a legitimate
> solution?

Why should he need to be told that? Asking for help *is* a legitimate
solution, just as legitimate as Reading The Fine Manual, searching on
Google, asking your work peers for mentoring, or doing a training course.

What's next? Are we going to start dumping on people for reading the
manual? "If you can't intuit the right method calls, you're cheating"?

"What, you had to *study* to learn the language? With a teacher and
everything? Cheater!"

Asking questions and sharing expertise is what we're here for, and it is an
absolutely critical part of open-source communities. The idea that people
cannot ask for help because they are working, or being interviewed for
work, should be anathema to us all.

Schooling is slightly different in that (1) whether we like it or not, many
schools enforce ridiculously stupid and strict rules against plagiarism, so
bizarrely the best way we can help students is to refuse to help them[1];
and (2) when people are learning, making their own mistakes and wrong turns
is often a good learning exercise. For students, it is the learning process
itself which is more important than the solution -- and the less
experienced the student, the more that is the case.

But heaven forbid that students copy recipes from the documentation, from
the ActiveState Cookbook, or Stackoverflow without attribution like
professionals do. The current anti-plagiarism (so-called plagiarism)
climate goes directly against the principles of free and open code and
information. Requiring every trivial thought and phrase to be wholly
original[2] or else paid for (whether paid for in money or in credit/
reputation points) is a direct assault against the intellectual Commons.

[1] The negative effects of an accusation of plagiarism for few lines of
allegedly copied code or text may be far, far worse than if the student
actually learns nothing. If you learn nothing, there's at least a chance
that your teacher will grade on a curve and you'll squeeze in a passing
grade, but allegedly plagiarise, even *your own work*[3] and you may lose
all your academic future.

[2] An impossibility.

[3] Quite possibly the stupidest thing that has come out of academia since
post-modernism.

-- 
Steve
“Cheer up,” they said, “things could be worse.” So I cheered up, and sure
enough, things got worse.

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Is it possible to use 'groupby' asynchronously?

2016-10-06 Thread Ian Kelly

On Thu, Oct 6, 2016 at 9:57 AM, Chris Angelico  wrote:
> On Thu, Oct 6, 2016 at 11:09 PM, Frank Millman  wrote:
>> Hi all
>>
>> I have used itertools.groupby before, and I love it. I used it to process a
>> csv file and 'break' on change of a particular field. It worked very well.
>>
>> Now I want to use it to process a database table. I can select the rows in
>> the desired sequence with no problem. However, I am using asyncio, so I am
>> reading the rows asynchronously.
>>
>> Before I spend hours trying to figure it out, can anyone confirm if this is
>> doable at all, or is groupby not designed for this.
>
> Most of itertools is going to assume synchronous iterables. However,
> the docs give pure-Python equivalents for quite a few:
>
> https://docs.python.org/3/library/itertools.html#itertools.groupby
>
> You may be able to tweak that into an actual "itertools.agroupby" or
> "aitertools.groupby".

There is already a third-party "aitertools" library, so there should
be no need to write one from scratch.

https://pypi.python.org/pypi/aitertools/0.1.0
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Python and ssh for remote login

2016-10-06 Thread Noah

On 6 Oct 2016 04:56, "Michael Torrie"  wrote:
>
> On 10/05/2016 11:46 AM, Noah wrote:
> > Hello folk,
> >
> > I would like to use a python script to ssh into a server using a
username
> > and password and perhaps ass port.
> >
> > Any ideas on how to script that.
>
> If paramiko doesn't fit your needs, traditionally this sort of work was
> done with the pexpect module for drying a TTY. There is a submodule of
> pexpect called pxssh for automating things.
>

Hi Micheal

Thank youn for your suggestion. I played around with paramiko today and the
results are owesome.

> http://pexpect.readthedocs.io/en/stable/api/pxssh.html
>
> Note that pexpect uses your normal ssh binary.  Paramiko is a complete
> implementation of the ssh protocol in python.  Both modules are useful
> and fill certain needs.

So i  am going to also try pexpect and everything pexpect and i will let
you know.

Thank you so much

Noah
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!

On Thursday, October 6, 2016 at 9:57:46 PM UTC+5:30, Navneet Siddhant wrote:
> On Thursday, October 6, 2016 at 9:42:47 PM UTC+5:30, Steve D'Aprano wrote:
> > On Fri, 7 Oct 2016 02:30 am, alister wrote:
> > 
> > > On Thu, 06 Oct 2016 08:22:05 -0700, desolate.soul.me wrote:
> > > 
> > >> So I've just started up with python and an assignment was given to me by
> > >> a company as an recruitment task.
> > >>
> > > so by your own admission you have just started with python yet you
> > > consider your self suitable for employment?
> > 
> > What's your problem Alister? Do you think that junior devs aren't allowed to
> > ask for help?
> > 
> > Desolate.Soul.Me has either applied for a job, and their interview test
> > is "do this task using Python", or he's been accepted in a new job, and the
> > same applies.
> > 
> > Whether it's a learning exercise, a test of skill + initiative, or actual
> > work given to a junior developer, Desolate.Soul.Me is perfectly entitled to
> > ask for help.
> > 
> > This isn't some artificially constrained academic homework, with stupidly
> > strict and hypocritical rules about so-called "plagiarism". This is the
> > real world where you take all the help you can get and you shouldn't feel
> > ashamed for asking for help. ESPECIALLY in the open source world, including
> > Python, where one of the community values is to share expertise.
> > 
> > My own employer has hired plenty of junior developers and given them
> > relatively minor tasks to do as a learning exercise. We're not going to
> > trust a junior developer with a critical piece of code, but we might say:
> > 
> > "Scrape this website. Use Python. Here's the Python For Beginners
> > book. Here's the Python documentation, and a few more forums where
> > you can ask for help. If you get stuck, and aren't getting useful
> > answers from the forums, you can ask Lisa. But not today, as she's
> > busy doing a critical release and can't be disturbed."
> > 
> > 
> > P.S. Desolate.Soul.Me, you might be taken a bit more seriously if you give a
> > name, or at least a moniker or nick-name which is easier for others to
> > refer to you by. It doesn't have to be your birthname, or legal name. What
> > do your friends and workmates call you?
> > 
> > 
> > I don't know Beautiful Soup, so I'm afraid I can't help.
> > 
> > 
> > 
> > -- 
> > Steve
> > “Cheer up,” they said, “things could be worse.” So I cheered up, and sure
> > enough, things got worse.
> 
> Yes ... Im allowed to take help as this group and python forums and other 
> links were suggested to me by them. Have been trying since morning not 
> getting the desired results , posted here as last resort .. and Im ok if I 
> dont find what m seeking.  Though m getting close , lets see what I could 
> achieve.

Btw my institute has referred me for this job in the company who has further 
emailed me with the assignment to complete after which I will have face to face 
interviews. Im still wondering as why I was told to do a python assignment when 
nowhere in my CV i have mentioned about python nor it was taught to us in the 
curriculum 

-- 
https://mail.python.org/mailman/listinfo/python-list

Re: BeautifulSoup help !!