date:20160708

[issue27181] Add geometric mean to `statistics` module

2016-07-08 Thread Raymond Hettinger


Raymond Hettinger added the comment:

I would like to see them spelled-out:  geometric_mean and harmonic_mean

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27181] Add geometric mean to `statistics` module

2016-07-08 Thread Steven D'Aprano


Steven D'Aprano added the comment:

Does anyone have any strong feeling about the name for these functions?

gmean and hmean;

geometric_mean and harmonic_mean

And "subcontrary_mean" is not an option :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27471] sre_constants.error: bad escape \d

2016-07-08 Thread Serhiy Storchaka


Serhiy Storchaka added the comment:

It is a deprection warning in 3.5.

$ python3.5 -Wd
>>> import re
>>> re.sub(re.escape(r'(\d+?)'), '(?:\d+?)', r'(\d+?)')
/usr/lib/python3.5/re.py:182: DeprecationWarning: bad escape \d
  return _compile(pattern, flags).sub(repl, string, count)
'(?:\\d+?)'

The warning was added in issue23622, and turned into exception in issue27030.

--
nosy: +serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

2016-07-08 Thread Connor Wolf


Connor Wolf added the comment:

> Python 3.5.1+ (default, Mar 30 2016, 22:46:26)

Whatever the stock 3.5 on ubuntu 16.04 x64 is.

I've actually been running into a whole horde of really bizarre issues related 
to what I /think/ is locking in stdout. 

Basically, I have a context where I have thousands and thousands of (relatively 
short lived) `multiprocessing.Process()` processes, and over time they all get 
wedged (basically, I have ~4-32 processes alive at any time, but they all get 
recycled every few minutes).

After doing some horrible 
(https://github.com/fake-name/ReadableWebProxy/blob/master/logSetup.py#L21-L78) 
hackery in the logging module, I'm not seeing processes get wedged there, but I 
do still encounter issues with what I can only assume is a lock in the print 
statement. I'm hooking into a wedged process using 
[pystuck](https://github.com/alonho/pystuck)

durr@rwpscrape:/media/Storage/Scripts/ReadableWebProxy⟫ pystuck --port 6675
Welcome to the pystuck interactive shell.
Use the 'modules' dictionary to access remote modules (like 'os', or '__main__')
Use the `%show threads` magic to display all thread stack traces.

In [1]: show threads
<_MainThread(MainThread, started 140574012434176)>
  File "runScrape.py", line 74, in 
go()
  File "runScrape.py", line 57, in go
runner.run()
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 453, 
in run
living = sum([manager.check_run_jobs() for manager in managers])
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 453, 
in 
living = sum([manager.check_run_jobs() for manager in managers])
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 364, 
in check_run_jobs
proc.start()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/context.py", line 267, in _Popen
return Popen(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
  File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
self.run()
  File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 145, 
in run
run.go()
  File "/media/Storage/Scripts/ReadableWebProxy/WebMirror/Runner.py", line 101, 
in go
self.log.info("RunInstance starting!")
  File "/usr/lib/python3.5/logging/__init__.py", line 1279, in info
self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.5/logging/__init__.py", line 1415, in _log
self.handle(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 1425, in handle
self.callHandlers(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 1487, in callHandlers
hdlr.handle(record)
  File "/usr/lib/python3.5/logging/__init__.py", line 855, in handle
self.emit(record)
  File "/media/Storage/Scripts/ReadableWebProxy/logSetup.py", line 134, in emit
print(outstr)


  File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.5/dist-packages/rpyc/utils/server.py", line 241, 
in start
self.accept()
  File "/usr/local/lib/python3.5/dist-packages/rpyc/utils/server.py", line 128, 
in accept
sock, addrinfo = self.listener.accept()
  File "/usr/lib/python3.5/socket.py", line 195, in accept
fd, addr = self._accept()


  File "/usr/local/lib/python3.5/dist-packages/pystuck/thread_probe.py", line 
15, in thread_frame_generator
yield (thread_, frame)


So, somehow the print() statement is blocking, which I have /no/ idea how to go 
about debugging. I assume there's a lock /in/ the print statement function 
call, and I'm probably going to look into wrapping both the print() call and 
the multiprocessing.Process() call  execution in a single, shared 
multiprocessing lock, but that
seems like a very patchwork solution to something that should just work.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

2016-07-08 Thread Gregory P. Smith


Gregory P. Smith added the comment:

My intent is not to block anything.  I'm Just explaining why I'm not motivated 
to spend much time on this issue myself.  Others are welcome to.

subprocess is not related to this issue, it has been fixed for use with threads 
(in 3.2 and higher) with an extremely widely used drop in replacement back-port 
for 2.7 https://pypi.python.org/pypi/subprocess32.  But even 2.7's poor 
subprocess implementation never triggered this specific issue in the first 
place (unless someone logged from a pre_exec_fn which would be a laughable 
thing to do anyways).

multiprocessing: It has spawn (as of 3.4) and forkserver methods both of which 
can help avoid this issue.  Caveats: spawn understandably has negative 
performance implications and forkserver requires the forkserver to be forked 
before threads potentially holding locks have been started.

As for the gross hacky monkey patching workaround: That was the approach I took 
in 
https://github.com/google/python-atfork/blob/master/atfork/stdlib_fixer.py#L51

Definitely a hack, but one that does work on existing interpreters.

Conner & lesha: Which Python version(s) are you using?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

2016-07-08 Thread lesha


lesha added the comment:

On a moment's reflection, a lot of my prior comment is wrong. Sorry about that. 

- glog does not, that I know of, sanitize locks on fork. You just shouldn't log 
after fork but before exec.

- Using `pthread_atfork` to clean up the `logging` lock might be enough to make 
it safe from the "just forked" context, but without adding fairly exhaustive 
tests around this logic, it would be fragile with respect to further 
improvements to `logging`. So even just making this one library safe is a 
considerable amount of work.

So I retract most of my previous opinion. Sorry.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6721] Locks in the standard library should be sanitized on fork

2016-07-08 Thread lesha

lesha added the comment:

I work on a multi-million-line C++ codebase that uses fork() from multithreaded
programs all over the place. We use `glog` ubiquitously.

This bug here that spans 6 years and has affected dozens of people
(conservatively) simply does not exist for us. That is because glog takes the
pragmatic approach of sanitizing its mutex on fork:

https://github.com/google/glog/blob/4d391fe692ae6b9e0105f473945c415a3ce5a401/src/base/mutex.h#L249

In my opinion, "thou shalt never fork() in a threaded program" is impractical
purism. It is good to be aware of the dangers that lie therein, but it is
completely possible to safely spawn **subprocesses** from multithreaded
programs on modern OSes like Linux.

Python's subprocess **ought** to be safe to use in threaded programs. Any
issues with this (aside from `pre_exec_fn`, obviously) are bugs in
`subprocess`.

Here is a C++ implementation of the concept that can be safely used in threaded
programs:

https://github.com/facebook/folly/blob/master/folly/Subprocess.cpp

Note that unlike Python's subprocess `pre_exec_fn`, the C++ analog is very loud
in warning you about the scary world in which your closure will execute:

https://github.com/facebook/folly/blob/master/folly/Subprocess.h#L252

The point to my message is simple: there is a pragmatic way to save hundreds of
debugging hours for users of Python. People are going to find it necessary to
do such things from time to time, so instead of telling them to redesign their
programs, it is better to give them a safer tool.

59 matches

Mail list logo