DanilZ added the comment:
Dear All,
Thanks for the great input. As described above it appears to be a MacOS problem.
--
___
Python tracker
<https://bugs.python.org/issue42
DanilZ added the comment:
Hi Ken,
Thanks for your comment.
Unfortunately at the time I can not upgrade to 3.8 to run this test. My whole
system depends on 3.7 and some peculiarities of 3.8 need to be dealt with.
It would be great if someone with OSX and 3.8 could test this out, otherwise
DanilZ added the comment:
I have managed to solve the problem by inserting in the beginning of my program:
import multiprocessing
multiprocessing.set_start_method('forkserver')
as this is explained here:
https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n
DanilZ added the comment:
Thank you so much for the input! I will study all the links you have sent:
Here is a screen recording of some additional experiments:
https://vimeo.com/user50681456/review/474733642/b712c12c2c
<https://vimeo.com/user50681456/review/474733642/b712c12
DanilZ added the comment:
FYI: I’ve tried all the three of the possible backends: ‘loky’ (default) /
’threading’ / ‘multiprocessing’. None of them solved the problem.
> On 2 Nov 2020, at 17:34, Ken Jin wrote:
>
> A temporary workaround might be to reduce n_jobs OR even better: use
DanilZ added the comment:
Here is a gif of what’s going on in my ActivityMonitor on a Mac while this code
is executed:
https://gfycat.com/unselfishthatgraysquirrel
<https://gfycat.com/unselfishthatgraysquirrel>
--
___
Python tracker
DanilZ added the comment:
Hi Ken, thanks for a quick reply.
Here are the requested specs.
System:
Python 3.7.6
OS X 10.15.7
Packages:
XGBoost 1.2.0
sklearn 0.22.2
pandas 1.0.5
numpy 1.18.1
I can see that you have reduced the code, which now excludes the
RandomizedSearchCV part
New submission from DanilZ :
Note: problem occurs only after performing the RandomizedSearchCV...
When applying a function in a multiprocess using concurrent.futures if the
function includes anything else other than print(), it is not executed and the
process freezes.
Here is the code
DanilZ added the comment:
I think you have correctly estimated the problem in the last part of your
message: "as it could possibly indicate an issue with running out of memory
when the dataframe is converted to pickle format (which often increases the
total size) within the pr
DanilZ added the comment:
.
> On 1 Oct 2020, at 03:11, Kyle Stanley wrote:
>
>
> Kyle Stanley added the comment:
>
> DanilZ, could you take a look at the superseding issue
> (https://bugs.python.org/issue37297) and see if your exception raised within
DanilZ added the comment:
After executing a single task inside a process the result is returned with
state=finished raised error.
Error happens when trying to load a big dataset (over 5 GB). Otherwise the same
dataset reduced to a smaller nrows executes and returns from result() without
11 matches
Mail list logo
