[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-10-01 Thread DanilZ
DanilZ added the comment: I think you have correctly estimated the problem in the last part of your message: "as it could possibly indicate an issue with running out of memory when the dataframe is converted to pickle format (which often increases the total size) within the process

[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-10-01 Thread DanilZ
DanilZ added the comment: . > On 1 Oct 2020, at 03:11, Kyle Stanley wrote: > > > Kyle Stanley added the comment: > > DanilZ, could you take a look at the superseding issue > (https://bugs.python.org/issue37297) and see if your exception raised within > the job is the same? > > If

[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-09-30 Thread Kyle Stanley
Kyle Stanley added the comment: DanilZ, could you take a look at the superseding issue (https://bugs.python.org/issue37297) and see if your exception raised within the job is the same? If it's not, I would suggest opening a separate issue (and linking to it in a comment here), as I don't

[issue37294] concurrent.futures.ProcessPoolExecutor state=finished raised error

2020-09-28 Thread DanilZ
DanilZ added the comment: After executing a single task inside a process the result is returned with state=finished raised error. Error happens when trying to load a big dataset (over 5 GB). Otherwise the same dataset reduced to a smaller nrows executes and returns from result() without