I have to say thank you very much for all your experience and hints. It helps me a lot just talking with real professionals! ;)
On 2021-08-19 23:32 "Stephen J. Turnbull" <turnbull.stephen...@u.tsukuba.ac.jp> wrote: > The remaining question is "how many chunks?" If that's relevant, ISTM > a few simple experiments will show where the sweet spot is. Try a > queue of 64 chunks, then 128 chunks, and refine guesses from there. That brings me to another side-question which brings me to an unanswered bug-report https://bugs.python.org/issue44901 This was reported as a bug by me because I see this as a problem with documentation. I simply tried to understand how processes transfering data between each other. I know they pickle. But how exactly? Which pickle protocol they use by default? Do they decide the protocol depending on the type/kind/structure of data? Do they compress the pickled data? e.g. I read a PEP about pickle version 5 which is relevant for large data like pandas.DataFrames. Knowing more about this would help me to understand the needed ressources for creating a process and transfering data to and from it. The RAM itself does not take into account. We assume here I have enough RAM. _______________________________________________ Python-ideas mailing list -- python-ideas@python.org To unsubscribe send an email to python-ideas-le...@python.org https://mail.python.org/mailman3/lists/python-ideas.python.org/ Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/K5I32ZKQIPHNX5OOWJ4IVKQ5A3ZA44DZ/ Code of Conduct: http://python.org/psf/codeofconduct/
