I looked into CloudPickle a while back, and would be supportive of the change.
On Mon, Jul 1, 2019 at 11:06 PM Valentyn Tymofieiev <valen...@google.com> wrote: > > I have checked that cloudpickle (an alternative to dill) is able to pickle > and unpickle typing types on Python 3.5, 3.6, which seems to be a recent > change, see: > https://github.com/cloudpipe/cloudpickle/issues/63#issuecomment-501624383. > > I am evaluating cloudpickle as a potential avenue to address several other > issues we found in Beam while working on Python 3 support, such as: > > https://issues.apache.org/jira/browse/BEAM-6522 > https://issues.apache.org/jira/browse/BEAM-7284 > https://issues.apache.org/jira/browse/BEAM-5878?focusedCommentId=16834554 > https://github.com/uqfoundation/dill/issues/300 > https://issues.apache.org/jira/browse/BEAM-7540 > > Once I have more information on cloudpickle vs dill in Beam, I'll bring it to > the mailing list. > > On Wed, May 15, 2019 at 5:25 AM Robert Bradshaw <rober...@google.com> wrote: >> >> (2) seems reasonable. >> >> On Tue, May 14, 2019 at 3:15 AM Udi Meiri <eh...@google.com> wrote: >> > >> > It seems like pickling of typing types is broken in 3.5 and 3.6, fixed in >> > 3.7: >> > https://github.com/python/typing/issues/511 >> > >> > Here are my attempts: >> > https://gist.github.com/udim/ec213305ca865390c391001e8778e91d >> > >> > >> > My ideas: >> > 1. I know that we override type object handling in pickler.py >> > (_nested_type_wrapper), and perhaps this mechanism can be used to pickle >> > typing classes correctly. The question is how. >> > >> > 2. Exclude/stub out these classes when pickling a pipeline - they are only >> > used for verification during pipeline construction anyway. This could be a >> > temporary solution for versions 3.5 and 3.6. >> > >> > Any ideas / opinions?
