Re: migrating gearpump-runner to new DoFn fails with NotSerializableException

2016-10-30 Thread Manu Zhang
Thanks Kenn. That is indeed the problem. Manu On Mon, Oct 31, 2016 at 12:12 PM Kenneth Knowles wrote: > Hi Manu, > > That class is generated by DoFnInvokers, which generates bytecode to > efficiently execute a DoFn. It should not be part of the serialized > payload,

Re: migrating gearpump-runner to new DoFn fails with NotSerializableException

2016-10-30 Thread Kenneth Knowles
Hi Manu, That class is generated by DoFnInvokers, which generates bytecode to efficiently execute a DoFn. It should not be part of the serialized payload, but should be instantiated on the service/worker/etc. If you are trying to serialize a DoFnInvoker, then my recommendation is to serialize

migrating gearpump-runner to new DoFn fails with NotSerializableException

2016-10-30 Thread Manu Zhang
Hi all, I'm migrating `OldDoFn` to `DoFn` in gearpump-runner. We serialize all the functions locally and ship to remote cluster. Hence, I try to make sure the functions are serializable. Unluckily, the integration-tests fail with `NotSerializableException` as follows. Anyone knows what that