Re: Error when using @DefaultCoder(AvroCoder.class)
Tried some code search in Beam repo but I didn't find the exact line of code that throws your exception. However, I believe for Java Classes you used in primitives (ParDo, CombineFn) and coders, it's very likely you need to make them serializable (i.e. implements Serializable). -Rui On Wed, Jul 8, 2020 at 6:23 AM Kirill Zhdanovich wrote: > > Hi! > I'm using Apache Beam Java(2.19.0) with Dataflow. I created class and > annotated it with DefaultCoder > > @DefaultCoder(AvroCoder.class) > public class ProductCatalog { > > When I trying to submit it to cluster I get an error: > > Caused by: java.io.NotSerializableException: ...common.ProductCatalog > > If I add `implements Serializable` to the class definition everything works > fine. In the Apache Beam guide, I don't see anything about using implements > Serializable. What I'm doing wrong? Thank you in advance for your help
Re: Error when using @DefaultCoder(AvroCoder.class)
Can you provide the full stacktrace? On Wed, Jul 8, 2020 at 12:33 PM Rui Wang wrote: > Tried some code search in Beam repo but I didn't find the exact line > of code that throws your exception. > > However, I believe for Java Classes you used in primitives (ParDo, > CombineFn) and coders, it's very likely you need to make them > serializable (i.e. implements Serializable). > > > -Rui > > On Wed, Jul 8, 2020 at 6:23 AM Kirill Zhdanovich > wrote: > > > > Hi! > > I'm using Apache Beam Java(2.19.0) with Dataflow. I created class and > annotated it with DefaultCoder > > > > @DefaultCoder(AvroCoder.class) > > public class ProductCatalog { > > > > When I trying to submit it to cluster I get an error: > > > > Caused by: java.io.NotSerializableException: ...common.ProductCatalog > > > > If I add `implements Serializable` to the class definition everything > works fine. In the Apache Beam guide, I don't see anything about using > implements Serializable. What I'm doing wrong? Thank you in advance for > your help >
Re: Error when using @DefaultCoder(AvroCoder.class)
Hello Luke, I will try to get "22 more" soon, not sure how to it though 23:11:11.339 [ERROR] [system.err] Exception in thread "main" java.lang.IllegalArgumentException: unable to serialize DoFnWithExecutionInformation{doFn=com.ikea.search.ab.bootstrap.Job$CreateSessionMetrics@444f44c5, mainOutputTag=Tag:400#6929f09b03d242ca>, sideInputMapping={}, schemaInformation=DoFnSchemaInformation{elementConverters=[]}} 23:11:11.339 [ERROR] [system.err] at org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:55) 23:11:11.339 [ERROR] [system.err] at org.apache.beam.runners.core.construction.ParDoTranslation.translateDoFn(ParDoTranslation.java:606) 23:11:11.339 [ERROR] [system.err] at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator$1.translateDoFn(PrimitiveParDoSingleFactory.java:197) 23:11:11.339 [ERROR] [system.err] at org.apache.beam.runners.core.construction.ParDoTranslation.payloadForParDoLike(ParDoTranslation.java:738) 23:11:11.339 [ERROR] [system.err] at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.payloadForParDoSingle(PrimitiveParDoSingleFactory.java:193) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.translate(PrimitiveParDoSingleFactory.java:154) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.runners.core.construction.PTransformTranslation$KnownTransformPayloadTranslator.translate(PTransformTranslation.java:417) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.runners.core.construction.PTransformTranslation.toProto(PTransformTranslation.java:225) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:157) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:87) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317) 23:11:11.340 [ERROR] [system.err] at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:187) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:795) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:186) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) 23:11:11.341 [ERROR] [system.err] at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) 23:11:11.341 [ERROR] [system.err] at com.ikea.search.ab.bootstrap.Job.testPipeline(Job.java:185) 23:11:11.341 [ERROR] [system.err] at com.ikea.search.ab.bootstrap.Job.main(Job.java:196) 23:11:11.342 [ERROR] [system.err] Caused by: java.io.NotSerializableException: com.ikea.search.ab.common.ProductCatalog$Product 23:11:11.342 [ERROR] [system.err] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) 23:11:11.342 [ERROR] [system.err] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) 23:11:11.342 [ERROR] [system.err] at java.util.HashMap.internalWriteEntries(HashMap.java:1790) 23:11:11.342 [ERROR] [system.err] at java.util.HashMap.writeObject(HashMap.java:1363) 23:11:11.342 [ERROR] [system.err] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 23:11:11.342 [ERROR] [system.err] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 23:11:11.342 [ERROR] [system.err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 23:11:11.342 [ERROR] [system.err] at java.lang.reflect.Method.invoke(Method.java:498) 23:11:11.342 [ERROR] [system.err] at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1140) 23:11:11.343 [ERROR] [system.err] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496) 23:11:11.343 [ERROR] [system.err] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) 23:11:11.343 [ERROR] [system.err] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) 23:11:11.343 [ERROR] [system.err] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) 23:11:11.343 [ERROR] [system.err] at java.io.ObjectOutputStream.writeSerialData(ObjectOutpu
Re: Error when using @DefaultCoder(AvroCoder.class)
The problem is that CreateSessionMetrics has an instance of com.ikea.search.ab.common.ProductCatalog$Product as part of its serialization closure. Does it have a member variable of that type, or refer to one that is part of its parent's class, or ...? Should it be marked transient? Marking it as serializable is necessary if you want to keep it part of that class's closure and is unrelated to the usage of @DefaultCoder. Here are some articles[1, 2, 3] about Java serialization and what is captured that could help if you want additional background. 1: https://www.cs.unh.edu/~charpov/programming-serial-lambda.html 2: http://www.lifeisafile.com/Serialization-in-spark/ 3: https://stackoverflow.com/questions/40259196/understanding-sparks-closures-and-their-serialization On Wed, Jul 8, 2020 at 1:22 PM Kirill Zhdanovich wrote: > Hello Luke, > I will try to get "22 more" soon, not sure how to it though > > 23:11:11.339 [ERROR] [system.err] Exception in thread "main" > java.lang.IllegalArgumentException: unable to serialize > DoFnWithExecutionInformation{doFn=com.ikea.search.ab.bootstrap.Job$CreateSessionMetrics@444f44c5, > mainOutputTag=Tag:400#6929f09b03d242ca>, > sideInputMapping={}, > schemaInformation=DoFnSchemaInformation{elementConverters=[]}} > 23:11:11.339 [ERROR] [system.err] at > org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:55) > 23:11:11.339 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.ParDoTranslation.translateDoFn(ParDoTranslation.java:606) > 23:11:11.339 [ERROR] [system.err] at > org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator$1.translateDoFn(PrimitiveParDoSingleFactory.java:197) > 23:11:11.339 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.ParDoTranslation.payloadForParDoLike(ParDoTranslation.java:738) > 23:11:11.339 [ERROR] [system.err] at > org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.payloadForParDoSingle(PrimitiveParDoSingleFactory.java:193) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.translate(PrimitiveParDoSingleFactory.java:154) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.PTransformTranslation$KnownTransformPayloadTranslator.translate(PTransformTranslation.java:417) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.PTransformTranslation.toProto(PTransformTranslation.java:225) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:157) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:87) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317) > 23:11:11.340 [ERROR] [system.err] at > org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:187) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:795) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:186) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) > 23:11:11.341 [ERROR] [system.err] at > org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) > 23:11:11.341 [ERROR] [system.err] at > com.ikea.search.ab.bootstrap.Job.testPipeline(Job.java:185) > 23:11:11.341 [ERROR] [system.err] at > com.ikea.search.ab.bootstrap.Job.main(Job.java:196) > 23:11:11.342 [ERROR] [system.err] Caused by: > java.io.NotSerializableException: > com.ikea.search.ab.common.ProductCatalog$Product > 23:11:11.342 [ERROR] [system.err] at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > 23:11:11.342 [ERROR] [system.err] at > java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > 23:11:11.342 [ERROR] [system.err] at > java.util.HashMap.internalWriteEntries(HashMap.java:1790) > 23:11:11.342 [ERROR] [system.err] at > java.util.HashMap.writeObject(HashMap.java:1363) > 23:11:11.342 [ERROR] [system.err] at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Metho
Re: Error when using @DefaultCoder(AvroCoder.class)
Cool! Thanks a lot for your help, Luke. On Wed, 8 Jul 2020 at 23:45, Luke Cwik wrote: > The problem is that CreateSessionMetrics has an instance of > com.ikea.search.ab.common.ProductCatalog$Product as part of its > serialization closure. Does it have a member variable of that type, or > refer to one that is part of its parent's class, or ...? Should it be > marked transient? > > Marking it as serializable is necessary if you want to keep it part of > that class's closure and is unrelated to the usage of @DefaultCoder. > > Here are some articles[1, 2, 3] about Java serialization and what is > captured that could help if you want additional background. > > 1: https://www.cs.unh.edu/~charpov/programming-serial-lambda.html > 2: http://www.lifeisafile.com/Serialization-in-spark/ > 3: > https://stackoverflow.com/questions/40259196/understanding-sparks-closures-and-their-serialization > > > On Wed, Jul 8, 2020 at 1:22 PM Kirill Zhdanovich > wrote: > >> Hello Luke, >> I will try to get "22 more" soon, not sure how to it though >> >> 23:11:11.339 [ERROR] [system.err] Exception in thread "main" >> java.lang.IllegalArgumentException: unable to serialize >> DoFnWithExecutionInformation{doFn=com.ikea.search.ab.bootstrap.Job$CreateSessionMetrics@444f44c5, >> mainOutputTag=Tag:400#6929f09b03d242ca>, >> sideInputMapping={}, >> schemaInformation=DoFnSchemaInformation{elementConverters=[]}} >> 23:11:11.339 [ERROR] [system.err] at >> org.apache.beam.sdk.util.SerializableUtils.serializeToByteArray(SerializableUtils.java:55) >> 23:11:11.339 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.ParDoTranslation.translateDoFn(ParDoTranslation.java:606) >> 23:11:11.339 [ERROR] [system.err] at >> org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator$1.translateDoFn(PrimitiveParDoSingleFactory.java:197) >> 23:11:11.339 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.ParDoTranslation.payloadForParDoLike(ParDoTranslation.java:738) >> 23:11:11.339 [ERROR] [system.err] at >> org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.payloadForParDoSingle(PrimitiveParDoSingleFactory.java:193) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.runners.dataflow.PrimitiveParDoSingleFactory$PayloadTranslator.translate(PrimitiveParDoSingleFactory.java:154) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.PTransformTranslation$KnownTransformPayloadTranslator.translate(PTransformTranslation.java:417) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.PTransformTranslation.toProto(PTransformTranslation.java:225) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:157) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:87) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:665) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:657) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:317) >> 23:11:11.340 [ERROR] [system.err] at >> org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:251) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:460) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.runners.dataflow.DataflowPipelineTranslator.translate(DataflowPipelineTranslator.java:187) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:795) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.runners.dataflow.DataflowRunner.run(DataflowRunner.java:186) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.sdk.Pipeline.run(Pipeline.java:315) >> 23:11:11.341 [ERROR] [system.err] at >> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) >> 23:11:11.341 [ERROR] [system.err] at >> com.ikea.search.ab.bootstrap.Job.testPipeline(Job.java:185) >> 23:11:11.341 [ERROR] [system.err] at >> com.ikea.search.ab.bootstrap.Job.main(Job.java:196) >> 23:11:11.342 [ERROR] [system.err] Caused by: >> java.io.NotSerializableException: >> com.ikea.search.ab.common.ProductCatalog$Product >> 23:11:11.342 [ERROR] [system.err] at >> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) >> 23:11:11.342 [ERROR] [system.err] at >> java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) >> 23:11:11.342 [ERROR] [system.err] at >> java.util.HashMap.internalWriteEntries(HashMap.java:1790)