[ https://issues.apache.org/jira/browse/BEAM-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shashank Prabhakara updated BEAM-3482: -------------------------------------- Attachment: m.patch > Java serialiazation exception when using BigQueryIO > --------------------------------------------------- > > Key: BEAM-3482 > URL: https://issues.apache.org/jira/browse/BEAM-3482 > Project: Beam > Issue Type: Bug > Components: sdk-java-gcp > Affects Versions: 2.3.0 > Reporter: Shashank Prabhakara > Assignee: Chamikara Jayalath > Priority: Major > Labels: easyfix, newbie > Attachments: m.patch > > > When writing data to BQ using BigQueryIO, we get the following exception when > checkpointing. To reproduce, use BigQueryIO.writeTableRows() using > sufficiently large dataset with latest code from master branch. > The following stack trace is when using using ApexRunner: > ERROR com.datatorrent.stram.engine.StreamingContainer: Operator set > [OperatorDeployInfo[id=20,name=BigQueryIO.Write/PrepareWrite/ParDo(Anonymous)/ParMultiDo(Anonymous),type=GENERIC,checkpoint=\{ffffffffffffffff, > 0, 0},inputs=[OperatorDeployInfo.Inpu > tDeployInfo[portName=input,streamId=stream9,sourceNodeId=19,sourcePortName=output,locality=CONTAINER_LOCAL,partitionMask=0,partitionKeys=<null>]],outputs=[OperatorDeployInfo.OutputDeployInfo[portName=output,streamId=stream56,bufferServer=<null>]]]] > stopped running due to > an exception. > com.esotericsoftware.kryo.KryoException: Error during Java serialization. > Serialization trace: > doFn (org.apache.beam.runners.apex.translation.operators.ApexParDoOperator) > at > com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:33) > at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:523) > at > com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:61) > at > com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:495) > at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:599) > at > com.datatorrent.common.util.FSStorageAgent.store(FSStorageAgent.java:190) > at > com.datatorrent.common.util.AsyncFSStorageAgent.save(AsyncFSStorageAgent.java:101) > at com.datatorrent.stram.engine.Node.checkpoint(Node.java:521) > at com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:461) > at > com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.java:1428) > Caused by: java.io.NotSerializableException: > org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinations$SideInputAccessorViaProcessContext > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) > at > java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) > at > java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) > at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) > at > com.esotericsoftware.kryo.serializers.JavaSerializer.write(JavaSerializer.java:30) > ... 9 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)