[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519 ] Aaron Dixon edited comment on BEAM-8388 at 1/16/20 9:51 PM: Just to add more context/help here. The exception I see after upgrading from Beam 2.16->2.17 is this (running in Dataflow): {code:java} java.lang.NoClassDefFoundError: org/apache/avro/data/TimeConversions$TimestampConversion # at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder (KafkaUnboundedSource.java:131) at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator (WorkerCustomSources.java:419) at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop (ReadOperation.java:178) at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start (ReadOperation.java:159) at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute (MapTaskExecutor.java:77) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process (StreamingDataflowWorker.java:1320) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000 (StreamingDataflowWorker.java:151) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run (StreamingDataflowWorker.java:1053) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.avro.data.TimeConversions$TimestampConversion # at java.net.URLClassLoader.findClass (URLClassLoader.java:381) at java.lang.ClassLoader.loadClass (ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code} # ## ## ``` was (Author: atdixon): Just to add more context/help here. The exception I see after upgrading from Beam 2.16->2.17 is this (running in Dataflow): ``` java.lang.NoClassDefFoundError: org/apache/avro/data/TimeConversions$TimestampConversion # ## at org.apache.beam.sdk.coders.AvroCoder. ([AvroCoder.java:269|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=269=flow-p1-9c38=1]) ## at org.apache.beam.sdk.coders.AvroCoder.of ([AvroCoder.java:121|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=121=flow-p1-9c38=1]) ## at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder ([KafkaUnboundedSource.java:131|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.io.kafka%2FKafkaUnboundedSource.java=131=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator ([WorkerCustomSources.java:419|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FWorkerCustomSources.java=419=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop ([ReadOperation.java:178|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=178=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start ([ReadOperation.java:159|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=159=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute ([MapTaskExecutor.java:77|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java=77=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process ([StreamingDataflowWorker.java:1320|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1320=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519 ] Aaron Dixon edited comment on BEAM-8388 at 1/16/20 9:51 PM: Just to add more context/help here. The exception I see after upgrading from Beam 2.16->2.17 is this (running in Dataflow): {code:java} java.lang.NoClassDefFoundError: org/apache/avro/data/TimeConversions$TimestampConversion # at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder (KafkaUnboundedSource.java:131) at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator (WorkerCustomSources.java:419) at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop (ReadOperation.java:178) at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start (ReadOperation.java:159) at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute (MapTaskExecutor.java:77) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process (StreamingDataflowWorker.java:1320) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000 (StreamingDataflowWorker.java:151) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run (StreamingDataflowWorker.java:1053) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.avro.data.TimeConversions$TimestampConversion # at java.net.URLClassLoader.findClass (URLClassLoader.java:381) at java.lang.ClassLoader.loadClass (ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code} was (Author: atdixon): Just to add more context/help here. The exception I see after upgrading from Beam 2.16->2.17 is this (running in Dataflow): {code:java} java.lang.NoClassDefFoundError: org/apache/avro/data/TimeConversions$TimestampConversion # at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder (KafkaUnboundedSource.java:131) at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator (WorkerCustomSources.java:419) at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop (ReadOperation.java:178) at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start (ReadOperation.java:159) at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute (MapTaskExecutor.java:77) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process (StreamingDataflowWorker.java:1320) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000 (StreamingDataflowWorker.java:151) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run (StreamingDataflowWorker.java:1053) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.avro.data.TimeConversions$TimestampConversion # at java.net.URLClassLoader.findClass (URLClassLoader.java:381) at java.lang.ClassLoader.loadClass (ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code} # ## ## ``` > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016551#comment-17016551 ] Tomo Suzuki edited comment on BEAM-8388 at 1/16/20 5:25 AM: Note for myself: * Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? * What's Beam's dependency that does not work with Avro 1.9. Which Avro classes do they expect from which line of code? Is there a new version of the dependencies released to adopt Avro 1.9? Read the PR comment carefully. * Can Avro 1.9 read data written by Avro 1.8? was (Author: suztomo): Note for myself: * Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? * What's Beam's dependency that does not work with Avro 1.9. Which Avro classes do they expect? Is there a new version of the dependencies released to adopt Avro 1.9? Read the PR comment carefully. * Can Avro 1.9 read data written by Avro 1.8? > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016551#comment-17016551 ] Tomo Suzuki edited comment on BEAM-8388 at 1/16/20 5:25 AM: Note for myself: * Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? * What's Beam's dependency that does not work with Avro 1.9. Which Avro classes do they expect? Is there a new version of the dependencies released to adopt Avro 1.9? Read the PR comment carefully. * Can Avro 1.9 read data written by Avro 1.8? was (Author: suztomo): Note for myself: Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)