[ https://issues.apache.org/jira/browse/BEAM-9144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020512#comment-17020512 ]
Tomo Suzuki commented on BEAM-9144: ----------------------------------- My problem was also caused by NoClassDefFoundError: !NoClassDefFoundError in word-count-beam.png|width=525,height=247! I got another advice from Luke: I had to add {{--dataflowWorkerJar}} option to specify Dataflow runtime JAR when submitting the job with the new SDK. {noformat} suztomo@suxtomo24:~/word-count-beam$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="--runner=DataflowRunner --project=suztomo-hello-beam \ --gcpTempLocation=gs://suztomo-hello-beam/tmp2 \ --dataflowWorkerJar=/usr/local/google/home/suztomo/beam6/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.20.0-SNAPSHOT.jar \ --inputFile=gs://apache-beam-samples/shakespeare/* --output=gs://suztomo-hello-beam/counts2" \ -Pdataflow-runner {noformat} This worked successfully to test my 2.20.0-SNAPSHOT for word-count-beam: !dataflowWorkerJar_succeeded.png|width=465,height=384! [~atdixon] Would you try this {{--dataflowWorkerJar}} option? You can generate the worker JAR file by {{./gradlew :runners:google-cloud-dataflow-java:worker:legacy-worker:shadowJar}} in Beam's source tree. Alternatively, I uploaded my copy to [https://github.com/suztomo/beam/blob/worker-jar/beam-runners-google-cloud-dataflow-java-legacy-worker-2.20.0-SNAPSHOT.jar]. [~lcwik] Thank you for quick response! > Beam's own Avro TimeConversion class in beam-sdk-java-core > ----------------------------------------------------------- > > Key: BEAM-9144 > URL: https://issues.apache.org/jira/browse/BEAM-9144 > Project: Beam > Issue Type: Bug > Components: sdk-java-core > Reporter: Tomo Suzuki > Assignee: Tomo Suzuki > Priority: Major > Fix For: 2.19.0 > > Attachments: NoClassDefFoundError in word-count-beam.png, > avro-beam-dependency-graph.png, dataflow-not-finish.png, > dataflowWorkerJar_succeeded.png, dataflow_step_job_id_OBFUSC-0.json > > Time Spent: 2h 40m > Remaining Estimate: 0h > > From Aaron's comment in > https://issues.apache.org/jira/browse/BEAM-8388?focusedCommentId=17016476&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17016476 > . > {quote}My org must use Avro 1.9.x (due to some Avro schema resolution issues > resolved in 1.9.x) so downgrading Avro is not possible for us. > Beam 2.16.0 is compatible with our usage of Avro 1.9.x – but upgrading to > 2.17.0 we are broken as 2.17.0 links to Java classes in Avro 1.8.x that are > not available in 1.9.x. > {quote} > The Java class is > {{org.apache.avro.data.TimeConversions.TimestampConversion}} in Avro 1.8. > It's renamed to {{org.apache.avro.data.JodaTimeConversions}} in Avro 1.9. > h1. Beam Java SDK cannot upgrade Avro to 1.9 > Beam has Spark runners and Spark has not yet upgraded to Avro 1.9. > Illustration of the dependency > !avro-beam-dependency-graph.png|width=799,height=385! > h1. Short-term Solution > As illustrated above, as long as Beam Java SDK uses only the intersection of > Avro classes, method, and fields between Avro 1.8 and 1.9, it will provide > flexibility in runtime Avro versions (as it did until Beam 2.16). > h2. Difference of the TimeConversion Classes > Avro 1.9's TimestampConversion overrides {{getRecommendedSchema}} method. > Details below: > Avro 1.8's TimeConversions.TimestampConversion: > {code:java} > public static class TimestampConversion extends Conversion<DateTime> { > @Override > public Class<DateTime> getConvertedType() { > return DateTime.class; > } > @Override > public String getLogicalTypeName() { > return "timestamp-millis"; > } > @Override > public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType > type) { > return new DateTime(millisFromEpoch, DateTimeZone.UTC); > } > @Override > public Long toLong(DateTime timestamp, Schema schema, LogicalType type) { > return timestamp.getMillis(); > } > } > {code} > Avro 1.9's JodaTimeConversions.TimestampConversion: > {code:java} > public static class TimestampConversion extends Conversion<DateTime> { > @Override > public Class<DateTime> getConvertedType() { > return DateTime.class; > } > @Override > public String getLogicalTypeName() { > return "timestamp-millis"; > } > @Override > public DateTime fromLong(Long millisFromEpoch, Schema schema, LogicalType > type) { > return new DateTime(millisFromEpoch, DateTimeZone.UTC); > } > @Override > public Long toLong(DateTime timestamp, Schema schema, LogicalType type) { > return timestamp.getMillis(); > } > @Override > public Schema getRecommendedSchema() { > return > LogicalTypes.timestampMillis().addToSchema(Schema.create(Schema.Type.LONG)); > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)