[ https://issues.apache.org/jira/browse/BEAM-9745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17109176#comment-17109176 ]
Chamikara Madhusanka Jayalath edited comment on BEAM-9745 at 5/16/20, 7:04 PM: ------------------------------------------------------------------------------- I noticed that we add harness jar to the container in addition to jars received through the retrieval service. [https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122] Could it be that we end up with conflicting versions of the same set of classes due to this ? was (Author: chamikara): I noticed that we add harness jar to the container in addition to jars received through the retrieval service. [https://github.com/apache/beam/blob/master/sdks/java/container/boot.go#L122] Could it be that we end up with conflicting versions of the same set of classes for Dataflow due to this ? > [beam_PostCommit_Java_PortabilityApi] Various GCP IO tests failing, unable to > deserialize Custom DoFns and Custom Coders. > ------------------------------------------------------------------------------------------------------------------------- > > Key: BEAM-9745 > URL: https://issues.apache.org/jira/browse/BEAM-9745 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, java-fn-execution, sdk-java-harness, > test-failures > Reporter: Daniel Oliveira > Assignee: Kenneth Knowles > Priority: P0 > Labels: currently-failing > > _Use this form to file an issue for test failure:_ > * [Jenkins > Job|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4657/] > * [Gradle Build > Scan|https://scans.gradle.com/s/c3izncsa4u24k/tests/by-project] > Initial investigation: > The bug appears to be popping up on BigQuery tests mostly, but also a > BigTable and a Datastore test. > Here's an example stacktrace of the two errors, showing _only_ the error > messages themselves. Source: > [https://scans.gradle.com/s/c3izncsa4u24k/tests/efn4wciuamvqq-ccxt3jvofvqbe] > {noformat} > java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error > received from SDK harness for instruction -191: > java.lang.IllegalArgumentException: unable to deserialize Custom DoFn With > Execution Info > ... > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3 > ... > Caused by: java.lang.RuntimeException: Error received from SDK harness for > instruction -191: java.lang.IllegalArgumentException: unable to deserialize > Custom DoFn With Execution Info > ... > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.sdk.io.gcp.bigquery.BatchLoads$3 > java.util.concurrent.ExecutionException: java.lang.RuntimeException: Error > received from SDK harness for instruction -206: > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes > ... > Caused by: > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes > ... > Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom > Coder Bytes > ... > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder > ... > Caused by: java.lang.RuntimeException: Error received from SDK harness for > instruction -206: > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes > ... > Caused by: > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > java.lang.IllegalArgumentException: unable to deserialize Custom Coder Bytes > ... > Caused by: java.lang.IllegalArgumentException: unable to deserialize Custom > Coder Bytes > ... > Caused by: java.lang.ClassNotFoundException: > org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder > ... > {noformat} > Update: Looks like this has been failing as far back as [Apr > 4|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4566/] > after a long period where the test was consistently timing out since [Mar > 31|https://builds.apache.org/job/beam_PostCommit_Java_PortabilityApi/4546/]. > So it's hard to narrow down what commit may have caused this. Plus, the test > was failing due to a completely different BigQuery failure before anyway, so > it seems like this test will need to be completely fixed from scratch, > instead of tracking down a specific breaking change. > ---- > _After you've filled out the above details, please [assign the issue to an > individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist]. > Assignee should [treat test failures as > high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test], > helping to fix the issue or find a more appropriate owner. See [Apache Beam > Post-Commit > Policies|https://beam.apache.org/contribute/postcommits-policies]._ -- This message was sent by Atlassian Jira (v8.3.4#803005)