[ https://issues.apache.org/jira/browse/BEAM-4839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom closed BEAM-4839. --------------------- Resolution: Fixed Fix Version/s: (was: Not applicable) 2.6.0 > EOF Exception writing non-english Characters to Spanner > ------------------------------------------------------- > > Key: BEAM-4839 > URL: https://issues.apache.org/jira/browse/BEAM-4839 > Project: Beam > Issue Type: Bug > Components: runner-core, runner-dataflow > Affects Versions: 2.3.0, 2.4.0, 2.5.0 > Environment: GCP and Local (High Sierra) > Reporter: Tom > Assignee: Kenneth Knowles > Priority: Minor > Fix For: 2.6.0 > > > I am having an issue with Apache Beam ^2.3 and Google Cloud Platform Spanner. > In short, I'm trying to write data into Spanner. Some of this data contains > non-English characters, which blows up the dataflow job when using Beam 2.3 > or higher. > Currently, I'm trying to use Apache Beam 2.5, google-api-client 1.23 and Java > 1.8. > This error occurs using Apache Beam 2.3, and not when using 2.2. When using > Apache Beam 2.2, we have to drop the google-http/api-client to 1.22 from 1.23. > {quote}<beam.version>2.5.0</beam.version>{quote} > {quote} > <dependency> > <groupId>org.apache.beam</groupId> > <artifactId>beam-sdks-java-core</artifactId> > <version>${beam.version}</version> > </dependency> > <dependency> > <groupId>org.apache.beam</groupId> > <artifactId>beam-runners-google-cloud-dataflow-java</artifactId> > <version>${beam.version}</version> > </dependency> > <dependency> > <groupId>org.apache.beam</groupId> > <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId> > <version>${beam.version}</version> > </dependency> > {quote} > The google libraries > {quote} > <google-clients.version>1.23.0</google-clients.version> > <google-package.version>0.26.0-alpha</google-package.version> > {quote} > {quote} > <dependency> > <groupId>com.google.cloud</groupId> > <artifactId>google-cloud</artifactId> > <version>${google-package.version}</version> > </dependency> > <dependency> > <groupId>com.google.http-client</groupId> > <artifactId>google-http-client</artifactId> > <version>${google-clients.version}</version> > <exclusions> > <!-- Exclude an old version of guava that is being pulled > in by a transitive dependency of google-api-client --> > <exclusion> > <groupId>com.google.guava</groupId> > <artifactId>guava-jdk5</artifactId> > </exclusion> > </exclusions> > </dependency> > {quote} > Here's the stack trace. You'll see that in this quick sample runner we tried > inserting 4 rows. Everything runs fine until I attempt to write mutations to > the table. > {quote} > [INFO] Scanning for projects... > [INFO] > [INFO] --------------------< com.testing:dataflowtwofive > >--------------------- > [INFO] Building dataflowtwofive 0.1 > [INFO] --------------------------------[ jar > ]--------------------------------- > [WARNING] The POM for com.google.oauth-client:google-oauth-client:jar:1.23.0 > is invalid, transitive dependencies (if any) will not be available, enable > debug logging for more details > [WARNING] The POM for > com.google.http-client:google-http-client-jackson:jar:1.23.0 is invalid, > transitive dependencies (if any) will not be available, enable debug logging > for more details > [WARNING] The POM for > com.google.apis:google-api-services-storage:jar:v1-rev114-1.23.0 is invalid, > transitive dependencies (if any) will not be available, enable debug logging > for more details > [INFO] > [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ > dataflowtwofive --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] Copying 0 resource > [INFO] > [INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ > dataflowtwofive --- > [INFO] Nothing to compile - all classes are up to date > [INFO] > [INFO] --- exec-maven-plugin:1.4.0:java (default-cli) @ dataflowtwofive --- > Jul 19, 2018 3:22:41 PM runners.SimpleTestRunner main > WARNING: ATTN: Coder for mutations: [class com.google.cloud.spanner.Mutation] > Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement > WARNING: ATTN: mutation: > [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.765000000Z, > ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.809000000Z, > GUID=f2360671-557d-4406-b693-c0d66..., LAHQ=a, LAISO=a2, LASPEZ=a3, SPRAS=a4}] > Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement > WARNING: ATTN: mutation: > [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.765000000Z, > ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.809000000Z, > GUID=cba60659-6986-4dc3-a523-116e2..., LAHQ=รก, LAISO=c2, LASPEZ=c3, SPRAS=c4}] > Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement > WARNING: ATTN: mutation: > [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.765000000Z, > ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.809000000Z, > GUID=889d9d14-df57-4d72-8748-8de0f..., LAHQ=d, LAISO=d2, LASPEZ=d3, SPRAS=d4}] > Jul 19, 2018 3:22:42 PM transforms.TestMutationBuilder processElement > WARNING: ATTN: mutation: > [\{ROW_INSERT_TIMESTAMP=2018-07-19T20:22:42.765000000Z, > ROW_UPDATE_TIMESTAMP=2018-07-19T20:22:42.809000000Z, > GUID=888a2a76-aff7-4f64-9e24-2c231..., LAHQ=b, LAISO=b2, LASPEZ=b3, SPRAS=b4}] > [WARNING] > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke > (NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke > (DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke (Method.java:498) > at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:293) > at java.lang.Thread.run (Thread.java:748) > Caused by: org.apache.beam.sdk.Pipeline$PipelineExecutionException: > java.lang.RuntimeException: java.io.EOFException > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish > (DirectRunner.java:349) > at > org.apache.beam.runners.direct.DirectRunner$DirectPipelineResult.waitUntilFinish > (DirectRunner.java:319) > at org.apache.beam.runners.direct.DirectRunner.run (DirectRunner.java:210) > at org.apache.beam.runners.direct.DirectRunner.run (DirectRunner.java:66) > at org.apache.beam.sdk.Pipeline.run (Pipeline.java:311) > at org.apache.beam.sdk.Pipeline.run (Pipeline.java:297) > at runners.SimpleTestRunner.main (SimpleTestRunner.java:53) > at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke > (NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke > (DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke (Method.java:498) > at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:293) > at java.lang.Thread.run (Thread.java:748) > Caused by: java.lang.RuntimeException: java.io.EOFException > at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode > (MutationGroupEncoder.java:271) > at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement > (SpannerIO.java:1030) > Caused by: java.io.EOFException > at java.io.DataInputStream.readFully (DataInputStream.java:197) > at java.io.DataInputStream.readFully (DataInputStream.java:169) > at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.readBytes > (MutationGroupEncoder.java:475) > at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodePrimitive > (MutationGroupEncoder.java:434) > at > org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeModification > (MutationGroupEncoder.java:326) > at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decodeMutation > (MutationGroupEncoder.java:280) > at org.apache.beam.sdk.io.gcp.spanner.MutationGroupEncoder.decode > (MutationGroupEncoder.java:264) > at org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn.processElement > (SpannerIO.java:1030) > at > org.apache.beam.sdk.io.gcp.spanner.SpannerIO$BatchFn$DoFnInvoker.invokeProcessElement > (Unknown Source) > at > org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.invokeProcessElement > (SimpleDoFnRunner.java:185) > at > org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimpleDoFnRunner.processElement > (SimpleDoFnRunner.java:146) > at > org.apache.beam.repackaged.beam_runners_direct_java.runners.core.SimplePushbackSideInputDoFnRunner.processElementInReadyWindows > (SimplePushbackSideInputDoFnRunner.java:87) > at org.apache.beam.runners.direct.ParDoEvaluator.processElement > (ParDoEvaluator.java:189) > at > org.apache.beam.runners.direct.DoFnLifecycleManagerRemovingTransformEvaluator.processElement > (DoFnLifecycleManagerRemovingTransformEvaluator.java:55) > at org.apache.beam.runners.direct.DirectTransformExecutor.processElements > (DirectTransformExecutor.java:161) > at org.apache.beam.runners.direct.DirectTransformExecutor.run > (DirectTransformExecutor.java:125) > at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511) > at java.util.concurrent.FutureTask.run (FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker > (ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run > (ThreadPoolExecutor.java:624) > at java.lang.Thread.run (Thread.java:748) > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 8.817 s > [INFO] Finished at: 2018-07-19T15:22:47-05:00 > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java > (default-cli) on project dataflowtwofive: An exception occured while > executing the Java class. null: InvocationTargetException: > java.lang.RuntimeException: java.io.EOFException -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > {quote} > If anything else is needed to help solve this issue, please let me know. -- This message was sent by Atlassian JIRA (v7.6.3#76005)