[jira] [Resolved] (BEAM-4061) Chaining SpannerIO#write() transforms
[ https://issues.apache.org/jira/browse/BEAM-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mairbek Khadikov resolved BEAM-4061. Resolution: Fixed > Chaining SpannerIO#write() transforms > - > > Key: BEAM-4061 > URL: https://issues.apache.org/jira/browse/BEAM-4061 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Major > Fix For: 2.5.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > It should be possible to chain several Cloud Spanner writes. In practice, we > can leverage Wait.on transform by returning a result object from > SpannerIO#write. > One particular example, when this feature is useful is full database import. > When data in parent tables should be injected before interleaved tables. See > more about table hierarchies in Spanner here > https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-4061) Chaining SpannerIO#write() transforms
[ https://issues.apache.org/jira/browse/BEAM-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mairbek Khadikov updated BEAM-4061: --- Fix Version/s: (was: 2.6.0) 2.5.0 > Chaining SpannerIO#write() transforms > - > > Key: BEAM-4061 > URL: https://issues.apache.org/jira/browse/BEAM-4061 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Major > Fix For: 2.5.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > It should be possible to chain several Cloud Spanner writes. In practice, we > can leverage Wait.on transform by returning a result object from > SpannerIO#write. > One particular example, when this feature is useful is full database import. > When data in parent tables should be injected before interleaved tables. See > more about table hierarchies in Spanner here > https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4061) Chaining SpannerIO#write() transforms
[ https://issues.apache.org/jira/browse/BEAM-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16540667#comment-16540667 ] Mairbek Khadikov commented on BEAM-4061: It was fixed with Beam 2.5.0 > Chaining SpannerIO#write() transforms > - > > Key: BEAM-4061 > URL: https://issues.apache.org/jira/browse/BEAM-4061 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Major > Fix For: 2.5.0 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > It should be possible to chain several Cloud Spanner writes. In practice, we > can leverage Wait.on transform by returning a result object from > SpannerIO#write. > One particular example, when this feature is useful is full database import. > When data in parent tables should be injected before interleaved tables. See > more about table hierarchies in Spanner here > https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3973) Allow to disable batch API in SpannerIO
[ https://issues.apache.org/jira/browse/BEAM-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16479727#comment-16479727 ] Mairbek Khadikov commented on BEAM-3973: Fixed with [https://github.com/apache/beam/pull/4946] > Allow to disable batch API in SpannerIO > --- > > Key: BEAM-3973 > URL: https://issues.apache.org/jira/browse/BEAM-3973 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API > provides abstractions to scale out reads from Spanner, but it requires the > query to be root-partitionable. The root-partitionable queries cover majority > of the use cases, however there are examples when running arbitrary query is > useful. For example, reading all the table names from the > information_schema.* and reading the content of those tables in the next > step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-3973) Allow to disable batch API in SpannerIO
[ https://issues.apache.org/jira/browse/BEAM-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mairbek Khadikov resolved BEAM-3973. Resolution: Fixed > Allow to disable batch API in SpannerIO > --- > > Key: BEAM-3973 > URL: https://issues.apache.org/jira/browse/BEAM-3973 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API > provides abstractions to scale out reads from Spanner, but it requires the > query to be root-partitionable. The root-partitionable queries cover majority > of the use cases, however there are examples when running arbitrary query is > useful. For example, reading all the table names from the > information_schema.* and reading the content of those tables in the next > step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-4264) Java PostCommit Spanner tests are failing due to "Instance not found"
[ https://issues.apache.org/jira/browse/BEAM-4264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mairbek Khadikov resolved BEAM-4264. Resolution: Fixed > Java PostCommit Spanner tests are failing due to "Instance not found" > - > > Key: BEAM-4264 > URL: https://issues.apache.org/jira/browse/BEAM-4264 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Reporter: Chamikara Jayalath >Assignee: Mairbek Khadikov >Priority: Blocker > Fix For: 2.5.0 > > Time Spent: 4.5h > Remaining Estimate: 0h > > First failure triggered by the commit: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/316/] > > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/316/testReport/junit/org.apache.beam.sdk.io.gcp.spanner/SpannerWriteIT/testWrite/] > > Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: Instance not found: > projects/apache-beam-testing/instances/mairbek-deleteme resource_type: > "type.googleapis.com/google.spanner.admin.instance.v1.Instance" > resource_name: "projects/apache-beam-testing/instances/mairbek-deleteme" > description: "Instance does not exist." at > io.grpc.Status.asRuntimeException(Status.java:540) at > io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:439) at > io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56) > at > com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor$1$1.onClose(SpannerErrorInterceptor.java:100) > at > io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:56) > at > com.google.cloud.spanner.spi.v1.WatchdogInterceptor$MonitoredCall$1.onClose(WatchdogInterceptor.java:190) > at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:428) at > io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76) at > io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:514) > at > io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:431) > at > io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:546) > at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52) at > io.grpc.internal.SerializingExecutor$TaskRunner.run(SerializingExecutor.java:152) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > ... 1 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure
[
https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mairbek Khadikov reopened BEAM-2837:
> Writing To Spanner From Google Cloud DataFlow - Failure
> ---
>
> Key: BEAM-2837
> URL: https://issues.apache.org/jira/browse/BEAM-2837
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: Google Cloud DataFlow
>Reporter: Al Yaros
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.2.0
>
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with
> cryptic error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>
> com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>
> com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>
> com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)*
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure
[
https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16441299#comment-16441299
]
Mairbek Khadikov commented on BEAM-2837:
It seems like it is still possible to run into the version conflict, when
google-cloud-dataflow-java-sdk-all is included.
> Writing To Spanner From Google Cloud DataFlow - Failure
> ---
>
> Key: BEAM-2837
> URL: https://issues.apache.org/jira/browse/BEAM-2837
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: Google Cloud DataFlow
>Reporter: Al Yaros
>Assignee: Mairbek Khadikov
>Priority: Major
> Fix For: 2.2.0
>
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with
> cryptic error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>
> com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>
> com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>
> com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)*
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-4061) Chaining SpannerIO#write() transforms
Mairbek Khadikov created BEAM-4061: -- Summary: Chaining SpannerIO#write() transforms Key: BEAM-4061 URL: https://issues.apache.org/jira/browse/BEAM-4061 Project: Beam Issue Type: Bug Components: io-java-gcp Reporter: Mairbek Khadikov Assignee: Mairbek Khadikov Fix For: 2.5.0 It should be possible to chain several Cloud Spanner writes. In practice, we can leverage Wait.on transform by returning a result object from SpannerIO#write. One particular example, when this feature is useful is full database import. When data in parent tables should be injected before interleaved tables. See more about table hierarchies in Spanner here https://cloud.google.com/spanner/docs/schema-and-data-model#creating_a_hierarchy_of_interleaved_tables -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3973) Allow to disable batch API in SpannerIO
[ https://issues.apache.org/jira/browse/BEAM-3973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mairbek Khadikov updated BEAM-3973: --- Description: In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API provides abstractions to scale out reads from Spanner, but it requires the query to be root-partitionable. The root-partitionable queries cover majority of the use cases, however there are examples when running arbitrary query is useful. For example, reading all the table names from the information_schema.* and reading the content of those tables in the next step. (was: In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API provides abstractions to scale out reads from Spanner, but it requires the query to be root-partitionable. The root-partitionable queries cover majority of the use cases, however there are examples when running arbitrary query is useful. For example, reading all the tables from the information_schema.* and reading the content of the schema schema in the next step. ) > Allow to disable batch API in SpannerIO > --- > > Key: BEAM-3973 > URL: https://issues.apache.org/jira/browse/BEAM-3973 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.4.0 >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Major > Fix For: 2.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API > provides abstractions to scale out reads from Spanner, but it requires the > query to be root-partitionable. The root-partitionable queries cover majority > of the use cases, however there are examples when running arbitrary query is > useful. For example, reading all the table names from the > information_schema.* and reading the content of those tables in the next > step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3973) Allow to disable batch API in SpannerIO
Mairbek Khadikov created BEAM-3973: -- Summary: Allow to disable batch API in SpannerIO Key: BEAM-3973 URL: https://issues.apache.org/jira/browse/BEAM-3973 Project: Beam Issue Type: Bug Components: io-java-gcp Affects Versions: 2.4.0 Reporter: Mairbek Khadikov Assignee: Mairbek Khadikov Fix For: 2.5.0 In 2.4.0, SpannerIO#read has been migrated to use batch API. The batch API provides abstractions to scale out reads from Spanner, but it requires the query to be root-partitionable. The root-partitionable queries cover majority of the use cases, however there are examples when running arbitrary query is useful. For example, reading all the tables from the information_schema.* and reading the content of the schema schema in the next step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3932) Spanner MutationSizeEstimator doesn't handle null array values
[ https://issues.apache.org/jira/browse/BEAM-3932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mairbek Khadikov updated BEAM-3932: --- Description: More details here https://stackoverflow.com/questions/49484770/dataflow-spanner-api-throws-illegalstateexception-when-value-of-array-type-colum > Spanner MutationSizeEstimator doesn't handle null array values > -- > > Key: BEAM-3932 > URL: https://issues.apache.org/jira/browse/BEAM-3932 > Project: Beam > Issue Type: Bug > Components: io-java-gcp >Affects Versions: 2.3.0 >Reporter: Mairbek Khadikov >Assignee: Mairbek Khadikov >Priority: Major > Fix For: 2.5.0 > > > More details here > https://stackoverflow.com/questions/49484770/dataflow-spanner-api-throws-illegalstateexception-when-value-of-array-type-colum -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-3932) Spanner MutationSizeEstimator doesn't handle null array values
Mairbek Khadikov created BEAM-3932: -- Summary: Spanner MutationSizeEstimator doesn't handle null array values Key: BEAM-3932 URL: https://issues.apache.org/jira/browse/BEAM-3932 Project: Beam Issue Type: Bug Components: io-java-gcp Affects Versions: 2.3.0 Reporter: Mairbek Khadikov Assignee: Mairbek Khadikov Fix For: 2.5.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure
[
https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16170773#comment-16170773
]
Mairbek Khadikov commented on BEAM-2837:
As a workaround, you could exclude _grpc-google-common-protos_ library that
comes with beam and add an explicit dependency
{code:java}
com.google.api.grpc
grpc-google-common-protos
0.1.9
org.apache.beam
beam-sdks-java-io-google-cloud-platform
${beam.version}
com.google.api.grpc
grpc-google-common-protos
{code}
> Writing To Spanner From Google Cloud DataFlow - Failure
> ---
>
> Key: BEAM-2837
> URL: https://issues.apache.org/jira/browse/BEAM-2837
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: Google Cloud DataFlow
>Reporter: Al Yaros
>Assignee: Mairbek Khadikov
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with
> cryptic error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>
> com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>
> com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>
> com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)*
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-2837) Writing To Spanner From Google Cloud DataFlow - Failure
[
https://issues.apache.org/jira/browse/BEAM-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16155916#comment-16155916
]
Mairbek Khadikov commented on BEAM-2837:
This is likely grpc-google-common-protos dependency coming from the pubsub
client protobuf library
[INFO] +-
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.1.0:compile
[INFO] | +- com.google.api.grpc:grpc-google-pubsub-v1:jar:0.1.0:compile
[INFO] | | +- com.google.api.grpc:grpc-google-common-protos:jar:0.1.0:compile
[INFO] | | \- com.google.api.grpc:grpc-google-iam-v1:jar:0.1.0:compile
Cloud Spanner client library expects grpc-google-common-protos:jar:0.1.9.
> Writing To Spanner From Google Cloud DataFlow - Failure
> ---
>
> Key: BEAM-2837
> URL: https://issues.apache.org/jira/browse/BEAM-2837
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
>Affects Versions: 2.1.0
> Environment: Google Cloud DataFlow
>Reporter: Al Yaros
>Assignee: Mairbek Khadikov
>
> Simple Java Program That reads from Pub\Sub and Writes to Spanner Fails with
> cryptic error message.
> Simple Program to Demonstrate the Error:
> [https://github.com/alyaros/ExamplePubSubToSpannerViaDataFlow]
> {code:java}
> *Caused by: org.apache.beam.sdk.util.UserCodeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
>
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> org.apache.beam.sdk.io.
> gcp.spanner.SpannerWriteGroupFn$DoFnInvoker.invokeSetup(Unknown Source)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.deserializeCopy(DoFnInstanceManagers.java:66)
>
> com.google.cloud.dataflow.worker.DoFnInstanceManagers$ConcurrentQueueInstanceManager.peek(DoFnInstanceManagers.java:48)
>
> com.google.cloud.dataflow.worker.UserParDoFnFactory.create(UserParDoFnFactory.java:104)
>
> com.google.cloud.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:66)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.createParDoOperation(MapTaskExecutorFactory.java:360)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:271)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory$3.typedApply(MapTaskExecutorFactory.java:253)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:55)
>
> com.google.cloud.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:43)
>
> com.google.cloud.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:78)
>
> com.google.cloud.dataflow.worker.MapTaskExecutorFactory.create(MapTaskExecutorFactory.java:142)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:925)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker.access$800(StreamingDataflowWorker.java:133)
>
> com.google.cloud.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:771)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)*
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-2538) Spanner IO ITs failing
[ https://issues.apache.org/jira/browse/BEAM-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086459#comment-16086459 ] Mairbek Khadikov commented on BEAM-2538: Stephen, can we close this? > Spanner IO ITs failing > -- > > Key: BEAM-2538 > URL: https://issues.apache.org/jira/browse/BEAM-2538 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions, sdk-java-gcp >Reporter: Stephen Sisk >Assignee: Mairbek Khadikov > > Both in local dev, and in the jenkins PostCommit_MavenInstall tests, the > spanner integration tests are currently failing. > There appear to be two different failures occurring. > https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/4245/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.spanner/SpannerReadIT/ > first error: > java.lang.NullPointerException > at > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.tearDown(SpannerReadIT.java:159) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at > org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > and the second error: > java.lang.NoClassDefFoundError: > org/apache/commons/text/RandomStringGenerator$Builder > at > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.generateDatabaseName(SpannerReadIT.java:164) > at > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.setUp(SpannerReadIT.java:95) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at > org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > org.apache.commons.text.Ra
[jira] [Commented] (BEAM-2538) Spanner IO ITs failing
[ https://issues.apache.org/jira/browse/BEAM-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16067464#comment-16067464 ] Mairbek Khadikov commented on BEAM-2538: Should be fixed with https://github.com/apache/beam/pull/3459 > Spanner IO ITs failing > -- > > Key: BEAM-2538 > URL: https://issues.apache.org/jira/browse/BEAM-2538 > Project: Beam > Issue Type: Bug > Components: sdk-java-extensions, sdk-java-gcp >Reporter: Stephen Sisk >Assignee: Mairbek Khadikov > > Both in local dev, and in the jenkins PostCommit_MavenInstall tests, the > spanner integration tests are currently failing. > There appear to be two different failures occurring. > https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/4245/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.spanner/SpannerReadIT/ > first error: > java.lang.NullPointerException > at > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.tearDown(SpannerReadIT.java:159) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at > org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > and the second error: > java.lang.NoClassDefFoundError: > org/apache/commons/text/RandomStringGenerator$Builder > at > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.generateDatabaseName(SpannerReadIT.java:164) > at > org.apache.beam.sdk.io.gcp.spanner.SpannerReadIT.setUp(SpannerReadIT.java:95) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at > org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundExcepti
[jira] [Commented] (BEAM-2394) Postcommit_Java_JDK_Version_Test is broken since SpannerWriteIT failed
[
https://issues.apache.org/jira/browse/BEAM-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037411#comment-16037411
]
Mairbek Khadikov commented on BEAM-2394:
Got it. We need to use a random database name. Or even better, use a random
table name.
I'll send out a PR.
> Postcommit_Java_JDK_Version_Test is broken since SpannerWriteIT failed
> --
>
> Key: BEAM-2394
> URL: https://issues.apache.org/jira/browse/BEAM-2394
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp, testing
>Reporter: Mark Liu
>Assignee: Mairbek Khadikov
>
> SpannerWriteIT.testWrite failed in Postcommit_Java_JDK_Version_Test since
> database didn't setup successfully.
> Error logs:
> {code}
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT
> 2017-05-31\T\12:21:30.032 [ERROR]
> testWrite(org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT) Time elapsed:
> 283.011 s <<< ERROR!
> java.lang.RuntimeException:
> (b2cfd106d806288f): com.google.cloud.spanner.SpannerException: NOT_FOUND:
> io.grpc.StatusRuntimeException: NOT_FOUND: Database not found:
> projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
> resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
> resource_name:
> "projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
> description: "Database does not exist."
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
> at
> com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
> at
> com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
> at
> com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
> at
> com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
> at
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
> at
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
> ...
> {code}
> Jenkins link:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/26/jdk=OpenJDK%207%20(on%20Ubuntu%20only),label=beam/
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/26/jdk=OpenJDK%208%20(on%20Ubuntu%20only),label=beam/
> Note: the root directory of JDK version test contains space, which is the
> main difference with Postcommit_Java_MavenInstall. It can be like:
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/OpenJDK
> 7 (on Ubuntu only)/label/beam/..."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-2394) Postcommit_Java_JDK_Version_Test is broken since SpannerWriteIT failed
[
https://issues.apache.org/jira/browse/BEAM-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037391#comment-16037391
]
Mairbek Khadikov commented on BEAM-2394:
Interesting, are these tests running concurrently?
> Postcommit_Java_JDK_Version_Test is broken since SpannerWriteIT failed
> --
>
> Key: BEAM-2394
> URL: https://issues.apache.org/jira/browse/BEAM-2394
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-gcp, testing
>Reporter: Mark Liu
>Assignee: Mairbek Khadikov
>
> SpannerWriteIT.testWrite failed in Postcommit_Java_JDK_Version_Test since
> database didn't setup successfully.
> Error logs:
> {code}
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT
> 2017-05-31\T\12:21:30.032 [ERROR]
> testWrite(org.apache.beam.sdk.io.gcp.spanner.SpannerWriteIT) Time elapsed:
> 283.011 s <<< ERROR!
> java.lang.RuntimeException:
> (b2cfd106d806288f): com.google.cloud.spanner.SpannerException: NOT_FOUND:
> io.grpc.StatusRuntimeException: NOT_FOUND: Database not found:
> projects/apache-beam-testing/instances/beam-test/databases/beam-testdb
> resource_type: "type.googleapis.com/google.spanner.admin.database.v1.Database"
> resource_name:
> "projects/apache-beam-testing/instances/beam-test/databases/beam-testdb"
> description: "Database does not exist."
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:71)
> at
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:58)
> at
> com.google.cloud.spanner.SessionPool$Waiter.take(SessionPool.java:376)
> at
> com.google.cloud.spanner.SessionPool$Waiter.access$2800(SessionPool.java:362)
> at
> com.google.cloud.spanner.SessionPool.getReadSession(SessionPool.java:697)
> at
> com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
> at
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.flushBatch(SpannerIO.java:322)
> at
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn.finishBundle(SpannerIO.java:281)
> ...
> {code}
> Jenkins link:
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/26/jdk=OpenJDK%207%20(on%20Ubuntu%20only),label=beam/
> https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_JDK_Versions_Test/26/jdk=OpenJDK%208%20(on%20Ubuntu%20only),label=beam/
> Note: the root directory of JDK version test contains space, which is the
> main difference with Postcommit_Java_MavenInstall. It can be like:
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_JDK_Versions_Test/jdk/OpenJDK
> 7 (on Ubuntu only)/label/beam/..."
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (BEAM-2379) SpannerIO tests are failing
[
https://issues.apache.org/jira/browse/BEAM-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029909#comment-16029909
]
Mairbek Khadikov commented on BEAM-2379:
Will send a fix PR
> SpannerIO tests are failing
> ---
>
> Key: BEAM-2379
> URL: https://issues.apache.org/jira/browse/BEAM-2379
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Mairbek Khadikov
> Fix For: Not applicable
>
>
> When performing a complete {{mvn clean install}}, the build is failing on
> {{SpannerIO}} tests:
> {code}
> [ERROR] Errors:
> [ERROR] SpannerIOTest.batching:129 » UserCode
> java.lang.IllegalArgumentException: A pr...
> [ERROR] SpannerIOTest.batchingGroups:155 » UserCode
> java.lang.IllegalArgumentException...
> [ERROR] SpannerIOTest.noBatching:178 » UserCode
> java.lang.IllegalArgumentException: A ...
> [ERROR] SpannerIOTest.singleMutationPipeline » UncheckedExecution
> org.apache.beam.sdk
> {code}
> These tests fail with the same reason, here's the complete stack trace:
> {code}
> [ERROR] batchingGroups(org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest)
> Time elapsed: 0.004 s <<< ERROR!
> org.apache.beam.sdk.util.UserCodeException:
> java.lang.IllegalArgumentException: A project ID is required for this service
> but could not be determined from the builder or the environment. Please set
> a project ID using the builder.
> at
> org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:36)
> at
> org.apache.beam.sdk.io.gcp.spanner.SpannerIO$SpannerWriteFn$DoFnInvoker.invokeSetup(Unknown
> Source)
> at
> org.apache.beam.sdk.transforms.DoFnTester.initializeState(DoFnTester.java:745)
> at
> org.apache.beam.sdk.transforms.DoFnTester.startBundle(DoFnTester.java:219)
> at
> org.apache.beam.sdk.transforms.DoFnTester.processBundle(DoFnTester.java:183)
> at
> org.apache.beam.sdk.io.gcp.spanner.SpannerIOTest.batchingGroups(SpannerIOTest.java:155)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.apache.beam.sdk.testing.TestPipeline$1.evaluate(TestPipeline.java:321)
> at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:239)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.
[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner
[ https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023881#comment-16023881 ] Mairbek Khadikov commented on BEAM-1542: Created https://github.com/apache/beam-site/pull/252 > Need Source/Sink for Spanner > > > Key: BEAM-1542 > URL: https://issues.apache.org/jira/browse/BEAM-1542 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Guy Molinari >Assignee: Mairbek Khadikov > > Is there a source/sink for Spanner in the works? If not I would gladly give > this a shot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-1542) Need Source/Sink for Spanner
[ https://issues.apache.org/jira/browse/BEAM-1542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15957937#comment-15957937 ] Mairbek Khadikov commented on BEAM-1542: Hi Guy, I work on Cloud Spanner at Google. Let us know if you need any help with the Sink. I'm going to leave few comments in the PR as well. If you are hitting the following exception Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.SpannerErrorInterceptor at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214) at com.google.common.cache.LocalCache.get(LocalCache.java:4053) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4986) at org.apache.beam.runners.direct.DoFnLifecycleManager.get(DoFnLifecycleManager.java:61) at org.apache.beam.runners.direct.ParDoEvaluatorFactory.createEvaluator(ParDoEvaluatorFactory.java:124) at org.apache.beam.runners.direct.ParDoEvaluatorFactory.forApplication(ParDoEvaluatorFactory.java:77) at org.apache.beam.runners.direct.TransformEvaluatorRegistry.forApplication(TransformEvaluatorRegistry.java:96) at org.apache.beam.runners.direct.TransformExecutor.run(TransformExecutor.java:100) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) It should be resolved with https://github.com/apache/beam/pull/2377 > Need Source/Sink for Spanner > > > Key: BEAM-1542 > URL: https://issues.apache.org/jira/browse/BEAM-1542 > Project: Beam > Issue Type: New Feature > Components: sdk-java-gcp >Reporter: Guy Molinari >Assignee: Guy Molinari > > Is there a source/sink for Spanner in the works? If not I would gladly give > this a shot. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
