Stephen Sisk created BEAM-2545: ---------------------------------- Summary: bigtable e2e tests failing - UNKNOWN: Stale requests/Error mutating row Key: BEAM-2545 URL: https://issues.apache.org/jira/browse/BEAM-2545 Project: Beam Issue Type: Bug Components: sdk-java-gcp Reporter: Stephen Sisk Assignee: Stephen Sisk
The BigtableWriteIT is taking a long time (~10min) and throwing errors. Example test run: https://builds.apache.org/view/Beam/job/beam_PostCommit_Java_MavenInstall/4264/org.apache.beam$beam-runners-google-cloud-dataflow-java/testReport/junit/org.apache.beam.sdk.io.gcp.bigtable/BigtableWriteIT/testE2EBigtableWrite/ (96dc5c8efaf8fa26): java.io.IOException: At least 25 errors occurred writing to Bigtable. First 10 errors: Error mutating row key000000175 with mutations [set_cell { family_name: "cf" value: "value000000175" } ]: UNKNOWN: Stale requests. Error mutating row key000000176 with mutations [set_cell { family_name: "cf" value: "value000000176" } ]: UNKNOWN: Stale requests. Error mutating row key000000177 with mutations [set_cell { family_name: "cf" value: "value000000177" } ]: UNKNOWN: Stale requests. Error mutating row key000000178 with mutations [set_cell { family_name: "cf" value: "value000000178" } ]: UNKNOWN: Stale requests. Error mutating row key000000179 with mutations [set_cell { family_name: "cf" value: "value000000179" } ]: UNKNOWN: Stale requests. Error mutating row key000000180 with mutations [set_cell { family_name: "cf" value: "value000000180" } ]: UNKNOWN: Stale requests. Error mutating row key000000181 with mutations [set_cell { family_name: "cf" value: "value000000181" } ]: UNKNOWN: Stale requests. Error mutating row key000000182 with mutations [set_cell { family_name: "cf" value: "value000000182" } ]: UNKNOWN: Stale requests. Error mutating row key000000183 with mutations [set_cell { family_name: "cf" value: "value000000183" } ]: UNKNOWN: Stale requests. Error mutating row key000000184 with mutations [set_cell { family_name: "cf" value: "value000000184" } ]: UNKNOWN: Stale requests. at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Write$BigtableWriterFn.checkForFailures(BigtableIO.java:655) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Write$BigtableWriterFn.finishBundle(BigtableIO.java:607) Stacktrace java.lang.RuntimeException: (96dc5c8efaf8fa26): java.io.IOException: At least 25 errors occurred writing to Bigtable. First 10 errors: Error mutating row key000000175 with mutations [set_cell { family_name: "cf" value: "value000000175" } ]: UNKNOWN: Stale requests. Error mutating row key000000176 with mutations [set_cell { family_name: "cf" value: "value000000176" } ]: UNKNOWN: Stale requests. Error mutating row key000000177 with mutations [set_cell { family_name: "cf" value: "value000000177" } ]: UNKNOWN: Stale requests. Error mutating row key000000178 with mutations [set_cell { family_name: "cf" value: "value000000178" } ]: UNKNOWN: Stale requests. Error mutating row key000000179 with mutations [set_cell { family_name: "cf" value: "value000000179" } ]: UNKNOWN: Stale requests. Error mutating row key000000180 with mutations [set_cell { family_name: "cf" value: "value000000180" } ]: UNKNOWN: Stale requests. Error mutating row key000000181 with mutations [set_cell { family_name: "cf" value: "value000000181" } ]: UNKNOWN: Stale requests. Error mutating row key000000182 with mutations [set_cell { family_name: "cf" value: "value000000182" } ]: UNKNOWN: Stale requests. Error mutating row key000000183 with mutations [set_cell { family_name: "cf" value: "value000000183" } ]: UNKNOWN: Stale requests. Error mutating row key000000184 with mutations [set_cell { family_name: "cf" value: "value000000184" } ]: UNKNOWN: Stale requests. at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Write$BigtableWriterFn.checkForFailures(BigtableIO.java:655) at org.apache.beam.sdk.io.gcp.bigtable.BigtableIO$Write$BigtableWriterFn.finishBundle(BigtableIO.java:607) at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:133) at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:89) at org.apache.beam.runners.dataflow.TestDataflowRunner.run(TestDataflowRunner.java:54) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297) at org.apache.beam.sdk.Pipeline.run(Pipeline.java:283) at org.apache.beam.sdk.io.gcp.bigtable.BigtableWriteIT.testE2EBigtableWrite(BigtableWriteIT.java:128) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.apache.maven.surefire.junitcore.pc.Scheduler$1.run(Scheduler.java:393) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v6.4.14#64029)