[ 
https://issues.apache.org/jira/browse/BEAM-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037346#comment-17037346
 ] 

Liu Wang edited comment on BEAM-9003 at 2/14/20 10:55 PM:
----------------------------------------------------------

Thank you for pointing that out!

I ran the test with 2.20.0 version.

Command:

python setup.py nosetests -test-pipeline-options="runner=TestDataflowRunner 
--dataflow_worker_jar='./../../runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.20.0-SNAPSHOT.jar'
 --project=google.com:clouddfe --temp_location=gs://clouddfe-test/staging$USER 
--output=gs://world-readable-mkcq69tkcu/$USER/result.txt 
--sdk_location=./build/apache-beam-2.20.0.dev0.tar.gz --num_workers=1 
--sleep_secs=20 --streaming " 
--tests=apache_beam.transforms.util_test.ReshuffleTest --attr=ValidatesRunner 
--nocapture

Link:

[https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-02-14_14_38_22-18393797308579978754?project=google.com:clouddfe]

 

Error message:

Error message from worker: java.lang.IllegalStateException: TimestampCombiner 
moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier 
time 294247-01-09T04:00:54.775Z (end of global window) for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) 
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
 org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) 
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
 
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
 
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
 
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException: 
TimestampCombiner moved element from 294247-01-10T04:00:54.775Z 
(TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global 
window) for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) 
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
 org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) 
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
 
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
 
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
 
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
java.lang.Thread.run(Thread.java:748)

 


was (Author: liuwang_jira):
Thank you for pointing that out!

I ran the test with 2.20.0 version.

Command:

python setup.py nosetests --test-pipeline-options="--runner=TestDataflowRunner 
--dataflow_worker_jar='./../../runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.20.0-SNAPSHOT.jar'
 --project=google.com:clouddfe --temp_location=gs://clouddfe-test/staging-$USER 
--output=gs://world-readable-mkcq69tkcu/$USER/result.txt 
--sdk_location=./build/apache-beam-2.20.0.dev0.tar.gz --num_workers=1 
--sleep_secs=20 --streaming " 
--tests=apache_beam.transforms.util_test.ReshuffleTest --attr=ValidatesRunner 
--nocapture

Link:

[https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-02-14_14_38_22-18393797308579978754?project=google.com:clouddfe]

 

Error message:

Error message from worker: java.lang.IllegalStateException: TimestampCombiner 
moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier 
time 294247-01-09T04:00:54.775Z (end of global window) for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) 
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
 org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) 
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
 
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
 
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
 
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException: 
TimestampCombiner moved element from 294247-01-10T04:00:54.775Z 
(TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global 
window) for window 
org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 
org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) 
org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154)
 org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) 
org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605)
 
org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94)
 
org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
 
org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80)
 
org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159)
 
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77)
 
org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153)
 
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081)
 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
java.lang.Thread.run(Thread.java:748)

 

> test_reshuffle_preserves_timestamps 
> (apache_beam.transforms.util_test.ReshuffleTest) does not work in Streaming 
> VR suite on Dataflow
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: BEAM-9003
>                 URL: https://issues.apache.org/jira/browse/BEAM-9003
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow, sdk-py-core
>            Reporter: Valentyn Tymofieiev
>            Assignee: Liu Wang
>            Priority: Major
>
> Per investigation in https://issues.apache.org/jira/browse/BEAM-8877, the 
> test times out and was recently added to VR test suite.
> [~liumomo315], I will sickbay this test for streaming, could you please help 
> triage the failure?
> Thank you!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to