[ https://issues.apache.org/jira/browse/BEAM-9003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037346#comment-17037346 ]
Liu Wang edited comment on BEAM-9003 at 2/14/20 10:55 PM: ---------------------------------------------------------- Thank you for pointing that out! I ran the test with 2.20.0 version. Command: python setup.py nosetests -test-pipeline-options="runner=TestDataflowRunner --dataflow_worker_jar='./../../runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.20.0-SNAPSHOT.jar' --project=google.com:clouddfe --temp_location=gs://clouddfe-test/staging$USER --output=gs://world-readable-mkcq69tkcu/$USER/result.txt --sdk_location=./build/apache-beam-2.20.0.dev0.tar.gz --num_workers=1 --sleep_secs=20 --streaming " --tests=apache_beam.transforms.util_test.ReshuffleTest --attr=ValidatesRunner --nocapture Link: [https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-02-14_14_38_22-18393797308579978754?project=google.com:clouddfe] Error message: Error message from worker: java.lang.IllegalStateException: TimestampCombiner moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global window) for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154) org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605) org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73) org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134) org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException: TimestampCombiner moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global window) for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154) org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605) org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73) org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134) org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:748) was (Author: liuwang_jira): Thank you for pointing that out! I ran the test with 2.20.0 version. Command: python setup.py nosetests --test-pipeline-options="--runner=TestDataflowRunner --dataflow_worker_jar='./../../runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.20.0-SNAPSHOT.jar' --project=google.com:clouddfe --temp_location=gs://clouddfe-test/staging-$USER --output=gs://world-readable-mkcq69tkcu/$USER/result.txt --sdk_location=./build/apache-beam-2.20.0.dev0.tar.gz --num_workers=1 --sleep_secs=20 --streaming " --tests=apache_beam.transforms.util_test.ReshuffleTest --attr=ValidatesRunner --nocapture Link: [https://pantheon.corp.google.com/dataflow/jobs/us-central1/2020-02-14_14_38_22-18393797308579978754?project=google.com:clouddfe] Error message: Error message from worker: java.lang.IllegalStateException: TimestampCombiner moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global window) for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154) org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605) org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73) org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134) org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:748) java.lang.IllegalStateException: TimestampCombiner moved element from 294247-01-10T04:00:54.775Z (TIMESTAMP_MAX_VALUE) to earlier time 294247-01-09T04:00:54.775Z (end of global window) for window org.apache.beam.sdk.transforms.windowing.GlobalWindow@cdd4d56 org.apache.beam.runners.core.WatermarkHold.shift(WatermarkHold.java:117) org.apache.beam.runners.core.WatermarkHold.addElementHold(WatermarkHold.java:154) org.apache.beam.runners.core.WatermarkHold.addHolds(WatermarkHold.java:98) org.apache.beam.runners.core.ReduceFnRunner.processElement(ReduceFnRunner.java:605) org.apache.beam.runners.core.ReduceFnRunner.processElements(ReduceFnRunner.java:349) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:94) org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowViaWindowSetFn.processElement(StreamingGroupAlsoByWindowViaWindowSetFn.java:42) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:120) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73) org.apache.beam.runners.core.LateDataDroppingDoFnRunner.processElement(LateDataDroppingDoFnRunner.java:80) org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:134) org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44) org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:201) org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:159) org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:77) org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1358) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:153) org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1081) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:748) > test_reshuffle_preserves_timestamps > (apache_beam.transforms.util_test.ReshuffleTest) does not work in Streaming > VR suite on Dataflow > ------------------------------------------------------------------------------------------------------------------------------------ > > Key: BEAM-9003 > URL: https://issues.apache.org/jira/browse/BEAM-9003 > Project: Beam > Issue Type: Bug > Components: runner-dataflow, sdk-py-core > Reporter: Valentyn Tymofieiev > Assignee: Liu Wang > Priority: Major > > Per investigation in https://issues.apache.org/jira/browse/BEAM-8877, the > test times out and was recently added to VR test suite. > [~liumomo315], I will sickbay this test for streaming, could you please help > triage the failure? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)