See <https://builds.apache.org/job/beam_PostCommit_Python_Verify/4478/display/redirect?page=changes>
Changes: [herohde] [BEAM-3897] Add Go wordcount example with multi output DoFns [herohde] [BEAM-3866] Remove windowed value requirement for External [markliu] Clean up terminal state check in TestDataflowRunner [tgroh] Make RemoteEnvironment public ------------------------------------------ [...truncated 1.13 MB...] root: INFO: 2018-03-22T02:22:43.709Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/InitializeWrite into write/Write/WriteImpl/DoOnce/Read root: INFO: 2018-03-22T02:22:43.741Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/FinalizeWrite/MapToVoidKey2 into write/Write/WriteImpl/PreFinalize/PreFinalize root: INFO: 2018-03-22T02:22:43.775Z: JOB_MESSAGE_DETAILED: Fusing consumer write/Write/WriteImpl/FinalizeWrite/MapToVoidKey2 into write/Write/WriteImpl/PreFinalize/PreFinalize root: INFO: 2018-03-22T02:22:43.810Z: JOB_MESSAGE_DEBUG: Workflow config is missing a default resource spec. root: INFO: 2018-03-22T02:22:43.842Z: JOB_MESSAGE_DEBUG: Adding StepResource setup and teardown to workflow graph. root: INFO: 2018-03-22T02:22:43.876Z: JOB_MESSAGE_DEBUG: Adding workflow start and stop steps. root: INFO: 2018-03-22T02:22:43.911Z: JOB_MESSAGE_DEBUG: Assigning stage ids. root: INFO: 2018-03-22T02:22:44.091Z: JOB_MESSAGE_DEBUG: Executing wait step start26 root: INFO: 2018-03-22T02:22:44.177Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/DoOnce/Read+write/Write/WriteImpl/InitializeWrite+write/Write/WriteImpl/PreFinalize/MapToVoidKey0+write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0+write/Write/WriteImpl/WriteBundles/MapToVoidKey0+write/Write/WriteImpl/PreFinalize/MapToVoidKey0+write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0+write/Write/WriteImpl/WriteBundles/MapToVoidKey0 root: INFO: 2018-03-22T02:22:44.209Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/GroupByKey/Create root: INFO: 2018-03-22T02:22:44.223Z: JOB_MESSAGE_DEBUG: Starting worker pool setup. root: INFO: 2018-03-22T02:22:44.225Z: JOB_MESSAGE_BASIC: Executing operation group/Create root: INFO: 2018-03-22T02:22:44.246Z: JOB_MESSAGE_BASIC: Starting 1 workers in us-central1-f... root: INFO: Job 2018-03-21_19_22_38-2399826686858835030 is in state JOB_STATE_RUNNING root: INFO: 2018-03-22T02:22:51.766Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 0 based on the rate of progress in the currently running step(s). root: INFO: 2018-03-22T02:22:51.840Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/GroupByKey/Session" materialized. root: INFO: 2018-03-22T02:22:51.863Z: JOB_MESSAGE_DEBUG: Value "group/Session" materialized. root: INFO: 2018-03-22T02:22:51.924Z: JOB_MESSAGE_BASIC: Executing operation read/Read+split+pair_with_one+group/Reify+group/Write root: INFO: 2018-03-22T02:22:57.073Z: JOB_MESSAGE_BASIC: Autoscaling: Resizing worker pool from 1 to 2. root: INFO: 2018-03-22T02:23:09.044Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 1 based on the rate of progress in the currently running step(s). root: INFO: 2018-03-22T02:23:09.075Z: JOB_MESSAGE_DETAILED: Resized worker pool to 1, though goal was 2. This could be a quota issue. root: INFO: 2018-03-22T02:23:22.955Z: JOB_MESSAGE_DETAILED: Workers have started successfully. root: INFO: 2018-03-22T02:23:24.974Z: JOB_MESSAGE_DETAILED: Autoscaling: Raised the number of workers to 2 based on the rate of progress in the currently running step(s). root: INFO: 2018-03-22T02:27:45.472Z: JOB_MESSAGE_BASIC: Executing operation group/Close root: INFO: 2018-03-22T02:27:48.714Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/DoOnce/Read.out" materialized. root: INFO: 2018-03-22T02:27:48.735Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize/MapToVoidKey0.out" materialized. root: INFO: 2018-03-22T02:27:48.758Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/MapToVoidKey0.out" materialized. root: INFO: 2018-03-22T02:27:48.789Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/WriteBundles/MapToVoidKey0.out" materialized. root: INFO: 2018-03-22T02:27:48.824Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0) root: INFO: 2018-03-22T02:27:48.856Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0) root: INFO: 2018-03-22T02:27:48.891Z: JOB_MESSAGE_BASIC: Executing operation write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0) root: INFO: 2018-03-22T02:27:48.930Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/PreFinalize/_DataflowIterableSideInput(MapToVoidKey0.out.0).output" materialized. root: INFO: 2018-03-22T02:27:48.961Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/FinalizeWrite/_DataflowIterableSideInput(MapToVoidKey0.out.0).output" materialized. root: INFO: 2018-03-22T02:27:48.992Z: JOB_MESSAGE_DEBUG: Value "write/Write/WriteImpl/WriteBundles/_DataflowIterableSideInput(MapToVoidKey0.out.0).output" materialized. root: INFO: 2018-03-22T02:27:49.050Z: JOB_MESSAGE_BASIC: Executing operation group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write root: INFO: 2018-03-22T02:27:51.524Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:51.530Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:52.915Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:53.335Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:54.725Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:54.953Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:56.160Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:56.554Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:57.341Z: JOB_MESSAGE_ERROR: Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py", line 609, in do_work work_executor.execute() File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py", line 167, in execute op.start() File "apache_beam/runners/worker/operations.py", line 339, in apache_beam.runners.worker.operations.DoOperation.start def start(self): File "apache_beam/runners/worker/operations.py", line 340, in apache_beam.runners.worker.operations.DoOperation.start with self.scoped_start_state: File "apache_beam/runners/worker/operations.py", line 372, in apache_beam.runners.worker.operations.DoOperation.start self.dofn_runner = common.DoFnRunner( File "apache_beam/runners/common.py", line 483, in apache_beam.runners.common.DoFnRunner.__init__ self.do_fn_invoker = DoFnInvoker.create_invoker( File "apache_beam/runners/common.py", line 203, in apache_beam.runners.common.DoFnInvoker.create_invoker return PerWindowInvoker( File "apache_beam/runners/common.py", line 313, in apache_beam.runners.common.PerWindowInvoker.__init__ input_args, input_kwargs, [si[global_window] for si in side_inputs]) File "/usr/local/lib/python2.7/dist-packages/apache_beam/transforms/sideinputs.py", line 62, in __getitem__ self._cache[window] = self._view_class._from_runtime_iterable( AttributeError: type object '_DataflowIterableSideInput' has no attribute '_from_runtime_iterable' root: INFO: 2018-03-22T02:27:57.385Z: JOB_MESSAGE_DEBUG: Executing failure step failure25 root: INFO: 2018-03-22T02:27:57.415Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: S08:group/Read+group/GroupByWindow+count+format+write/Write/WriteImpl/WriteBundles/WriteBundles+write/Write/WriteImpl/Pair+write/Write/WriteImpl/WindowInto(WindowIntoFn)+write/Write/WriteImpl/GroupByKey/Reify+write/Write/WriteImpl/GroupByKey/Write failed., A work item was attempted 4 times without success. Each time the worker eventually lost contact with the service. The work item was attempted on: beamapp-jenkins-032202223-03211922-31e7-harness-fhjp, beamapp-jenkins-032202223-03211922-31e7-harness-fhjp, beamapp-jenkins-032202223-03211922-31e7-harness-fhjp, beamapp-jenkins-032202223-03211922-31e7-harness-2s1j root: INFO: 2018-03-22T02:27:57.537Z: JOB_MESSAGE_DETAILED: Cleaning up. root: INFO: 2018-03-22T02:27:57.595Z: JOB_MESSAGE_DEBUG: Starting worker pool teardown. root: INFO: 2018-03-22T02:27:57.622Z: JOB_MESSAGE_BASIC: Stopping worker pool... root: INFO: 2018-03-22T02:29:16.637Z: JOB_MESSAGE_DETAILED: Autoscaling: Reduced the number of workers to 0 based on the rate of progress in the currently running step(s). root: INFO: 2018-03-22T02:29:17.083Z: JOB_MESSAGE_DEBUG: Tearing down pending resources... root: INFO: Job 2018-03-21_19_22_38-2399826686858835030 is in state JOB_STATE_FAILED --------------------- >> end captured logging << --------------------- ---------------------------------------------------------------------- Ran 3 tests in 418.859s FAILED (errors=1) Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-21_19_22_34-9863111785664501396?project=apache-beam-testing Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-21_19_22_35-9367960069665094921?project=apache-beam-testing Found: https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-03-21_19_22_38-2399826686858835030?project=apache-beam-testing Build step 'Execute shell' marked build as failure Not sending mail to unregistered user hero...@google.com Not sending mail to unregistered user aal...@gmail.com Not sending mail to unregistered user mark...@google.com