[
https://issues.apache.org/jira/browse/BEAM-14091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Victor Chen updated BEAM-14091:
-------------------------------
Description:
* Bug: the show and collect commands under Interactive Beam would eventually
call a PipelineFragment to execute the InteractiveRunner corresponding to the
given PCollection. The PipelineFragment by default does not wait for pipeline
to finish execution, and when a pipeline runs the FlinkRunner remotely, the
Interactive Environment is not updated with the results of the pipeline run.
Subsequent calls to ib.show/ib.collect would rerun the pipeline instead of
directly retrieving the results from the Interactive Environment.
* Setting the blocking to be True when using the FlinkRunner inside the
RecordingManager.record() method when calling the .run() method of the
PipelineFragment properly saves completed pipeline results in the Interactive
Environment and prevents a need to rerun the pipelines again on subsequent
calls to ib.show/ib.collect.
was:
* Bug: the show and collect commands under Interactive Beam would eventually
call a PipelineFragment to execute the InteractiveRunner corresponding to the
given PCollection. The PipelineFragment by default does not wait for pipeline
to finish execution, and when a pipeline runs remotely, the Interactive
Environment is not updated with the results of the pipeline run. Subsequent
calls to ib.show/ib.collect would rerun the pipeline instead of directly
retrieving the results from the Interactive Environment.
* Setting the blocking to always be True inside the RecordingManager.record()
method when calling the .run() method of the PipelineFragment properly saves
completed pipeline results in the Interactive Environment and prevents a need
to rerun the pipelines again on subsequent calls to ib.show/ib.collect.
> Fixing Interactive Beam show/collect for remote runners
> -------------------------------------------------------
>
> Key: BEAM-14091
> URL: https://issues.apache.org/jira/browse/BEAM-14091
> Project: Beam
> Issue Type: Bug
> Components: runner-py-interactive
> Reporter: Victor Chen
> Assignee: Victor Chen
> Priority: P2
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> * Bug: the show and collect commands under Interactive Beam would eventually
> call a PipelineFragment to execute the InteractiveRunner corresponding to the
> given PCollection. The PipelineFragment by default does not wait for pipeline
> to finish execution, and when a pipeline runs the FlinkRunner remotely, the
> Interactive Environment is not updated with the results of the pipeline run.
> Subsequent calls to ib.show/ib.collect would rerun the pipeline instead of
> directly retrieving the results from the Interactive Environment.
> * Setting the blocking to be True when using the FlinkRunner inside the
> RecordingManager.record() method when calling the .run() method of the
> PipelineFragment properly saves completed pipeline results in the Interactive
> Environment and prevents a need to rerun the pipelines again on subsequent
> calls to ib.show/ib.collect.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
