thank you!
Shen
On Tue, Aug 15, 2017 at 1:29 PM, Robert Bradshaw <
rober...@google.com.invalid> wrote:
> On Tue, Aug 15, 2017 at 10:21 AM, Shen Li wrote:
> > Hi Thomas,
> >
> > Does it mean future Pipeline implementations would allow applications to
> > set the runner after a pipeline has been
On Tue, Aug 15, 2017 at 10:21 AM, Shen Li wrote:
> Hi Thomas,
>
> Does it mean future Pipeline implementations would allow applications to
> set the runner after a pipeline has been constructed?
Correct, that's the intent.
>
> Thanks,
> Shen
>
> On Tue, Aug 15, 2017 at 12:36 PM, Thomas Groh
> w
Hi Thomas,
Does it mean future Pipeline implementations would allow applications to
set the runner after a pipeline has been constructed?
Thanks,
Shen
On Tue, Aug 15, 2017 at 12:36 PM, Thomas Groh
wrote:
> This style of method doesn't fit with the current approach of pipeline
> construction, w
This style of method doesn't fit with the current approach of pipeline
construction, where the PipelineRunner need not be specified until the
pipeline is run; as such, the runner can't observe the construction of the
Pipeline, as it may not exist during the construction of the Pipeline.
On Tue, Au
... And remember it and make available inside PCollection (which
application produced this collection).
On Tue, Aug 15, 2017, 8:39 AM Eugene Kirpichov wrote:
> In general, no - but the implementation of PAssertionSite exemplifies the
> approach. I guess it could be useful to make this a general
In general, no - but the implementation of PAssertionSite exemplifies the
approach. I guess it could be useful to make this a general beam feature
and remember it for all transforms. It would probably be best to implement
inside Pipeline.apply().
On Tue, Aug 15, 2017, 7:02 AM Shen Li wrote:
> Hi
Hi Eugene,
Thanks for sharing the info. That PAssertionSite tracks where an assertion
error occurred. Do you know if it is possible to get the class name and
line number where a PTransform was added?
Thanks,
Shen
On Mon, Aug 14, 2017 at 10:54 PM, Eugene Kirpichov <
kirpic...@google.com.invalid>
Hi Shen,
Responding just to one part of your message - "remember the line at which
the PTransform was added": take a look at
https://github.com/apache/beam/pull/2247 which does this for PAssert.
On Mon, Aug 14, 2017 at 7:32 PM Shen Li wrote:
> In 0.5.0 or earlier releases, PipelineRunner provide
In 0.5.0 or earlier releases, PipelineRunner provides an
apply(PTransform, InputT) method which allows runner
implementation to perform actions when the PTransform is added to the
pipeline. In later releases, that apply(...) method has been replaced by a
Pipeline#replaceAll() method, where the runn