This is not exactly a problem. Having the runner to explicitly declare
its capabilities is of course a possibility. But do we want to modify
each runner, in case all the functionality is actually provided by a
common library? This is about usability. I can imagine (and easier if we
can provide
Good question. My last sentence was not clear. We do not need to
automatically propagate the capabilities offered by runners-core to a
particular runner. The runner can (and should) own the claim of what its
capabilities are.
Kenn
On Thu, Feb 20, 2020 at 10:05 PM Luke Cwik wrote:
> Which part o
Which part of the proposal do you think is solving a problem we may not
have?
On Thu, Feb 20, 2020 at 8:19 PM Kenneth Knowles wrote:
> I would rather say that "runners-core" is a utility library with some
> helpful things. Like other libraries. The runner still decides how to use
> the library.
I would rather say that "runners-core" is a utility library with some
helpful things. Like other libraries. The runner still decides how to use
the library. That was the idea, anyhow. A runner could have a bunch of "if"
statements around how it uses some generic runners-core utility, etc. I
think a
On 2/20/20 8:24 PM, Robert Bradshaw wrote:
On Thu, Feb 13, 2020 at 12:42 PM Jan Lukavský wrote:
Hi,
+1 for adding pipeline required features. I think being able to reject pipeline
with unknown requirement is pretty much needed, mostly because that enables
runners to completely decouple from
On Thu, Feb 13, 2020 at 12:42 PM Jan Lukavský wrote:
>
> Hi,
>
> +1 for adding pipeline required features. I think being able to reject
> pipeline with unknown requirement is pretty much needed, mostly because that
> enables runners to completely decouple from SDKs, while being able to
> recogn
Cool, thanks. :-)
Jan
On 2/14/20 11:35 PM, Robert Bradshaw wrote:
Oh, sorry. Try it again
https://docs.google.com/document/d/1CyVElQDYHBRfXu6k1VSXv3Yok_4r8c4V0bkh2nFAWYc/edit?usp=sharing
On Fri, Feb 14, 2020 at 2:04 PM Jan Lukavský wrote:
Hi Robert,
the doc seems to be locked.
Jan
On 2/14
Oh, sorry. Try it again
https://docs.google.com/document/d/1CyVElQDYHBRfXu6k1VSXv3Yok_4r8c4V0bkh2nFAWYc/edit?usp=sharing
On Fri, Feb 14, 2020 at 2:04 PM Jan Lukavský wrote:
>
> Hi Robert,
>
> the doc seems to be locked.
>
> Jan
>
> On 2/14/20 10:56 PM, Robert Bradshaw wrote:
> > I created https:/
Hi Robert,
the doc seems to be locked.
Jan
On 2/14/20 10:56 PM, Robert Bradshaw wrote:
I created https://github.com/apache/beam/pull/10873 to add the minimal
set of fields to the existing protos, and also created
https://docs.google.com/document/d/1CyVElQDYHBRfXu6k1VSXv3Yok_4r8c4V0bkh2nFAWYc/e
I created https://github.com/apache/beam/pull/10873 to add the minimal
set of fields to the existing protos, and also created
https://docs.google.com/document/d/1CyVElQDYHBRfXu6k1VSXv3Yok_4r8c4V0bkh2nFAWYc/edit
to start enumerating some of the URNs we may want to have. It would be
a good milestone
> we can take advantage of these pipeline features to get rid of the
categories of @ValidatesRunner tests, because we could have just simply
@ValidatesRunner and each test would be matched against runner capabilities
+1, I think the potential to formally integrate our idea of compatibility
and uni
Wrt per DoFn/ParDo level, there's the similar case of wether the DoFn has
an Urn for requiring something or it's an annotation for saying the DoFn
provides something (eg. Provides K-anonymization with k defined)
The general theme of this thread seems to be trying to ensure a runner can
reject a pi
On Thu, Feb 13, 2020 at 12:42 PM Jan Lukavský wrote:
> Hi,
>
> +1 for adding pipeline required features. I think being able to reject
> pipeline with unknown requirement is pretty much needed, mostly because
> that enables runners to completely decouple from SDKs, while being able to
> recognize
Hi,
+1 for adding pipeline required features. I think being able to reject
pipeline with unknown requirement is pretty much needed, mostly because
that enables runners to completely decouple from SDKs, while being able
to recognize when a pipeline constructed with incomplatible version of
SDK
+1 to deferring for now. Since they should not be modified after adoption,
it makes sense not to get ahead of ourselves.
On Thu, Feb 13, 2020, 10:59 AM Robert Bradshaw wrote:
> On Thu, Feb 13, 2020 at 10:12 AM Robert Burke wrote:
> >
> > One thing that doesn't appear to have been suggested yet
On Thu, Feb 13, 2020 at 10:12 AM Robert Burke wrote:
>
> One thing that doesn't appear to have been suggested yet is we could "batch"
> urns together under a "super urn" so that adding one super urn is like adding
> each of the represented batch of features. This prevents needing to send
> doze
One thing that doesn't appear to have been suggested yet is we could
"batch" urns together under a "super urn" so that adding one super urn is
like adding each of the represented batch of features. This prevents
needing to send dozens of urns to be individually sent over.
The super urns would nee
On Wed, Feb 12, 2020 at 2:24 PM Kenneth Knowles wrote:
>
>
> On Wed, Feb 12, 2020 at 12:04 PM Robert Bradshaw
> wrote:
>
>> On Wed, Feb 12, 2020 at 11:08 AM Luke Cwik wrote:
>> >
>> > We can always detect on the runner/SDK side whether there is an unknown
>> field[1] within a payload and fail t
On Wed, Feb 12, 2020 at 12:04 PM Robert Bradshaw
wrote:
> On Wed, Feb 12, 2020 at 11:08 AM Luke Cwik wrote:
> >
> > We can always detect on the runner/SDK side whether there is an unknown
> field[1] within a payload and fail to process it but this is painful in two
> situations:
> > 1) It doesn'
On Wed, Feb 12, 2020 at 11:08 AM Luke Cwik wrote:
>
> We can always detect on the runner/SDK side whether there is an unknown
> field[1] within a payload and fail to process it but this is painful in two
> situations:
> 1) It doesn't provide for a good error message since you can't say what the
On Wed, Feb 12, 2020 at 7:57 AM Robert Bradshaw wrote:
> On Tue, Feb 11, 2020 at 7:25 PM Kenneth Knowles wrote:
> >
> > On Tue, Feb 11, 2020 at 8:38 AM Robert Bradshaw
> wrote:
> >>
> >> On Mon, Feb 10, 2020 at 7:35 PM Kenneth Knowles
> wrote:
> >> >
> >> > On the runner requirements side: if
On Tue, Feb 11, 2020 at 7:25 PM Kenneth Knowles wrote:
>
> On Tue, Feb 11, 2020 at 8:38 AM Robert Bradshaw
wrote:
>>
>> On Mon, Feb 10, 2020 at 7:35 PM Kenneth Knowles wrote:
>> >
>> > On the runner requirements side: if you have such a list at the
pipeline level, it is an opportunity for the li
On Tue, Feb 11, 2020 at 8:38 AM Robert Bradshaw wrote:
> On Mon, Feb 10, 2020 at 7:35 PM Kenneth Knowles wrote:
> >
> > On the runner requirements side: if you have such a list at the pipeline
> level, it is an opportunity for the list to be inconsistent with the
> contents of the pipeline. For
On Mon, Feb 10, 2020 at 7:35 PM Kenneth Knowles wrote:
>
> On the runner requirements side: if you have such a list at the pipeline
> level, it is an opportunity for the list to be inconsistent with the contents
> of the pipeline. For example, if a DoFn is marked "requires stable input" but
> n
On the runner requirements side: if you have such a list at the pipeline
level, it is an opportunity for the list to be inconsistent with the
contents of the pipeline. For example, if a DoFn is marked "requires stable
input" but not listed at the pipeline level, then the runner may run it
without e
I like the capabilities/requirements idea. Would these capabilities be at a
level that it would make sense to document in the capabilities matrix? i.e.
could the URNs be the values of "X" Pablo described here [1].
Brian
[1]
https://lists.apache.org/thread.html/e93ac64d484551d61e559e1ba0cf4a15b760
With an eye towards cross-language (which includes cross-version)
pipelines and services (specifically looking at Dataflow) supporting
portable pipelines, there's been a desire to stabilize the portability
protos. There are currently many cleanups we'd like to do [1] (some
essential, others nice to
27 matches
Mail list logo
