Reminder that this is something we ideally address before the next release...
Considering the discussion so far, my preference is that we get away from unknown options and discover valid options from the runner (by expanding the job service). Once the SDK is aware of all valid options, it is possible to provide meaningful feedback to the user (validate or help), and correctly handle scopes and types. I would prefer we don't introduce a (quirky) way of passing unknown options that forces users to type JSON into the command line (or similar acrobatics). To someone wanting to run a pipeline, all options are equally important, whether they are application specific, SDK specific or runner specific. It should be possible to *optionally* qualify/scope (to cover cases where there is ambiguity), but otherwise I prefer the format we currently have. Regarding type inference: Correct handling of numeric types matters, see following issue with protobuf (not JSON): https://issues.apache.org/jira/browse/BEAM-5509 Thomas On Thu, Oct 18, 2018 at 6:55 AM Robert Bradshaw <rober...@google.com> wrote: > On Wed, Oct 17, 2018 at 11:35 PM Lukasz Cwik <lc...@google.com> wrote: > >> >> On Tue, Oct 16, 2018 at 11:51 AM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> On Tue, Oct 16, 2018 at 7:03 PM Lukasz Cwik <lc...@google.com> wrote: >>> > >>> > For all unknown options, the SDK can require that all flag values be >>> specified explicitly as a valid JSON type. >>> > starts with { -> object >>> > starts with [ -> list >>> > starts with " -> string >>> > is null / true / false -> null / true / false >>> > otherwise is number. >>> > >>> > This isn't great for strings but works well for all the other types. >>> > >>> > Thus for known options, the additional typing information would >>> disambiguate whether something should be a >>> string/boolean/number/object/list but for unknown options we would expect >>> the user to use valid JSON explicitly and write: >>> > --foo={"object": "value"} >>> > --foo=["value", "value2"] >>> > --foo="string value" >>> >>> Due to shell escaping, one would have to write >>> >>> --foo=\"string value\" >>> >>> or actually, due to the space >>> >>> --foo='"string value"' >>> >>> or some other variation on that, which is really unfortunate. (The JSON >>> list/objects would need similar quoting, but that's less surprising.) Also, >>> does this mean we'd only have one kind of number (not integer vs. float, >>> i.e. --parallelism=5.0 works)? I suppose that is JSON. >>> >> >> Yes, I was suspecting that users would need to type the second variant as >> \"...\" I found more burdensome then '"..."' >> >> >>> >>> > --foo=3.5 --foo=-4 >>> > --foo=true --foo=false >>> > --foo=null >>> > This also works if the flag is repeated, so --foo=3.5 --foo=-4 is >>> [3.5, -4] >>> >>> The thing that sparked this discussion was what to do when unknown foo >>> is repeated, but only one value given. >>> >> >> If the person only specifies one value, then they have to disambiguate >> and put it in a list, only if they specify more then one value will they >> have to turn it into a list. >> >> I believe we could come up with other schemes on how to convert unknown >> options to JSON where we prefer strings over non-string types like >> null/true/false/numbers/list/object and require the user to escape out of >> the string default but anything that is too different from strict JSON >> would cause headaches when attempting to explain the format to users. I >> think a happy middle ground would be that we will only require escaping for >> strings which are ambiguous, so things like true, null, false, ... to be >> treated as strings would require the user to escape them. >> > > I'd prefer to avoid inferring the type of an unknown argument based on its > contents, which can lead to surprises. We could declare every unknown type > to be repeated string, and let any parsing/validation occur on the runner. > If desired, we could pass these around as a single "runner options" dict > that runners could inspect and use to populate the actual dict rather than > mixing parsed and unparsed options. > > >> >> >>> > On Tue, Oct 16, 2018 at 7:56 AM Thomas Weise <t...@apache.org> wrote: >>> >> >>> >> Discovering options from the job server seems preferable over >>> replicating runner options in SDKs. >>> >> >>> >> Runners evolve on their own, and with portability the SDK does not >>> need to know anything about the runner. >>> >> >>> >> Regarding --runner-option. It is true that this looks less user >>> friendly. On the other hand it eliminates the possibility of name >>> collisions. >>> >> >>> >> But if options are discovered, the SDK can perform full validation. >>> It would only be necessary to use explicit scoping when there is ambiguity. >>> >> >>> >> Thomas >>> >> >>> >> >>> >> On Tue, Oct 16, 2018 at 3:48 AM Maximilian Michels <m...@apache.org> >>> wrote: >>> >>> >>> >>> Fetching options directly from the Runner's JobServer seems like the >>> >>> ideal solution. I agree with Robert that it creates additional >>> >>> complexity for SDK authors, so the `--runner-option` flag would be an >>> >>> easy and explicit way to specify additional Runner options. >>> >>> >>> >>> The format I prefer would be: --runner_option=key1=val1 >>> >>> --runner_option=key2=val2 >>> >>> >>> >>> Now, from the perspective of end users, I think it is neither >>> convenient >>> >>> nor reasonable to require the use of the `--runner-option` flag. To >>> the >>> >>> user it seems nebulous why some pipeline options live in the >>> top-level >>> >>> option namespace while others need to be nested within an option. >>> This >>> >>> is amplified by there being two Runners the user needs to be aware >>> of, >>> >>> i.e. PortableRunner and the actual Runner (Dataflow/Flink/Spark..). >>> >>> >>> >>> I feel like we would eventually replicate all options in the SDK >>> because >>> >>> otherwise users have to use the `--runner-option`, but at least we >>> can >>> >>> specify options which have not been replicated yet. >>> >>> >>> >>> -Max >>> >>> >>> >>> On 16.10.18 10:27, Robert Bradshaw wrote: >>> >>> > Yes, we don't know how to parse and/or validate it. >>> >>> > >>> >>> > On Tue, Oct 16, 2018 at 1:14 AM Lukasz Cwik <lc...@google.com >>> >>> > <mailto:lc...@google.com>> wrote: >>> >>> > >>> >>> > I see, is the issue that we currently are using a JSON >>> >>> > representation for options when being serialized and when we >>> get >>> >>> > some unknown option, we don't know how to convert it into its >>> JSON form? >>> >>> > >>> >>> > On Mon, Oct 15, 2018 at 2:41 PM Robert Bradshaw < >>> rober...@google.com >>> >>> > <mailto:rober...@google.com>> wrote: >>> >>> > >>> >>> > On Mon, Oct 15, 2018 at 11:30 PM Lukasz Cwik < >>> lc...@google.com >>> >>> > <mailto:lc...@google.com>> wrote: >>> >>> > > >>> >>> > > On Mon, Oct 15, 2018 at 1:17 PM Robert Bradshaw >>> >>> > <rober...@google.com <mailto:rober...@google.com>> wrote: >>> >>> > >> >>> >>> > >> On Mon, Oct 15, 2018 at 7:50 PM Lukasz Cwik >>> >>> > <lc...@google.com <mailto:lc...@google.com>> wrote: >>> >>> > >> > >>> >>> > >> > I agree with the sentiment for better error checking. >>> >>> > >> > >>> >>> > >> > We can try to make it such that the SDK can "fetch" >>> the >>> >>> > set of options that the runner supports by making a call >>> to the >>> >>> > Job API. The API could return a list of option names >>> >>> > (descriptions for --help purposes and also potentially the >>> >>> > expected format) which would remove the worry around >>> "unknown" >>> >>> > options. Yes I understand to be able to make the Job API >>> call, >>> >>> > we may need to parse some options from the args parameters >>> first >>> >>> > and then parse the unknown options after they are fetched. >>> >>> > >> >>> >>> > >> This is an interesting idea, but seems it could get >>> quite >>> >>> > complicated. >>> >>> > >> E.g. for delegating runners, one would first read the >>> options to >>> >>> > >> determine which runner to fetch the options from, which >>> >>> > would then >>> >>> > >> return a set of options that possibly depends on the >>> values >>> >>> > of some of >>> >>> > >> its options... >>> >>> > >> >>> >>> > >> > Alternatively, we can choose an explicit format >>> upfront. >>> >>> > >> > To expand on the exact format for >>> --runner_option=..., >>> >>> > here are some different ideas: >>> >>> > >> > 1) Specified multiple times, each one is an explicit >>> flag >>> >>> > >> > --runner_option=--blah=bar --runner_option=--foo=baz1 >>> >>> > --runner_option=--foo=baz2 >>> >>> > >> >>> >>> > >> I'm -1 on this format. We should move away from the >>> idea >>> >>> > that options >>> >>> > >> == flags (as that doesn't compose well with other >>> libraries >>> >>> > that do >>> >>> > >> their own flags parsing). The ability to parse a set of >>> >>> > flags into >>> >>> > >> options is just a convenience that an author may (or >>> may >>> >>> > not) choose >>> >>> > >> to use (e.g. when running pipelines a long-lived >>> process like a >>> >>> > >> service or a notebook, the command line flags are >>> almost >>> >>> > certainly not >>> >>> > >> the right interface). >>> >>> > >> >>> >>> > >> > 2) specified multiple times, we drop the explicit >>> flag >>> >>> > >> > --runner_option=blah=bar --runner_option=foo=baz1 >>> >>> > --runner_option=foo=baz2 >>> >>> > >> >>> >>> > >> This or (4) is my preference. >>> >>> > >> >>> >>> > >> > 3) we use a string which the runner can choose to >>> >>> > interpret however they want (JSON/XML shown below) >>> >>> > >> > --runner_option='{"blah": "bar", "foo": ["baz1", >>> "baz2"]}' >>> >>> > >> > >>> >>> > >>> --runner_option='<options><blah>bar</blah><foo>baz1</foo><foo>baz2</foo></options>' >>> >>> > >> >>> >>> > >> This would make validation hard. Also, I think it makes >>> >>> > sense for some >>> >>> > >> runner options to be "shared" (parallelism") by >>> convention, >>> >>> > so letting >>> >>> > >> it be a free-form string wouldn't allow different >>> runners to >>> >>> > inspect >>> >>> > >> different bits. >>> >>> > >> >>> >>> > >> We should consider if we should use urns for >>> namespacing, and >>> >>> > >> assigning semantic meaning to strings, here. >>> >>> > >> >>> >>> > >> > 4) we use a string which must be a specific format >>> such as >>> >>> > JSON (allows the SDK to do simple validation): >>> >>> > >> > --runner_option='{"blah": "bar", "foo": ["baz1", >>> "baz2"]}' >>> >>> > >> >>> >>> > >> I like this in that at least some validation can be >>> >>> > performed, and >>> >>> > >> expectations of how to format richer types. On the >>> other >>> >>> > hand it gets >>> >>> > >> a bit verbose, given that most (I'd imagine) options >>> will be >>> >>> > simple. >>> >>> > >> As with normal options, >>> >>> > >> >>> >>> > >> --option1=value1 --option2=value2 >>> >>> > >> >>> >>> > >> is shorthand for {"option1": value1, "option2": >>> value2}. >>> >>> > >> >>> >>> > > I lean to 4 the most. With 2, you run into issues of >>> what >>> >>> > does --runner_option=foo=["a", "b"] >>> --runner_option=foo=["c", >>> >>> > "d"] mean? >>> >>> > > Is it an error or list of lists or concatenated. Similar >>> >>> > issues for map types represented via JSON object {...} >>> >>> > >>> >>> > We can err to be on the safe side unless/until an argument >>> can >>> >>> > be made >>> >>> > that merging is more natural. I just think this will be >>> excessively >>> >>> > verbose to use. >>> >>> > >>> >>> > >> > I would strongly suggest that we go with the "fetch" >>> >>> > approach, since this makes the set of options discoverable >>> and >>> >>> > helps users find errors much earlier in their pipeline. >>> >>> > >> >>> >>> > >> This seems like an advanced feature that SDKs may want >>> to >>> >>> > support, but >>> >>> > >> I wouldn't want to require this complexity for >>> bootstrapping >>> >>> > an SDK. >>> >>> > >> >>> >>> > > SDKs that are starting off wouldn't need to "fetch" >>> options, >>> >>> > they could choose to not support runner options or they >>> could >>> >>> > choose to pass all options through to the runner blindly. >>> >>> > Fetching the options only provides the SDK the ability to >>> >>> > provide error checking upfront and useful error/help >>> messages. >>> >>> > >>> >>> > But how to even pass all options through blindly is >>> exactly the >>> >>> > difficulty we're running into here. >>> >>> > >>> >>> > >> Regarding always keeping runner options separate, +1, >>> though >>> >>> > I'm not >>> >>> > >> sure the line is always clear. >>> >>> > >> >>> >>> > >> >>> >>> > >> > On Mon, Oct 15, 2018 at 8:04 AM Robert Bradshaw >>> >>> > <rober...@google.com <mailto:rober...@google.com>> wrote: >>> >>> > >> >> >>> >>> > >> >> On Mon, Oct 15, 2018 at 3:58 PM Maximilian Michels >>> >>> > <m...@apache.org <mailto:m...@apache.org>> wrote: >>> >>> > >> >> > >>> >>> > >> >> > I agree that the current approach breaks the >>> pipeline >>> >>> > options contract >>> >>> > >> >> > because "unknown" options get parsed in the same >>> way as >>> >>> > options which >>> >>> > >> >> > have been defined by the user. >>> >>> > >> >> >>> >>> > >> >> FWIW, I think we're already breaking this >>> "contract." >>> >>> > Unknown options >>> >>> > >> >> are silently ignored; with this change we just >>> change how >>> >>> > we record >>> >>> > >> >> them. It still feels a bit hacky though. >>> >>> > >> >> >>> >>> > >> >> > I'm not sure the `experiments` flag works for us. >>> AFAIK >>> >>> > it only allows >>> >>> > >> >> > true/false flags. We want to pass all types of >>> pipeline >>> >>> > options to the >>> >>> > >> >> > Runner. >>> >>> > >> >> >>> >>> > >> >> Experiments is an arbitrary set of strings, which >>> can be >>> >>> > of the form >>> >>> > >> >> "param=value" if that's useful. (Dataflow does >>> this.) >>> >>> > There is, again, >>> >>> > >> >> no namespacing on the param names, but we could >>> user urns >>> >>> > or impose >>> >>> > >> >> some other structure here. >>> >>> > >> >> >>> >>> > >> >> > How to solve this? >>> >>> > >> >> > >>> >>> > >> >> > 1) Add all options of all Runners to each SDK >>> >>> > >> >> > We added some of the FlinkRunner options to the >>> Python >>> >>> > SDK but realized >>> >>> > >> >> > syncing is rather cumbersome in the long term. >>> However, >>> >>> > we want the most >>> >>> > >> >> > important options to be validated on the client >>> side. >>> >>> > >> >> >>> >>> > >> >> I don't think this is sustainable in the long run. >>> >>> > However, thinking >>> >>> > >> >> about this, in the worse case validation happens >>> after >>> >>> > construction >>> >>> > >> >> but before execution (as with much of our other >>> >>> > validation) so it >>> >>> > >> >> isn't that bad. >>> >>> > >> >> >>> >>> > >> >> > 2) Pass "unknown" options via a separate list in >>> the >>> >>> > Proto which can >>> >>> > >> >> > only be accessed internally by the Runners. This >>> still >>> >>> > allows passing >>> >>> > >> >> > arbitrary options but we wouldn't leak unknown >>> options >>> >>> > and display them >>> >>> > >> >> > as top-level options. >>> >>> > >> >> >>> >>> > >> >> I think there needs to be a way for the user to >>> >>> > communicate values >>> >>> > >> >> directly to the runner regardless of the SDK. My >>> >>> > preference would be >>> >>> > >> >> to make this explicit, e.g. (repeated) >>> >>> > --runner_option=..., rather >>> >>> > >> >> than scooping up all unknown flags at command line >>> >>> > parsing time. >>> >>> > >> >> Perhaps an SDK that is aware of some runners could >>> choose >>> >>> > to lift >>> >>> > >> >> these as top-level options, but still pass them as >>> runner >>> >>> > options. >>> >>> > >> >> >>> >>> > >> >> > On 13.10.18 02:34, Charles Chen wrote: >>> >>> > >> >> > > The current release branch >>> >>> > >> >> > > >>> >>> > (https://github.com/apache/beam/commits/release-2.8.0) >>> was cut >>> >>> > after the >>> >>> > >> >> > > revert went in. Sent out >>> >>> > https://github.com/apache/beam/pull/6683 as a >>> >>> > >> >> > > revert of the revert. Regarding your comment >>> above, >>> >>> > I can help out with >>> >>> > >> >> > > the design / PR reviews for common Python code >>> as you >>> >>> > suggest. >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at 4:48 PM Thomas Weise >>> >>> > <t...@apache.org <mailto:t...@apache.org> >>> >>> > >> >> > > <mailto:t...@apache.org <mailto:t...@apache.org>>> >>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > Thanks, will tag you and looking forward to >>> >>> > feedback so we can >>> >>> > >> >> > > ensure that changes work for everyone. >>> >>> > >> >> > > >>> >>> > >> >> > > Looking at the PR, I see agreement from Max >>> to >>> >>> > revert the change on >>> >>> > >> >> > > the release branch, but not in master. >>> Would you >>> >>> > mind to restore it >>> >>> > >> >> > > in master? >>> >>> > >> >> > > >>> >>> > >> >> > > Thanks >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at 4:40 PM Ahmet Altay >>> >>> > <al...@google.com <mailto:al...@google.com> >>> >>> > >> >> > > <mailto:al...@google.com >>> >>> > <mailto:al...@google.com>>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at 11:31 AM, >>> Charles >>> >>> > Chen <c...@google.com <mailto:c...@google.com> >>> >>> > >> >> > > <mailto:c...@google.com >>> >>> > <mailto:c...@google.com>>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > What I mean is that a user may find >>> that >>> >>> > it works for them >>> >>> > >> >> > > to pass "--myarg blah" and access >>> it as >>> >>> > "options.myarg" >>> >>> > >> >> > > without explicitly defining a >>> "my_arg" >>> >>> > flag due to the added >>> >>> > >> >> > > logic. This is not the intended >>> behavior >>> >>> > and we may want to >>> >>> > >> >> > > change this implementation detail >>> in the >>> >>> > future. However, >>> >>> > >> >> > > having this logic in a released >>> version >>> >>> > makes it hard to >>> >>> > >> >> > > change this behavior since users may >>> >>> > erroneously depend on >>> >>> > >> >> > > this undocumented behavior. >>> Instead, we >>> >>> > should namespace / >>> >>> > >> >> > > scope this so that it is obvious >>> that >>> >>> > this is meant for >>> >>> > >> >> > > runner (and not Beam user) >>> consumption. >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at 10:48 AM >>> Thomas Weise >>> >>> > >> >> > > <t...@apache.org <mailto: >>> t...@apache.org> >>> >>> > <mailto:t...@apache.org <mailto:t...@apache.org>>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > Can you please elaborate more >>> what >>> >>> > practical problems >>> >>> > >> >> > > this introduces for users? >>> >>> > >> >> > > >>> >>> > >> >> > > I can see that this change >>> allows a >>> >>> > user to specify a >>> >>> > >> >> > > runner specific option, which >>> in the >>> >>> > future may change >>> >>> > >> >> > > because we decide to scope >>> >>> > differently. If this only >>> >>> > >> >> > > affects users of the portable >>> Flink >>> >>> > runner (like us), >>> >>> > >> >> > > then no need to revert, because >>> at >>> >>> > this early stage we >>> >>> > >> >> > > prefer something that works over >>> >>> > being blocked. >>> >>> > >> >> > > >>> >>> > >> >> > > It would also be really great >>> if some >>> >>> > of the core Python >>> >>> > >> >> > > SDK developers could help out >>> with >>> >>> > the design aspects >>> >>> > >> >> > > and PR reviews of changes that >>> affect >>> >>> > common Python >>> >>> > >> >> > > code. Anyone who specifically >>> wants >>> >>> > to be tagged on >>> >>> > >> >> > > relevant JIRAs and PRs? >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > I would be happy to be tagged, and I >>> can also >>> >>> > help with >>> >>> > >> >> > > including other relevant folks whenever >>> >>> > possible. In general I >>> >>> > >> >> > > think Robert, Charles, myself are good >>> >>> > candidates. >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > Thanks >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at 10:20 AM >>> >>> > Ahmet Altay >>> >>> > >> >> > > <al...@google.com >>> >>> > <mailto:al...@google.com> <mailto:al...@google.com >>> >>> > <mailto:al...@google.com>>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at >>> 10:11 AM, >>> >>> > Charles Chen >>> >>> > >> >> > > <c...@google.com >>> >>> > <mailto:c...@google.com> <mailto:c...@google.com >>> >>> > <mailto:c...@google.com>>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > For context, I made >>> comments on >>> >>> > >> >> > > https://github.com/apache/beam/pull/6600 noting >>> >>> > >> >> > > that the changes being >>> made >>> >>> > were not good for >>> >>> > >> >> > > Beam >>> >>> > backwards-compatibility. The change as is >>> >>> > >> >> > > allows users to use >>> pipeline >>> >>> > options without >>> >>> > >> >> > > explicitly defining >>> them, >>> >>> > which is not the type >>> >>> > >> >> > > of usage we would like >>> to >>> >>> > encourage since we >>> >>> > >> >> > > prefer to be explicit >>> >>> > whenever possible. If >>> >>> > >> >> > > users write pipelines >>> with >>> >>> > this sort of pattern, >>> >>> > >> >> > > they will potentially >>> >>> > encounter pain when >>> >>> > >> >> > > upgrading to a later >>> version >>> >>> > since this is an >>> >>> > >> >> > > implementation detail >>> and not >>> >>> > an officially >>> >>> > >> >> > > supported pattern. I >>> agree >>> >>> > with the comments >>> >>> > >> >> > > above that this is >>> ultimately >>> >>> > a scoping issue. >>> >>> > >> >> > > I would not have a >>> problem >>> >>> > with these changes if >>> >>> > >> >> > > they were explicitly >>> scoped >>> >>> > under either a >>> >>> > >> >> > > runner or unparsed >>> options >>> >>> > namespace. >>> >>> > >> >> > > >>> >>> > >> >> > > As a second note, since >>> the >>> >>> > 2.8.0 release is >>> >>> > >> >> > > being cut right now, >>> because >>> >>> > of these >>> >>> > >> >> > > backwards-compatibility >>> >>> > concerns, I would >>> >>> > >> >> > > suggest reverting these >>> >>> > changes, at least until >>> >>> > >> >> > > 2.8.0 is cut, so we can >>> have >>> >>> > a discussion here >>> >>> > >> >> > > before committing to and >>> >>> > releasing any API-level >>> >>> > >> >> > > changes. >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > +1 I would like to revert >>> the >>> >>> > changes in order not >>> >>> > >> >> > > rush this into the release. >>> Once >>> >>> > this discussion >>> >>> > >> >> > > results in an agreement >>> changes >>> >>> > can be brought back. >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, 2018 at >>> 9:26 >>> >>> > AM Henning Rohde >>> >>> > >> >> > > <hero...@google.com >>> >>> > <mailto:hero...@google.com> <mailto:hero...@google.com >>> >>> > <mailto:hero...@google.com>>> >>> >>> > >> >> > > wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > Agree that pipeline >>> >>> > options lack some >>> >>> > >> >> > > mechanism for >>> scoping. It >>> >>> > is also not always >>> >>> > >> >> > > possible distinguish >>> >>> > options meant to be >>> >>> > >> >> > > consumed at pipeline >>> >>> > construction time, by >>> >>> > >> >> > > the runner, by the >>> SDK >>> >>> > harness, by the user >>> >>> > >> >> > > code or any >>> combination >>> >>> > -- and this causes >>> >>> > >> >> > > confusion every now >>> and then. >>> >>> > >> >> > > >>> >>> > >> >> > > For Dataflow, we >>> have >>> >>> > been using >>> >>> > >> >> > > "experiments" for >>> >>> > arbitrary runner-specific >>> >>> > >> >> > > options. It's >>> simply a >>> >>> > string list pipeline >>> >>> > >> >> > > option that all SDKs >>> >>> > support and, for Go at >>> >>> > >> >> > > least, is sent to >>> >>> > portable runners. Flink >>> >>> > >> >> > > can do the same in >>> the >>> >>> > short term to move >>> >>> > >> >> > > forward. >>> >>> > >> >> > > >>> >>> > >> >> > > Henning >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > On Fri, Oct 12, >>> 2018 at >>> >>> > 8:50 AM Thomas Weise >>> >>> > >> >> > > <t...@apache.org >>> >>> > <mailto:t...@apache.org> <mailto:t...@apache.org >>> >>> > <mailto:t...@apache.org>>> wrote: >>> >>> > >> >> > > >>> >>> > >> >> > > [moving to the >>> list] >>> >>> > >> >> > > >>> >>> > >> >> > > The requirement >>> >>> > driving this part of the >>> >>> > >> >> > > change was to >>> allow a >>> >>> > user to specify >>> >>> > >> >> > > pipeline >>> options that >>> >>> > a runner supports >>> >>> > >> >> > > without having >>> to >>> >>> > declare those in each >>> >>> > >> >> > > language SDK. >>> >>> > >> >> > > >>> >>> > >> >> > > In the specific >>> >>> > scenario, we have >>> >>> > >> >> > > options that the >>> >>> > Flink runner supports >>> >>> > >> >> > > (and can >>> validate), >>> >>> > that are not >>> >>> > >> >> > > enumerated in >>> the >>> >>> > Python SDK. >>> >>> > >> >> > > >>> >>> > >> >> > > I think we have >>> a >>> >>> > bigger problem scoping >>> >>> > >> >> > > pipeline >>> options. For >>> >>> > example, the >>> >>> > >> >> > > runner options >>> are >>> >>> > dumped into the SDK >>> >>> > >> >> > > worker. There >>> is also >>> >>> > a possibility of >>> >>> > >> >> > > name >>> collisions. So I >>> >>> > think this would >>> >>> > >> >> > > benefit from >>> broader >>> >>> > feedback. >>> >>> > >> >> > > >>> >>> > >> >> > > Thanks, >>> >>> > >> >> > > Thomas >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > ---------- >>> Forwarded >>> >>> > message --------- >>> >>> > >> >> > > From: *Charles >>> Chen* >>> >>> > >> >> > > >>> >>> > <notificati...@github.com <mailto: >>> notificati...@github.com> >>> >>> > >> >> > > >>> >>> > <mailto:notificati...@github.com >>> >>> > <mailto:notificati...@github.com>>> >>> >>> > >> >> > > Date: Fri, Oct >>> 12, >>> >>> > 2018 at 8:36 AM >>> >>> > >> >> > > Subject: Re: >>> >>> > [apache/beam] [BEAM-5442] >>> >>> > >> >> > > Store duplicate >>> >>> > unknown options in a >>> >>> > >> >> > > list argument >>> (#6600) >>> >>> > >> >> > > To: apache/beam >>> >>> > <b...@noreply.github.com <mailto:b...@noreply.github.com> >>> >>> > >> >> > > >>> >>> > <mailto:b...@noreply.github.com <mailto: >>> b...@noreply.github.com>>> >>> >>> > >> >> > > Cc: Thomas Weise >>> >>> > <thomas.we...@gmail.com <mailto:thomas.we...@gmail.com> >>> >>> > >> >> > > >>> >>> > <mailto:thomas.we...@gmail.com <mailto: >>> thomas.we...@gmail.com>>>, >>> >>> > >> >> > > Mention >>> >>> > <ment...@noreply.github.com <mailto: >>> ment...@noreply.github.com> >>> >>> > >> >> > > >>> >>> > <mailto:ment...@noreply.github.com >>> >>> > <mailto:ment...@noreply.github.com>>> >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > CC: @tweise >>> >>> > <https://github.com/tweise> >>> >>> > >> >> > > >>> >>> > >> >> > > — >>> >>> > >> >> > > You are >>> receiving >>> >>> > this because you were >>> >>> > >> >> > > mentioned. >>> >>> > >> >> > > Reply to this >>> email >>> >>> > directly, view it on >>> >>> > >> >> > > GitHub >>> >>> > >> >> > > >>> >>> > < >>> https://github.com/apache/beam/pull/6600#issuecomment-429367754>, >>> >>> > >> >> > > or mute the >>> thread >>> >>> > >> >> > > >>> >>> > < >>> https://github.com/notifications/unsubscribe-auth/AAQGDwwt15R85eq9pySUisyxq2HYz-Vyks5ukLcLgaJpZM4XMo-T >>> >. >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >> >> > > >>> >>> > >>> >>
