Yes, we moves a few steps forward - was waiting before speaking of it but seems it led to it.
Idea is to prepend a prefix per IO in the key. Currently the easiest working solution is to sanitize the transform/fn name (since we validate the unicity by default) and we are done. If not znough we can pass the actual prefix to use in the pipeline but I doubt we need to be that fancy. Logic would just be like https://github.com/apache/tomee/blob/master/container/openejb-core/src/main/java/org/apache/openejb/config/ConfigurationFactory.java#L1542 at the end. Le 12 janv. 2018 20:39, "Lukasz Cwik" <lc...@google.com> a écrit : Your original proposal was too vague so I was extrapolating on what I thought you meant. At this point it seems like what I extrapolated and what your talking about are far enough apart that I can't extrapolate what you mean and you'll need to be significantly more detailed in what your suggesting. With the system property approach, how would a user use the same IO multiple times? On Fri, Jan 12, 2018 at 10:18 AM, Romain Manni-Bucau <rmannibu...@gmail.com> wrote: > > 2018-01-12 19:12 GMT+01:00 Lukasz Cwik <lc...@google.com>: > >> >> >> On Fri, Jan 12, 2018 at 10:01 AM, Romain Manni-Bucau < >> rmannibu...@gmail.com> wrote: >> >>> >>> 2018-01-12 18:54 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>> >>>> Some concerns would be: >>>> * How does a user discover what options can be set and what values they >>>> take? >>>> >>> >>> Tempted to say "same as today", what's the regression you would see? A >>> --verbose can be nice thought to dump the DAG at startup (before >>> translation) - but with this impl or not ;). >>> >> PipelineOptionsFactory supports passing in "--help" which prints all the >> descriptions of what options it is aware of that can be set to the >> terminal. Also, users can find all PipelineOptions interfaces and all >> sub-interfaces using an IDE or javadoc. >> > > Yep, same except it would be enriched with the (included optional > prefixes). > > But still misses the DAG dump which is valuable to go one step further and > allows a fully configurable through options DAG - read from system > properties or not, this is just the properties source ending in a String[]. > > >> >> >>> >>> >>>> * System properties are global so how would you prevent system >>>> properties from conflicting within Apache Beam and with other non Apache >>>> Beam libraries that may rely on system properties? >>>> >>> >>> The prefixes I mentionned. Far enough in all libs so why not lib, >>> anything special I missed? >>> >> Thats a good amount of additional typing per option, even if its just the >> 5 characters "beam.". >> > > You do it in your launcher but enables you to do it. Today it is up to > each dev to store its own config without any normalization which is a lost > of value IMHO we can easily fix. > > >> >>> >>>> * There are users who launch more then one pipeline from the same >>>> application, how would system properties work there? >>>> >>> >>> And these users don't use the system properties but a custom way to >>> build String[] with their own configs so no issues I think. As you can or >>> not use the beamTestPipelineOptions system property, you would use this way >>> to configure it or not. >>> But having it would be nice for a lot of users + tests which wouldnt >>> really need this beamTestPipelineOptions (thinking out loud). >>> >> That is only for integration testing and using multiple system properties >> that then get mapped to PipelineOptions just for integration testing seems >> like we will add a whole level of complexity for little gain. >> > > Can be, so we keep the previous values which are way sufficient from what > I saw in batch usages and kind of lack in a built-in fashion in Beam for > now. > > >> >>> >>>> >>>> Note that the system property usage is only for integration testing and >>>> not meant for users writing pipelines. >>>> >>>> On Thu, Jan 11, 2018 at 9:38 PM, Romain Manni-Bucau < >>>> rmannibu...@gmail.com> wrote: >>>> >>>>> Hmm, here is my thought: allowing main options settings is super >>>>> important and one of the most important user experience point. I d even >>>>> say >>>>> any IO config shouldnt be an auto value but an option and through the >>>>> transform name you should be able to prefux the option name and override a >>>>> particular IO config through the command line - just to say how important >>>>> it is and avois hack by IO for that. >>>>> >>>>> The fact there is this global system property, json integration, and >>>>> in this PR another system property which is a bit weird for a user IMHO >>>>> can >>>>> mean we nees to update the config design. If each option has an equivalent >>>>> system property then no need of all these hacks. >>>>> >>>>> Ex: tempLocation would support -DtempLocation, -Dbeam.tempLocation, >>>>> -D<options simplename|qualified name>.templocation (+ transform name >>>>> prefix >>>>> if we add IO config one day) >>>>> >>>>> Then jackson is only here to parse the options....which is already >>>>> done manually since options use a custom serializer so it can be dropped, >>>>> no? >>>>> >>>>> Is this reasonning wrong? >>>>> >>>>> Le 12 janv. 2018 00:11, "Lukasz Cwik" <lc...@google.com> a écrit : >>>>> >>>>>> Robert Bradshaw had the idea of migrating away from using >>>>>> main(String[] args) and just refactoring the code to test the WordCount >>>>>> PTransform allowing one to write a traditional JUnit test that didn't >>>>>> call >>>>>> main(String[] args). This would change what the contents of our examples >>>>>> are and make them more amenable to testing which could be good guidance >>>>>> for >>>>>> developers as well. >>>>>> >>>>>> On Thu, Jan 11, 2018 at 3:08 PM, Lukasz Cwik <lc...@google.com> >>>>>> wrote: >>>>>> >>>>>>> 1) TestPipeline#convertToArgs is meant to convert >>>>>>> TestPipelineOptions into a String[] containing --arg=value for >>>>>>> integration >>>>>>> tests that only have a main(String[] arg) entry point like WordCount. >>>>>>> There >>>>>>> is this PR[1] that is outstanding that is attempting to clean this up >>>>>>> and >>>>>>> simplify it so that we aren't doing this haphazard conversion. >>>>>>> [1] https://github.com/apache/beam/pull/4346 >>>>>>> >>>>>>> I haven't yet commented on the PR but I was going to suggest that >>>>>>> the user add String[] getMainArgs to TestPipelineOptions and not do the >>>>>>> convertToArgs hackiness for ITs that use main(). >>>>>>> >>>>>>> On Thu, Jan 11, 2018 at 1:53 PM, Romain Manni-Bucau < >>>>>>> rmannibu...@gmail.com> wrote: >>>>>>> >>>>>>>> Some inputs - maybe comments? - would be appreciated on: >>>>>>>> >>>>>>>> 1. Why using json in https://github.com/apache/b >>>>>>>> eam/blob/master/sdks/java/core/src/main/java/org/apache/beam >>>>>>>> /sdk/testing/TestPipeline.java#L480 and not iterate over the >>>>>>>> options? >>>>>>>> 2. Also this is likely used to do a fromArgs and create another >>>>>>>> pipeline options later no? Is this api that useful here or does it >>>>>>>> need an >>>>>>>> options composition - fully object? >>>>>>>> >>>>>>>> Le 11 janv. 2018 19:09, "Lukasz Cwik" <lc...@google.com> a écrit : >>>>>>>> >>>>>>>>> Give links to the code segments that you want background on. >>>>>>>>> >>>>>>>>> On Wed, Jan 10, 2018 at 12:44 AM, Romain Manni-Bucau < >>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> updated my junit 5 PR to show that: https://github.com/apach >>>>>>>>>> e/beam/pull/4360/files#diff-578d1770f8b47ebbc1e74a2c19de9a6aR28 >>>>>>>>>> >>>>>>>>>> It doesn't remove jackson yet but exposes a nicer user interface >>>>>>>>>> for the config. >>>>>>>>>> >>>>>>>>>> I'm not fully clear on all the jackson usage yet, there are some >>>>>>>>>> round trips (PO -> json -> PO) which are weird without more >>>>>>>>>> knowledge. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Romain Manni-Bucau >>>>>>>>>> @rmannibucau <https://twitter.com/rmannibucau> | Blog >>>>>>>>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> >>>>>>>>>> >>>>>>>>>> 2018-01-09 22:57 GMT+01:00 Lukasz Cwik <lc...@google.com>: >>>>>>>>>> >>>>>>>>>>> Removing large dependencies is great but I can only assume it >>>>>>>>>>> will be a lot of work so if you can get it to work in a backwards >>>>>>>>>>> compatible way great. >>>>>>>>>>> >>>>>>>>>>> On Tue, Jan 9, 2018 at 1:52 PM, Romain Manni-Bucau < >>>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> It conflicts easily between libs and containers. Shade is not a >>>>>>>>>>>> good option too - see the thread on this topic :(. >>>>>>>>>>>> >>>>>>>>>>>> At the end i see using the cli solution closer to user - vs >>>>>>>>>>>> framework dev for json - and hurting less in terms of classpath so >>>>>>>>>>>> pby sthg >>>>>>>>>>>> to test no? >>>>>>>>>>>> >>>>>>>>>>>> Le 9 janv. 2018 22:47, "Lukasz Cwik" <lc...@google.com> a >>>>>>>>>>>> écrit : >>>>>>>>>>>> >>>>>>>>>>>>> Romain, how has Jackson been a classpath breaker? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Tue, Jan 9, 2018 at 1:20 PM, Romain Manni-Bucau < >>>>>>>>>>>>> rmannibu...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hmm, beam already owns the cli parsing - that is what I meant >>>>>>>>>>>>>> - it only misses the arg delimiter (ie quoting) and adding it is >>>>>>>>>>>>>> easy no? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Le 9 janv. 2018 21:19, "Robert Bradshaw" <rober...@google.com> >>>>>>>>>>>>>> a écrit : >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Jan 9, 2018 at 11:48 AM, Romain Manni-Bucau >>>>>>>>>>>>>>> <rmannibu...@gmail.com> wrote: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Le 9 janv. 2018 19:50, "Robert Bradshaw" < >>>>>>>>>>>>>>> rober...@google.com> a écrit : >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > From what I understand: >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > 1) The command line argument values (both simple and more >>>>>>>>>>>>>>> complex) are >>>>>>>>>>>>>>> > all JSON-representable. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > And must be all CLI representable >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Sorry, I should have said "all pipeline options." In any >>>>>>>>>>>>>>> case, one can >>>>>>>>>>>>>>> always do JSON -> string and the resulting string is usable >>>>>>>>>>>>>>> as a >>>>>>>>>>>>>>> command line argument. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > 2) The command line is a mapping of keys to these values. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Which makes your 1 not true since json supports more ;) >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > As such, it seems quite natural to represent the whole set >>>>>>>>>>>>>>> as a single >>>>>>>>>>>>>>> > JSON map, rather than using a different, custom encoding >>>>>>>>>>>>>>> for the top >>>>>>>>>>>>>>> > level (whose custom escaping would have to be carried into >>>>>>>>>>>>>>> the inner >>>>>>>>>>>>>>> > JSON values). Note that JSON has the advantage that one >>>>>>>>>>>>>>> never needs to >>>>>>>>>>>>>>> > explain or define it, and parsers/serializers already >>>>>>>>>>>>>>> exists for all >>>>>>>>>>>>>>> > languages (e.g. if one has a driver script in another >>>>>>>>>>>>>>> language for >>>>>>>>>>>>>>> > launching a java pipeline, it's easy to communicate all >>>>>>>>>>>>>>> the args). >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > Same reasonning applies to CLI AFAIK >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The spec of what a valid command line argument list is is >>>>>>>>>>>>>>> surprisingly >>>>>>>>>>>>>>> inconsistent across platforms, languages, and programs. And >>>>>>>>>>>>>>> your >>>>>>>>>>>>>>> proposal seems to be getting into what the delimiter is, and >>>>>>>>>>>>>>> how to >>>>>>>>>>>>>>> escape it, and possibly then how to escape the escape >>>>>>>>>>>>>>> character. All >>>>>>>>>>>>>>> of this is sidestepped by pointing at an existing spec. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > We can't get rid of Jackson in the core because of (1) so >>>>>>>>>>>>>>> there's >>>>>>>>>>>>>>> > little value in adding complexity to remove it from (2). >>>>>>>>>>>>>>> The fact that >>>>>>>>>>>>>>> > Java doesn't ship anything in its expansive standard >>>>>>>>>>>>>>> library for this >>>>>>>>>>>>>>> > is unfortuante, so we have to take a dependency on >>>>>>>>>>>>>>> something. >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > We actually can as shown before >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> How, if JSON is integral to the parsing of the argument >>>>>>>>>>>>>>> values >>>>>>>>>>>>>>> themselves? (Or is the argument that it's not, but each >>>>>>>>>>>>>>> extender of >>>>>>>>>>>>>>> PipelineOptions is responsible for choosing a JSON parser >>>>>>>>>>>>>>> and doing it >>>>>>>>>>>>>>> themselves.) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > and jackson is not a simple negligible lib >>>>>>>>>>>>>>> > but a classpath breaker :( >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Perhaps there are better libraries we could use instead? Is >>>>>>>>>>>>>>> the >>>>>>>>>>>>>>> ecosystem so bad that it encourages projects to roll their >>>>>>>>>>>>>>> own of >>>>>>>>>>>>>>> everything rather than share code? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > On Tue, Jan 9, 2018 at 10:00 AM, Lukasz Cwik < >>>>>>>>>>>>>>> lc...@google.com> wrote: >>>>>>>>>>>>>>> >> Ah, you don't want JSON within JSON. I see, if thats the >>>>>>>>>>>>>>> case just migrate >>>>>>>>>>>>>>> >> all of them to string tokenization and drop the Jackson >>>>>>>>>>>>>>> usage for string >>>>>>>>>>>>>>> >> -> >>>>>>>>>>>>>>> >> string[] conversion. >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> >> On Mon, Jan 8, 2018 at 10:06 PM, Romain Manni-Bucau >>>>>>>>>>>>>>> >> <rmannibu...@gmail.com> >>>>>>>>>>>>>>> >> wrote: >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> Lukasz, the use case is to znsure the config used can >>>>>>>>>>>>>>> still map the CLI >>>>>>>>>>>>>>> >>> and that options dont start to abuse json so it is >>>>>>>>>>>>>>> exactly the opposite >>>>>>>>>>>>>>> >>> of >>>>>>>>>>>>>>> >>> "we can be fancy" and is closer to "we can be robust". >>>>>>>>>>>>>>> Also the default >>>>>>>>>>>>>>> >>> should be easy and not json (i just want to set the >>>>>>>>>>>>>>> runner, --runner=xxx >>>>>>>>>>>>>>> >>> is >>>>>>>>>>>>>>> >>> easier than a json version). If the value doesnt start >>>>>>>>>>>>>>> with a [ we can >>>>>>>>>>>>>>> >>> use >>>>>>>>>>>>>>> >>> the tokenizer else jackson, wdyt? >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> Le 8 janv. 2018 22:59, "Lukasz Cwik" <lc...@google.com> >>>>>>>>>>>>>>> a écrit : >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> Now that I think about this more. I looked at some of >>>>>>>>>>>>>>> the examples in the >>>>>>>>>>>>>>> >>> pom.xml and they don't seem to be tricky to write in >>>>>>>>>>>>>>> JSON. >>>>>>>>>>>>>>> >>> I also looked at the Jenkins job configurations >>>>>>>>>>>>>>> (specifically the >>>>>>>>>>>>>>> >>> performance tests) and they pass around maps which they >>>>>>>>>>>>>>> convert to the >>>>>>>>>>>>>>> >>> required format without needing a user to write it >>>>>>>>>>>>>>> themselves. >>>>>>>>>>>>>>> >>> Using gradle, we will be able to trivially to do the >>>>>>>>>>>>>>> same thing (convert >>>>>>>>>>>>>>> >>> maps to json without needing the person to write it by >>>>>>>>>>>>>>> hand) like groovy >>>>>>>>>>>>>>> >>> does. >>>>>>>>>>>>>>> >>> Since we are migrating away from Maven it doesn't seem >>>>>>>>>>>>>>> worthwhile to >>>>>>>>>>>>>>> >>> spend >>>>>>>>>>>>>>> >>> time on to make it easier to write the args in the Maven >>>>>>>>>>>>>>> poms. >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> Is there another use case that is being missed? >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> On Mon, Jan 8, 2018 at 1:38 PM, Romain Manni-Bucau >>>>>>>>>>>>>>> >>> <rmannibu...@gmail.com> >>>>>>>>>>>>>>> >>> wrote: >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> Good point for \t and ,. >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> Any objection to use jackson as a fallback for that >>>>>>>>>>>>>>> purpose - for >>>>>>>>>>>>>>> >>>> backwqrd compat only - and make it optional then? Will >>>>>>>>>>>>>>> create the ticket >>>>>>>>>>>>>>> >>>> if >>>>>>>>>>>>>>> >>>> not. >>>>>>>>>>>>>>> >>>> >>>>>>>>>>>>>>> >>>> Le 8 janv. 2018 20:32, "Robert Bradshaw" < >>>>>>>>>>>>>>> rober...@google.com> a écrit : >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> Part of the motivation to use JSON for more complex >>>>>>>>>>>>>>> options was that >>>>>>>>>>>>>>> >>>>> it avoids having to define (and document, test, have >>>>>>>>>>>>>>> users learn, ...) >>>>>>>>>>>>>>> >>>>> yet another format for expressing lists, maps, etc. >>>>>>>>>>>>>>> >>>>> >>>>>>>>>>>>>>> >>>>> On Mon, Jan 8, 2018 at 11:19 AM, Lukasz Cwik < >>>>>>>>>>>>>>> lc...@google.com> wrote: >>>>>>>>>>>>>>> >>>>> > Ken, this is specifically about running integration >>>>>>>>>>>>>>> tests and not a >>>>>>>>>>>>>>> >>>>> > users >>>>>>>>>>>>>>> >>>>> > main(). >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > Note, that PipelineOptions JSON format was used >>>>>>>>>>>>>>> because it was a >>>>>>>>>>>>>>> >>>>> > convenient >>>>>>>>>>>>>>> >>>>> > serialized form that is easy to explain to people >>>>>>>>>>>>>>> what is required. >>>>>>>>>>>>>>> >>>>> > Using a different string tokenizer and calling >>>>>>>>>>>>>>> >>>>> > PipelineOptionsFactory.fromArgs() with the parsed >>>>>>>>>>>>>>> strings seems like >>>>>>>>>>>>>>> >>>>> > it >>>>>>>>>>>>>>> >>>>> > would be better. >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > These are the supported formats for fromArgs(): >>>>>>>>>>>>>>> >>>>> > * --project=MyProject (simple property, will >>>>>>>>>>>>>>> set the "project" >>>>>>>>>>>>>>> >>>>> > property >>>>>>>>>>>>>>> >>>>> > to "MyProject") >>>>>>>>>>>>>>> >>>>> > * --readOnly=true (for boolean properties, will >>>>>>>>>>>>>>> set the >>>>>>>>>>>>>>> >>>>> > "readOnly" >>>>>>>>>>>>>>> >>>>> > property to "true") >>>>>>>>>>>>>>> >>>>> > * --readOnly (shorthand for boolean properties, >>>>>>>>>>>>>>> will set the >>>>>>>>>>>>>>> >>>>> > "readOnly" >>>>>>>>>>>>>>> >>>>> > property to "true") >>>>>>>>>>>>>>> >>>>> > * --x=1 --x=2 --x=3 (list style simple >>>>>>>>>>>>>>> property, will set the >>>>>>>>>>>>>>> >>>>> > "x" >>>>>>>>>>>>>>> >>>>> > property to [1, 2, 3]) >>>>>>>>>>>>>>> >>>>> > * --x=1,2,3 (shorthand list style simple >>>>>>>>>>>>>>> property, will set the >>>>>>>>>>>>>>> >>>>> > "x" >>>>>>>>>>>>>>> >>>>> > property to [1, 2, 3]) >>>>>>>>>>>>>>> >>>>> > * --complexObject='{"key1":"value1",...} (JSON >>>>>>>>>>>>>>> format for all >>>>>>>>>>>>>>> >>>>> > other >>>>>>>>>>>>>>> >>>>> > complex types) >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > Using a string tokenizer that minimizes the number >>>>>>>>>>>>>>> of required escape >>>>>>>>>>>>>>> >>>>> > characters would be good so we could use newline >>>>>>>>>>>>>>> characters as our >>>>>>>>>>>>>>> >>>>> > only >>>>>>>>>>>>>>> >>>>> > token. I would avoid ',\t ' as tokens since they are >>>>>>>>>>>>>>> more likely to >>>>>>>>>>>>>>> >>>>> > appear. >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>>>> > On Mon, Jan 8, 2018 at 10:33 AM, Kenneth Knowles < >>>>>>>>>>>>>>> k...@google.com> >>>>>>>>>>>>>>> >>>>> > wrote: >>>>>>>>>>>>>>> >>>>> >> >>>>>>>>>>>>>>> >>>>> >> We do have a plain command line syntax, and whoever >>>>>>>>>>>>>>> writes the >>>>>>>>>>>>>>> >>>>> >> main(String[]) function is responsible for invoking >>>>>>>>>>>>>>> the parser. It >>>>>>>>>>>>>>> >>>>> >> isn't >>>>>>>>>>>>>>> >>>>> >> quite as nice as standard arg parse libraries, but >>>>>>>>>>>>>>> it isn't too bad. >>>>>>>>>>>>>>> >>>>> >> It >>>>>>>>>>>>>>> >>>>> >> would be great to improve, though. >>>>>>>>>>>>>>> >>>>> >> >>>>>>>>>>>>>>> >>>>> >> Jackson is for machine-to-machine communication or >>>>>>>>>>>>>>> other situations >>>>>>>>>>>>>>> >>>>> >> where >>>>>>>>>>>>>>> >>>>> >> command line parsing doesn't work so well. >>>>>>>>>>>>>>> >>>>> >> >>>>>>>>>>>>>>> >>>>> >> Are we using these some other way? >>>>>>>>>>>>>>> >>>>> >> >>>>>>>>>>>>>>> >>>>> >> Kenn >>>>>>>>>>>>>>> >>>>> >> >>>>>>>>>>>>>>> >>>>> >> On Sun, Jan 7, 2018 at 7:21 AM, Romain Manni-Bucau >>>>>>>>>>>>>>> >>>>> >> <rmannibu...@gmail.com> >>>>>>>>>>>>>>> >>>>> >> wrote: >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> Le 7 janv. 2018 12:53, "Jean-Baptiste Onofré" < >>>>>>>>>>>>>>> j...@nanthrax.net> a >>>>>>>>>>>>>>> >>>>> >>> écrit : >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> Hi Romain, >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> I guess you are assuming that pipeline options are >>>>>>>>>>>>>>> flat command >>>>>>>>>>>>>>> >>>>> >>> line >>>>>>>>>>>>>>> >>>>> >>> like >>>>>>>>>>>>>>> >>>>> >>> argument, right ? >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> Actually, theoretically, pipeline options can be >>>>>>>>>>>>>>> represented as >>>>>>>>>>>>>>> >>>>> >>> json, >>>>>>>>>>>>>>> >>>>> >>> that's why we use jackson. >>>>>>>>>>>>>>> >>>>> >>> The pipeline options can be >>>>>>>>>>>>>>> serialized/deserialized as json. >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> Yes but if users (or dev ;)) start to use that >>>>>>>>>>>>>>> then it is trivial >>>>>>>>>>>>>>> >>>>> >>> to >>>>>>>>>>>>>>> >>>>> >>> break the cli handling and fromArgs parsing or, if >>>>>>>>>>>>>>> not, break the >>>>>>>>>>>>>>> >>>>> >>> user >>>>>>>>>>>>>>> >>>>> >>> experience. So at the end it is a kind of "yes but >>>>>>>>>>>>>>> no", right? >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> PS: already see some advanced users having a >>>>>>>>>>>>>>> headache trying to >>>>>>>>>>>>>>> >>>>> >>> pass >>>>>>>>>>>>>>> >>>>> >>> pipeline options in json so using the plain >>>>>>>>>>>>>>> command line syntax can >>>>>>>>>>>>>>> >>>>> >>> be more >>>>>>>>>>>>>>> >>>>> >>> friendly too. >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> The purpose is to remove the jackson dependencies ? >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> Later yes, seeing core dep tree I identify a lot >>>>>>>>>>>>>>> of dep which can >>>>>>>>>>>>>>> >>>>> >>> conflict in some env and are not really needed or >>>>>>>>>>>>>>> bring much being >>>>>>>>>>>>>>> >>>>> >>> in the >>>>>>>>>>>>>>> >>>>> >>> core - like avro as mentionned in another thread. >>>>>>>>>>>>>>> Can need a >>>>>>>>>>>>>>> >>>>> >>> sanitizing >>>>>>>>>>>>>>> >>>>> >>> round. Short term it was really a "why is it that >>>>>>>>>>>>>>> complicated" ;). >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> Regards >>>>>>>>>>>>>>> >>>>> >>> JB >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> On 01/07/2018 11:38 AM, Romain Manni-Bucau wrote: >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> Hi guys, >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> not sure i fully get why jackson is used to parse >>>>>>>>>>>>>>> pipeline options >>>>>>>>>>>>>>> >>>>> >>>> in >>>>>>>>>>>>>>> >>>>> >>>> the testing integration >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> why not using a token parser like [1] which >>>>>>>>>>>>>>> allows to map 1-1 with >>>>>>>>>>>>>>> >>>>> >>>> the >>>>>>>>>>>>>>> >>>>> >>>> user interface (command line) the options? >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> [1] >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> https://github.com/Talend/comp >>>>>>>>>>>>>>> onent-runtime/blob/master/component-server/src/main/java/org >>>>>>>>>>>>>>> /talend/sdk/component/server/lang/StringPropertiesTokenizer. >>>>>>>>>>>>>>> java >>>>>>>>>>>>>>> >>>>> >>>> >>>>>>>>>>>>>>> >>>>> >>>> Romain Manni-Bucau >>>>>>>>>>>>>>> >>>>> >>>> @rmannibucau <https://twitter.com/rmannibucau> | >>>>>>>>>>>>>>> Blog >>>>>>>>>>>>>>> >>>>> >>>> <https://rmannibucau.metawerx.net/> | Old Blog >>>>>>>>>>>>>>> >>>>> >>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>>>>>>>>> >>>>> >>>> <https://github.com/rmannibucau> >>>>>>>>>>>>>>> >>>>> >>>> | LinkedIn <https://www.linkedin.com/in/r >>>>>>>>>>>>>>> mannibucau> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> -- >>>>>>>>>>>>>>> >>>>> >>> Jean-Baptiste Onofré >>>>>>>>>>>>>>> >>>>> >>> jbono...@apache.org >>>>>>>>>>>>>>> >>>>> >>> http://blog.nanthrax.net >>>>>>>>>>>>>>> >>>>> >>> Talend - http://www.talend.com >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >>> >>>>>>>>>>>>>>> >>>>> >> >>>>>>>>>>>>>>> >>>>> > >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >>> >>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>> >>> >> >
