Thanks for the more shared details Venkata, I did not used spark widely myself, but if there are more examples like follows that approach it makes sense to comply and be consistent.
Regarding the planned release schedule on the 2.0 wiki page [1] the expected releases are 1.19 -> 1.20 -> 2.0. I am not sure about how realistic is that or there are any chance there will be a 1.21, but even if not, deprecating the current behavior for even 1.20 would not hurt IMO. WDYT? Looking for other opinios as well of course. Regards, Ferenc [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release On Tuesday, February 27th, 2024 at 07:08, Venkatakrishnan Sowrirajan <vsowr...@asu.edu> wrote: > > > Thanks for sharing your thoughts, Ferenc. > > Just my 2 cents, coming from Spark background that uses "spark.hadoop." > prefix to handle all Hadoop configs, I prefer the "flink.hadoop." prefix > and "flink.yarn." prefix. Generally, users use both these systems and I > prefer to be consistent that way. Having said that, Flink doesn't need to > be consistent with Spark so I am fine with the other approach as well. > > I believe in order to make backwards incompatible changes, Flink needs the > change to be in deprecated status for at least 2 minor versions which means > we will already have 2.0, therefore this can probably go in 3.0 only. > > It is still good to deprecate the current behavior and fix with the right > behavior and get rid of this in 3.0 totally. > > Looking for more thoughts from others in the community to make sure that I > don't miss anything. Once the discussion settles, I can start a FLIP with > the new proposal. > > Thanks > Venkat > > > On Mon, Feb 26, 2024, 1:09 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > wrote: > > > Hi Venkata krishnan, > > > > Thanks for starting a discussion on this topic. I completely > > agree with you on that, this behavior can create confusion and > > cause debugging sessions that could be spared with aligning how Flink > > parses external properties. > > > > Personally, I find the Yarn props prefixing more intuitive, but > > I do not have strong opinions other than prefixing configs for > > external systems should follow the same semantics and behavior. > > > > It would make sense to align these in Flink 2.0 IMO, but I would > > be curious about other opinions. > > > > On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan < > > vsowr...@asu.edu> wrote: > > > > > Gentle ping on the ^^ question to surface this back up again. Any > > > thoughts? > > > > > > Regards > > > Venkata krishnan > > > > > > On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan > > > vsowr...@asu.edu > > > > > > wrote: > > > > > > > Hi Flink devs, > > > > > > > > Flink supports overriding "hadoop" and "yarn" configuration. As part of > > > > the override mechanism, users have to prefix `hadoop` configs with " > > > > flink.hadoop." and the prefix will be removed, while with `yarn` > > > > configs > > > > users have to prefix it with "flink.yarn." but "flink." only is > > > > removed, > > > > not "flink.yarn.". > > > > > > > > Following is an example: > > > > > > > > 1. "Hadoop" config > > > > > > > > Hadoop config key = hadoop.tmp.dir => Flink config = > > > > flink.hadoop.hadoop.tmp.dir => Hadoop's configuration object would have > > > > hadoop.tmp.dir*.* > > > > > > > > 2. "YARN" config > > > > > > > > YARN config key = yarn.application.classpath => Flink config = > > > > flink.yarn.yarn.application.classpath => YARN's configuration object > > > > would have yarn.yarn.application.classpath*.* > > > > > > > > Although this is documented > > > > https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/*flink-yarn-__;Iw!!IKRxdwAv5BmarQ!ewtlUgGysGDiWgKPr9D1bsGDp-jLagZqppUvXAtqvbO5lNMg7QTr4y5L4OL-hTFPO1qTR1nvh4ALBEQtm0RnE6X1WTEkp0Sb$ > > <key> > > > > > > properly, it feels unintuitive and it tripped me, took quite a while to > > > > understand why the above YARN configuration override was not working as > > > > expected. Is this something that should be fixed? The problem with > > > > fixing > > > > it is, it will become backwards incompatible. Therefore, can this be > > > > addressed as part of Flink-2.0? > > > > > > > > Any thoughts? > > > > > > > > Regards > > > > Venkata krishnan
