Thanks for your response. Sorry for the late reply, Ferenc. Yes, totally agree with you on deprecating this behavior as part of 1.20. Let me follow it up with a FLIP to deprecate the current behavior and with a proposed solution. We can discuss further in the [DISCUSS] thread of the FLIP.
Regards Venkata krishnan On Mon, Feb 26, 2024 at 11:25 PM Ferenc Csaky <ferenc.cs...@pm.me.invalid> wrote: > Thanks for the more shared details Venkata, I did not used spark widely > myself, but if there are more examples > like follows that approach it makes sense to comply and be > consistent. > > Regarding the planned release schedule on the 2.0 wiki page [1] the > expected releases are 1.19 -> 1.20 -> 2.0. I am not sure about how > realistic is that or there are any chance there will be a 1.21, but even if > not, deprecating the current behavior for even 1.20 would not hurt IMO. > > WDYT? > > Looking for other opinios as well of course. > > Regards, > Ferenc > > [1] > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/2.0*Release__;Kw!!IKRxdwAv5BmarQ!ZafwWMtIzLzlT2CxT8XSct-ZjnETPvL0KFMGi13v1KWCEE7GXBq4NXZK-kjhRG1AZfiU0aALIPsSkEDcwzDHxrhej3L0i2YI$ > > > On Tuesday, February 27th, 2024 at 07:08, Venkatakrishnan Sowrirajan < > vsowr...@asu.edu> wrote: > > > > > > > Thanks for sharing your thoughts, Ferenc. > > > > Just my 2 cents, coming from Spark background that uses "spark.hadoop." > > prefix to handle all Hadoop configs, I prefer the "flink.hadoop." prefix > > and "flink.yarn." prefix. Generally, users use both these systems and I > > prefer to be consistent that way. Having said that, Flink doesn't need to > > be consistent with Spark so I am fine with the other approach as well. > > > > I believe in order to make backwards incompatible changes, Flink needs > the > > change to be in deprecated status for at least 2 minor versions which > means > > we will already have 2.0, therefore this can probably go in 3.0 only. > > > > It is still good to deprecate the current behavior and fix with the right > > behavior and get rid of this in 3.0 totally. > > > > Looking for more thoughts from others in the community to make sure that > I > > don't miss anything. Once the discussion settles, I can start a FLIP with > > the new proposal. > > > > Thanks > > Venkat > > > > > > On Mon, Feb 26, 2024, 1:09 AM Ferenc Csaky ferenc.cs...@pm.me.invalid > > > > wrote: > > > > > Hi Venkata krishnan, > > > > > > Thanks for starting a discussion on this topic. I completely > > > agree with you on that, this behavior can create confusion and > > > cause debugging sessions that could be spared with aligning how Flink > > > parses external properties. > > > > > > Personally, I find the Yarn props prefixing more intuitive, but > > > I do not have strong opinions other than prefixing configs for > > > external systems should follow the same semantics and behavior. > > > > > > It would make sense to align these in Flink 2.0 IMO, but I would > > > be curious about other opinions. > > > > > > On Saturday, February 24th, 2024 at 07:36, Venkatakrishnan Sowrirajan < > > > vsowr...@asu.edu> wrote: > > > > > > > Gentle ping on the ^^ question to surface this back up again. Any > > > > thoughts? > > > > > > > > Regards > > > > Venkata krishnan > > > > > > > > On Fri, Feb 16, 2024 at 7:32 PM Venkatakrishnan Sowrirajan > > > > vsowr...@asu.edu > > > > > > > > wrote: > > > > > > > > > Hi Flink devs, > > > > > > > > > > Flink supports overriding "hadoop" and "yarn" configuration. As > part of > > > > > the override mechanism, users have to prefix `hadoop` configs with > " > > > > > flink.hadoop." and the prefix will be removed, while with `yarn` > > > > > configs > > > > > users have to prefix it with "flink.yarn." but "flink." only is > > > > > removed, > > > > > not "flink.yarn.". > > > > > > > > > > Following is an example: > > > > > > > > > > 1. "Hadoop" config > > > > > > > > > > Hadoop config key = hadoop.tmp.dir => Flink config = > > > > > flink.hadoop.hadoop.tmp.dir => Hadoop's configuration object would > have > > > > > hadoop.tmp.dir*.* > > > > > > > > > > 2. "YARN" config > > > > > > > > > > YARN config key = yarn.application.classpath => Flink config = > > > > > flink.yarn.yarn.application.classpath => YARN's configuration > object > > > > > would have yarn.yarn.application.classpath*.* > > > > > > > > > > Although this is documented > > > > > > > https://urldefense.com/v3/__https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/deployment/config/*flink-yarn-__;Iw!!IKRxdwAv5BmarQ!ewtlUgGysGDiWgKPr9D1bsGDp-jLagZqppUvXAtqvbO5lNMg7QTr4y5L4OL-hTFPO1qTR1nvh4ALBEQtm0RnE6X1WTEkp0Sb$ > > > <key> > > > > > > > > properly, it feels unintuitive and it tripped me, took quite a > while to > > > > > understand why the above YARN configuration override was not > working as > > > > > expected. Is this something that should be fixed? The problem with > > > > > fixing > > > > > it is, it will become backwards incompatible. Therefore, can this > be > > > > > addressed as part of Flink-2.0? > > > > > > > > > > Any thoughts? > > > > > > > > > > Regards > > > > > Venkata krishnan >
