In this case, we required hive-site.xml which is available only on the client side and not on all the TM nodes. But the workaround you mentioned for HADOOP_CONF_DIR should work.
On Tue, Sep 12, 2023, 7:35 PM Biao Geng <biaoge...@gmail.com> wrote: > +1 for the FLIP. > Another side thought is that in my experience, when users want to make > Flink TM use a specific hadoop/hive configuration, an easier way is to ship > the corresponding conf dir and set the env > variable via containerized.taskmanager.env.HADOOP_CONF_DIR. > > Best, > Biao Geng > > Archit Goyal <argo...@linkedin.com.invalid> 于2023年9月13日周三 08:00写道： > > > Hi All, > > > > If there are no further concerns about this FLIP, I will start a vote > > thread tomorrow. > > > > Thanks, > > Archit Goyal > > > > > > From: Venkatakrishnan Sowrirajan <vsowr...@asu.edu> > > Date: Thursday, August 24, 2023 at 10:21 PM > > To: dev@flink.apache.org <dev@flink.apache.org> > > Subject: Re: [Discuss] FLIP-355: Add parent dir of files to classpath > > using yarn.provided.lib.dirs > > Thanks Yang Wang. > > > > In that case, whenever you get a chance could you please review the PR? > > > > > > On Wed, Aug 23, 2023, 8:15 PM Yang Wang <wangyang0...@apache.org> wrote: > > > > > +1 for this FLIP > > > > > > Maybe a FLIP is an overkill for this enhancement. > > > > > > > > > Best, > > > Yang > > > > > > Venkatakrishnan Sowrirajan <vsowr...@asu.edu> 于2023年8月23日周三 01:44写道： > > > > > > > Thanks for the FLIP, Archit. > > > > > > > > This is definitely quite a useful addition to the > > > *yarn.provided.lib.dirs* > > > > . +1. > > > > > > > > IMO, except for the fact that *yarn.provided.lib.dirs* (platform > > specific > > > > jars can be cached) takes only directories whereas *yarn.ship-files* > > > (user > > > > files) takes both files and dirs, the overall logic in terms of > > > > constructing the classpath in both the cases should be roughly the > > same. > > > > > > > > Referencing the PR ( > > > > > > https://urldefense.com/v3/__https://nam06.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fgithub.com*2Fapache*2Fflink*2Fpull*2F23164__*3B!!IKRxdwAv5BmarQ!cgTpodngoQAdd-qu3CvbQeAwENiu1nf0eahTH-v1NhUsSn4Y7M7sVGQYSnBjB2XgaOlyzGe7XEiU3-cAOoy84Kw*24&data=05*7C01*7Cargoyal*40linkedin.com*7C1e626c31ecaf408575b008dba52b1c6a*7C72f988bf86f141af91ab2d7cd011db47*7C0*7C0*7C638285376817262437*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*7C*7C*7C&sdata=*2F8QiBnfVJlLZ9atF1WamCsMbAaK0TzEgcCvmd85uSnk*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUlJQ!!IKRxdwAv5BmarQ!anzL1056sJNn-x90OFKxrO_kLtu8STP0j1s8uxyruNAXzhrzTf9sIRbb28d34GQlnza8cVlB8jWs1vgPXw$ > > < > > > https://urldefense.com/v3/__https://github.com/apache/flink/pull/23164__;!!IKRxdwAv5BmarQ!cgTpodngoQAdd-qu3CvbQeAwENiu1nf0eahTH-v1NhUsSn4Y7M7sVGQYSnBjB2XgaOlyzGe7XEiU3-cAOoy84Kw$ > > > > > > ) with the > > > > initial implementation you created as well here. > > > > > > > > Regards > > > > Venkata krishnan > > > > > > > > > > > > On Tue, Aug 22, 2023 at 10:09 AM Archit Goyal > > > <argo...@linkedin.com.invalid > > > > > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Gentle ping if I can get a review on the FLIP. > > > > > > > > > > Thanks, > > > > > Archit Goyal > > > > > > > > > > From: Archit Goyal <argo...@linkedin.com> > > > > > Date: Thursday, August 17, 2023 at 5:51 PM > > > > > To: dev@flink.apache.org <dev@flink.apache.org> > > > > > Subject: [Discuss] FLIP-355: Add parent dir of files to classpath > > using > > > > > yarn.provided.lib.dirs > > > > > Hi All, > > > > > > > > > > I am opening this thread to discuss the proposal to add parent > > > > directories > > > > > of files to classpath when using yarn.provided.lib.dirs. This is > > > > documented > > > > > in FLIP-355 < > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://nam06.safelinks.protection.outlook.com/?url=https*3A*2F*2Furldefense.com*2Fv3*2F__https*3A*2F*2Fcwiki.apache.org*2Fconfluence*2Fdisplay*2FFLINK*2FFLIP*355*3A*Add*parent*dir*of*files*to*classpath*using*yarn.provided.lib.dirs__*3BKyUrKysrKysrKys!!IKRxdwAv5BmarQ!fFlyBKWuWwYcWfOcpLflTTi36tyHPiENIUry9J0ygaZY0VURnIs0glu5yafV0w0tfSsnOb9ZxDD9Cjw2TApTekVU*24&data=05*7C01*7Cargoyal*40linkedin.com*7C1e626c31ecaf408575b008dba52b1c6a*7C72f988bf86f141af91ab2d7cd011db47*7C0*7C0*7C638285376817262437*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C3000*7C*7C*7C&sdata=FfyKaJDglL1caECVWX0nX8SwhHMnejfFMLjtpo5pizo*3D&reserved=0__;JSUlJSUlJSUlJSUlKioqKioqKioqKiolJSUlJSUlJSUlJSUlJSUlJSU!!IKRxdwAv5BmarQ!anzL1056sJNn-x90OFKxrO_kLtu8STP0j1s8uxyruNAXzhrzTf9sIRbb28d34GQlnza8cVlB8jVjx2XcPQ$ > > < > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP*355*3A*Add*parent*dir*of*files*to*classpath*using*yarn.provided.lib.dirs__;KyUrKysrKysrKys!!IKRxdwAv5BmarQ!fFlyBKWuWwYcWfOcpLflTTi36tyHPiENIUry9J0ygaZY0VURnIs0glu5yafV0w0tfSsnOb9ZxDD9Cjw2TApTekVU$ > > > > > > > > >. > > > > > > > > > > This FLIP mentions about enhancing YARN's classpath configuration > to > > > > > include parent directories of files in yarn.provided.lib.dirs. > > > > > > > > > > Please feel free to reply to this email thread and share your > > opinions. > > > > > > > > > > Thanks, > > > > > Archit Goyal > > > > > > > > > > > > > > >
