Re: Classloading issues with Flink Operator / Kubernetes Native
Version 1.15.2, there is no /opt/flink/usrlib folder created El mar, 20 sept 2022 a las 20:53, Yaroslav Tkachenko () escribió: > Interesting, do you see the /opt/flink/usrlib folder created as well? > Also, what Flink version do you use? > > Thanks. > > On Tue, Sep 20, 2022 at 4:04 PM Javier Vegas wrote: > >> >> jarURI: local:///opt/flink/lib/MYJARNAME.jar >> >> El mar, 20 sept 2022 a las 0:25, Yaroslav Tkachenko (< >> yaros...@goldsky.com>) escribió: >> >>> Hi Javier, >>> >>> What do you specify as a jarURI? >>> >>> On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas wrote: >>> I am doing the same thing (migrating from standalone to operator in native mode) and also have my jar in /opt/flink/lib but for me it works fine, no class loading errors on app startup. El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (< yaros...@goldsky.com>) escribió: > Application mode. I've done a bit more research and created > https://issues.apache.org/jira/browse/FLINK-29288, planning to work > on a PR today. > > TLDR: currently Flink operator always creates /opt/flink/usrlib folder > and forces you to specify the jarURI parameter, which is passed as > pipeline.jars / pipeline.classpaths configuration options. This leads to > the jar being loaded twice by different classloaders (system and user > ones). > > On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl > wrote: > >> Are you deploying the job in session or application mode? Could you >> provide the stacktrace. I'm wondering whether that would be helpful to >> pin >> a code location for further investigation. >> So far, I couldn't come up with a definite answer about placing the >> jar in the lib directory. Initially, I would have thought that it's fine >> considering that all dependencies are included and the job jar itself >> ends >> up on the user classpath. I'm curious whether Chesnay (CC'd) has an >> answer >> to that one. >> >> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko < >> yaros...@goldsky.com> wrote: >> >>> Hey everyone, >>> >>> I’m migrating a Flink Kubernetes standalone job to the Flink >>> operator (with Kubernetes native mode). >>> >>> I have a lot of classloading issues when trying to run with >>> the operator in native mode. For example, I have a Postgres driver as a >>> dependency (I can confirm the files are included in the uber jar), but I >>> still get "java.sql.SQLException: No suitable driver found for >>> jdbc:postgresql:..." exception. >>> >>> In the Kubernetes standalone setup my uber jar is placed in the >>> /opt/flink/lib folder, this is what I specify as "jarURI" in the >>> operator >>> config. Is this supported? Should I only be using /opt/flink/usrlib? >>> >>> Thanks for any suggestions. >>> >>
Re: Classloading issues with Flink Operator / Kubernetes Native
Interesting, do you see the /opt/flink/usrlib folder created as well? Also, what Flink version do you use? Thanks. On Tue, Sep 20, 2022 at 4:04 PM Javier Vegas wrote: > > jarURI: local:///opt/flink/lib/MYJARNAME.jar > > El mar, 20 sept 2022 a las 0:25, Yaroslav Tkachenko () > escribió: > >> Hi Javier, >> >> What do you specify as a jarURI? >> >> On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas wrote: >> >>> I am doing the same thing (migrating from standalone to operator in >>> native mode) and also have my jar in /opt/flink/lib but for me it works >>> fine, no class loading errors on app startup. >>> >>> El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (< >>> yaros...@goldsky.com>) escribió: >>> Application mode. I've done a bit more research and created https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a PR today. TLDR: currently Flink operator always creates /opt/flink/usrlib folder and forces you to specify the jarURI parameter, which is passed as pipeline.jars / pipeline.classpaths configuration options. This leads to the jar being loaded twice by different classloaders (system and user ones). On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl wrote: > Are you deploying the job in session or application mode? Could you > provide the stacktrace. I'm wondering whether that would be helpful to pin > a code location for further investigation. > So far, I couldn't come up with a definite answer about placing the > jar in the lib directory. Initially, I would have thought that it's fine > considering that all dependencies are included and the job jar itself ends > up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer > to that one. > > On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko < > yaros...@goldsky.com> wrote: > >> Hey everyone, >> >> I’m migrating a Flink Kubernetes standalone job to the Flink operator >> (with Kubernetes native mode). >> >> I have a lot of classloading issues when trying to run with >> the operator in native mode. For example, I have a Postgres driver as a >> dependency (I can confirm the files are included in the uber jar), but I >> still get "java.sql.SQLException: No suitable driver found for >> jdbc:postgresql:..." exception. >> >> In the Kubernetes standalone setup my uber jar is placed in the >> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator >> config. Is this supported? Should I only be using /opt/flink/usrlib? >> >> Thanks for any suggestions. >> >
Re: Classloading issues with Flink Operator / Kubernetes Native
jarURI: local:///opt/flink/lib/MYJARNAME.jar El mar, 20 sept 2022 a las 0:25, Yaroslav Tkachenko () escribió: > Hi Javier, > > What do you specify as a jarURI? > > On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas wrote: > >> I am doing the same thing (migrating from standalone to operator in >> native mode) and also have my jar in /opt/flink/lib but for me it works >> fine, no class loading errors on app startup. >> >> El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko (< >> yaros...@goldsky.com>) escribió: >> >>> Application mode. I've done a bit more research and created >>> https://issues.apache.org/jira/browse/FLINK-29288, planning to work on >>> a PR today. >>> >>> TLDR: currently Flink operator always creates /opt/flink/usrlib folder >>> and forces you to specify the jarURI parameter, which is passed as >>> pipeline.jars / pipeline.classpaths configuration options. This leads to >>> the jar being loaded twice by different classloaders (system and user >>> ones). >>> >>> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl >>> wrote: >>> Are you deploying the job in session or application mode? Could you provide the stacktrace. I'm wondering whether that would be helpful to pin a code location for further investigation. So far, I couldn't come up with a definite answer about placing the jar in the lib directory. Initially, I would have thought that it's fine considering that all dependencies are included and the job jar itself ends up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer to that one. On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko < yaros...@goldsky.com> wrote: > Hey everyone, > > I’m migrating a Flink Kubernetes standalone job to the Flink operator > (with Kubernetes native mode). > > I have a lot of classloading issues when trying to run with > the operator in native mode. For example, I have a Postgres driver as a > dependency (I can confirm the files are included in the uber jar), but I > still get "java.sql.SQLException: No suitable driver found for > jdbc:postgresql:..." exception. > > In the Kubernetes standalone setup my uber jar is placed in the > /opt/flink/lib folder, this is what I specify as "jarURI" in the operator > config. Is this supported? Should I only be using /opt/flink/usrlib? > > Thanks for any suggestions. >
Re: Classloading issues with Flink Operator / Kubernetes Native
Hi Javier, What do you specify as a jarURI? On Mon, Sep 19, 2022 at 3:56 PM Javier Vegas wrote: > I am doing the same thing (migrating from standalone to operator in native > mode) and also have my jar in /opt/flink/lib but for me it works fine, no > class loading errors on app startup. > > El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko () > escribió: > >> Application mode. I've done a bit more research and created >> https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a >> PR today. >> >> TLDR: currently Flink operator always creates /opt/flink/usrlib folder >> and forces you to specify the jarURI parameter, which is passed as >> pipeline.jars / pipeline.classpaths configuration options. This leads to >> the jar being loaded twice by different classloaders (system and user >> ones). >> >> On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl >> wrote: >> >>> Are you deploying the job in session or application mode? Could you >>> provide the stacktrace. I'm wondering whether that would be helpful to pin >>> a code location for further investigation. >>> So far, I couldn't come up with a definite answer about placing the jar >>> in the lib directory. Initially, I would have thought that it's fine >>> considering that all dependencies are included and the job jar itself ends >>> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer >>> to that one. >>> >>> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko >>> wrote: >>> Hey everyone, I’m migrating a Flink Kubernetes standalone job to the Flink operator (with Kubernetes native mode). I have a lot of classloading issues when trying to run with the operator in native mode. For example, I have a Postgres driver as a dependency (I can confirm the files are included in the uber jar), but I still get "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..." exception. In the Kubernetes standalone setup my uber jar is placed in the /opt/flink/lib folder, this is what I specify as "jarURI" in the operator config. Is this supported? Should I only be using /opt/flink/usrlib? Thanks for any suggestions. >>>
Re: Classloading issues with Flink Operator / Kubernetes Native
I am doing the same thing (migrating from standalone to operator in native mode) and also have my jar in /opt/flink/lib but for me it works fine, no class loading errors on app startup. El vie, 16 sept 2022 a las 9:28, Yaroslav Tkachenko () escribió: > Application mode. I've done a bit more research and created > https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a > PR today. > > TLDR: currently Flink operator always creates /opt/flink/usrlib folder and > forces you to specify the jarURI parameter, which is passed as > pipeline.jars / pipeline.classpaths configuration options. This leads to > the jar being loaded twice by different classloaders (system and user > ones). > > On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl > wrote: > >> Are you deploying the job in session or application mode? Could you >> provide the stacktrace. I'm wondering whether that would be helpful to pin >> a code location for further investigation. >> So far, I couldn't come up with a definite answer about placing the jar >> in the lib directory. Initially, I would have thought that it's fine >> considering that all dependencies are included and the job jar itself ends >> up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer >> to that one. >> >> On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko >> wrote: >> >>> Hey everyone, >>> >>> I’m migrating a Flink Kubernetes standalone job to the Flink operator >>> (with Kubernetes native mode). >>> >>> I have a lot of classloading issues when trying to run with the operator >>> in native mode. For example, I have a Postgres driver as a dependency (I >>> can confirm the files are included in the uber jar), but I still get >>> "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..." >>> exception. >>> >>> In the Kubernetes standalone setup my uber jar is placed in the >>> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator >>> config. Is this supported? Should I only be using /opt/flink/usrlib? >>> >>> Thanks for any suggestions. >>> >>
Re: Classloading issues with Flink Operator / Kubernetes Native
Application mode. I've done a bit more research and created https://issues.apache.org/jira/browse/FLINK-29288, planning to work on a PR today. TLDR: currently Flink operator always creates /opt/flink/usrlib folder and forces you to specify the jarURI parameter, which is passed as pipeline.jars / pipeline.classpaths configuration options. This leads to the jar being loaded twice by different classloaders (system and user ones). On Fri, Sep 16, 2022 at 2:30 AM Matthias Pohl wrote: > Are you deploying the job in session or application mode? Could you > provide the stacktrace. I'm wondering whether that would be helpful to pin > a code location for further investigation. > So far, I couldn't come up with a definite answer about placing the jar in > the lib directory. Initially, I would have thought that it's fine > considering that all dependencies are included and the job jar itself ends > up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer > to that one. > > On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko > wrote: > >> Hey everyone, >> >> I’m migrating a Flink Kubernetes standalone job to the Flink operator >> (with Kubernetes native mode). >> >> I have a lot of classloading issues when trying to run with the operator >> in native mode. For example, I have a Postgres driver as a dependency (I >> can confirm the files are included in the uber jar), but I still get >> "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..." >> exception. >> >> In the Kubernetes standalone setup my uber jar is placed in the >> /opt/flink/lib folder, this is what I specify as "jarURI" in the operator >> config. Is this supported? Should I only be using /opt/flink/usrlib? >> >> Thanks for any suggestions. >> >
Re: Classloading issues with Flink Operator / Kubernetes Native
Are you deploying the job in session or application mode? Could you provide the stacktrace. I'm wondering whether that would be helpful to pin a code location for further investigation. So far, I couldn't come up with a definite answer about placing the jar in the lib directory. Initially, I would have thought that it's fine considering that all dependencies are included and the job jar itself ends up on the user classpath. I'm curious whether Chesnay (CC'd) has an answer to that one. On Tue, Sep 13, 2022 at 1:40 AM Yaroslav Tkachenko wrote: > Hey everyone, > > I’m migrating a Flink Kubernetes standalone job to the Flink operator > (with Kubernetes native mode). > > I have a lot of classloading issues when trying to run with the operator > in native mode. For example, I have a Postgres driver as a dependency (I > can confirm the files are included in the uber jar), but I still get > "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..." > exception. > > In the Kubernetes standalone setup my uber jar is placed in the > /opt/flink/lib folder, this is what I specify as "jarURI" in the operator > config. Is this supported? Should I only be using /opt/flink/usrlib? > > Thanks for any suggestions. >
Classloading issues with Flink Operator / Kubernetes Native
Hey everyone, I’m migrating a Flink Kubernetes standalone job to the Flink operator (with Kubernetes native mode). I have a lot of classloading issues when trying to run with the operator in native mode. For example, I have a Postgres driver as a dependency (I can confirm the files are included in the uber jar), but I still get "java.sql.SQLException: No suitable driver found for jdbc:postgresql:..." exception. In the Kubernetes standalone setup my uber jar is placed in the /opt/flink/lib folder, this is what I specify as "jarURI" in the operator config. Is this supported? Should I only be using /opt/flink/usrlib? Thanks for any suggestions.
