Then I don't really know what else to suggest in this direction. This approach should work in general, if you have control over the class path and you make sure all the dependencies play well with each other, as the parent-first classloading doesn't provide you with any crutches as the child-first one.
As mentioned previously, using a CLI instead of REST API to submit the job should avoid this issue whatsoever, as the job driver is executed in a separate JVM. There is some future work (also mentioned above) to address the known issues with the web-submission, but I'm afraid there will always be some edge cases, that we won't be able to handle, because we have no control of the user code. D. On Thu, Dec 23, 2021 at 12:54 PM Lior Liviev <lior.liv...@earnix.com> wrote: > We use hadoop in EMR 6.4 (if I'm not mistaken, emr has it's own version of > hadoop so we don't define it) and we use flink 13.1 > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Thursday, December 23, 2021 1:44 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Please try to post the whole stacktrace the next time, but in general it > sounds like you've conflicting avro versions on the classpath (actually I'd > expect a version < 1.8 [1] in your user jar / maybe loaded from hadoop?). > What Flink version are you using? Are you using Flink with Hadoop (which > version / how you link the hadoop deps)? > > [1] https://issues.apache.org/jira/browse/AVRO-1497 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FAVRO-1497&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079045391%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ADcpi0%2BNDdTHf5iPA%2F6HC9ax%2FtN6ZqwvJ5CucQeEdB0%3D&reserved=0> > > On Thu, Dec 23, 2021 at 12:20 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > I get this: Caused by: java.lang.NoSuchMethodError: > org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType; > And I'm using avro 1.10 > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Thursday, December 23, 2021 12:37 PM > *To:* Lior Liviev <lior.liv...@earnix.com>; user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > I guess I'd need more context to answer that. Have you checked the JM logs > for more details? > > On Thu, Dec 23, 2021 at 9:01 AM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Is there any reason I'm getting "Could not execute application" after I > put the Jar in /lib? > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Wednesday, December 22, 2021 2:04 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > 1. Yes, I'm not aware of a way to avoid it right now when you're > submitting job via REST API. > 2. Hopefully not, the classes should be always loaded from the parent > loader if they can be found on classpath ... but as I've told you before, > this is a hacky solution which is healing symptoms instead of addressing > the cause and definitely not a recommended way to submit jobs > > D. > > On Wed, Dec 22, 2021 at 12:58 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Hello David, > > I have some questions regarding our conversation: > > 1. When I put the JAR in $FLINK/lib, do I need to use your REST API to > load it? > 2. If I have my JAR in the folder AND I load same JAR via REST API, > will I run into problems? (class loading strategy is set to parent-first) > > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:53 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > "+ *setting class-loading strategy to parent-first *could" ... otherwise > the classes will be always loaded from the jar provided via the REST API > > On Tue, Dec 21, 2021 at 5:46 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > So again, after putting the jar in that folder I don’t need to configure > anything else? > > Get Outlook for iOS > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079045391%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=L1OefEw4bSxRseobx%2BkeVdCkQTQSrfBo3sJiDp%2Fujcs%3D&reserved=0> > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:39:10 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > hmm, with this approach I can only think about not really nice > solutions... I guess putting a jar into `/lib` folder + setting > class-loading strategy to parent-first could do the trick (load everything > in the "main" class loader), but then using this endpoint / deployment path > for submission kind of seems to lack purpose. > > on the other hand, I don't really think we provide a nice way to achieve > this with session cluster right now :( > > D. > > On Tue, Dec 21, 2021 at 5:27 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Yes, I’m using "/jars/:jarid/run" > > Get Outlook for iOS > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079045391%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=L1OefEw4bSxRseobx%2BkeVdCkQTQSrfBo3sJiDp%2Fujcs%3D&reserved=0> > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:08:51 PM > *To:* Lior Liviev <lior.liv...@earnix.com>; user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Please always include the ML in the reply-list, so other can participate > in the discussion / learn from the findings > > we are aware of multiple issues when web-submission can result in > classloader / thread local leaks, which could potentially result in the > behavior you're describing. We're working on addressing them. > > FLINK-25022 [1]: The most critical one leaking thread locals. > FLINK-25027 [2]: Is only a memory improvement for a particular situation (a > lot of small batch jobs) and could be fixed by accounting for when setting > Metaspace size. > FLINK-25023 [3]: Can leak the classloader of the first job submitted via > rest API. (constant overhead for Metaspace) > > In general, web-submission is different from a normal submission in way, > that the "main method" of the uploaded jar is executed on JobManager and > it's really hard to isolate it's execution from possible side effects. > > Could you by any chance try to submit jobs with the Flink CLI instead? > That should be more robust when it comes to the class loading issues. > > Which endpoint are you using for submitting the job? "/jars/:jarid/run"? > > [1] https://issues.apache.org/jira/browse/FLINK-25022 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25022&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079201634%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rXecRRkk6dNhUPkqu%2BG9OS39xT3TU%2BpAUNPDqfe537o%3D&reserved=0> > [2] https://issues.apache.org/jira/browse/FLINK-25027 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25027&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079201634%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=N4MUl%2FfRrA7UXwDyNICsfUXqePi4BYMBKd%2FfiZqS99w%3D&reserved=0> > [3] https://issues.apache.org/jira/browse/FLINK-25023 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25023&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079201634%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IYd7fckO12YzklV27zQziEYJGbtMkoALfGQLGIi0KZc%3D&reserved=0> > > On Tue, Dec 21, 2021 at 4:49 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Yes, I use the REST API. I'm running into OOM Metaspace, and I think it's > a class-loading problem, so that's why I'm thinking of putting the jar in > flink/lib > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 5:43 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Hi Lior, > > can you please provide details about the steps (I'm not sure what load jar > / execute with the API means)? are you submitting the job using the REST > API or Flink CLI? I assume you're using a session cluster. > > also what is the concern here? do you run into any class-loading related > issues? > > D. > > On Tue, Dec 21, 2021 at 3:48 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Hello, I have existing fixed cluster (*not* a new one with every job > execution) and a single Jar +multiple executions with different params. > > Currently my procedure is: 1. Download Jar 2. Load Jar with API 3. > Execute with API. > I plan to avoid dynamic class loading by applying method described in: > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079201634%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=24zpEBnVzwv1Q4ntDyxCBdevpMOdOdKoxQkgSYuRwHc%3D&reserved=0> > Debugging Classloading | Apache Flink > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079201634%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=24zpEBnVzwv1Q4ntDyxCBdevpMOdOdKoxQkgSYuRwHc%3D&reserved=0> > Debugging Classloading # Overview of Classloading in Flink # When running > Flink applications, the JVM will load various classes over time. These > classes can be divided into three groups based on their origin: The Java > Classpath: This is Java’s common classpath, and it includes the JDK > libraries, and all code in Flink’s /lib folder (the classes of Apache Flink > and some dependencies). > nightlies.apache.org > <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnightlies.apache.org%2F&data=04%7C01%7CLior.Liviev%40earnix.com%7C2f61c10e891c46f4d4d908d9c609a9d6%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637758567079201634%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=v34iqUZNxwAnCKuIp9Q9a%2FPbbZ1c07NAy1Zx49%2B%2FIl0%3D&reserved=0> > My question is: > > After putting the Jar in $FLINK/lib, do I need to load Jar and execute it > the old way, or what? > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. >
