1. Yes, I'm not aware of a way to avoid it right now when you're submitting job via REST API. 2. Hopefully not, the classes should be always loaded from the parent loader if they can be found on classpath ... but as I've told you before, this is a hacky solution which is healing symptoms instead of addressing the cause and definitely not a recommended way to submit jobs
D. On Wed, Dec 22, 2021 at 12:58 PM Lior Liviev <lior.liv...@earnix.com> wrote: > Hello David, > > I have some questions regarding our conversation: > > 1. When I put the JAR in $FLINK/lib, do I need to use your REST API to > load it? > 2. If I have my JAR in the folder AND I load same JAR via REST API, > will I run into problems? (class loading strategy is set to parent-first) > > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:53 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > "+ *setting class-loading strategy to parent-first *could" ... otherwise > the classes will be always loaded from the jar provided via the REST API > > On Tue, Dec 21, 2021 at 5:46 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > So again, after putting the jar in that folder I don’t need to configure > anything else? > > Get Outlook for iOS > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Gy0dL71pm%2BdHsTLfTNjQMWWmUiWx1d1rKkJkHFZw%2BFI%3D&reserved=0> > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:39:10 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > hmm, with this approach I can only think about not really nice > solutions... I guess putting a jar into `/lib` folder + setting > class-loading strategy to parent-first could do the trick (load everything > in the "main" class loader), but then using this endpoint / deployment path > for submission kind of seems to lack purpose. > > on the other hand, I don't really think we provide a nice way to achieve > this with session cluster right now :( > > D. > > On Tue, Dec 21, 2021 at 5:27 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Yes, I’m using "/jars/:jarid/run" > > Get Outlook for iOS > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Gy0dL71pm%2BdHsTLfTNjQMWWmUiWx1d1rKkJkHFZw%2BFI%3D&reserved=0> > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:08:51 PM > *To:* Lior Liviev <lior.liv...@earnix.com>; user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Please always include the ML in the reply-list, so other can participate > in the discussion / learn from the findings > > we are aware of multiple issues when web-submission can result in > classloader / thread local leaks, which could potentially result in the > behavior you're describing. We're working on addressing them. > > FLINK-25022 [1]: The most critical one leaking thread locals. > FLINK-25027 [2]: Is only a memory improvement for a particular situation (a > lot of small batch jobs) and could be fixed by accounting for when setting > Metaspace size. > FLINK-25023 [3]: Can leak the classloader of the first job submitted via > rest API. (constant overhead for Metaspace) > > In general, web-submission is different from a normal submission in way, > that the "main method" of the uploaded jar is executed on JobManager and > it's really hard to isolate it's execution from possible side effects. > > Could you by any chance try to submit jobs with the Flink CLI instead? > That should be more robust when it comes to the class loading issues. > > Which endpoint are you using for submitting the job? "/jars/:jarid/run"? > > [1] https://issues.apache.org/jira/browse/FLINK-25022 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25022&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2Fmy%2BfhYBMFND7K%2BPxjQi1U78V7gWWvHnArC0BPL1dZ8%3D&reserved=0> > [2] https://issues.apache.org/jira/browse/FLINK-25027 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25027&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jAerDuCYlW7aghp%2F0MyLPIgxa4vcJA8u%2FMjkGcW2ejE%3D&reserved=0> > [3] https://issues.apache.org/jira/browse/FLINK-25023 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25023&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rMew16AU%2Bupdx5%2F%2FYlGmbwrv%2FaGGTracbOMSHPnZs1k%3D&reserved=0> > > On Tue, Dec 21, 2021 at 4:49 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Yes, I use the REST API. I'm running into OOM Metaspace, and I think it's > a class-loading problem, so that's why I'm thinking of putting the jar in > flink/lib > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 5:43 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Hi Lior, > > can you please provide details about the steps (I'm not sure what load jar > / execute with the API means)? are you submitting the job using the REST > API or Flink CLI? I assume you're using a session cluster. > > also what is the concern here? do you run into any class-loading related > issues? > > D. > > On Tue, Dec 21, 2021 at 3:48 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Hello, I have existing fixed cluster (*not* a new one with every job > execution) and a single Jar +multiple executions with different params. > > Currently my procedure is: 1. Download Jar 2. Load Jar with API 3. > Execute with API. > I plan to avoid dynamic class loading by applying method described in: > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MEh86jZ5yiBIxPjnzEWyxjFsEwsFIPjJpGyU8uSQSj8%3D&reserved=0> > Debugging Classloading | Apache Flink > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MEh86jZ5yiBIxPjnzEWyxjFsEwsFIPjJpGyU8uSQSj8%3D&reserved=0> > Debugging Classloading # Overview of Classloading in Flink # When running > Flink applications, the JVM will load various classes over time. These > classes can be divided into three groups based on their origin: The Java > Classpath: This is Java’s common classpath, and it includes the JDK > libraries, and all code in Flink’s /lib folder (the classes of Apache Flink > and some dependencies). > nightlies.apache.org > <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnightlies.apache.org%2F&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BEAPS94tefRLKPE0DrhJ%2BIk7fREFyr5ix6zxet8wPQ4%3D&reserved=0> > My question is: > > After putting the Jar in $FLINK/lib, do I need to load Jar and execute it > the old way, or what? > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. >
