Hello David, I have some questions regarding our conversation:
1. When I put the JAR in $FLINK/lib, do I need to use your REST API to load it? 2. If I have my JAR in the folder AND I load same JAR via REST API, will I run into problems? (class loading strategy is set to parent-first) ________________________________ From: David Morávek <[email protected]> Sent: Tuesday, December 21, 2021 6:53 PM To: Lior Liviev <[email protected]> Cc: user <[email protected]> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source "+ setting class-loading strategy to parent-first could" ... otherwise the classes will be always loaded from the jar provided via the REST API On Tue, Dec 21, 2021 at 5:46 PM Lior Liviev <[email protected]<mailto:[email protected]>> wrote: So again, after putting the jar in that folder I don’t need to configure anything else? Get Outlook for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Gy0dL71pm%2BdHsTLfTNjQMWWmUiWx1d1rKkJkHFZw%2BFI%3D&reserved=0> ________________________________ From: David Morávek <[email protected]<mailto:[email protected]>> Sent: Tuesday, December 21, 2021 6:39:10 PM To: Lior Liviev <[email protected]<mailto:[email protected]>> Cc: user <[email protected]<mailto:[email protected]>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source hmm, with this approach I can only think about not really nice solutions... I guess putting a jar into `/lib` folder + setting class-loading strategy to parent-first could do the trick (load everything in the "main" class loader), but then using this endpoint / deployment path for submission kind of seems to lack purpose. on the other hand, I don't really think we provide a nice way to achieve this with session cluster right now :( D. On Tue, Dec 21, 2021 at 5:27 PM Lior Liviev <[email protected]<mailto:[email protected]>> wrote: Yes, I’m using "/jars/:jarid/run" Get Outlook for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Gy0dL71pm%2BdHsTLfTNjQMWWmUiWx1d1rKkJkHFZw%2BFI%3D&reserved=0> ________________________________ From: David Morávek <[email protected]<mailto:[email protected]>> Sent: Tuesday, December 21, 2021 6:08:51 PM To: Lior Liviev <[email protected]<mailto:[email protected]>>; user <[email protected]<mailto:[email protected]>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Please always include the ML in the reply-list, so other can participate in the discussion / learn from the findings we are aware of multiple issues when web-submission can result in classloader / thread local leaks, which could potentially result in the behavior you're describing. We're working on addressing them. FLINK-25022 [1]: The most critical one leaking thread locals. FLINK-25027 [2]: Is only a memory improvement for a particular situation (a lot of small batch jobs) and could be fixed by accounting for when setting Metaspace size. FLINK-25023 [3]: Can leak the classloader of the first job submitted via rest API. (constant overhead for Metaspace) In general, web-submission is different from a normal submission in way, that the "main method" of the uploaded jar is executed on JobManager and it's really hard to isolate it's execution from possible side effects. Could you by any chance try to submit jobs with the Flink CLI instead? That should be more robust when it comes to the class loading issues. Which endpoint are you using for submitting the job? "/jars/:jarid/run"? [1] https://issues.apache.org/jira/browse/FLINK-25022<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25022&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2Fmy%2BfhYBMFND7K%2BPxjQi1U78V7gWWvHnArC0BPL1dZ8%3D&reserved=0> [2] https://issues.apache.org/jira/browse/FLINK-25027<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25027&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jAerDuCYlW7aghp%2F0MyLPIgxa4vcJA8u%2FMjkGcW2ejE%3D&reserved=0> [3] https://issues.apache.org/jira/browse/FLINK-25023<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25023&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rMew16AU%2Bupdx5%2F%2FYlGmbwrv%2FaGGTracbOMSHPnZs1k%3D&reserved=0> On Tue, Dec 21, 2021 at 4:49 PM Lior Liviev <[email protected]<mailto:[email protected]>> wrote: Yes, I use the REST API. I'm running into OOM Metaspace, and I think it's a class-loading problem, so that's why I'm thinking of putting the jar in flink/lib ________________________________ From: David Morávek <[email protected]<mailto:[email protected]>> Sent: Tuesday, December 21, 2021 5:43 PM To: Lior Liviev <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Hi Lior, can you please provide details about the steps (I'm not sure what load jar / execute with the API means)? are you submitting the job using the REST API or Flink CLI? I assume you're using a session cluster. also what is the concern here? do you run into any class-loading related issues? D. On Tue, Dec 21, 2021 at 3:48 PM Lior Liviev <[email protected]<mailto:[email protected]>> wrote: Hello, I have existing fixed cluster (not a new one with every job execution) and a single Jar +multiple executions with different params. Currently my procedure is: 1. Download Jar 2. Load Jar with API 3. Execute with API. I plan to avoid dynamic class loading by applying method described in: https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MEh86jZ5yiBIxPjnzEWyxjFsEwsFIPjJpGyU8uSQSj8%3D&reserved=0> Debugging Classloading | Apache Flink<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=MEh86jZ5yiBIxPjnzEWyxjFsEwsFIPjJpGyU8uSQSj8%3D&reserved=0> Debugging Classloading # Overview of Classloading in Flink # When running Flink applications, the JVM will load various classes over time. These classes can be divided into three groups based on their origin: The Java Classpath: This is Java’s common classpath, and it includes the JDK libraries, and all code in Flink’s /lib folder (the classes of Apache Flink and some dependencies). nightlies.apache.org<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnightlies.apache.org%2F&data=04%7C01%7CLior.Liviev%40earnix.com%7C4cd19ff4ed48425a662c08d9c4a26503%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757024043468973%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=BEAPS94tefRLKPE0DrhJ%2BIk7fREFyr5ix6zxet8wPQ4%3D&reserved=0> My question is: After putting the Jar in $FLINK/lib, do I need to load Jar and execute it the old way, or what? Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious.
