I guess I'd need more context to answer that. Have you checked the JM logs for more details?
On Thu, Dec 23, 2021 at 9:01 AM Lior Liviev <lior.liv...@earnix.com> wrote: > Is there any reason I'm getting "Could not execute application" after I > put the Jar in /lib? > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Wednesday, December 22, 2021 2:04 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > 1. Yes, I'm not aware of a way to avoid it right now when you're > submitting job via REST API. > 2. Hopefully not, the classes should be always loaded from the parent > loader if they can be found on classpath ... but as I've told you before, > this is a hacky solution which is healing symptoms instead of addressing > the cause and definitely not a recommended way to submit jobs > > D. > > On Wed, Dec 22, 2021 at 12:58 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Hello David, > > I have some questions regarding our conversation: > > 1. When I put the JAR in $FLINK/lib, do I need to use your REST API to > load it? > 2. If I have my JAR in the folder AND I load same JAR via REST API, > will I run into problems? (class loading strategy is set to parent-first) > > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:53 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > "+ *setting class-loading strategy to parent-first *could" ... otherwise > the classes will be always loaded from the jar provided via the REST API > > On Tue, Dec 21, 2021 at 5:46 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > So again, after putting the jar in that folder I don’t need to configure > anything else? > > Get Outlook for iOS > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jLI9gIYxxyCOFZtEIs8Ytp84XDUKznjfXxteVNKX8mo%3D&reserved=0> > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:39:10 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > hmm, with this approach I can only think about not really nice > solutions... I guess putting a jar into `/lib` folder + setting > class-loading strategy to parent-first could do the trick (load everything > in the "main" class loader), but then using this endpoint / deployment path > for submission kind of seems to lack purpose. > > on the other hand, I don't really think we provide a nice way to achieve > this with session cluster right now :( > > D. > > On Tue, Dec 21, 2021 at 5:27 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Yes, I’m using "/jars/:jarid/run" > > Get Outlook for iOS > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=jLI9gIYxxyCOFZtEIs8Ytp84XDUKznjfXxteVNKX8mo%3D&reserved=0> > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 6:08:51 PM > *To:* Lior Liviev <lior.liv...@earnix.com>; user <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Please always include the ML in the reply-list, so other can participate > in the discussion / learn from the findings > > we are aware of multiple issues when web-submission can result in > classloader / thread local leaks, which could potentially result in the > behavior you're describing. We're working on addressing them. > > FLINK-25022 [1]: The most critical one leaking thread locals. > FLINK-25027 [2]: Is only a memory improvement for a particular situation (a > lot of small batch jobs) and could be fixed by accounting for when setting > Metaspace size. > FLINK-25023 [3]: Can leak the classloader of the first job submitted via > rest API. (constant overhead for Metaspace) > > In general, web-submission is different from a normal submission in way, > that the "main method" of the uploaded jar is executed on JobManager and > it's really hard to isolate it's execution from possible side effects. > > Could you by any chance try to submit jobs with the Flink CLI instead? > That should be more robust when it comes to the class loading issues. > > Which endpoint are you using for submitting the job? "/jars/:jarid/run"? > > [1] https://issues.apache.org/jira/browse/FLINK-25022 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25022&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=VSeqCMxUuGf0s43SdZkMkWlfQlHYjlHTEnQnKmUjd%2Fg%3D&reserved=0> > [2] https://issues.apache.org/jira/browse/FLINK-25027 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25027&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=IaW%2Bq9U0aW5A%2B86xuIQv984WApprwiNyNdSguhLixUk%3D&reserved=0> > [3] https://issues.apache.org/jira/browse/FLINK-25023 > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25023&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=r%2BZIyx1nnwpO3wqnfHulQYIUUfB5vZe6vkC6N7IGS8U%3D&reserved=0> > > On Tue, Dec 21, 2021 at 4:49 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Yes, I use the REST API. I'm running into OOM Metaspace, and I think it's > a class-loading problem, so that's why I'm thinking of putting the jar in > flink/lib > ------------------------------ > *From:* David Morávek <d...@apache.org> > *Sent:* Tuesday, December 21, 2021 5:43 PM > *To:* Lior Liviev <lior.liv...@earnix.com> > *Cc:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Re: Avoiding Dynamic Classloading for User Code > > > *CAUTION*: external source > Hi Lior, > > can you please provide details about the steps (I'm not sure what load jar > / execute with the API means)? are you submitting the job using the REST > API or Flink CLI? I assume you're using a session cluster. > > also what is the concern here? do you run into any class-loading related > issues? > > D. > > On Tue, Dec 21, 2021 at 3:48 PM Lior Liviev <lior.liv...@earnix.com> > wrote: > > Hello, I have existing fixed cluster (*not* a new one with every job > execution) and a single Jar +multiple executions with different params. > > Currently my procedure is: 1. Download Jar 2. Load Jar with API 3. > Execute with API. > I plan to avoid dynamic class loading by applying method described in: > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uEK%2BChY0COeJjs17q1xfQROSeniUAHHLtDCnGkntpzY%3D&reserved=0> > Debugging Classloading | Apache Flink > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uEK%2BChY0COeJjs17q1xfQROSeniUAHHLtDCnGkntpzY%3D&reserved=0> > Debugging Classloading # Overview of Classloading in Flink # When running > Flink applications, the JVM will load various classes over time. These > classes can be divided into three groups based on their origin: The Java > Classpath: This is Java’s common classpath, and it includes the JDK > libraries, and all code in Flink’s /lib folder (the classes of Apache Flink > and some dependencies). > nightlies.apache.org > <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnightlies.apache.org%2F&data=04%7C01%7CLior.Liviev%40earnix.com%7C1a9c1b07797b464d41a508d9c5432d54%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637757714588342072%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2B5%2Bn%2BTyZJ%2FLmQY1JvFQ1zYEymZqkg03Ty9cPQTFfwK0%3D&reserved=0> > My question is: > > After putting the Jar in $FLINK/lib, do I need to load Jar and execute it > the old way, or what? > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. > > Do not click on links or open attachments unless you recognize the sender. > Please use the report button if you believe this email is suspicious. >
