And everything happens at launch ________________________________ From: Lior Liviev <lior.liv...@earnix.com> Sent: Sunday, December 26, 2021 7:38 PM To: David Morávek <d...@apache.org> Cc: user <user@flink.apache.org> Subject: Re: Avoiding Dynamic Classloading for User Code
CAUTION: external source Hey David, I placed the jar in flink folder but now I see some weird exceptions that weren't before: 1. In flink logs I see non stop Failed to access job archive location for path hdfs:/completed-jobs. java.io.FileNotFoundException: File hdfs:/completed-jobs does not exist. And after couple of minutes it recovered and was able to read the file. 2. At launch I see: No jobs included in application. And as the first case it recovers and was able to find the jobs. Do you have any idea what can be the problem? ________________________________ From: David Morávek <d...@apache.org> Sent: Thursday, December 23, 2021 2:06 PM To: Lior Liviev <lior.liv...@earnix.com> Cc: user <user@flink.apache.org> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Then I don't really know what else to suggest in this direction. This approach should work in general, if you have control over the class path and you make sure all the dependencies play well with each other, as the parent-first classloading doesn't provide you with any crutches as the child-first one. As mentioned previously, using a CLI instead of REST API to submit the job should avoid this issue whatsoever, as the job driver is executed in a separate JVM. There is some future work (also mentioned above) to address the known issues with the web-submission, but I'm afraid there will always be some edge cases, that we won't be able to handle, because we have no control of the user code. D. On Thu, Dec 23, 2021 at 12:54 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: We use hadoop in EMR 6.4 (if I'm not mistaken, emr has it's own version of hadoop so we don't define it) and we use flink 13.1 ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Thursday, December 23, 2021 1:44 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> Cc: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Please try to post the whole stacktrace the next time, but in general it sounds like you've conflicting avro versions on the classpath (actually I'd expect a version < 1.8 [1] in your user jar / maybe loaded from hadoop?). What Flink version are you using? Are you using Flink with Hadoop (which version / how you link the hadoop deps)? [1] https://issues.apache.org/jira/browse/AVRO-1497<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FAVRO-1497&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405467761%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=34UKv2Fdd1JohBxJ%2FSlWcY9la53RefeCfg7NDmUu6io%3D&reserved=0> On Thu, Dec 23, 2021 at 12:20 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: I get this: Caused by: java.lang.NoSuchMethodError: org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType; And I'm using avro 1.10 ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Thursday, December 23, 2021 12:37 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>>; user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source I guess I'd need more context to answer that. Have you checked the JM logs for more details? On Thu, Dec 23, 2021 at 9:01 AM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: Is there any reason I'm getting "Could not execute application" after I put the Jar in /lib? ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Wednesday, December 22, 2021 2:04 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> Cc: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source 1. Yes, I'm not aware of a way to avoid it right now when you're submitting job via REST API. 2. Hopefully not, the classes should be always loaded from the parent loader if they can be found on classpath ... but as I've told you before, this is a hacky solution which is healing symptoms instead of addressing the cause and definitely not a recommended way to submit jobs D. On Wed, Dec 22, 2021 at 12:58 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: Hello David, I have some questions regarding our conversation: 1. When I put the JAR in $FLINK/lib, do I need to use your REST API to load it? 2. If I have my JAR in the folder AND I load same JAR via REST API, will I run into problems? (class loading strategy is set to parent-first) ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Tuesday, December 21, 2021 6:53 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> Cc: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source "+ setting class-loading strategy to parent-first could" ... otherwise the classes will be always loaded from the jar provided via the REST API On Tue, Dec 21, 2021 at 5:46 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: So again, after putting the jar in that folder I don’t need to configure anything else? Get Outlook for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405467761%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=f%2FrCUZ%2BC4DsQOF1%2BsqP7mDTD7BDYDeSvHaKiXn7G8AA%3D&reserved=0> ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Tuesday, December 21, 2021 6:39:10 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> Cc: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source hmm, with this approach I can only think about not really nice solutions... I guess putting a jar into `/lib` folder + setting class-loading strategy to parent-first could do the trick (load everything in the "main" class loader), but then using this endpoint / deployment path for submission kind of seems to lack purpose. on the other hand, I don't really think we provide a nice way to achieve this with session cluster right now :( D. On Tue, Dec 21, 2021 at 5:27 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: Yes, I’m using "/jars/:jarid/run" Get Outlook for iOS<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Faka.ms%2Fo0ukef&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405467761%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=f%2FrCUZ%2BC4DsQOF1%2BsqP7mDTD7BDYDeSvHaKiXn7G8AA%3D&reserved=0> ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Tuesday, December 21, 2021 6:08:51 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>>; user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Please always include the ML in the reply-list, so other can participate in the discussion / learn from the findings we are aware of multiple issues when web-submission can result in classloader / thread local leaks, which could potentially result in the behavior you're describing. We're working on addressing them. FLINK-25022 [1]: The most critical one leaking thread locals. FLINK-25027 [2]: Is only a memory improvement for a particular situation (a lot of small batch jobs) and could be fixed by accounting for when setting Metaspace size. FLINK-25023 [3]: Can leak the classloader of the first job submitted via rest API. (constant overhead for Metaspace) In general, web-submission is different from a normal submission in way, that the "main method" of the uploaded jar is executed on JobManager and it's really hard to isolate it's execution from possible side effects. Could you by any chance try to submit jobs with the Flink CLI instead? That should be more robust when it comes to the class loading issues. Which endpoint are you using for submitting the job? "/jars/:jarid/run"? [1] https://issues.apache.org/jira/browse/FLINK-25022<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25022&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405623999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=n9L6dHUhktU2bl4FOZBaYj1oMABPF9DeKWq%2BBWafJhw%3D&reserved=0> [2] https://issues.apache.org/jira/browse/FLINK-25027<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25027&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405623999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2Fe%2FRGSjSOLHzUAu2jxb19TSj6dyHBn5%2B6degW6kY87c%3D&reserved=0> [3] https://issues.apache.org/jira/browse/FLINK-25023<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-25023&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405623999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=eO9R4tV5YfwHALy9kzOAQeYtT%2BC1TC8MRlwZ2rXpK%2BA%3D&reserved=0> On Tue, Dec 21, 2021 at 4:49 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: Yes, I use the REST API. I'm running into OOM Metaspace, and I think it's a class-loading problem, so that's why I'm thinking of putting the jar in flink/lib ________________________________ From: David Morávek <d...@apache.org<mailto:d...@apache.org>> Sent: Tuesday, December 21, 2021 5:43 PM To: Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: Avoiding Dynamic Classloading for User Code CAUTION: external source Hi Lior, can you please provide details about the steps (I'm not sure what load jar / execute with the API means)? are you submitting the job using the REST API or Flink CLI? I assume you're using a session cluster. also what is the concern here? do you run into any class-loading related issues? D. On Tue, Dec 21, 2021 at 3:48 PM Lior Liviev <lior.liv...@earnix.com<mailto:lior.liv...@earnix.com>> wrote: Hello, I have existing fixed cluster (not a new one with every job execution) and a single Jar +multiple executions with different params. Currently my procedure is: 1. Download Jar 2. Load Jar with API 3. Execute with API. I plan to avoid dynamic class loading by applying method described in: https://nightlies.apache.org/flink/flink-docs-master/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405623999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Zo%2FP2Ca98W4ercKJVxBJr037aljT43IwgTXvzTZcPOo%3D&reserved=0> Debugging Classloading | Apache Flink<https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fnightlies.apache.org%2Fflink%2Fflink-docs-master%2Fdocs%2Fops%2Fdebugging%2Fdebugging_classloading%2F%23avoiding-dynamic-classloading-for-user-code&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405623999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Zo%2FP2Ca98W4ercKJVxBJr037aljT43IwgTXvzTZcPOo%3D&reserved=0> Debugging Classloading # Overview of Classloading in Flink # When running Flink applications, the JVM will load various classes over time. These classes can be divided into three groups based on their origin: The Java Classpath: This is Java’s common classpath, and it includes the JDK libraries, and all code in Flink’s /lib folder (the classes of Apache Flink and some dependencies). nightlies.apache.org<https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fnightlies.apache.org%2F&data=04%7C01%7CLior.Liviev%40earnix.com%7Cb10fb472d26c4bd1dccc08d9c896972e%7Cae9992508a9f4ae58a5dce9de7084b84%7C0%7C0%7C637761371405623999%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Ttie1fBwQ%2BuumLXdNc2H0Qq%2Bzd9UE62GUcgLvc8idT0%3D&reserved=0> My question is: After putting the Jar in $FLINK/lib, do I need to load Jar and execute it the old way, or what? Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious. Do not click on links or open attachments unless you recognize the sender. Please use the report button if you believe this email is suspicious.
