Hi Fil, I've replied in the JIRA Cheers, Igal.
On Tue, Mar 8, 2022 at 6:08 PM Filip Karnicki <filip.karni...@gmail.com> wrote: > Hi Roman, Igal (@ below) > > Thank you for your answer. I don't think I'll have access to flink's lib > folder given it's a shared Cloudera cluster. The only thing I could think > of is to not include com.google.protobuf in the > classloader.parent-first-patterns.additional setting, and > including protobuf-java 3.7.1 in the uber jar. > > I created a jira for this just now + a discuss thread on the dev group > https://issues.apache.org/jira/browse/FLINK-26537 > > Hi @Igal Shilman <igal.shil...@gmail.com> , is the plugin solution > outlined by Roman something that fits in better with Statefun than having > the creators of uber .jars be responsible for using a statefun-compatible > protobuf-java? > > Kind regards > Fil > > On Tue, 8 Mar 2022 at 14:02, Roman Khachatryan <ro...@apache.org> wrote: > >> Hi Filip, >> >> Have you tried putting protobuf-java 3.7.1 into the Flink's lib/ folder? >> Or maybe re-writing the dependencies you mentioned to be loaded as >> plugins? [1] >> >> I don't see any other ways to solve this problem. >> Probably Chesnay or Seth will suggest a better solution. >> >> [1] >> >> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/ >> >> >> Regards, >> Roman >> >> On Fri, Mar 4, 2022 at 9:54 AM Filip Karnicki <filip.karni...@gmail.com> >> wrote: >> > >> > Hi All! >> > >> > We're running a statefun uber jar on a shared cloudera flink cluster, >> the latter of which launches with some ancient protobuf dependencies >> because of reasons[1]. >> > >> > Setting the following flink-config settings on the entire cluster >> > >> > classloader.parent-first-patterns.additional: >> org.apache.flink.statefun;org.apache.kafka;com.google.protobuf >> > >> > causes these old protobuf dependencies to get loaded over statefun's >> protobuf-java 3.7.1, and NoSuchMethod exceptions occur. >> > >> > We hacked together a version of statefun that doesn't perform the check >> whether the classloader settings contain the three patterns from above, and >> as long as our job uses protobouf-java 3.7.1 and the com.google.protobuf >> pattern is not present in the classloader.parent-first-patterns.additional >> setting, then all is well. >> > >> > Aside from removing old hadoop from the classpath, which may not be >> possible given that it's a shared cluster, is there anything we can do >> other than adding a configurable override not to perform the config check >> in StatefulFunctionsConfigValidator to an upcoming statefun core release? >> > >> > Many thanks >> > Fil >> > >> > >> > [1] We're still trying to find out if it's absolutely necessary to have >> these on the classpath. >> >
