lhotari edited a comment on issue #6484: URL: https://github.com/apache/pulsar/issues/6484#issuecomment-618827497
Shading protobuf might not solve all issues. The generated protobuf classes seem to have dependencies to specific protobuf version like already mentioned in https://github.com/apache/pulsar/issues/6484#issuecomment-600413743. Here's an example of a stack trace when using Pulsar client 2.5.1 and a Protobuf class compiled with Protobuf protoc version 3.11.4 : ``` java.lang.NoSuchMethodError: 'com.google.protobuf.Descriptors$FileDescriptor com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(java.lang.String[], com.google.protobuf.Descriptors$FileDescriptor[])' at my.code.proto.SomeOuterClass.<clinit>(SomeOuterClass.java:44) at my.code.proto.SomeApi.<clinit>(SomeApi.java:75) at my.code.proto.Something.getDescriptor(Something.java:112) ... 139 more ``` <details> <summary>Full stacktrace</summary> <pre> java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.pulsar.shade.org.apache.avro.protobuf.ProtobufData.getSchema(ProtobufData.java:189) at org.apache.pulsar.client.impl.schema.ProtobufSchema.createProtobufAvroSchema(ProtobufSchema.java:64) at org.apache.pulsar.client.impl.schema.ProtobufSchema.of(ProtobufSchema.java:120) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.pulsar.client.internal.DefaultImplementation.lambda$newProtobufSchema$25(DefaultImplementation.java:235) at org.apache.pulsar.client.internal.ReflectionUtils.catchExceptions(ReflectionUtils.java:35) at org.apache.pulsar.client.internal.DefaultImplementation.newProtobufSchema(DefaultImplementation.java:232) at org.apache.pulsar.client.api.Schema.PROTOBUF(Schema.java:220) ... Caused by: java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.pulsar.shade.org.apache.avro.protobuf.ProtobufData.getSchema(ProtobufData.java:183) ... 134 more Caused by: java.lang.NoSuchMethodError: 'com.google.protobuf.Descriptors$FileDescriptor com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(java.lang.String[], com.google.protobuf.Descriptors$FileDescriptor[])' at my.code.proto.SomeOuterClass.<clinit>(SomeOuterClass.java:44) at my.code.proto.SomeApi.<clinit>(SomeApi.java:75) at my.code.proto.Something.getDescriptor(Something.java:112) ... 139 more </pre> </details> It seems that Apache Avro depends on a specific Protobuf version and it's not possible to upgrade it since it's shaded. Is there a plain "unshaded" Pulsar client available? It seems that Avro 1.9.1 depends on Protobuf 3.6.1 . That was released 31-Jul-2018 . The newest Protobuf version is currently 3.11.4 . Is there a way to use a newer Protobuf version with Pulsar? UPDATE: it seems that my issue would also be resolved if pulsar-client-admin-2.5.x.jar would contain a shaded Protobuf version. In my case, the unshaded classes got loaded from pulsar-client-admin-2.5.1.jar and that caused the issue and exception. However my question is that why is protobuf shaded in the first place? Could it be an optional dependency instead so that the user would have to include protobuf if the user is using protobuf features of Pulsar. Anyhow the user's code is dependent on the protobuf version that the user protobuf classes have been compiled with. I investigated the issue and Protobuf versions <3.8.0 aren't compatible with Protobuf generated classes that have been compiled with Protobuf protoc >=3.8.0 (I tracked the breaking change in Protobuf Java to be dependent on this change https://github.com/protocolbuffers/protobuf/commit/d0f91c863ae0fbb75b41460c8bbb786ade197a0f#diff-8168b8744060c287ceee789b0eb8cbb4R403-R405) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
