[ https://issues.apache.org/jira/browse/SPARK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223688#comment-16223688 ]
Sean Owen commented on SPARK-22380: ----------------------------------- The problem is that the tightly-coupled Hadoop dependencies depend on a 2.x version, as far as I know. You'd have to research what's compatible with Hadoop. You can always shade your dependencies. > Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0 > ----------------------------------------------------------------------- > > Key: SPARK-22380 > URL: https://issues.apache.org/jira/browse/SPARK-22380 > Project: Spark > Issue Type: Dependency upgrade > Components: Deploy > Affects Versions: 1.6.1, 2.2.0 > Environment: Cloudera 5.13.x > Spark 2.2.0.cloudera1-1.cdh5.12.0.p0.142354 > And anything beyond Spark 2.2.0 > Reporter: Maziyar PANAHI > Priority: Blocker > > Hi, > This upgrade is needed when we try to use CoreNLP 3.8 with Spark (1.6+ and > 2.2+) due to incompatibilities in the protobuf version used by > com.google.protobuf and the one is used in latest Stanford CoreNLP (3.8). The > version of protobuf has been set to 2.5.0 in the global properties, and this > is stated in the pom.xml file. > The error that refers to this dependency: > {code:java} > java.lang.VerifyError: Bad type on operand stack > Exception Details: > Location: > > com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;I)Ljava/lang/Object; > @3: invokevirtual > Reason: > Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current > frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite' > Current Frame: > bci: @3 > flags: { } > locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', > 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer } > stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage', > 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer } > Bytecode: > 0x0000000: 2a2b 1cb6 0024 b0 > at edu.stanford.nlp.simple.Document.<init>(Document.java:433) > at edu.stanford.nlp.simple.Sentence.<init>(Sentence.java:118) > at edu.stanford.nlp.simple.Sentence.<init>(Sentence.java:126) > ... 56 elided > {code} > Is it possible to upgrade this dependency to the latest (3.4) or any > workaround besides manually removing protobuf-java-2.5.0.jar and adding > protobuf-java-3.4.0.jar? > You can follow the discussion of how this upgrade would fix the issue: > https://github.com/stanfordnlp/CoreNLP/issues/556 > Many thanks, > Maziyar -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org