[jira] [Commented] (SPARK-22380) Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0
[
https://issues.apache.org/jira/browse/SPARK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223700#comment-16223700
]
Maziyar PANAHI commented on SPARK-22380:
Hi Sean,
Tanks for your reply. In fact I am more looking for workaround this issue than
upgrading this dependency in Hadoop as the current version is compatible with
all the builds as you said.
Will you show me a way of how to shade dependencies for Hadoop? Or add version
3.4 and ignore 2.5 inside the Spark App? I am using Cloudera distribution of
Spark 2 to be precise.
Thanks,
Maziyar
> Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0
> ---
>
> Key: SPARK-22380
> URL: https://issues.apache.org/jira/browse/SPARK-22380
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Deploy
>Affects Versions: 1.6.1, 2.2.0
> Environment: Cloudera 5.13.x
> Spark 2.2.0.cloudera1-1.cdh5.12.0.p0.142354
> And anything beyond Spark 2.2.0
>Reporter: Maziyar PANAHI
>Priority: Blocker
>
> Hi,
> This upgrade is needed when we try to use CoreNLP 3.8 with Spark (1.6+ and
> 2.2+) due to incompatibilities in the protobuf version used by
> com.google.protobuf and the one is used in latest Stanford CoreNLP (3.8). The
> version of protobuf has been set to 2.5.0 in the global properties, and this
> is stated in the pom.xml file.
> The error that refers to this dependency:
> {code:java}
> java.lang.VerifyError: Bad type on operand stack
> Exception Details:
> Location:
>
> com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;I)Ljava/lang/Object;
> @3: invokevirtual
> Reason:
> Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current
> frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
> Current Frame:
> bci: @3
> flags: { }
> locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage',
> 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer }
> stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage',
> 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer }
> Bytecode:
> 0x000: 2a2b 1cb6 0024 b0
> at edu.stanford.nlp.simple.Document.(Document.java:433)
> at edu.stanford.nlp.simple.Sentence.(Sentence.java:118)
> at edu.stanford.nlp.simple.Sentence.(Sentence.java:126)
> ... 56 elided
> {code}
> Is it possible to upgrade this dependency to the latest (3.4) or any
> workaround besides manually removing protobuf-java-2.5.0.jar and adding
> protobuf-java-3.4.0.jar?
> You can follow the discussion of how this upgrade would fix the issue:
> https://github.com/stanfordnlp/CoreNLP/issues/556
> Many thanks,
> Maziyar
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22380) Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0
[
https://issues.apache.org/jira/browse/SPARK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223688#comment-16223688
]
Sean Owen commented on SPARK-22380:
---
The problem is that the tightly-coupled Hadoop dependencies depend on a 2.x
version, as far as I know. You'd have to research what's compatible with
Hadoop. You can always shade your dependencies.
> Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0
> ---
>
> Key: SPARK-22380
> URL: https://issues.apache.org/jira/browse/SPARK-22380
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Deploy
>Affects Versions: 1.6.1, 2.2.0
> Environment: Cloudera 5.13.x
> Spark 2.2.0.cloudera1-1.cdh5.12.0.p0.142354
> And anything beyond Spark 2.2.0
>Reporter: Maziyar PANAHI
>Priority: Blocker
>
> Hi,
> This upgrade is needed when we try to use CoreNLP 3.8 with Spark (1.6+ and
> 2.2+) due to incompatibilities in the protobuf version used by
> com.google.protobuf and the one is used in latest Stanford CoreNLP (3.8). The
> version of protobuf has been set to 2.5.0 in the global properties, and this
> is stated in the pom.xml file.
> The error that refers to this dependency:
> {code:java}
> java.lang.VerifyError: Bad type on operand stack
> Exception Details:
> Location:
>
> com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;I)Ljava/lang/Object;
> @3: invokevirtual
> Reason:
> Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current
> frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
> Current Frame:
> bci: @3
> flags: { }
> locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage',
> 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer }
> stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage',
> 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer }
> Bytecode:
> 0x000: 2a2b 1cb6 0024 b0
> at edu.stanford.nlp.simple.Document.(Document.java:433)
> at edu.stanford.nlp.simple.Sentence.(Sentence.java:118)
> at edu.stanford.nlp.simple.Sentence.(Sentence.java:126)
> ... 56 elided
> {code}
> Is it possible to upgrade this dependency to the latest (3.4) or any
> workaround besides manually removing protobuf-java-2.5.0.jar and adding
> protobuf-java-3.4.0.jar?
> You can follow the discussion of how this upgrade would fix the issue:
> https://github.com/stanfordnlp/CoreNLP/issues/556
> Many thanks,
> Maziyar
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
