Pratik Malani created THRIFT-5645:
-------------------------------------
Summary: libthrift 0.14.0 onwards doesn't works with Spark 3.3.0
Key: THRIFT-5645
URL: https://issues.apache.org/jira/browse/THRIFT-5645
Project: Thrift
Issue Type: Bug
Components: Java - Library
Affects Versions: 0.17.0, 0.16.0, 0.14.2, 0.14.1, 0.15.0, 0.14.0
Reporter: Pratik Malani
Attachments: image-2022-10-03-14-45-30-224.png
libthrift:0.13.0 is affected with
[CVE-2020-13949|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-13949]
Currently I am using Spark 3.3.0 and Hive 2.3.9 with Hadoop 3.3.4.
When we do an upgrade to use libthrift:0.14.0 and above jar, below exception is
thrown while starting the Spark Thriftserver.
{noformat}
org.apache.hive.service.ServiceException: Failed to Start HiveServer2
at
org.apache.hive.service.CompositeService.start(CompositeService.java:79)
at
org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:104)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.start(HiveThriftServer2.scala:154)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:64)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:104)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NoSuchMethodError:
org.apache.thrift.server.TThreadPoolServer$Args.requestTimeout(I)Lorg/apache/thrift/server/TThreadPoolServer$Args;
at
org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.initializeServer(ThriftBinaryCLIService.java:101)
at
org.apache.hive.service.cli.thrift.ThriftCLIService.start(ThriftCLIService.java:176)
at
org.apache.hive.service.CompositeService.start(CompositeService.java:69)
at
org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:104)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.start(HiveThriftServer2.scala:154)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.startWithContext(HiveThriftServer2.scala:64)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:104)
at
org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at
org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{noformat}
After detailed investigation, found out that since 0.14.0 version, in class
org.apache.thrift.server.TThreadPoolServer, requestTimeout property has been
removed.
Even in latest hive release (3.1.3), it is still referencing to the
requestTimeout attribute has been removed.
[https://jar-download.com/artifacts/org.apache.hive/hive-service/3.1.3/source-code/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java]
!image-2022-10-03-14-45-30-224.png|width=722,height=92!
Wanted to know as part of what change and why this was removed? Is there any
alternative to use?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
