Hi Sanket, While migrating to Hive 3.1.3 may resolve many issues, the link below suggests that there might still be some vulnerabilities present. Do you think the specific vulnerability you're concerned about can be addressed with Hive 3.1.3?
https://mvnrepository.com/artifact/org.apache.hive/hive-exec/3.1.3 Regards, Yasukazu 2023年9月8日(金) 12:36 Agrawal, Sanket <sankeagra...@deloitte.com.invalid>: > Hi Chao, > > > > The reason to migrate to Hive 3.1.3 is to remove a vulnerability from > hive-exec-2.3.9.jar. > > > > Thanks > > Sanket > > > > *From:* Chao Sun <sunc...@apache.org> > *Sent:* Thursday, September 7, 2023 10:23 PM > *To:* Agrawal, Sanket <sankeagra...@deloitte.com.invalid> > *Cc:* Yeachan Park <yeachan...@gmail.com>; user@spark.apache.org > *Subject:* [EXT] Re: Spark 3.4.1 and Hive 3.1.3 > > > > Hi Sanket, > > > > Spark 3.4.1 currently only works with Hive 2.3.9, and it would require a > lot of work to upgrade the Hive version to 3.x and up. > > > > Normally though, you only need the Hive client in Spark to talk to > HiveMetastore (HMS) for things like table or partition metadata > information. In this case, Hive 2.3.9 used by Spark is already capable of > communicating with HMS of other versions like Hive 3.x. So, could you share > a bit of context why you want to use Hive 3.1.3 with Spark? > > > > Chao > > > > > > On Thu, Sep 7, 2023 at 6:22 AM Agrawal, Sanket < > sankeagra...@deloitte.com.invalid> wrote: > > Hi > > > > I Tried using the maven option and it’s working. But we are not allowed to > download jars at runtime from maven because of some security restrictions. > > > > So, I tried again with downloading hive 3.1.3 and giving the location of > jars and it worked this time. But now in our docker image we have 40 new > Critical vulnerabilities due to Hive (scanned by AWS Inspector). > > > > So, The only solution I see here is to build *Spark 3.4.1* *with Hive > 3.1.3*. But when I do so the build is failing while compiling the files > in /spark/sql/hive. But when I am trying to build *Spark 3.4.1* *with > Hive 2.3.9* the build is completed successfully. > > > > Has anyone tried building Spark 3.4.1 with Hive 3.1.3 or higher? > > > > Thanks, > > Sanket A. > > > > *From:* Yeachan Park <yeachan...@gmail.com> > *Sent:* Tuesday, September 5, 2023 8:52 PM > *To:* Agrawal, Sanket <sankeagra...@deloitte.com> > *Cc:* user@spark.apache.org > *Subject:* [EXT] Re: Spark 3.4.1 and Hive 3.1.3 > > > > What's the full traceback when you run the same thing via spark-shell? So > something like: > > > > $SPARK_HOME/bin/spark-shell \ > --conf "spark.sql.hive.metastore.version=3.1.3" \ > --conf "spark.sql.hive.metastore.jars=path" \ > --conf "spark.sql.hive.metastore.jars.path=/opt/hive/lib/*.jar" > > > > W.r.t building hive, there's no need - either download it from > https://downloads.apache.org/hive/hive-3.1.3/ > <https://secure-web.cisco.com/1IsuHM1ALR8L3m2ZVx4VbSWxlL34thDBf_dHELqydfQIj7R90KvNhGSEkXqyHXmOfSenFAtnuzzarKHiNMbSqX72Kh4feX6b6QpNP16REgegIZLutUZ_MJcQ_CPPCNre-OeveW0hgCfi_nmR5aLeG-SGHSeTfMF42qJd4xndcM5FFxQe4Tfg8gAP2UCVxyvhQut40U9xDaIjcJD5_IT1y7whzw4xcxp2s_lhL7VAEBHOrWdMTG2MI8qdm7HzyE_By32O6XDkc0YaMdQLcAomZ5l5Ssp0DKwoVMntgNZe_adWv-yvUSNuwpqb-af55AjSgXf3Vy2ajVN0tBPY2Li_igjTilrrRoKugtNZsaOTpx3Ex5RUFdu0g2TK8bombxiVsncFiGVvmvOCCewuE-dEV44b6EveOyoqNcbE6AHgI9-6wcy5qtrScU5wruVO6z3_-tvpH26RFVw7fYla-mMeqX2PLhsnwqFvcU1lRc0Hiq9J93VmLyr3Y-mDYKqlFUL6EqRGhT7hY9Szurj5BSHzoDw/https%3A%2F%2Fdownloads.apache.org%2Fhive%2Fhive-3.1.3%2F> > or use the maven option like Yasukazu suggested. If you do want to build it > make sure you are using Java 8 to do so. > > > > On Tue, Sep 5, 2023 at 12:00 PM Agrawal, Sanket <sankeagra...@deloitte.com> > wrote: > > Hi, > > > > I tried pointing to hive 3.1.3 using the below command. But still getting > error. I see that the spark-hive-thriftserver_2.12/3.4.1 and > spark-hive_2.12/3.4.1 have dependency on hive 2.3.9 > > > > Command: pyspark --conf "spark.sql.hive.metastore.version=3.1.3" --conf > "spark.sql.hive.metastore.jars=path" --conf > "spark.sql.hive.metastore.jars.path=file://opt/hive/lib/*.jar" > > > > Error: > > > > > > Also, when I am trying to build spark with Hive 3.1.3 I am getting > following error. > > > > If anyone can give me some direction then it would of great help. > > > > Thanks, > > Sanket > > > > *From:* Yeachan Park <yeachan...@gmail.com> > *Sent:* Tuesday, September 5, 2023 1:32 AM > *To:* Agrawal, Sanket <sankeagra...@deloitte.com.invalid> > *Cc:* user@spark.apache.org > *Subject:* [EXT] Re: Spark 3.4.1 and Hive 3.1.3 > > > > Hi, > > > > Why not download/build the hive 3.1.3 bundle and tell Spark to use that? > See https://spark.apache.org/docs/latest/sql-data-sources-hive-tables.html > <https://secure-web.cisco.com/1v3sayeGSek80FVpl91pY_-yK4C4shE3LRRtmMS7Th08V9Ka4HMy009tMFFeXWLOUKRl2IpbeIxoppyDNCEYm7q9QnWfObVCCq-DJEmdtoG4XJGEioT8hpRCd0PTuXEai8_zSk-2RzByf5ksuQ49_QPiGQi-33wS3GvNkBIjWvPuZstPHlxYrhYzOqZU2xIzhtr_VbqkG0N-7YHs1O8dyR-Xomli8_SgFh-RPPUuwb5nH-Yj-Ro6FTJ0hRlnOjmvu6c9im6V2WPg6rmWXr7KuN2zyuxzzsxr0fOYKgLLhSKUNi__wZY9jfzKlLalS88DZKx5fkK15vfWW-FTULz20KGDETRmLryWFZDaeTgYyDQ1-fqR-9G4IaVOvj9DmXRqkYRlMWE1n3Jq8BOABFJoyOdJF5RE3irkrOOYdk2Q5ip_qCwtd6qMKQH-QqlyNqWdbrGS1xPKdP1lZv25dJJ7KsM7kbO8eqyKlk0YJp5C1mVPZr4UfFu885lNXi-6D-3eudTU6B5m3-ynoieZC94eUGw/https%3A%2F%2Fspark.apache.org%2Fdocs%2Flatest%2Fsql-data-sources-hive-tables.html> > > > > Basically, set: > > spark.sql.hive.metastore.version 3.1.3 > > spark.sql.hive.metastore.jars path > > spark.sql.hive.metastore.jars.path <path_to_hive_3.1.3_jars> > > > > On Mon, Sep 4, 2023 at 7:42 PM Agrawal, Sanket < > sankeagra...@deloitte.com.invalid> wrote: > > Hi, > > > > Has anyone tried building Spark 3.4.1 with Hive 3.1.3. I tried by making > below changes in spark pom.xml but it’s failing. > > > > Pom.xml > > > > Error: > > > > Can anyone help me with the required configurations? > > > > Thanks, > > SA > > This message (including any attachments) contains confidential information > intended for a specific individual and purpose, and is protected by law. If > you are not the intended recipient, you should delete this message and any > disclosure, copying, or distribution of this message, or the taking of any > action based on it, by you is strictly prohibited. > > Deloitte refers to a Deloitte member firm, one of its related entities, or > Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a > separate legal entity and a member of DTTL. DTTL does not provide services > to clients. Please see www.deloitte.com/about to learn more. > > v.E.1 > >
