Hi @Wenchen Fan <[email protected]> Thanks for your response. I believe we have not had enough time to "DISCUSS" this matter.
Currently in order to make Spark take advantage of Hive, I create a soft link in $SPARK_HOME/conf. FYI, my spark version is 3.4.0 and Hive is 3.1.1 /opt/spark/conf/hive-site.xml -> /data6/hduser/hive-3.1.1/conf/hive-site.xml This works fine for me in my lab. So in the future if we opt to use the setting "spark.sql.legacy.createHiveTableByDefault" to False, there will not be a need for this logical link.? On the face of it, this looks fine but in real life it may require a number of changes to the old scripts. Hence my concern. As a matter of interest has anyone liaised with the Hive team to ensure they have introduced the additional changes you outlined? HTH Mich Talebzadeh, Technologist | Architect | Data Engineer | Generative AI | FinCrime London United Kingdom view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> https://en.everybodywiki.com/Mich_Talebzadeh *Disclaimer:* The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". On Sun, 28 Apr 2024 at 09:34, Wenchen Fan <[email protected]> wrote: > @Mich Talebzadeh <[email protected]> thanks for sharing your > concern! > > Note: creating Spark native data source tables is usually Hive compatible > as well, unless we use features that Hive does not support (TIMESTAMP NTZ, > ANSI INTERVAL, etc.). I think it's a better default to create Spark native > table in this case, instead of creating Hive table and fail. > > On Sat, Apr 27, 2024 at 12:46 PM Cheng Pan <[email protected]> wrote: > >> +1 (non-binding) >> >> Thanks, >> Cheng Pan >> >> On Sat, Apr 27, 2024 at 9:29 AM Holden Karau <[email protected]> >> wrote: >> > >> > +1 >> > >> > Twitter: https://twitter.com/holdenkarau >> > Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 >> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> > >> > >> > On Fri, Apr 26, 2024 at 12:06 PM L. C. Hsieh <[email protected]> wrote: >> >> >> >> +1 >> >> >> >> On Fri, Apr 26, 2024 at 10:01 AM Dongjoon Hyun <[email protected]> >> wrote: >> >> > >> >> > I'll start with my +1. >> >> > >> >> > Dongjoon. >> >> > >> >> > On 2024/04/26 16:45:51 Dongjoon Hyun wrote: >> >> > > Please vote on SPARK-46122 to set >> spark.sql.legacy.createHiveTableByDefault >> >> > > to `false` by default. The technical scope is defined in the >> following PR. >> >> > > >> >> > > - DISCUSSION: >> >> > > https://lists.apache.org/thread/ylk96fg4lvn6klxhj6t6yh42lyqb8wmd >> >> > > - JIRA: https://issues.apache.org/jira/browse/SPARK-46122 >> >> > > - PR: https://github.com/apache/spark/pull/46207 >> >> > > >> >> > > The vote is open until April 30th 1AM (PST) and passes >> >> > > if a majority +1 PMC votes are cast, with a minimum of 3 +1 votes. >> >> > > >> >> > > [ ] +1 Set spark.sql.legacy.createHiveTableByDefault to false by >> default >> >> > > [ ] -1 Do not change spark.sql.legacy.createHiveTableByDefault >> because ... >> >> > > >> >> > > Thank you in advance. >> >> > > >> >> > > Dongjoon >> >> > > >> >> > >> >> > --------------------------------------------------------------------- >> >> > To unsubscribe e-mail: [email protected] >> >> > >> >> >> >> --------------------------------------------------------------------- >> >> To unsubscribe e-mail: [email protected] >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: [email protected] >> >>
