+1 On Thu, Apr 25, 2024 at 8:16 PM Yuming Wang <yumw...@apache.org> wrote:
> +1 > > On Fri, Apr 26, 2024 at 8:25 AM Nimrod Ofek <ofek.nim...@gmail.com> wrote: > >> Of course, I can't think of a scenario of thousands of tables with single >> in memory Spark cluster with in memory catalog. >> Thanks for the help! >> >> בתאריך יום ה׳, 25 באפר׳ 2024, 23:56, מאת Mich Talebzadeh < >> mich.talebza...@gmail.com>: >> >>> >>> >>> Agreed. In scenarios where most of the interactions with the catalog are >>> related to query planning, saving and metadata management, the choice of >>> catalog implementation may have less impact on query runtime performance. >>> This is because the time spent on metadata operations is generally >>> minimal compared to the time spent on actual data fetching, processing, and >>> computation. >>> However, if we consider scalability and reliability concerns, especially >>> as the size and complexity of data and query workload grow. While an >>> in-memory catalog may offer excellent performance for smaller workloads, >>> it will face limitations in handling larger-scale deployments with >>> thousands of tables, partitions, and users. Additionally, durability and >>> persistence are crucial considerations, particularly in production >>> environments where data integrity >>> and availability are crucial. In-memory catalog implementations may lack >>> durability, meaning that metadata changes could be lost in the event of a >>> system failure or restart. Therefore, while in-memory catalog >>> implementations can provide speed and efficiency for certain use cases, we >>> ought to consider the requirements for scalability, reliability, and data >>> durability when choosing a catalog solution for production deployments. In >>> many cases, a combination of in-memory and disk-based catalog solutions may >>> offer the best balance of performance and resilience for demanding large >>> scale workloads. >>> >>> >>> HTH >>> >>> >>> Mich Talebzadeh, >>> Technologist | Architect | Data Engineer | Generative AI | FinCrime >>> London >>> United Kingdom >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* The information provided is correct to the best of my >>> knowledge but of course cannot be guaranteed . It is essential to note >>> that, as with any advice, quote "one test result is worth one-thousand >>> expert opinions (Werner >>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>> >>> >>> On Thu, 25 Apr 2024 at 16:32, Nimrod Ofek <ofek.nim...@gmail.com> wrote: >>> >>>> Of course, but it's in memory and not persisted which is much faster, >>>> and as I said- I believe that most of the interaction with it is during the >>>> planning and save and not actual query run operations, and they are short >>>> and minimal compared to data fetching and manipulation so I don't believe >>>> it will have big impact on query run... >>>> >>>> בתאריך יום ה׳, 25 באפר׳ 2024, 17:52, מאת Mich Talebzadeh < >>>> mich.talebza...@gmail.com>: >>>> >>>>> Well, I will be surprised because Derby database is single threaded >>>>> and won't be much of a use here. >>>>> >>>>> Most Hive metastore in the commercial world utilise postgres or Oracle >>>>> for metastore that are battle proven, replicated and backed up. >>>>> >>>>> Mich Talebzadeh, >>>>> Technologist | Architect | Data Engineer | Generative AI | FinCrime >>>>> London >>>>> United Kingdom >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* The information provided is correct to the best of my >>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>> expert opinions (Werner >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>> >>>>> >>>>> On Thu, 25 Apr 2024 at 15:39, Nimrod Ofek <ofek.nim...@gmail.com> >>>>> wrote: >>>>> >>>>>> Yes, in memory hive catalog backed by local Derby DB. >>>>>> And again, I presume that most metadata related parts are during >>>>>> planning and not actual run, so I don't see why it should strongly affect >>>>>> query performance. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> >>>>>> בתאריך יום ה׳, 25 באפר׳ 2024, 17:29, מאת Mich Talebzadeh < >>>>>> mich.talebza...@gmail.com>: >>>>>> >>>>>>> With regard to your point below >>>>>>> >>>>>>> "The thing I'm missing is this: let's say that the output format I >>>>>>> choose is delta lake or iceberg or whatever format that uses parquet. >>>>>>> Where >>>>>>> does the catalog implementation (which holds metadata afaik, same >>>>>>> metadata >>>>>>> that iceberg and delta lake save for their tables about their columns) >>>>>>> comes into play and why should it affect performance? " >>>>>>> >>>>>>> The catalog implementation comes into play regardless of the output >>>>>>> format chosen (Delta Lake, Iceberg, Parquet, etc.) because it is >>>>>>> responsible for managing metadata about the datasets, tables, schemas, >>>>>>> and >>>>>>> other objects stored in aforementioned formats. Even though Delta Lake >>>>>>> and >>>>>>> Iceberg have their metadata management mechanisms internally, they still >>>>>>> rely on the catalog for providing a unified interface for accessing and >>>>>>> manipulating metadata across different storage formats. >>>>>>> >>>>>>> "Another thing is that if I understand correctly, and I might be >>>>>>> totally wrong here, the internal spark catalog is a local installation >>>>>>> of >>>>>>> hive metastore anyway, so I'm not sure what the catalog has to do with >>>>>>> anything" >>>>>>> >>>>>>> .I don't understand this. Do you mean a Derby database? >>>>>>> >>>>>>> HTH >>>>>>> >>>>>>> Mich Talebzadeh, >>>>>>> Technologist | Architect | Data Engineer | Generative AI | FinCrime >>>>>>> London >>>>>>> United Kingdom >>>>>>> >>>>>>> >>>>>>> view my Linkedin profile >>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>> >>>>>>> >>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>> >>>>>>> >>>>>>> >>>>>>> *Disclaimer:* The information provided is correct to the best of my >>>>>>> knowledge but of course cannot be guaranteed . It is essential to note >>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>> expert opinions (Werner >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>> >>>>>>> >>>>>>> On Thu, 25 Apr 2024 at 14:38, Nimrod Ofek <ofek.nim...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks for the detailed answer. >>>>>>>> The thing I'm missing is this: let's say that the output format I >>>>>>>> choose is delta lake or iceberg or whatever format that uses parquet. >>>>>>>> Where >>>>>>>> does the catalog implementation (which holds metadata afaik, same >>>>>>>> metadata >>>>>>>> that iceberg and delta lake save for their tables about their columns) >>>>>>>> comes into play and why should it affect performance? >>>>>>>> Another thing is that if I understand correctly, and I might be >>>>>>>> totally wrong here, the internal spark catalog is a local installation >>>>>>>> of >>>>>>>> hive metastore anyway, so I'm not sure what the catalog has to do with >>>>>>>> anything. >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>>> בתאריך יום ה׳, 25 באפר׳ 2024, 16:14, מאת Mich Talebzadeh < >>>>>>>> mich.talebza...@gmail.com>: >>>>>>>> >>>>>>>>> My take regarding your question is that your mileage varies so to >>>>>>>>> speak. >>>>>>>>> >>>>>>>>> 1) Hive provides a more mature and widely adopted catalog solution >>>>>>>>> that integrates well with other components in the Hadoop ecosystem, >>>>>>>>> such as >>>>>>>>> HDFS, HBase, and YARN. IIf you are Hadoop centric S(say on-premise), >>>>>>>>> using >>>>>>>>> Hive may offer better compatibility and interoperability. >>>>>>>>> 2) Hive provides a SQL-like interface that is familiar to users >>>>>>>>> who are accustomed to traditional RDBMs. If your use case involves >>>>>>>>> complex >>>>>>>>> SQL queries or existing SQL-based workflows, using Hive may be >>>>>>>>> advantageous. >>>>>>>>> 3) If you are looking for performance, spark's native catalog >>>>>>>>> tends to offer better performance for certain workloads, particularly >>>>>>>>> those >>>>>>>>> that involve iterative processing or complex data transformations.(my >>>>>>>>> understanding). Spark's in-memory processing capabilities and >>>>>>>>> optimizations >>>>>>>>> make it well-suited for interactive analytics and machine learning >>>>>>>>> tasks.(my favourite) >>>>>>>>> 4) Integration with Spark Workflows: If you primarily use Spark >>>>>>>>> for data processing and analytics, using Spark's native catalog may >>>>>>>>> simplify workflow management and reduce overhead, Spark's tight >>>>>>>>> integration with its catalog allows for seamless interaction with >>>>>>>>> Spark >>>>>>>>> applications and libraries. >>>>>>>>> 5) There seems to be some similarity with spark catalog and >>>>>>>>> Databricks unity catalog, so that may favour the choice. >>>>>>>>> >>>>>>>>> HTH >>>>>>>>> >>>>>>>>> Mich Talebzadeh, >>>>>>>>> Technologist | Architect | Data Engineer | Generative AI | >>>>>>>>> FinCrime >>>>>>>>> London >>>>>>>>> United Kingdom >>>>>>>>> >>>>>>>>> >>>>>>>>> view my Linkedin profile >>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>> >>>>>>>>> >>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> *Disclaimer:* The information provided is correct to the best of >>>>>>>>> my knowledge but of course cannot be guaranteed . It is essential to >>>>>>>>> note >>>>>>>>> that, as with any advice, quote "one test result is worth one-thousand >>>>>>>>> expert opinions (Werner >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, 25 Apr 2024 at 12:30, Nimrod Ofek <ofek.nim...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> I will also appreciate some material that describes the >>>>>>>>>> differences between Spark native tables vs hive tables and why each >>>>>>>>>> should >>>>>>>>>> be used... >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> Nimrod >>>>>>>>>> >>>>>>>>>> בתאריך יום ה׳, 25 באפר׳ 2024, 14:27, מאת Mich Talebzadeh < >>>>>>>>>> mich.talebza...@gmail.com>: >>>>>>>>>> >>>>>>>>>>> I see a statement made as below and I quote >>>>>>>>>>> >>>>>>>>>>> "The proposal of SPARK-46122 is to switch the default value of >>>>>>>>>>> this >>>>>>>>>>> configuration from `true` to `false` to use Spark native tables >>>>>>>>>>> because >>>>>>>>>>> we support better." >>>>>>>>>>> >>>>>>>>>>> Can you please elaborate on the above specifically with regard >>>>>>>>>>> to the phrase ".. because >>>>>>>>>>> we support better." >>>>>>>>>>> >>>>>>>>>>> Are you referring to the performance of Spark catalog (I believe >>>>>>>>>>> it is internal) or integration with Spark? >>>>>>>>>>> >>>>>>>>>>> HTH >>>>>>>>>>> >>>>>>>>>>> Mich Talebzadeh, >>>>>>>>>>> Technologist | Architect | Data Engineer | Generative AI | >>>>>>>>>>> FinCrime >>>>>>>>>>> London >>>>>>>>>>> United Kingdom >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> view my Linkedin profile >>>>>>>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> *Disclaimer:* The information provided is correct to the best >>>>>>>>>>> of my knowledge but of course cannot be guaranteed . It is >>>>>>>>>>> essential to >>>>>>>>>>> note that, as with any advice, quote "one test result is worth >>>>>>>>>>> one-thousand expert opinions (Werner >>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>Von Braun >>>>>>>>>>> <https://en.wikipedia.org/wiki/Wernher_von_Braun>)". >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, 25 Apr 2024 at 11:17, Wenchen Fan <cloud0...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> +1 >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Apr 25, 2024 at 2:46 PM Kent Yao <y...@apache.org> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 >>>>>>>>>>>>> >>>>>>>>>>>>> Nit: the umbrella ticket is SPARK-44111, not SPARK-44444. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Kent Yao >>>>>>>>>>>>> >>>>>>>>>>>>> Dongjoon Hyun <dongjoon.h...@gmail.com> 于2024年4月25日周四 14:39写道： >>>>>>>>>>>>> > >>>>>>>>>>>>> > Hi, All. >>>>>>>>>>>>> > >>>>>>>>>>>>> > It's great to see community activities to polish 4.0.0 more >>>>>>>>>>>>> and more. >>>>>>>>>>>>> > Thank you all. >>>>>>>>>>>>> > >>>>>>>>>>>>> > I'd like to bring SPARK-46122 (another SQL topic) to you >>>>>>>>>>>>> from the subtasks >>>>>>>>>>>>> > of SPARK-44444 (Prepare Apache Spark 4.0.0), >>>>>>>>>>>>> > >>>>>>>>>>>>> > - https://issues.apache.org/jira/browse/SPARK-46122 >>>>>>>>>>>>> > Set `spark.sql.legacy.createHiveTableByDefault` to >>>>>>>>>>>>> `false` by default >>>>>>>>>>>>> > >>>>>>>>>>>>> > This legacy configuration is about `CREATE TABLE` SQL syntax >>>>>>>>>>>>> without >>>>>>>>>>>>> > `USING` and `STORED AS`, which is currently mapped to `Hive` >>>>>>>>>>>>> table. >>>>>>>>>>>>> > The proposal of SPARK-46122 is to switch the default value >>>>>>>>>>>>> of this >>>>>>>>>>>>> > configuration from `true` to `false` to use Spark native >>>>>>>>>>>>> tables because >>>>>>>>>>>>> > we support better. >>>>>>>>>>>>> > >>>>>>>>>>>>> > In other words, Spark will use the value of >>>>>>>>>>>>> `spark.sql.sources.default` >>>>>>>>>>>>> > as the table provider instead of `Hive` like the other Spark >>>>>>>>>>>>> APIs. Of course, >>>>>>>>>>>>> > the users can get all the legacy behavior by setting back to >>>>>>>>>>>>> `true`. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Historically, this behavior change was merged once at Apache >>>>>>>>>>>>> Spark 3.0.0 >>>>>>>>>>>>> > preparation via SPARK-30098 already, but reverted during the >>>>>>>>>>>>> 3.0.0 RC period. >>>>>>>>>>>>> > >>>>>>>>>>>>> > 2019-12-06: SPARK-30098 Use default datasource as provider >>>>>>>>>>>>> for CREATE TABLE >>>>>>>>>>>>> > 2020-05-16: SPARK-31707 Revert SPARK-30098 Use default >>>>>>>>>>>>> datasource as >>>>>>>>>>>>> > provider for CREATE TABLE command >>>>>>>>>>>>> > >>>>>>>>>>>>> > At Apache Spark 3.1.0, we had another discussion about this >>>>>>>>>>>>> and defined it >>>>>>>>>>>>> > as one of legacy behavior via this configuration via reused >>>>>>>>>>>>> ID, SPARK-30098. >>>>>>>>>>>>> > >>>>>>>>>>>>> > 2020-12-01: >>>>>>>>>>>>> https://lists.apache.org/thread/8c8k1jk61pzlcosz3mxo4rkj5l23r204 >>>>>>>>>>>>> > 2020-12-03: SPARK-30098 Add a configuration to use default >>>>>>>>>>>>> datasource as >>>>>>>>>>>>> > provider for CREATE TABLE command >>>>>>>>>>>>> > >>>>>>>>>>>>> > Last year, we received two additional requests twice to >>>>>>>>>>>>> switch this because >>>>>>>>>>>>> > Apache Spark 4.0.0 is a good time to make a decision for the >>>>>>>>>>>>> future direction. >>>>>>>>>>>>> > >>>>>>>>>>>>> > 2023-02-27: SPARK-42603 as an independent idea. >>>>>>>>>>>>> > 2023-11-27: SPARK-46122 as a part of Apache Spark 4.0.0 idea >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > WDYT? The technical scope is defined in the following PR >>>>>>>>>>>>> which is one line of main >>>>>>>>>>>>> > code, one line of migration guide, and a few lines of test >>>>>>>>>>>>> code. >>>>>>>>>>>>> > >>>>>>>>>>>>> > - https://github.com/apache/spark/pull/46207 >>>>>>>>>>>>> > >>>>>>>>>>>>> > Dongjoon. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>>>>>>>> >>>>>>>>>>>>>
