>From my experience of dealing with metadata for other applications like Hive if needed an external database for Spark metadata would be useful.
However, the maintenance and upgrade of that database should be external to Spark (left to the user) and as usual some form of reliable API or JDBC connection will be needed from Spark to this persistent storage. Maybe a NoSQL DB will do. HTH, Mich view my Linkedin profile <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Tue, 11 May 2021 at 08:58, Wenchen Fan <cloud0...@gmail.com> wrote: > That's my expectation as well. Spark needs a reliable catalog. > backup/restore is just implementation details about how you make your > catalog reliable, which should be transparent to Spark. > > On Sat, May 8, 2021 at 6:54 AM ayan guha <guha.a...@gmail.com> wrote: > >> Just a consideration: >> >> Is there a value in backup/restore metadata within spark? I would >> strongly argue if the metadata is valuable enough and persistent enough, >> why dont just use external metastore? It is fairly straightforward process. >> Also regardless you are in cloud or not, database bkp is a routine and >> established pattern in most organizations. >> You can also enhance HA and DR by having replicas across zones and >> regions etc etc >> >> Thoughts? >> >> >> >> >> On Sat, 8 May 2021 at 7:02 am, Tianchen Zhang <dustinzhang2...@gmail.com> >> wrote: >> >>> For now we are thinking about adding two methods in Catalog API, not SQL >>> commands: >>> 1. spark.catalog.backup, which backs up the current catalog. >>> 2. spark.catalog.restore(file), which reads the DFS file and recreates >>> the entities described in that file. >>> >>> Can you please give an example of exposing client APIs to the end users >>> in this approach? The users can only call backup or restore, right? >>> >>> Thanks, >>> Tianchen >>> >>> On Fri, May 7, 2021 at 12:27 PM Wenchen Fan <cloud0...@gmail.com> wrote: >>> >>>> If a catalog implements backup/restore, it can easily expose some >>>> client APIs to the end-users (e.g. REST API), I don't see a strong reason >>>> to expose the APIs to Spark. Do you plan to add new SQL commands in Spark >>>> to backup/restore a catalog? >>>> >>>> On Tue, May 4, 2021 at 2:39 AM Tianchen Zhang < >>>> dustinzhang2...@gmail.com> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Currently the user-facing Catalog API doesn't support backup/restore >>>>> metadata. Our customers are asking for such functionalities. Here is a >>>>> usage example: >>>>> 1. Read all metadata of one Spark cluster >>>>> 2. Save them into a Parquet file on DFS >>>>> 3. Read the Parquet file and restore all metadata in another Spark >>>>> cluster >>>>> >>>>> From the current implementation, Catalog API has the list methods >>>>> (listDatabases, listFunctions, etc.) but they don't return enough >>>>> information in order to restore an entity (for example, listDatabases lose >>>>> "properties" of the database and we need "describe database extended" to >>>>> get them). And it only supports createTable (not any other entity >>>>> creations). The only way we can backup/restore an entity is using Spark >>>>> SQL. >>>>> >>>>> We want to introduce the backup and restore from an API level. We are >>>>> thinking of doing this simply by adding backup() and restore() in >>>>> CatalogImpl, as ExternalCatalog already includes all the methods we need >>>>> to >>>>> retrieve and recreate entities. We are wondering if there is any concern >>>>> or >>>>> drawback of this approach. Please advise. >>>>> >>>>> Thank you in advance, >>>>> Tianchen >>>>> >>>> -- >> Best Regards, >> Ayan Guha >> >
