Hello Zoi, I have the following suggestion:
Like we have an abstraction for the "supported Platforms" we should have an abstraction for the metadata related to the platform we work with. Here is an example from Kafka world: Multiple Kafka-Clusters are working with one SchemaRegistry within one Region. This is a special case, yes, each cluster could have its own SR. And this is the thing. We could have an organization with 3 supported Platforms, lets say: Spark, Flink, and PGSQL based processing. This organization uses e.g. https://open-metadata.org/ and a second organization uses only Apache Flink on top of an Apache Kafka cluster - no MD service. We should be able to handle such a situation. Proposal: Platform Configuration holds a reference to a MD-Service, or none. The MD-Service is an abstraction over the real implementation, and the implementation of a particular connector does the MD-Matching and Discovery. This opens the door for a "direct API based" integration of any given MD-Service and also, to implement a flexible Metadata discovery mechanism for Platforms and Users who do not even have a MD-Service. What do you think? Cheers, Mirko Am Fr., 26. Sept. 2025 um 08:49 Uhr schrieb Zoi Kaoudi <[email protected]>: > Cool! Thanks Kaustubh, I will take a look > Best > -- > Zoi > Στις Παρασκευή 26 Σεπτεμβρίου 2025 στις 02:43:09 π.μ. CEST, ο χρήστης > Kaustubh Beedkar <[email protected]> έγραψε: > > Id suggest to use openmetadata https://open-metadata.org/. Not as is, > but integrate its connectors/catalog. > > Other projects we could look at > - Apache Atlas > - Datahub > > Best, > Kaustubh > > > > Sent from my iPhone > > > On 26 Sep 2025, at 1:14 AM, Zoi Kaoudi <[email protected]> wrote: > > > > Dear Wayang community, > > how do you think we should go about getting metadata from the data > sources used by Wayang? It has popped up in different projects where things > like table names, columns etc. are required for the application (e.g., GUI) > and the current solution we have built is just a workaround by getting > directly the metadata from the source. But in a realistic scenario it may > not be possible that the application has access to the data sources, right? > > My question is shall we go for an independent metadata catalog? Do you > have some tools in mind? Or shall Wayang handle metadata from a diverse set > of sources and expose them with an API to the users? > > Any thoughts, ideas? > > Best > > - > > Zoi > -- Dr. Mirko Kämpf *Gründer & Coach * *maindset.ACADEMY* Müchelner Str. 23 06259 Frankleben
