Re: Converting ResolvedSchema to JSON and Protobuf Schemas

> meaning that double and integer I meant to write: "meaning that double and bigint ... " :) On Tue, Nov 15, 2022 at 8:54 AM Andrew Otto wrote: > > Also thanks for showing me your pattern with the SchemaConversions and > stuff. Feels pretty clean and worked like a charm :) > Glad to hear it, tha

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

> Also thanks for showing me your pattern with the SchemaConversions and stuff. Feels pretty clean and worked like a charm :) Glad to hear it, that is very cool! > converts number to double always. I wonder, did you make this up? Yes, we chose the the mapping. We chose to do number -> double and

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Yes, you are right. Schemas are not so nice in Json. When implementing and testing my converter from DataType to JsonSchema I noticed that your converter from JsonSchema to DataType converts number to double always. I wonder, did you make this up? Because the table that specifies the mapping

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Interesting, yeah I think you'll have to implement code to recurse through the (Row) DataType and somehow auto generate the JSONSchema you want. We abstracted the conversions from JSONSchema to other type systems in this JsonSchemaConverter

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

I want to register the result-schema in a schema registry, as I am pushing the result-data to a Kafka topic. The result-schema is not known at compile-time, so I need to find a way to compute it at runtime from the resulting Flink Schema. -Theo (resent - again sorry, I forgot to add the others

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Hey, thank you for your reply. Your converter looks very interesting. However, Flink comes with the JsonRowSchemaConverter that converts a JSONSchema-String to a TypeInformation already. From there you can convert the TypeInformation to, say, a DataType (Although I must admit I only got this do

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

> I want to convert the schema of a Flink table to both Protobuf *schema* and JSON *schema* Oh, you want to convert from Flink Schema TO JSONSchema? Interesting. That would indeed be something that is not usually done. Just curious, why do you want to do this? On Wed, Nov 9, 2022 at 8:46 AM And

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Hello! I see you are talking about JSONSchema, not just JSON itself. We're trying to do a similar thing at Wikimedia and have developed some tooling around this. JsonSchemaFlinkConverter

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Thanks for your reply Yaroslav! The way I do it with Avro seems similar to what you pointed out: ResolvedSchema resultSchema = resultTable.getResolvedSchema(); DataType type = resultSchema.toSinkRowDataType(); org.apache.avro.Schema converted = AvroSchemaConverter.convertToSchema(type.getLogicalT

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Got it. I'm not sure why you mentioned ResolvedSchema in the first place, usually in the Table API you work with RowType / RowData. - For Avro I use AvroSchemaConverter.convertToSchema method to get an Avro schema from a RowType, then use org.apache.flink.formats.avro.RowDataToAvroConverters to se

Re: Converting ResolvedSchema to JSON and Protobuf Schemas

Hey Theo, have you looked at the flink-json and flink-protobuf packages? On Tue, Nov 8, 2022 at 5:21 AM Theodor Wübker wrote: > Hello, > > I have a streaming use case, where I execute a query on a Table. I take > the ResolvedSchema of the table and convert it to an Avro-Schema using the > AvroSc

Converting ResolvedSchema to JSON and Protobuf Schemas

Hello, I have a streaming use case, where I execute a query on a Table. I take the ResolvedSchema of the table and convert it to an Avro-Schema using the AvroSchemaConverter. Now I want to do the same for JSON and Protobuf. However, it seems there is nothing similar to AvroSchemaConverter - I