Hi, Dale.
Thanks for your professional explanation ;)
--
Best！
Xuyang
在 2023-11-22 00:39:47，"Dale Lane" <[email protected]> 写道：
FYI in case it’s relevant for this discussion
> I'm not sure what is the ` Avro JSON` means
Avro supports two encoding mechanisms – binary encoding, and JSON encoding. [1]
In other words, an Avro record, although normally represented as binary data,
can instead be represented as a JSON document.
I contributed an improvement to the Flink Avro connector to add support for
JSON-encoded Avro as part of FLINK-33058 [2], although this is not yet merged.
I must admit, I’m not familiar with OpenSearch or what the Flink OpenSearch
connector offers, so my apologies if I’m just adding noise to the conversation
here!
Kind regards
Dale
[1] - https://avro.apache.org/docs/1.11.1/specification/#encodings
[2] - https://issues.apache.org/jira/browse/FLINK-33058
From: Xuyang <[email protected]>
Date: Tuesday, 21 November 2023 at 15:17
To: Praveen Chandna <[email protected]>
Cc: [email protected] <[email protected]>
Subject: [EXTERNAL] Re:Flink OpenSearch Connector - Avro JSON to JSON
Hi, Praveen. `OpenSearch supports writing data in the JSON format, but in Flink
its default data format is Avro JSON. ` I'm not sure what is the ` Avro JSON`
means. In Opensearch connector, there are multi formats it supports sucn as
"avro",
ZjQcmQRYFpfptBannerStart
|
|
|
This Message Is From an External Sender
|
|
This message came from outside your organization.
|
|
Report Suspicious
|
|
|
ZjQcmQRYFpfptBannerEnd
Hi, Praveen.
`OpenSearch supports writing data in the JSON format, but in Flink its default
data format is Avro JSON.`
I'm not sure what is the ` Avro JSON` means. In Opensearch connector, there are
multi formats it supports sucn as "avro", "format" and etc by adding
corresponding jars, and the default format[1] is "json" you want.
`OpenSearch Java Rest Client support writing the Java object directly to the
openSearch using the below code.`
Currently, the Opensearch connector connector always write the data
serialized[2]. You can modify here directly in Opensearch connector, re-package
it and try to use it.
[1]
https://github.com/apache/flink-connector-opensearch/blob/ab36cebc12db3aa0fa9df8a770b1845a78afe5bf/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/table/OpensearchConnectorOptions.java#L136
[2]
https://github.com/apache/flink-connector-opensearch/blob/ab36cebc12db3aa0fa9df8a770b1845a78afe5bf/flink-connector-opensearch/src/main/java/org/apache/flink/connector/opensearch/table/RowOpensearchEmitter.java#L110C32-L110C45
--
Best！
Xuyang
At 2023-11-21 13:32:35, "Praveen Chandna via user" <[email protected]>
wrote:
Hello Team
Please helps to answer the below query.
OpenSearch supports writing data in the JSON format, but in Flink its default
data format is Avro JSON. What is the best practice to write data to Open
Search using Flink OpenSearch Connector? Do we need to manually convert Avro
JSON to JSON format or is there any in-built support in Flink to write JSON to
openSearch.
How can I write the existing complex nested JSON data into OpenSearch using the
Flink connector rather than writing each key-value(or filed/value).
OpenSearch Java Rest Client support writing the Java object directly to the
openSearch using the below code.
IndexRequest<UserDataObject> indexRequest =new
IndexRequest.Builder<UserDataObject>().index(index).document(indexData).build();
https://opensearch.org/docs/latest/clients/java/
Thanks !!
// Regards
Praveen Chandna
Unless otherwise stated above:
IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
