[
https://issues.apache.org/jira/browse/DRILL-5976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arina Ielchiieva reassigned DRILL-5976:
---------------------------------------
Assignee: (was: AnilKumar B)
> Kafka MessageReader config optimization
> ---------------------------------------
>
> Key: DRILL-5976
> URL: https://issues.apache.org/jira/browse/DRILL-5976
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: AnilKumar B
> Priority: Major
>
> Kafka storage plugin currently supports JSON message format, but going
> forward it will support Avro etc.
> Also there might be multiple scenarios where user have to implement their own
> MessageReader (Corresponding their custom Deserializer/Decoder).
> So, this JIRA is to brainstorm on whether to go with MessageReader as a
> *Plugin config* OR system/session option.
> Paul's suggestion as part review comment is as below.
> {noformat}
> Suppose this is two or three releases from now and we support other forms of
> Kafka messages. Different topics use different formats.
> If the message format is a system/session option, then I need to switch the
> option before each query. Very cumbersome and error prone.
> Instead, perhaps this information should be part of the storage plugin
> config. Then, I can define different plugins: one for each message format.
> Further, can I have multiple Kafka servers? If so, would I need different
> plugin configs for each?
> So, should we be thinking about encoding most properties as plugin config
> properties?
> Now, the plugin might have a format property, one of which is json. The JSON
> config properties would be defined in the json format within the overall
> storage plugin config.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
