[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691151#comment-16691151
]
Eugene commented on GRIFFIN-213:
--------------------------------
two comments:
1.customization connector allows users to connect any data source and involves
third-party plugins, how could we guarantee safety and security of griffin
pipeline, is there policy or permission check?
2.'custom' seems not suitable connector type which describes data source like
'kafka' 'hive' 'text', do you think about it?
> Support pluggable datasource connectors
> ---------------------------------------
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
> Reporter: Nikolay Sokolov
> Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
