[ https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16690723#comment-16690723 ]
ASF GitHub Bot commented on GRIFFIN-213: ---------------------------------------- GitHub user chemikadze opened a pull request: https://github.com/apache/incubator-griffin/pull/456 [GRIFFIN-213] Custom connector support Provide ability to extend batch and streaming data integrations with custom user-provided connectors. Introduces new data connector type, `CUSTOM`, parameterized with `class` property. Also adds support for custom data connector enum on service side. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-213 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/456.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #456 ---- commit d487347a363f172cfc9e26225d5687cc3f95ab73 Author: Nikolay Sokolov <chemikadze@...> Date: 2018-11-17T22:37:36Z [GRIFFIN-213] Custom connector support Provide ability to extend batch and streaming data integrations with custom user-provided connectors. Introduces new data connector type, `CUSTOM`, parameterized with `class` property. Also adds support for custom data connector enum on service side. ---- > Support pluggable datasource connectors > --------------------------------------- > > Key: GRIFFIN-213 > URL: https://issues.apache.org/jira/browse/GRIFFIN-213 > Project: Griffin (Incubating) > Issue Type: Improvement > Reporter: Nikolay Sokolov > Priority: Minor > > As of Griffin 0.3, code modification is required, in order to add new data > connectors. > Proposal is to add new data connector type, CUSTOM, that would allow to > specify class name of data connector implementation to use. Additional jars > with custom connector implementations would be provided in spark > configuration template. > Class name would be specified in "class" config of data connector. For > example: > {code:json} > "connectors": [ > { > "type": "CUSTOM", > "config": { > "class": "org.example.griffin.JDBCConnector" > // extra connector-specific parameters > } > } > ] > {code} > Proposed contract for implementations is based on current convention: > - for batch > ** class should be a subclass of BatchDataConnector > ** if should have method with signature: > {code:java} > public static BatchDataConnector apply(ctx: BatchDataConnectorContext) > {code} > - for streaming > ** class should be a subclass of StreamingDataConnector > ** it should have method with signature: > {code:java} > public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext) > {code} > Signatures of context objects: > {code:scala} > case class BatchDataConnectorContext(@transient sparkSession: SparkSession, > dcParam: DataConnectorParam, > timestampStorage: TimestampStorage) > case class StreamingDataConnectorContext(@transient sparkSession: > SparkSession, > @transient ssc: StreamingContext, > dcParam: DataConnectorParam, > timestampStorage: TimestampStorage, > streamingCacheClientOpt: > Option[StreamingCacheClient]) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)