TL:DR: We want to develop support for Apache CDAP batch and streaming plugins to enrich Apache Beam connectors to external applications. Please review the design[1] to help us bring CDAP plugins integrations into Apache Beam.
Hi all, I along with a few community members thought of an idea to create an Apache Beam IO package for Apache CDAP. CDAP IO package will enable creating integrations of Apache CDAP plugins<https://github.com/data-integrations> to Apache Beam to extend application connectors offered by Apache Beam. CDAP IO connector will support batch source and sinks via HadoopFormatIO. CDAP IO connector will support streaming sources (SparkReceiverIO to proxy Custom Spark Receivers for usage in Apache Beam). The proposed design and implementation details are described in the design document[1]. Initially, we are thinking of creating the following integrations: * Salesforce * ServiceNow (batch) * Zendesk (batch) * Salesforce Marketing Cloud (batch) * Hubspot Please share your feedback both on the idea and the design doc. Thanks! [1] https://docs.google.com/document/d/1T-bhd0Qk7DBePIfgHEPagYiA1oLP4z5kYEd0S1SOGxQ/edit?usp=sharing
