Hi Devin, a warm welcome to the StreamPipes community. Yes, you are right. So far we have support for Flink processors, but not for Pulsar functions. The Pulsar integration currently only is via an adapter [1] and a sink [2] to read/write data from/to Pulsar. Currently, we mainly focus on the standalone java implementations, but it would be great to have a better Pulsar integration. It would be very cool to learn from your experience with Pulsar.
In general we have a wrapper concept, to provide a wrapper for different processing engines (e.g. Standalone [3], Flink [4], Siddhi [5]). Based on those wrappers it is then possible to implement extensions, to connect data streams and process events. Here is an example of a filter processor using the Java API [6]. In [7] it is described how you can setup the development environment. I hope this helps. Please write when you have any questions. Cheers, Philipp [1] https://github.com/apache/incubator-streampipes/tree/dev/streampipes-extensions/streampipes-connect-adapters-iiot/src/main/java/org/apache/streampipes/connect/iiot/protocol/stream/pulsar [2] https://github.com/apache/incubator-streampipes/tree/dev/streampipes-extensions/streampipes-sinks-brokers-jvm/src/main/java/org/apache/streampipes/sinks/brokers/jvm/pulsar [3] https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-standalone [4] https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-flink [5] https://github.com/apache/incubator-streampipes/tree/dev/streampipes-wrapper-siddhi/src/main/java/org/apache/streampipes/wrapper/siddhi [6] https://github.com/apache/incubator-streampipes/blob/dev/streampipes-extensions/streampipes-processors-filters-jvm/src/main/java/org/apache/streampipes/processors/filters/jvm/processor/numericalfilter/NumericalFilterProcessor.java [7] https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=148647899#content/view/148647899 Von: Devin Bost <[email protected]> Datum: Samstag, 28. Mai 2022 um 07:34 An: [email protected] <[email protected]> Betreff: Contributing Pulsar Function support Hi, I'm an Apache Pulsar contributor, and I'm very interested in your project! I've wanted a GUI for building serverless flows involving Flink and Pulsar for a long time, and your project seems to be the perfect platform for that. I noticed that you have support currently for Flink processors, and you have a Pulsar connector, but it doesn't appear that you have support for Pulsar function processors yet. My team has created a collection of generic Pulsar functions for performing common tasks, and it would be awesome if we could get StreamPipes to allow teams to use the UI to build flows with them. Examples of functions include things like JSON schema validation, sieves (dropping out columns), Pulsar message filters, and many others. What classes would be involved in adding support for Pulsar functions? We also have some topics involved in governance and automation for our functions, so I'd also like to write an interception layer to allow us to kick off those tasks when functions are created. If someone could please point out the code paths that I'd want to look at in StreamPipes, that would help a lot! Devin G. Bost
