Hi Grainier, we have a Github Actions workflow [1] for core+extensions that usually builds and pushes the latest development containers for dev on each push. You are using the latest CLI from dev and the tmpl_env or .env version is set to 0.67.0-SNAPSHOT, right?
I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local backend with the latest changes from dev and your PR) and it works great 😊 The only thing I had to change was a typo in line 59 (use outEvent instead of event). In the controller, the supportedProtocols and supportedFormats are no longer needed, as they can now be declared for a whole module in the Init class (registerFormats/Protocols). I need to update this in the documentation. Dominik https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml -----Original Message----- From: Grainier Perera <[email protected]> Sent: Friday, May 22, 2020 11:58 AM To: [email protected] Subject: Re: Data processor to evaluate JavaScript Hi Dominik, I've implemented the following[1]. However, when I try to install the processor it gives me the following error. Any clue? I've used CLI to start sp. So, maybe the snapshot docker image doesn't have the recent changes? If so what are the steps to build the images with changes in-place? [1] https://github.com/apache/incubator-streampipes-extensions/pull/15 ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of class org.apache.streampipes.model.output.OutputStrategy, it does not have a default constructor at io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178) at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261) at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695) at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388) at io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown Source) at io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown Source) at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at java.util.stream.ReferencePipeline.forEach(Unknown Source) at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299) at org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100) at org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169) at org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84) at org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96) at org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37) at org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93) at org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) Thanks, Grainier. On Thu, 21 May 2020 at 03:39, Dominik Riemer <[email protected]> wrote: > Hi Grainier, > > sounds good! > I've just pushed a commit that provides an initial version of the > user-defined output strategy. It lets you select the input schema from > the incoming stream and add/remove event properties. There is one > limitation, the component currently only works for flat event > structures, but it shouldn't be a problem to have a more advanced > schema editor for this output strategy soon (we already have this in > the StreamPipes connect code, but unfortunately, the pipeline editor > has yet to be migrated from AngularJS to Angular 2+ to reuse these > components). This is something we plan to do within the next weeks. > > I added an example how to use the strategy to the examples repo: > > https://github.com/apache/incubator-streampipes-examples/blob/dev/stre > ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap > ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro > ller.java > > Let me know what you think and if you need anything more/else! > > Dominik > > On 2020/05/19 17:14:21, Grainier Perera <[email protected]> > wrote: > > Hi Dominik, > > > > Awesome. This is really cool & exactly what I was looking for. I'll > > use this with my implementation, and keep this thread posted with > > the > updates. > > ps: let us know when you add output strategy as well. > > > > Thanks, > > Grainier. > > > > On Tue, 19 May 2020 at 03:05, Dominik Riemer <[email protected]> wrote: > > > > > Hi Grainier, > > > > > > I just pushed an initial version of the static property to add > javascript > > > code. > > > It's a simple codeblock with linting enabled and a (currently very > simple) > > > autocomplete feature that suggests existing properties from the > > > input > event > > > (currently, only if you press ctrl-space directly after typing "event" > > > followed by a dot, I'll fix that soon ;-) > > > > > > You can find an example how to use it here: > > > > > > > https://github.com/apache/incubator-streampipes-examples/blob/dev/stre > ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap > ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro > ller.java > > > > > > Feedback and ideas for improvement are welcome! > > > > > > Dominik > > > > > > > > > On 2020/05/17 12:36:05, Grainier Perera > > > <[email protected]> > > > wrote: > > > > Hi Dominik, > > > > > > > > Thank you. I'll work on `Implement JS evaluator` [1]. > > > > > > > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135 > > > > > > > > Grainier. > > > > > > > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <[email protected]> > wrote: > > > > > > > > > Hi Grainier, > > > > > > > > > > great! I created an umbrella issue for that: > > > > > https://issues.apache.org/jira/browse/STREAMPIPES-132 > > > > > > > > > > I'll now start to implement the new static property for > > > > > entering > code. > > > > > > > > > > Dominik > > > > > > > > > > On 2020/05/15 01:29:44, Grainier Perera > > > > > <[email protected] > > > > > > > wrote: > > > > > > Hi, > > > > > > > > > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you > > > > > > said, > > > that'll > > > > > be > > > > > > re-usable in the future processors as well. So let's proceed > > > > > > with > > > that. > > > > > > > > > > > > Regards, > > > > > > Grainier > > > > > > > > > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer > > > > > > <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > cool, that sounds very good! > > > > > > > > > > > > > > Concerning the output, I think there are two ways to > > > > > > > achieve > this: > > > > > > > - A collection static property in conjunction with a > > > > > > > runtime > > > resolvable > > > > > > > output strategy that creates the output schema based on > > > > > > > the > input > > > from > > > > > the > > > > > > > collection > > > > > > > - Or we create a new output strategy, e.g., let's call it > > > > > > > UserDefinedOutputStrategy, which would render a UI > > > > > > > component > to let > > > > > users > > > > > > > add/remove/change event properties from the input schema. > > > > > > > I > also > > > like > > > > > the > > > > > > > idea of preselecting all input properties. > > > > > > > > > > > > > > I personally tend towards the second option as such a > > > > > > > strategy > > > might be > > > > > > > useful for other pipeline elements as well and it would > > > > > > > ease > the > > > model > > > > > > > definition. We could also extend this component in the > > > > > > > future, > > > e.g., by > > > > > > > letting users enrich metadata such as measurement units > directly > > > in a > > > > > > > processor. > > > > > > > > > > > > > > But I'm also open to the first option and we can also just > start > > > > > > > implementing to see what works best. > > > > > > > > > > > > > > I'd also be happy to help with the implementation, e.g., I > could > > > work > > > > > on > > > > > > > the new static property for code input or the output strategy! > > > > > > > > > > > > > > Dominik > > > > > > > > > > > > > > On 2020/05/14 11:59:36, Grainier Perera < > [email protected] > > > > > > > > > > > wrote: > > > > > > > > Hi, > > > > > > > > > > > > > > > > +1 to the idea of having a collection property > > > > > > > > +containing the > > > fields > > > > > of > > > > > > > the > > > > > > > > output properties. Also, If we can show the user with > available > > > > > fields > > > > > > > with > > > > > > > > metadata (of input event) populated in that same > > > > > > > > collection > > > property, > > > > > > > then > > > > > > > > the user can add/remove/updates fields easily. Let's say > > > > > > > > they > > > just > > > > > have > > > > > > > to > > > > > > > > return a map containing all the fields they defined in > > > > > > > > output > > > > > properties > > > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": > > > > > > > > 301, > > > > > > > > "newTempInCelsius": 28 } ). > > > > > > > > > > > > > > > > [image: sample.png] > > > > > > > > > > > > > > > > Regards, > > > > > > > > Grainier. > > > > > > > > > > > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > sounds reasonable. > > > > > > > > > > > > > > > > > > I share the thoughts of Philipp regarding the output > schema. > > > But > > > > > as you > > > > > > > > > said Grainier, we can also improve :) > > > > > > > > > > > > > > > > > > btw: I really like the idea to also allow the more > technical > > > user > > > > > to > > > > > > > > > easily integrate their function snippets without > > > > > > > > > having to care about the actual execution. This gets a > > > > > > > > > more FaaS > like > > > > > feeling > > > > > > > for > > > > > > > > > streaming. > > > > > > > > > > > > > > > > > > Patrick > > > > > > > > > > > > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder < > > > > > [email protected]>: > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > it would be very useful to have such a processor. > > > > > > > > > > > > > > > > > > > > I like the ideas for 1 & 2 and I think we can do it > > > > > > > > > > this > way. > > > > > > > > > > > > > > > > > > > > Actually I think that the definition of the output > > > > > > > > > > scheme > > > could > > > > > be > > > > > > > quite > > > > > > > > > tricky. > > > > > > > > > > Because users can do anything with the event. > > > > > > > > > > > > > > > > > > > > How about we provide an option property? > > > > > > > > > > Users can select whether they want to define the > > > > > > > > > > whole > event > > > or > > > > > just > > > > > > > > > append properties. > > > > > > > > > > > > > > > > > > > > For the configurations of the properties we could do > > > something > > > > > > > similar > > > > > > > > > then we have in the PLC4xS7Adapter. > > > > > > > > > > A collection property containing the fields of the > > > > > > > > > > output > > > > > properties. > > > > > > > > > > Therefore, we have to decide what information we > > > > > > > > > > have to > > > > > provide. I > > > > > > > > > think if we would support all options it becomes very > cluttered > > > > > for the > > > > > > > > > user. > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > Philipp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera < > > > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> Hi Dominik, > > > > > > > > > >> > > > > > > > > > >> I agree. We have to do some enhancements make this > processor > > > > > user > > > > > > > > > friendly. > > > > > > > > > >> Like you said it's nice to have; > > > > > > > > > >> 1. A new static property that supports code > highlighting & > > > > > syntax > > > > > > > > > >> checking. We can simply do such client-side > > > > > > > > > >> validations > > > using > > > > > > > > > highlight.js > > > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc... > > > > > > > > > >> 2. A mechanism to show the user with available > > > > > > > > > >> fields > > > they can > > > > > > > use to > > > > > > > > > >> write code (similar to what you've done with usable > > > templates in > > > > > > > > > >> EmailPublisher using HtmlInputParameter). > > > > > > > > > >> 3. A mechanism to map output schema. > > > > > > > > > >> > > > > > > > > > >> I think 1 & 2 go together and for the 3rd > > > > > > > > > >> requirement > given > > > > > that we > > > > > > > are > > > > > > > > > >> targetting a more technical user group for this > processor, > > > we > > > > > can > > > > > > > let > > > > > > > > > them > > > > > > > > > >> define the output. Otherwise, it would be quite > difficult to > > > > > derive > > > > > > > > > return > > > > > > > > > >> type from the JS. Moreover, I'm planning to use > > > > > > > > > >> Java's > > > builtin > > > > > > > > > ScriptEngine > > > > > > > > > >> to eval. Of course, there'll be limitations. But, > > > > > > > > > >> we can > > > always > > > > > > > improve > > > > > > > > > >> them :) > > > > > > > > > >> > > > > > > > > > >> [1] https://highlightjs.org/ [2] > > > > > > > > > >> https://jshint.com/ > > > > > > > > > >> > > > > > > > > > >> Regards, > > > > > > > > > >> Grainier. > > > > > > > > > >> > > > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer < > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Hi Grainier, > > > > > > > > > >>> > > > > > > > > > >>> in my opinion, this would definitely be a very > interesting > > > > > > > addition! We > > > > > > > > > >>> already briefly discussed such a feature in the > > > > > > > > > >>> past, > > > before we > > > > > > > went > > > > > > > > > to the > > > > > > > > > >>> ASF. I’d say that a JS evaluator would be > > > > > > > > > >>> definitely > > > useful to > > > > > a > > > > > > > > > slightly > > > > > > > > > >>> more technical target user group, but would > > > > > > > > > >>> provide > great > > > > > > > flexibility > > > > > > > > > for, > > > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss > > > > > > > > > >>> what > is > > > > > needed to > > > > > > > > > >>> implement this! I’d guess that we (maybe) need to > > > > > > > > > >>> add > some > > > > > > > > > enhancements to > > > > > > > > > >>> the core to make this processor good from a > > > > > > > > > >>> usability > > > > > perspective. > > > > > > > > > >>> > > > > > > > > > >>> From a conceptual point of view, I guess the JS > processor > > > would > > > > > > > have no > > > > > > > > > >>> specific input requirements and a single static > property > > > that > > > > > > > allows > > > > > > > > > users > > > > > > > > > >>> to enter the code (maybe it would be cool to add a > > > > > > > > > >>> new > > > static > > > > > > > property > > > > > > > > > here > > > > > > > > > >>> that supports code highlighting or even syntax > checking). > > > The > > > > > only > > > > > > > open > > > > > > > > > >>> issue I see is the output schema – we would > > > > > > > > > >>> somehow > need to > > > > > > > extract the > > > > > > > > > >>> output from the JS function. This could probably > partially > > > be > > > > > done > > > > > > > > > using > > > > > > > > > >>> the CustomTransformOutput (see > > > > > > > > > >>> > > > > > > > > > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategie > > > s/), > > > > > > > > > >>> but we would somehow need to derive type > > > > > > > > > >>> information > from > > > the > > > > > JS > > > > > > > > > function > > > > > > > > > >>> or introduce a feature to let users define/refine > > > > > > > > > >>> the > > > output > > > > > > > manually > > > > > > > > > >>> (e.g., to add semantic metadata). > > > > > > > > > >>> > > > > > > > > > >>> What do you think would be best? And would you > > > > > > > > > >>> evaluate > > > the JS > > > > > > > code in > > > > > > > > > >>> Java or somehow else? > > > > > > > > > >>> > > > > > > > > > >>> Also, as I’ve just started to improve the SDK > > > > > > > > > >>> (e.g., > > > supporting > > > > > > > > > optional > > > > > > > > > >>> static properties and an easier way to define and > extract > > > model > > > > > > > > > >>> parameters), we can collect your requirements for > > > > > > > > > >>> the > JS > > > > > processor > > > > > > > and > > > > > > > > > >>> extend the SDK if needed, just say what you would > > > > > > > > > >>> like > to > > > have > > > > > 😉 > > > > > > > > > >>> > > > > > > > > > >>> Dominik > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> From: Grainier Perera <[email protected]> > > > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM > > > > > > > > > >>> To: [email protected] > > > > > > > > > >>> Subject: Data processor to evaluate JavaScript > > > > > > > > > >>> > > > > > > > > > >>> Hi all, > > > > > > > > > >>> > > > > > > > > > >>> I'm planning to implement a JavaScript eval data > > > processor. As > > > > > the > > > > > > > name > > > > > > > > > >>> suggests, users will be able to write some > > > > > > > > > >>> JavaScript > code > > > > > which > > > > > > > takes > > > > > > > > > in > > > > > > > > > >>> an event (as a map), do some processing on the > > > > > > > > > >>> event, > and > > > > > return a > > > > > > > new > > > > > > > > > map > > > > > > > > > >>> which then gets converted to an sp-event. > > > > > > > > > >>> > > > > > > > > > >>> example js: > > > > > > > > > >>> function process(event) { > > > > > > > > > >>> // do processing here. > > > > > > > > > >>> // return processed event. > > > > > > > > > >>> return {id: http://event.id, tempInCelsius: > > > > > (event.tempInKelvin > > > > > > > - > > > > > > > > > >>> 273.15)}; > > > > > > > > > >>> }; > > > > > > > > > >>> > > > > > > > > > >>> Will this be a good addition to pipeline-elements? > What do > > > you > > > > > > > think? > > > > > > > > > >>> > > > > > > > > > >>> Regards, > > > > > > > > > >>> Grainier. > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
