Hi Dominik, My bad. There were some cached images & that was the issue. Also fixed the PR as per your comments. Please review and merge [1].
[1] https://github.com/apache/incubator-streampipes-extensions/pull/15 Thanks & Regards, Grainier On Fri, 22 May 2020 at 16:25, Dominik Riemer <[email protected]> wrote: > Hi Grainier, > > we have a Github Actions workflow [1] for core+extensions that usually > builds and pushes the latest development containers for dev on each push. > You are using the latest CLI from dev and the tmpl_env or .env version is > set to 0.67.0-SNAPSHOT, right? > > I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local > backend with the latest changes from dev and your PR) and it works great 😊 > The only thing I had to change was a typo in line 59 (use outEvent instead > of event). > > In the controller, the supportedProtocols and supportedFormats are no > longer needed, as they can now be declared for a whole module in the Init > class (registerFormats/Protocols). I need to update this in the > documentation. > > Dominik > > > > > > https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml > > -----Original Message----- > From: Grainier Perera <[email protected]> > Sent: Friday, May 22, 2020 11:58 AM > To: [email protected] > Subject: Re: Data processor to evaluate JavaScript > > Hi Dominik, > > I've implemented the following[1]. However, when I try to install the > processor it gives me the following error. Any clue? I've used CLI to start > sp. So, maybe the snapshot docker image doesn't have the recent changes? If > so what are the steps to build the images with changes in-place? > > [1] https://github.com/apache/incubator-streampipes-extensions/pull/15 > > ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy > io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of > class org.apache.streampipes.model.output.OutputStrategy, it does not have > a default constructor at > io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178) > at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261) > at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695) > at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388) > at > io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown > Source) > at > io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown > Source) > at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at > java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) > at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at > java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at > java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) at > java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown > Source) > at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at > java.util.stream.ReferencePipeline.forEach(Unknown Source) at > io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299) > at > > org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100) > at > > org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169) > at > > org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84) > at > > org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96) > at > > org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37) > at > > org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93) > at > > org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > Thanks, > Grainier. > > On Thu, 21 May 2020 at 03:39, Dominik Riemer <[email protected]> wrote: > > > Hi Grainier, > > > > sounds good! > > I've just pushed a commit that provides an initial version of the > > user-defined output strategy. It lets you select the input schema from > > the incoming stream and add/remove event properties. There is one > > limitation, the component currently only works for flat event > > structures, but it shouldn't be a problem to have a more advanced > > schema editor for this output strategy soon (we already have this in > > the StreamPipes connect code, but unfortunately, the pipeline editor > > has yet to be migrated from AngularJS to Angular 2+ to reuse these > > components). This is something we plan to do within the next weeks. > > > > I added an example how to use the strategy to the examples repo: > > > > https://github.com/apache/incubator-streampipes-examples/blob/dev/stre > > ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap > > ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro > > ller.java > > > > Let me know what you think and if you need anything more/else! > > > > Dominik > > > > On 2020/05/19 17:14:21, Grainier Perera <[email protected]> > > wrote: > > > Hi Dominik, > > > > > > Awesome. This is really cool & exactly what I was looking for. I'll > > > use this with my implementation, and keep this thread posted with > > > the > > updates. > > > ps: let us know when you add output strategy as well. > > > > > > Thanks, > > > Grainier. > > > > > > On Tue, 19 May 2020 at 03:05, Dominik Riemer <[email protected]> > wrote: > > > > > > > Hi Grainier, > > > > > > > > I just pushed an initial version of the static property to add > > javascript > > > > code. > > > > It's a simple codeblock with linting enabled and a (currently very > > simple) > > > > autocomplete feature that suggests existing properties from the > > > > input > > event > > > > (currently, only if you press ctrl-space directly after typing > "event" > > > > followed by a dot, I'll fix that soon ;-) > > > > > > > > You can find an example how to use it here: > > > > > > > > > > https://github.com/apache/incubator-streampipes-examples/blob/dev/stre > > ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap > > ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro > > ller.java > > > > > > > > Feedback and ideas for improvement are welcome! > > > > > > > > Dominik > > > > > > > > > > > > On 2020/05/17 12:36:05, Grainier Perera > > > > <[email protected]> > > > > wrote: > > > > > Hi Dominik, > > > > > > > > > > Thank you. I'll work on `Implement JS evaluator` [1]. > > > > > > > > > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135 > > > > > > > > > > Grainier. > > > > > > > > > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <[email protected]> > > wrote: > > > > > > > > > > > Hi Grainier, > > > > > > > > > > > > great! I created an umbrella issue for that: > > > > > > https://issues.apache.org/jira/browse/STREAMPIPES-132 > > > > > > > > > > > > I'll now start to implement the new static property for > > > > > > entering > > code. > > > > > > > > > > > > Dominik > > > > > > > > > > > > On 2020/05/15 01:29:44, Grainier Perera > > > > > > <[email protected] > > > > > > > > > wrote: > > > > > > > Hi, > > > > > > > > > > > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you > > > > > > > said, > > > > that'll > > > > > > be > > > > > > > re-usable in the future processors as well. So let's proceed > > > > > > > with > > > > that. > > > > > > > > > > > > > > Regards, > > > > > > > Grainier > > > > > > > > > > > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer > > > > > > > <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > cool, that sounds very good! > > > > > > > > > > > > > > > > Concerning the output, I think there are two ways to > > > > > > > > achieve > > this: > > > > > > > > - A collection static property in conjunction with a > > > > > > > > runtime > > > > resolvable > > > > > > > > output strategy that creates the output schema based on > > > > > > > > the > > input > > > > from > > > > > > the > > > > > > > > collection > > > > > > > > - Or we create a new output strategy, e.g., let's call it > > > > > > > > UserDefinedOutputStrategy, which would render a UI > > > > > > > > component > > to let > > > > > > users > > > > > > > > add/remove/change event properties from the input schema. > > > > > > > > I > > also > > > > like > > > > > > the > > > > > > > > idea of preselecting all input properties. > > > > > > > > > > > > > > > > I personally tend towards the second option as such a > > > > > > > > strategy > > > > might be > > > > > > > > useful for other pipeline elements as well and it would > > > > > > > > ease > > the > > > > model > > > > > > > > definition. We could also extend this component in the > > > > > > > > future, > > > > e.g., by > > > > > > > > letting users enrich metadata such as measurement units > > directly > > > > in a > > > > > > > > processor. > > > > > > > > > > > > > > > > But I'm also open to the first option and we can also just > > start > > > > > > > > implementing to see what works best. > > > > > > > > > > > > > > > > I'd also be happy to help with the implementation, e.g., I > > could > > > > work > > > > > > on > > > > > > > > the new static property for code input or the output > strategy! > > > > > > > > > > > > > > > > Dominik > > > > > > > > > > > > > > > > On 2020/05/14 11:59:36, Grainier Perera < > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > +1 to the idea of having a collection property > > > > > > > > > +containing the > > > > fields > > > > > > of > > > > > > > > the > > > > > > > > > output properties. Also, If we can show the user with > > available > > > > > > fields > > > > > > > > with > > > > > > > > > metadata (of input event) populated in that same > > > > > > > > > collection > > > > property, > > > > > > > > then > > > > > > > > > the user can add/remove/updates fields easily. Let's say > > > > > > > > > they > > > > just > > > > > > have > > > > > > > > to > > > > > > > > > return a map containing all the fields they defined in > > > > > > > > > output > > > > > > properties > > > > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": > > > > > > > > > 301, > > > > > > > > > "newTempInCelsius": 28 } ). > > > > > > > > > > > > > > > > > > [image: sample.png] > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Grainier. > > > > > > > > > > > > > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > sounds reasonable. > > > > > > > > > > > > > > > > > > > > I share the thoughts of Philipp regarding the output > > schema. > > > > But > > > > > > as you > > > > > > > > > > said Grainier, we can also improve :) > > > > > > > > > > > > > > > > > > > > btw: I really like the idea to also allow the more > > technical > > > > user > > > > > > to > > > > > > > > > > easily integrate their function snippets without > > > > > > > > > > having to care about the actual execution. This gets a > > > > > > > > > > more FaaS > > like > > > > > > feeling > > > > > > > > for > > > > > > > > > > streaming. > > > > > > > > > > > > > > > > > > > > Patrick > > > > > > > > > > > > > > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder < > > > > > > [email protected]>: > > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > it would be very useful to have such a processor. > > > > > > > > > > > > > > > > > > > > > > I like the ideas for 1 & 2 and I think we can do it > > > > > > > > > > > this > > way. > > > > > > > > > > > > > > > > > > > > > > Actually I think that the definition of the output > > > > > > > > > > > scheme > > > > could > > > > > > be > > > > > > > > quite > > > > > > > > > > tricky. > > > > > > > > > > > Because users can do anything with the event. > > > > > > > > > > > > > > > > > > > > > > How about we provide an option property? > > > > > > > > > > > Users can select whether they want to define the > > > > > > > > > > > whole > > event > > > > or > > > > > > just > > > > > > > > > > append properties. > > > > > > > > > > > > > > > > > > > > > > For the configurations of the properties we could do > > > > something > > > > > > > > similar > > > > > > > > > > then we have in the PLC4xS7Adapter. > > > > > > > > > > > A collection property containing the fields of the > > > > > > > > > > > output > > > > > > properties. > > > > > > > > > > > Therefore, we have to decide what information we > > > > > > > > > > > have to > > > > > > provide. I > > > > > > > > > > think if we would support all options it becomes very > > cluttered > > > > > > for the > > > > > > > > > > user. > > > > > > > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > > > > > > > Philipp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera < > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > >> Hi Dominik, > > > > > > > > > > >> > > > > > > > > > > >> I agree. We have to do some enhancements make this > > processor > > > > > > user > > > > > > > > > > friendly. > > > > > > > > > > >> Like you said it's nice to have; > > > > > > > > > > >> 1. A new static property that supports code > > highlighting & > > > > > > syntax > > > > > > > > > > >> checking. We can simply do such client-side > > > > > > > > > > >> validations > > > > using > > > > > > > > > > highlight.js > > > > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc... > > > > > > > > > > >> 2. A mechanism to show the user with available > > > > > > > > > > >> fields > > > > they can > > > > > > > > use to > > > > > > > > > > >> write code (similar to what you've done with usable > > > > templates in > > > > > > > > > > >> EmailPublisher using HtmlInputParameter). > > > > > > > > > > >> 3. A mechanism to map output schema. > > > > > > > > > > >> > > > > > > > > > > >> I think 1 & 2 go together and for the 3rd > > > > > > > > > > >> requirement > > given > > > > > > that we > > > > > > > > are > > > > > > > > > > >> targetting a more technical user group for this > > processor, > > > > we > > > > > > can > > > > > > > > let > > > > > > > > > > them > > > > > > > > > > >> define the output. Otherwise, it would be quite > > difficult to > > > > > > derive > > > > > > > > > > return > > > > > > > > > > >> type from the JS. Moreover, I'm planning to use > > > > > > > > > > >> Java's > > > > builtin > > > > > > > > > > ScriptEngine > > > > > > > > > > >> to eval. Of course, there'll be limitations. But, > > > > > > > > > > >> we can > > > > always > > > > > > > > improve > > > > > > > > > > >> them :) > > > > > > > > > > >> > > > > > > > > > > >> [1] https://highlightjs.org/ [2] > > > > > > > > > > >> https://jshint.com/ > > > > > > > > > > >> > > > > > > > > > > >> Regards, > > > > > > > > > > >> Grainier. > > > > > > > > > > >> > > > > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer < > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > >> > > > > > > > > > > >>> Hi Grainier, > > > > > > > > > > >>> > > > > > > > > > > >>> in my opinion, this would definitely be a very > > interesting > > > > > > > > addition! We > > > > > > > > > > >>> already briefly discussed such a feature in the > > > > > > > > > > >>> past, > > > > before we > > > > > > > > went > > > > > > > > > > to the > > > > > > > > > > >>> ASF. I’d say that a JS evaluator would be > > > > > > > > > > >>> definitely > > > > useful to > > > > > > a > > > > > > > > > > slightly > > > > > > > > > > >>> more technical target user group, but would > > > > > > > > > > >>> provide > > great > > > > > > > > flexibility > > > > > > > > > > for, > > > > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss > > > > > > > > > > >>> what > > is > > > > > > needed to > > > > > > > > > > >>> implement this! I’d guess that we (maybe) need to > > > > > > > > > > >>> add > > some > > > > > > > > > > enhancements to > > > > > > > > > > >>> the core to make this processor good from a > > > > > > > > > > >>> usability > > > > > > perspective. > > > > > > > > > > >>> > > > > > > > > > > >>> From a conceptual point of view, I guess the JS > > processor > > > > would > > > > > > > > have no > > > > > > > > > > >>> specific input requirements and a single static > > property > > > > that > > > > > > > > allows > > > > > > > > > > users > > > > > > > > > > >>> to enter the code (maybe it would be cool to add a > > > > > > > > > > >>> new > > > > static > > > > > > > > property > > > > > > > > > > here > > > > > > > > > > >>> that supports code highlighting or even syntax > > checking). > > > > The > > > > > > only > > > > > > > > open > > > > > > > > > > >>> issue I see is the output schema – we would > > > > > > > > > > >>> somehow > > need to > > > > > > > > extract the > > > > > > > > > > >>> output from the JS function. This could probably > > partially > > > > be > > > > > > done > > > > > > > > > > using > > > > > > > > > > >>> the CustomTransformOutput (see > > > > > > > > > > >>> > > > > > > > > > > > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategie > > > > s/), > > > > > > > > > > >>> but we would somehow need to derive type > > > > > > > > > > >>> information > > from > > > > the > > > > > > JS > > > > > > > > > > function > > > > > > > > > > >>> or introduce a feature to let users define/refine > > > > > > > > > > >>> the > > > > output > > > > > > > > manually > > > > > > > > > > >>> (e.g., to add semantic metadata). > > > > > > > > > > >>> > > > > > > > > > > >>> What do you think would be best? And would you > > > > > > > > > > >>> evaluate > > > > the JS > > > > > > > > code in > > > > > > > > > > >>> Java or somehow else? > > > > > > > > > > >>> > > > > > > > > > > >>> Also, as I’ve just started to improve the SDK > > > > > > > > > > >>> (e.g., > > > > supporting > > > > > > > > > > optional > > > > > > > > > > >>> static properties and an easier way to define and > > extract > > > > model > > > > > > > > > > >>> parameters), we can collect your requirements for > > > > > > > > > > >>> the > > JS > > > > > > processor > > > > > > > > and > > > > > > > > > > >>> extend the SDK if needed, just say what you would > > > > > > > > > > >>> like > > to > > > > have > > > > > > 😉 > > > > > > > > > > >>> > > > > > > > > > > >>> Dominik > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> From: Grainier Perera <[email protected]> > > > > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM > > > > > > > > > > >>> To: [email protected] > > > > > > > > > > >>> Subject: Data processor to evaluate JavaScript > > > > > > > > > > >>> > > > > > > > > > > >>> Hi all, > > > > > > > > > > >>> > > > > > > > > > > >>> I'm planning to implement a JavaScript eval data > > > > processor. As > > > > > > the > > > > > > > > name > > > > > > > > > > >>> suggests, users will be able to write some > > > > > > > > > > >>> JavaScript > > code > > > > > > which > > > > > > > > takes > > > > > > > > > > in > > > > > > > > > > >>> an event (as a map), do some processing on the > > > > > > > > > > >>> event, > > and > > > > > > return a > > > > > > > > new > > > > > > > > > > map > > > > > > > > > > >>> which then gets converted to an sp-event. > > > > > > > > > > >>> > > > > > > > > > > >>> example js: > > > > > > > > > > >>> function process(event) { > > > > > > > > > > >>> // do processing here. > > > > > > > > > > >>> // return processed event. > > > > > > > > > > >>> return {id: http://event.id, tempInCelsius: > > > > > > (event.tempInKelvin > > > > > > > > - > > > > > > > > > > >>> 273.15)}; > > > > > > > > > > >>> }; > > > > > > > > > > >>> > > > > > > > > > > >>> Will this be a good addition to pipeline-elements? > > What do > > > > you > > > > > > > > think? > > > > > > > > > > >>> > > > > > > > > > > >>> Regards, > > > > > > > > > > >>> Grainier. > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
