Hi all, I tried to use the Processor JavaScript Eval in a deployment of StreamPipes (Lite), running in Docker. Unfortunately, I get an error in the UI when starting the pipeline. There is no specific error message, just an error that the JavaScript Eval did not start. I also did not see anything in the logs. Did anyone else have a similar issue?
The processor runs in the container pipeline-elements-all-jvm using the CLI. Running the same pipeline locally in my IDE works well, so I guess there is a problem when building the project. Is it possible that some libraries are not bundled correctly? Philipp > On 22. May 2020, at 16:57, Grainier Perera <[email protected]> wrote: > > Hi Dominik, > > My bad. There were some cached images & that was the issue. Also fixed the > PR as per your comments. Please review and merge [1]. > > [1] https://github.com/apache/incubator-streampipes-extensions/pull/15 > > Thanks & Regards, > Grainier > > On Fri, 22 May 2020 at 16:25, Dominik Riemer <[email protected]> wrote: > >> Hi Grainier, >> >> we have a Github Actions workflow [1] for core+extensions that usually >> builds and pushes the latest development containers for dev on each push. >> You are using the latest CLI from dev and the tmpl_env or .env version is >> set to 0.67.0-SNAPSHOT, right? >> >> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local >> backend with the latest changes from dev and your PR) and it works great 😊 >> The only thing I had to change was a typo in line 59 (use outEvent instead >> of event). >> >> In the controller, the supportedProtocols and supportedFormats are no >> longer needed, as they can now be declared for a whole module in the Init >> class (registerFormats/Protocols). I need to update this in the >> documentation. >> >> Dominik >> >> >> >> >> >> https://github.com/apache/incubator-streampipes/blob/dev/.github/workflows/build.yml >> >> -----Original Message----- >> From: Grainier Perera <[email protected]> >> Sent: Friday, May 22, 2020 11:58 AM >> To: [email protected] >> Subject: Re: Data processor to evaluate JavaScript >> >> Hi Dominik, >> >> I've implemented the following[1]. However, when I try to install the >> processor it gives me the following error. Any clue? I've used CLI to start >> sp. So, maybe the snapshot docker image doesn't have the recent changes? If >> so what are the steps to build the images with changes in-place? >> >> [1] https://github.com/apache/incubator-streampipes-extensions/pull/15 >> >> ERROR: https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy >> io.fogsy.empire.pinto.RDFMappingException: Could not create an instance of >> class org.apache.streampipes.model.output.OutputStrategy, it does not have >> a default constructor at >> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178) >> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261) >> at io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695) >> at io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:388) >> at >> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Unknown >> Source) >> at >> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Unknown >> Source) >> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at >> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source) >> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at >> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) at >> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) at >> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown >> Source) >> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at >> java.util.stream.ReferencePipeline.forEach(Unknown Source) at >> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299) >> at >> >> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonLd(JsonLdTransformer.java:100) >> at >> >> org.apache.streampipes.manager.verification.ElementVerifier.transform(ElementVerifier.java:169) >> at >> >> org.apache.streampipes.manager.verification.ElementVerifier.verifyAndAdd(ElementVerifier.java:84) >> at >> >> org.apache.streampipes.manager.operations.Operations.verifyAndAddElement(Operations.java:96) >> at >> >> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAddEndpointItem(EndpointItemParser.java:37) >> at >> >> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddElement(PipelineElementImport.java:93) >> at >> >> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(PipelineElementImport.java:89) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) >> >> Thanks, >> Grainier. >> >> On Thu, 21 May 2020 at 03:39, Dominik Riemer <[email protected]> wrote: >> >>> Hi Grainier, >>> >>> sounds good! >>> I've just pushed a commit that provides an initial version of the >>> user-defined output strategy. It lets you select the input schema from >>> the incoming stream and add/remove event properties. There is one >>> limitation, the component currently only works for flat event >>> structures, but it shouldn't be a problem to have a more advanced >>> schema editor for this output strategy soon (we already have this in >>> the StreamPipes connect code, but unfortunately, the pipeline editor >>> has yet to be migrated from AngularJS to Angular 2+ to reuse these >>> components). This is something we plan to do within the next weeks. >>> >>> I added an example how to use the strategy to the examples repo: >>> >>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre >>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap >>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro >>> ller.java >>> >>> Let me know what you think and if you need anything more/else! >>> >>> Dominik >>> >>> On 2020/05/19 17:14:21, Grainier Perera <[email protected]> >>> wrote: >>>> Hi Dominik, >>>> >>>> Awesome. This is really cool & exactly what I was looking for. I'll >>>> use this with my implementation, and keep this thread posted with >>>> the >>> updates. >>>> ps: let us know when you add output strategy as well. >>>> >>>> Thanks, >>>> Grainier. >>>> >>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <[email protected]> >> wrote: >>>> >>>>> Hi Grainier, >>>>> >>>>> I just pushed an initial version of the static property to add >>> javascript >>>>> code. >>>>> It's a simple codeblock with linting enabled and a (currently very >>> simple) >>>>> autocomplete feature that suggests existing properties from the >>>>> input >>> event >>>>> (currently, only if you press ctrl-space directly after typing >> "event" >>>>> followed by a dot, I'll fix that soon ;-) >>>>> >>>>> You can find an example how to use it here: >>>>> >>>>> >>> https://github.com/apache/incubator-streampipes-examples/blob/dev/stre >>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/ap >>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleContro >>> ller.java >>>>> >>>>> Feedback and ideas for improvement are welcome! >>>>> >>>>> Dominik >>>>> >>>>> >>>>> On 2020/05/17 12:36:05, Grainier Perera >>>>> <[email protected]> >>>>> wrote: >>>>>> Hi Dominik, >>>>>> >>>>>> Thank you. I'll work on `Implement JS evaluator` [1]. >>>>>> >>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135 >>>>>> >>>>>> Grainier. >>>>>> >>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer <[email protected]> >>> wrote: >>>>>> >>>>>>> Hi Grainier, >>>>>>> >>>>>>> great! I created an umbrella issue for that: >>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132 >>>>>>> >>>>>>> I'll now start to implement the new static property for >>>>>>> entering >>> code. >>>>>>> >>>>>>> Dominik >>>>>>> >>>>>>> On 2020/05/15 01:29:44, Grainier Perera >>>>>>> <[email protected] >>>> >>>>>>> wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you >>>>>>>> said, >>>>> that'll >>>>>>> be >>>>>>>> re-usable in the future processors as well. So let's proceed >>>>>>>> with >>>>> that. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Grainier >>>>>>>> >>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer >>>>>>>> <[email protected]> >>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> cool, that sounds very good! >>>>>>>>> >>>>>>>>> Concerning the output, I think there are two ways to >>>>>>>>> achieve >>> this: >>>>>>>>> - A collection static property in conjunction with a >>>>>>>>> runtime >>>>> resolvable >>>>>>>>> output strategy that creates the output schema based on >>>>>>>>> the >>> input >>>>> from >>>>>>> the >>>>>>>>> collection >>>>>>>>> - Or we create a new output strategy, e.g., let's call it >>>>>>>>> UserDefinedOutputStrategy, which would render a UI >>>>>>>>> component >>> to let >>>>>>> users >>>>>>>>> add/remove/change event properties from the input schema. >>>>>>>>> I >>> also >>>>> like >>>>>>> the >>>>>>>>> idea of preselecting all input properties. >>>>>>>>> >>>>>>>>> I personally tend towards the second option as such a >>>>>>>>> strategy >>>>> might be >>>>>>>>> useful for other pipeline elements as well and it would >>>>>>>>> ease >>> the >>>>> model >>>>>>>>> definition. We could also extend this component in the >>>>>>>>> future, >>>>> e.g., by >>>>>>>>> letting users enrich metadata such as measurement units >>> directly >>>>> in a >>>>>>>>> processor. >>>>>>>>> >>>>>>>>> But I'm also open to the first option and we can also just >>> start >>>>>>>>> implementing to see what works best. >>>>>>>>> >>>>>>>>> I'd also be happy to help with the implementation, e.g., I >>> could >>>>> work >>>>>>> on >>>>>>>>> the new static property for code input or the output >> strategy! >>>>>>>>> >>>>>>>>> Dominik >>>>>>>>> >>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera < >>> [email protected] >>>>>> >>>>>>>>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> +1 to the idea of having a collection property >>>>>>>>>> +containing the >>>>> fields >>>>>>> of >>>>>>>>> the >>>>>>>>>> output properties. Also, If we can show the user with >>> available >>>>>>> fields >>>>>>>>> with >>>>>>>>>> metadata (of input event) populated in that same >>>>>>>>>> collection >>>>> property, >>>>>>>>> then >>>>>>>>>> the user can add/remove/updates fields easily. Let's say >>>>>>>>>> they >>>>> just >>>>>>> have >>>>>>>>> to >>>>>>>>>> return a map containing all the fields they defined in >>>>>>>>>> output >>>>>>> properties >>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin": >>>>>>>>>> 301, >>>>>>>>>> "newTempInCelsius": 28 } ). >>>>>>>>>> >>>>>>>>>> [image: sample.png] >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Grainier. >>>>>>>>>> >>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener < >>> [email protected]> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> sounds reasonable. >>>>>>>>>>> >>>>>>>>>>> I share the thoughts of Philipp regarding the output >>> schema. >>>>> But >>>>>>> as you >>>>>>>>>>> said Grainier, we can also improve :) >>>>>>>>>>> >>>>>>>>>>> btw: I really like the idea to also allow the more >>> technical >>>>> user >>>>>>> to >>>>>>>>>>> easily integrate their function snippets without >>>>>>>>>>> having to care about the actual execution. This gets a >>>>>>>>>>> more FaaS >>> like >>>>>>> feeling >>>>>>>>> for >>>>>>>>>>> streaming. >>>>>>>>>>> >>>>>>>>>>> Patrick >>>>>>>>>>> >>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder < >>>>>>> [email protected]>: >>>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> it would be very useful to have such a processor. >>>>>>>>>>>> >>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it >>>>>>>>>>>> this >>> way. >>>>>>>>>>>> >>>>>>>>>>>> Actually I think that the definition of the output >>>>>>>>>>>> scheme >>>>> could >>>>>>> be >>>>>>>>> quite >>>>>>>>>>> tricky. >>>>>>>>>>>> Because users can do anything with the event. >>>>>>>>>>>> >>>>>>>>>>>> How about we provide an option property? >>>>>>>>>>>> Users can select whether they want to define the >>>>>>>>>>>> whole >>> event >>>>> or >>>>>>> just >>>>>>>>>>> append properties. >>>>>>>>>>>> >>>>>>>>>>>> For the configurations of the properties we could do >>>>> something >>>>>>>>> similar >>>>>>>>>>> then we have in the PLC4xS7Adapter. >>>>>>>>>>>> A collection property containing the fields of the >>>>>>>>>>>> output >>>>>>> properties. >>>>>>>>>>>> Therefore, we have to decide what information we >>>>>>>>>>>> have to >>>>>>> provide. I >>>>>>>>>>> think if we would support all options it becomes very >>> cluttered >>>>>>> for the >>>>>>>>>>> user. >>>>>>>>>>>> >>>>>>>>>>>> What do you think? >>>>>>>>>>>> >>>>>>>>>>>> Philipp >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera < >>>>>>>>> [email protected]> >>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi Dominik, >>>>>>>>>>>>> >>>>>>>>>>>>> I agree. We have to do some enhancements make this >>> processor >>>>>>> user >>>>>>>>>>> friendly. >>>>>>>>>>>>> Like you said it's nice to have; >>>>>>>>>>>>> 1. A new static property that supports code >>> highlighting & >>>>>>> syntax >>>>>>>>>>>>> checking. We can simply do such client-side >>>>>>>>>>>>> validations >>>>> using >>>>>>>>>>> highlight.js >>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc... >>>>>>>>>>>>> 2. A mechanism to show the user with available >>>>>>>>>>>>> fields >>>>> they can >>>>>>>>> use to >>>>>>>>>>>>> write code (similar to what you've done with usable >>>>> templates in >>>>>>>>>>>>> EmailPublisher using HtmlInputParameter). >>>>>>>>>>>>> 3. A mechanism to map output schema. >>>>>>>>>>>>> >>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd >>>>>>>>>>>>> requirement >>> given >>>>>>> that we >>>>>>>>> are >>>>>>>>>>>>> targetting a more technical user group for this >>> processor, >>>>> we >>>>>>> can >>>>>>>>> let >>>>>>>>>>> them >>>>>>>>>>>>> define the output. Otherwise, it would be quite >>> difficult to >>>>>>> derive >>>>>>>>>>> return >>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use >>>>>>>>>>>>> Java's >>>>> builtin >>>>>>>>>>> ScriptEngine >>>>>>>>>>>>> to eval. Of course, there'll be limitations. But, >>>>>>>>>>>>> we can >>>>> always >>>>>>>>> improve >>>>>>>>>>>>> them :) >>>>>>>>>>>>> >>>>>>>>>>>>> [1] https://highlightjs.org/ [2] >>>>>>>>>>>>> https://jshint.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Grainier. >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer < >>>>> [email protected] >>>>>>>> >>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Grainier, >>>>>>>>>>>>>> >>>>>>>>>>>>>> in my opinion, this would definitely be a very >>> interesting >>>>>>>>> addition! We >>>>>>>>>>>>>> already briefly discussed such a feature in the >>>>>>>>>>>>>> past, >>>>> before we >>>>>>>>> went >>>>>>>>>>> to the >>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be >>>>>>>>>>>>>> definitely >>>>> useful to >>>>>>> a >>>>>>>>>>> slightly >>>>>>>>>>>>>> more technical target user group, but would >>>>>>>>>>>>>> provide >>> great >>>>>>>>> flexibility >>>>>>>>>>> for, >>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss >>>>>>>>>>>>>> what >>> is >>>>>>> needed to >>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to >>>>>>>>>>>>>> add >>> some >>>>>>>>>>> enhancements to >>>>>>>>>>>>>> the core to make this processor good from a >>>>>>>>>>>>>> usability >>>>>>> perspective. >>>>>>>>>>>>>> >>>>>>>>>>>>>> From a conceptual point of view, I guess the JS >>> processor >>>>> would >>>>>>>>> have no >>>>>>>>>>>>>> specific input requirements and a single static >>> property >>>>> that >>>>>>>>> allows >>>>>>>>>>> users >>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a >>>>>>>>>>>>>> new >>>>> static >>>>>>>>> property >>>>>>>>>>> here >>>>>>>>>>>>>> that supports code highlighting or even syntax >>> checking). >>>>> The >>>>>>> only >>>>>>>>> open >>>>>>>>>>>>>> issue I see is the output schema – we would >>>>>>>>>>>>>> somehow >>> need to >>>>>>>>> extract the >>>>>>>>>>>>>> output from the JS function. This could probably >>> partially >>>>> be >>>>>>> done >>>>>>>>>>> using >>>>>>>>>>>>>> the CustomTransformOutput (see >>>>>>>>>>>>>> >>>>>>>>> >>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie >>>>> s/), >>>>>>>>>>>>>> but we would somehow need to derive type >>>>>>>>>>>>>> information >>> from >>>>> the >>>>>>> JS >>>>>>>>>>> function >>>>>>>>>>>>>> or introduce a feature to let users define/refine >>>>>>>>>>>>>> the >>>>> output >>>>>>>>> manually >>>>>>>>>>>>>> (e.g., to add semantic metadata). >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do you think would be best? And would you >>>>>>>>>>>>>> evaluate >>>>> the JS >>>>>>>>> code in >>>>>>>>>>>>>> Java or somehow else? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK >>>>>>>>>>>>>> (e.g., >>>>> supporting >>>>>>>>>>> optional >>>>>>>>>>>>>> static properties and an easier way to define and >>> extract >>>>> model >>>>>>>>>>>>>> parameters), we can collect your requirements for >>>>>>>>>>>>>> the >>> JS >>>>>>> processor >>>>>>>>> and >>>>>>>>>>>>>> extend the SDK if needed, just say what you would >>>>>>>>>>>>>> like >>> to >>>>> have >>>>>>> 😉 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dominik >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> From: Grainier Perera <[email protected]> >>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM >>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data >>>>> processor. As >>>>>>> the >>>>>>>>> name >>>>>>>>>>>>>> suggests, users will be able to write some >>>>>>>>>>>>>> JavaScript >>> code >>>>>>> which >>>>>>>>> takes >>>>>>>>>>> in >>>>>>>>>>>>>> an event (as a map), do some processing on the >>>>>>>>>>>>>> event, >>> and >>>>>>> return a >>>>>>>>> new >>>>>>>>>>> map >>>>>>>>>>>>>> which then gets converted to an sp-event. >>>>>>>>>>>>>> >>>>>>>>>>>>>> example js: >>>>>>>>>>>>>> function process(event) { >>>>>>>>>>>>>> // do processing here. >>>>>>>>>>>>>> // return processed event. >>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius: >>>>>>> (event.tempInKelvin >>>>>>>>> - >>>>>>>>>>>>>> 273.15)}; >>>>>>>>>>>>>> }; >>>>>>>>>>>>>> >>>>>>>>>>>>>> Will this be a good addition to pipeline-elements? >>> What do >>>>> you >>>>>>>>> think? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Grainier. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >>
