Cool. As for the base image discussion: I guess this is worth discussing in a different thread. So feel free to open.
Patrick > Am 17.06.2020 um 11:21 schrieb Philipp Zehnder <[email protected]>: > > Hi Dominik, > > unfortunately this was not the problem, but this could become a problem in > the future ;) > The problem was the java version. So far we used Java 8 in the docker images, > after updating it to Java 9 it worked. > > So far I just changed the docker versions of the images using the JS > processor and it should work now. > I saw that we use java 11 for the arm images. Which version should we use for > x86? I think it makes sense to harmonize the base image for all containers. > So the question is, which base image should we use? Any suggestions? > > Philipp > >> On 15. Jun 2020, at 23:11, Dominik Riemer <[email protected]> wrote: >> >> Hi Philipp, >> >> coming back to this (and with later I really didn't mean ten days, sorry for >> that) ;-) >> I could also reproduce this now - it seems to be only an issue when running >> in Docker. >> >> When looking at the container logs when running the element in Docker, I get >> a NullPointerException in line 45 of JsEval (which could indicate that >> either engine is null or code is null). >> >> So could this be an issue with the Java version we're using in the Docker >> images? The currently used version is AdoptOpenJDK 11. >> >> I found a similar issue on so: >> https://stackoverflow.com/questions/60916424/getenginebynamejavascript-returning-null-on-java-11 >> >> Could this be the cause? >> >> Dominik >> >> On 2020/06/05 09:02:10, "Dominik Riemer" <[email protected]> wrote: >>> Hi Philipp, >>> >>> I can try to reproduce it later. >>> >>> Dominik >>> >>> -----Original Message----- >>> From: Philipp Zehnder <[email protected]> >>> Sent: Friday, June 5, 2020 11:01 AM >>> To: [email protected] >>> Subject: Re: Data processor to evaluate JavaScript >>> >>> Hi, >>> >>> I just tried it again and removed all docker images from the VM did a fresh >>> pull. >>> Unfortunately, with the same result. I tried the processor in both >>> containers (pipeline-elements-all-jvm / processors-enricher-jvm). >>> >>> Does anyone else have an idea what the problem could be? >>> >>> Philipp >>> >>>> On 4. Jun 2020, at 12:00, Grainier Perera <[email protected]> wrote: >>>> >>>> Hmm, That's odd. It doesn't use any 3rd party libraries for the PE (we >>>> use javax.script.*). However, I came across a similar situation with >>>> this RDFMappingException[1]. The reason for that is JSEval use newly >>>> introduced .requiredCodeblock() controller prop, and for some reason, >>>> there's a cached Docker image without it. But when I cleaned that and >>>> re-build, it worked. Could this be the reason? >>>> >>>> [1] io.fogsy.empire.pinto.RDFMappingException: Could not create an >>>> instance of class org.apache.streampipes.model.output.OutputStrategy, >>>> it does not have a default constructor >>>> >>>> Grainier Perera. >>>> >>>> >>>> On Thu, 4 Jun 2020 at 15:17, Philipp Zehnder <[email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I tried to use the Processor JavaScript Eval in a deployment of >>>>> StreamPipes (Lite), running in Docker. >>>>> Unfortunately, I get an error in the UI when starting the pipeline. >>>>> There is no specific error message, just an error that the JavaScript >>>>> Eval did not start. >>>>> I also did not see anything in the logs. Did anyone else have a >>>>> similar issue? >>>>> >>>>> The processor runs in the container pipeline-elements-all-jvm using >>>>> the CLI. >>>>> >>>>> Running the same pipeline locally in my IDE works well, so I guess >>>>> there is a problem when building the project. >>>>> Is it possible that some libraries are not bundled correctly? >>>>> >>>>> Philipp >>>>> >>>>>> On 22. May 2020, at 16:57, Grainier Perera >>>>>> <[email protected]> >>>>> wrote: >>>>>> >>>>>> Hi Dominik, >>>>>> >>>>>> My bad. There were some cached images & that was the issue. Also >>>>>> fixed >>>>> the >>>>>> PR as per your comments. Please review and merge [1]. >>>>>> >>>>>> [1] >>>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15 >>>>>> >>>>>> Thanks & Regards, >>>>>> Grainier >>>>>> >>>>>> On Fri, 22 May 2020 at 16:25, Dominik Riemer <[email protected]> wrote: >>>>>> >>>>>>> Hi Grainier, >>>>>>> >>>>>>> we have a Github Actions workflow [1] for core+extensions that >>>>>>> usually builds and pushes the latest development containers for dev >>>>>>> on each >>>>> push. >>>>>>> You are using the latest CLI from dev and the tmpl_env or .env >>>>>>> version >>>>> is >>>>>>> set to 0.67.0-SNAPSHOT, right? >>>>>>> >>>>>>> I just tested the JsEval processor (with CLI 0.67.0-SNAPSHOT, local >>>>>>> backend with the latest changes from dev and your PR) and it works >>>>> great 😊 >>>>>>> The only thing I had to change was a typo in line 59 (use outEvent >>>>> instead >>>>>>> of event). >>>>>>> >>>>>>> In the controller, the supportedProtocols and supportedFormats are >>>>>>> no longer needed, as they can now be declared for a whole module in >>>>>>> the >>>>> Init >>>>>>> class (registerFormats/Protocols). I need to update this in the >>>>>>> documentation. >>>>>>> >>>>>>> Dominik >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>> https://github.com/apache/incubator-streampipes/blob/dev/.github/work >>>>> flows/build.yml >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Grainier Perera <[email protected]> >>>>>>> Sent: Friday, May 22, 2020 11:58 AM >>>>>>> To: [email protected] >>>>>>> Subject: Re: Data processor to evaluate JavaScript >>>>>>> >>>>>>> Hi Dominik, >>>>>>> >>>>>>> I've implemented the following[1]. However, when I try to install >>>>>>> the processor it gives me the following error. Any clue? I've used >>>>>>> CLI to >>>>> start >>>>>>> sp. So, maybe the snapshot docker image doesn't have the recent >>>>> changes? If >>>>>>> so what are the steps to build the images with changes in-place? >>>>>>> >>>>>>> [1] >>>>>>> https://github.com/apache/incubator-streampipes-extensions/pull/15 >>>>>>> >>>>>>> ERROR: >>>>>>> https://streampipes.org/vocabulary/v1/UserDefinedOutputStrategy >>>>>>> io.fogsy.empire.pinto.RDFMappingException: Could not create an >>>>>>> instance >>>>> of >>>>>>> class org.apache.streampipes.model.output.OutputStrategy, it does >>>>>>> not >>>>> have >>>>>>> a default constructor at >>>>>>> io.fogsy.empire.pinto.RDFMapper.newInstance(RDFMapper.java:178) >>>>>>> at io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:261) >>>>>>> at >>>>>>> io.fogsy.empire.pinto.RDFMapper.valueToObject(RDFMapper.java:695) >>>>>>> at >>>>>>> io.fogsy.empire.pinto.RDFMapper.lambda$toObject$3(RDFMapper.java:38 >>>>>>> 8) >>>>>>> at >>>>>>> >>>>> io.fogsy.empire.pinto.RDFMapper$$Lambda$802/0000000000096040.apply(Un >>>>> known >>>>>>> Source) >>>>>>> at >>>>>>> >>>>> io.fogsy.empire.pinto.RDFMapper$$Lambda$803/0000000000096580.apply(Un >>>>> known >>>>>>> Source) >>>>>>> at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) at >>>>>>> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown >>>>> Source) >>>>>>> at java.util.stream.AbstractPipeline.copyInto(Unknown Source) at >>>>>>> java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) >>>>>>> at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown >>>>> Source) at >>>>>>> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unkn >>>>>>> own >>>>>>> Source) >>>>>>> at java.util.stream.AbstractPipeline.evaluate(Unknown Source) at >>>>>>> java.util.stream.ReferencePipeline.forEach(Unknown Source) at >>>>>>> io.fogsy.empire.pinto.RDFMapper.readValue(RDFMapper.java:299) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.serializers.jsonld.JsonLdTransformer.fromJsonL >>>>> d(JsonLdTransformer.java:100) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.manager.verification.ElementVerifier.transform >>>>> (ElementVerifier.java:169) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.manager.verification.ElementVerifier.verifyAnd >>>>> Add(ElementVerifier.java:84) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.manager.operations.Operations.verifyAndAddElem >>>>> ent(Operations.java:96) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.manager.endpoint.EndpointItemParser.parseAndAd >>>>> dEndpointItem(EndpointItemParser.java:37) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.rest.impl.PipelineElementImport.verifyAndAddEl >>>>> ement(PipelineElementImport.java:93) >>>>>>> at >>>>>>> >>>>>>> >>>>> org.apache.streampipes.rest.impl.PipelineElementImport.addElement(Pip >>>>> elineElementImport.java:89) >>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) >>>>>>> >>>>>>> Thanks, >>>>>>> Grainier. >>>>>>> >>>>>>> On Thu, 21 May 2020 at 03:39, Dominik Riemer <[email protected]> wrote: >>>>>>> >>>>>>>> Hi Grainier, >>>>>>>> >>>>>>>> sounds good! >>>>>>>> I've just pushed a commit that provides an initial version of the >>>>>>>> user-defined output strategy. It lets you select the input schema >>>>>>>> from the incoming stream and add/remove event properties. There is >>>>>>>> one limitation, the component currently only works for flat event >>>>>>>> structures, but it shouldn't be a problem to have a more advanced >>>>>>>> schema editor for this output strategy soon (we already have this >>>>>>>> in the StreamPipes connect code, but unfortunately, the pipeline >>>>>>>> editor has yet to be migrated from AngularJS to Angular 2+ to >>>>>>>> reuse these components). This is something we plan to do within the >>>>>>>> next weeks. >>>>>>>> >>>>>>>> I added an example how to use the strategy to the examples repo: >>>>>>>> >>>>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/ >>>>>>>> stre >>>>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or >>>>>>>> g/ap >>>>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo >>>>>>>> ntro >>>>>>>> ller.java >>>>>>>> >>>>>>>> Let me know what you think and if you need anything more/else! >>>>>>>> >>>>>>>> Dominik >>>>>>>> >>>>>>>> On 2020/05/19 17:14:21, Grainier Perera >>>>>>>> <[email protected]> >>>>>>>> wrote: >>>>>>>>> Hi Dominik, >>>>>>>>> >>>>>>>>> Awesome. This is really cool & exactly what I was looking for. >>>>>>>>> I'll use this with my implementation, and keep this thread posted >>>>>>>>> with the >>>>>>>> updates. >>>>>>>>> ps: let us know when you add output strategy as well. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Grainier. >>>>>>>>> >>>>>>>>> On Tue, 19 May 2020 at 03:05, Dominik Riemer <[email protected]> >>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Grainier, >>>>>>>>>> >>>>>>>>>> I just pushed an initial version of the static property to add >>>>>>>> javascript >>>>>>>>>> code. >>>>>>>>>> It's a simple codeblock with linting enabled and a (currently >>>>>>>>>> very >>>>>>>> simple) >>>>>>>>>> autocomplete feature that suggests existing properties from the >>>>>>>>>> input >>>>>>>> event >>>>>>>>>> (currently, only if you press ctrl-space directly after typing >>>>>>> "event" >>>>>>>>>> followed by a dot, I'll fix that soon ;-) >>>>>>>>>> >>>>>>>>>> You can find an example how to use it here: >>>>>>>>>> >>>>>>>>>> >>>>>>>> https://github.com/apache/incubator-streampipes-examples/blob/dev/ >>>>>>>> stre >>>>>>>> ampipes-pipeline-elements-examples-processors-jvm/src/main/java/or >>>>>>>> g/ap >>>>>>>> ache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleCo >>>>>>>> ntro >>>>>>>> ller.java >>>>>>>>>> >>>>>>>>>> Feedback and ideas for improvement are welcome! >>>>>>>>>> >>>>>>>>>> Dominik >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 2020/05/17 12:36:05, Grainier Perera >>>>>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> Hi Dominik, >>>>>>>>>>> >>>>>>>>>>> Thank you. I'll work on `Implement JS evaluator` [1]. >>>>>>>>>>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/STREAMPIPES-135 >>>>>>>>>>> >>>>>>>>>>> Grainier. >>>>>>>>>>> >>>>>>>>>>> On Sun, 17 May 2020 at 18:02, Dominik Riemer >>>>>>>>>>> <[email protected]> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Grainier, >>>>>>>>>>>> >>>>>>>>>>>> great! I created an umbrella issue for that: >>>>>>>>>>>> https://issues.apache.org/jira/browse/STREAMPIPES-132 >>>>>>>>>>>> >>>>>>>>>>>> I'll now start to implement the new static property for >>>>>>>>>>>> entering >>>>>>>> code. >>>>>>>>>>>> >>>>>>>>>>>> Dominik >>>>>>>>>>>> >>>>>>>>>>>> On 2020/05/15 01:29:44, Grainier Perera >>>>>>>>>>>> <[email protected] >>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> I'm +1 to the idea of `UserDefinedOutputStrategy`. As you >>>>>>>>>>>>> said, >>>>>>>>>> that'll >>>>>>>>>>>> be >>>>>>>>>>>>> re-usable in the future processors as well. So let's proceed >>>>>>>>>>>>> with >>>>>>>>>> that. >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Grainier >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, 15 May 2020 at 01:37, Dominik Riemer >>>>>>>>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> cool, that sounds very good! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Concerning the output, I think there are two ways to achieve >>>>>>>> this: >>>>>>>>>>>>>> - A collection static property in conjunction with a runtime >>>>>>>>>> resolvable >>>>>>>>>>>>>> output strategy that creates the output schema based on the >>>>>>>> input >>>>>>>>>> from >>>>>>>>>>>> the >>>>>>>>>>>>>> collection >>>>>>>>>>>>>> - Or we create a new output strategy, e.g., let's call it >>>>>>>>>>>>>> UserDefinedOutputStrategy, which would render a UI component >>>>>>>> to let >>>>>>>>>>>> users >>>>>>>>>>>>>> add/remove/change event properties from the input schema. >>>>>>>>>>>>>> I >>>>>>>> also >>>>>>>>>> like >>>>>>>>>>>> the >>>>>>>>>>>>>> idea of preselecting all input properties. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I personally tend towards the second option as such a >>>>>>>>>>>>>> strategy >>>>>>>>>> might be >>>>>>>>>>>>>> useful for other pipeline elements as well and it would ease >>>>>>>> the >>>>>>>>>> model >>>>>>>>>>>>>> definition. We could also extend this component in the >>>>>>>>>>>>>> future, >>>>>>>>>> e.g., by >>>>>>>>>>>>>> letting users enrich metadata such as measurement units >>>>>>>> directly >>>>>>>>>> in a >>>>>>>>>>>>>> processor. >>>>>>>>>>>>>> >>>>>>>>>>>>>> But I'm also open to the first option and we can also just >>>>>>>> start >>>>>>>>>>>>>> implementing to see what works best. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'd also be happy to help with the implementation, e.g., I >>>>>>>> could >>>>>>>>>> work >>>>>>>>>>>> on >>>>>>>>>>>>>> the new static property for code input or the output >>>>>>> strategy! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dominik >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 2020/05/14 11:59:36, Grainier Perera < >>>>>>>> [email protected] >>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 to the idea of having a collection property containing >>>>>>>>>>>>>>> +the >>>>>>>>>> fields >>>>>>>>>>>> of >>>>>>>>>>>>>> the >>>>>>>>>>>>>>> output properties. Also, If we can show the user with >>>>>>>> available >>>>>>>>>>>> fields >>>>>>>>>>>>>> with >>>>>>>>>>>>>>> metadata (of input event) populated in that same collection >>>>>>>>>> property, >>>>>>>>>>>>>> then >>>>>>>>>>>>>>> the user can add/remove/updates fields easily. Let's say >>>>>>>>>>>>>>> they >>>>>>>>>> just >>>>>>>>>>>> have >>>>>>>>>>>>>> to >>>>>>>>>>>>>>> return a map containing all the fields they defined in >>>>>>>>>>>>>>> output >>>>>>>>>>>> properties >>>>>>>>>>>>>>> (i.e return {"existingId": 1, "existingTempInKelvin": >>>>>>>>>>>>>>> 301, >>>>>>>>>>>>>>> "newTempInCelsius": 28 } ). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [image: sample.png] >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Grainier. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, 14 May 2020 at 12:33, Patrick Wiener < >>>>>>>> [email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> sounds reasonable. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I share the thoughts of Philipp regarding the output >>>>>>>> schema. >>>>>>>>>> But >>>>>>>>>>>> as you >>>>>>>>>>>>>>>> said Grainier, we can also improve :) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> btw: I really like the idea to also allow the more >>>>>>>> technical >>>>>>>>>> user >>>>>>>>>>>> to >>>>>>>>>>>>>>>> easily integrate their function snippets without having to >>>>>>>>>>>>>>>> care about the actual execution. This gets a more FaaS >>>>>>>> like >>>>>>>>>>>> feeling >>>>>>>>>>>>>> for >>>>>>>>>>>>>>>> streaming. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Patrick >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Am 14.05.2020 um 08:26 schrieb Philipp Zehnder < >>>>>>>>>>>> [email protected]>: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> it would be very useful to have such a processor. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I like the ideas for 1 & 2 and I think we can do it this >>>>>>>> way. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Actually I think that the definition of the output scheme >>>>>>>>>> could >>>>>>>>>>>> be >>>>>>>>>>>>>> quite >>>>>>>>>>>>>>>> tricky. >>>>>>>>>>>>>>>>> Because users can do anything with the event. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> How about we provide an option property? >>>>>>>>>>>>>>>>> Users can select whether they want to define the whole >>>>>>>> event >>>>>>>>>> or >>>>>>>>>>>> just >>>>>>>>>>>>>>>> append properties. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> For the configurations of the properties we could do >>>>>>>>>> something >>>>>>>>>>>>>> similar >>>>>>>>>>>>>>>> then we have in the PLC4xS7Adapter. >>>>>>>>>>>>>>>>> A collection property containing the fields of the output >>>>>>>>>>>> properties. >>>>>>>>>>>>>>>>> Therefore, we have to decide what information we have to >>>>>>>>>>>> provide. I >>>>>>>>>>>>>>>> think if we would support all options it becomes very >>>>>>>> cluttered >>>>>>>>>>>> for the >>>>>>>>>>>>>>>> user. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What do you think? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Philipp >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 14. May 2020, at 06:55, Grainier Perera < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Dominik, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I agree. We have to do some enhancements make this >>>>>>>> processor >>>>>>>>>>>> user >>>>>>>>>>>>>>>> friendly. >>>>>>>>>>>>>>>>>> Like you said it's nice to have; 1. A new static >>>>>>>>>>>>>>>>>> property that supports code >>>>>>>> highlighting & >>>>>>>>>>>> syntax >>>>>>>>>>>>>>>>>> checking. We can simply do such client-side validations >>>>>>>>>> using >>>>>>>>>>>>>>>> highlight.js >>>>>>>>>>>>>>>>>> (BSD) [1], jshint (MIT) [2], etc... >>>>>>>>>>>>>>>>>> 2. A mechanism to show the user with available fields >>>>>>>>>> they can >>>>>>>>>>>>>> use to >>>>>>>>>>>>>>>>>> write code (similar to what you've done with usable >>>>>>>>>> templates in >>>>>>>>>>>>>>>>>> EmailPublisher using HtmlInputParameter). >>>>>>>>>>>>>>>>>> 3. A mechanism to map output schema. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I think 1 & 2 go together and for the 3rd requirement >>>>>>>> given >>>>>>>>>>>> that we >>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>> targetting a more technical user group for this >>>>>>>> processor, >>>>>>>>>> we >>>>>>>>>>>> can >>>>>>>>>>>>>> let >>>>>>>>>>>>>>>> them >>>>>>>>>>>>>>>>>> define the output. Otherwise, it would be quite >>>>>>>> difficult to >>>>>>>>>>>> derive >>>>>>>>>>>>>>>> return >>>>>>>>>>>>>>>>>> type from the JS. Moreover, I'm planning to use Java's >>>>>>>>>> builtin >>>>>>>>>>>>>>>> ScriptEngine >>>>>>>>>>>>>>>>>> to eval. Of course, there'll be limitations. But, we can >>>>>>>>>> always >>>>>>>>>>>>>> improve >>>>>>>>>>>>>>>>>> them :) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] https://highlightjs.org/ [2] https://jshint.com/ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>> Grainier. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, 14 May 2020 at 00:53, Dominik Riemer < >>>>>>>>>> [email protected] >>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Grainier, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> in my opinion, this would definitely be a very >>>>>>>> interesting >>>>>>>>>>>>>> addition! We >>>>>>>>>>>>>>>>>>> already briefly discussed such a feature in the past, >>>>>>>>>> before we >>>>>>>>>>>>>> went >>>>>>>>>>>>>>>> to the >>>>>>>>>>>>>>>>>>> ASF. I’d say that a JS evaluator would be >>>>>>>>>>>>>>>>>>> definitely >>>>>>>>>> useful to >>>>>>>>>>>> a >>>>>>>>>>>>>>>> slightly >>>>>>>>>>>>>>>>>>> more technical target user group, but would >>>>>>>>>>>>>>>>>>> provide >>>>>>>> great >>>>>>>>>>>>>> flexibility >>>>>>>>>>>>>>>> for, >>>>>>>>>>>>>>>>>>> e.g., data harmonization tasks. So let’s discuss >>>>>>>>>>>>>>>>>>> what >>>>>>>> is >>>>>>>>>>>> needed to >>>>>>>>>>>>>>>>>>> implement this! I’d guess that we (maybe) need to >>>>>>>>>>>>>>>>>>> add >>>>>>>> some >>>>>>>>>>>>>>>> enhancements to >>>>>>>>>>>>>>>>>>> the core to make this processor good from a >>>>>>>>>>>>>>>>>>> usability >>>>>>>>>>>> perspective. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> From a conceptual point of view, I guess the JS >>>>>>>> processor >>>>>>>>>> would >>>>>>>>>>>>>> have no >>>>>>>>>>>>>>>>>>> specific input requirements and a single static >>>>>>>> property >>>>>>>>>> that >>>>>>>>>>>>>> allows >>>>>>>>>>>>>>>> users >>>>>>>>>>>>>>>>>>> to enter the code (maybe it would be cool to add a >>>>>>>>>>>>>>>>>>> new >>>>>>>>>> static >>>>>>>>>>>>>> property >>>>>>>>>>>>>>>> here >>>>>>>>>>>>>>>>>>> that supports code highlighting or even syntax >>>>>>>> checking). >>>>>>>>>> The >>>>>>>>>>>> only >>>>>>>>>>>>>> open >>>>>>>>>>>>>>>>>>> issue I see is the output schema – we would >>>>>>>>>>>>>>>>>>> somehow >>>>>>>> need to >>>>>>>>>>>>>> extract the >>>>>>>>>>>>>>>>>>> output from the JS function. This could probably >>>>>>>> partially >>>>>>>>>> be >>>>>>>>>>>> done >>>>>>>>>>>>>>>> using >>>>>>>>>>>>>>>>>>> the CustomTransformOutput (see >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategie >>>>>>>>>> s/), >>>>>>>>>>>>>>>>>>> but we would somehow need to derive type >>>>>>>>>>>>>>>>>>> information >>>>>>>> from >>>>>>>>>> the >>>>>>>>>>>> JS >>>>>>>>>>>>>>>> function >>>>>>>>>>>>>>>>>>> or introduce a feature to let users define/refine >>>>>>>>>>>>>>>>>>> the >>>>>>>>>> output >>>>>>>>>>>>>> manually >>>>>>>>>>>>>>>>>>> (e.g., to add semantic metadata). >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> What do you think would be best? And would you >>>>>>>>>>>>>>>>>>> evaluate >>>>>>>>>> the JS >>>>>>>>>>>>>> code in >>>>>>>>>>>>>>>>>>> Java or somehow else? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Also, as I’ve just started to improve the SDK >>>>>>>>>>>>>>>>>>> (e.g., >>>>>>>>>> supporting >>>>>>>>>>>>>>>> optional >>>>>>>>>>>>>>>>>>> static properties and an easier way to define and >>>>>>>> extract >>>>>>>>>> model >>>>>>>>>>>>>>>>>>> parameters), we can collect your requirements for >>>>>>>>>>>>>>>>>>> the >>>>>>>> JS >>>>>>>>>>>> processor >>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>> extend the SDK if needed, just say what you would >>>>>>>>>>>>>>>>>>> like >>>>>>>> to >>>>>>>>>> have >>>>>>>>>>>> 😉 >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Dominik >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> From: Grainier Perera <[email protected]> >>>>>>>>>>>>>>>>>>> Sent: Wednesday, May 13, 2020 2:41 PM >>>>>>>>>>>>>>>>>>> To: [email protected] >>>>>>>>>>>>>>>>>>> Subject: Data processor to evaluate JavaScript >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> I'm planning to implement a JavaScript eval data >>>>>>>>>> processor. As >>>>>>>>>>>> the >>>>>>>>>>>>>> name >>>>>>>>>>>>>>>>>>> suggests, users will be able to write some >>>>>>>>>>>>>>>>>>> JavaScript >>>>>>>> code >>>>>>>>>>>> which >>>>>>>>>>>>>> takes >>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>> an event (as a map), do some processing on the >>>>>>>>>>>>>>>>>>> event, >>>>>>>> and >>>>>>>>>>>> return a >>>>>>>>>>>>>> new >>>>>>>>>>>>>>>> map >>>>>>>>>>>>>>>>>>> which then gets converted to an sp-event. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> example js: >>>>>>>>>>>>>>>>>>> function process(event) { >>>>>>>>>>>>>>>>>>> // do processing here. >>>>>>>>>>>>>>>>>>> // return processed event. >>>>>>>>>>>>>>>>>>> return {id: http://event.id, tempInCelsius: >>>>>>>>>>>> (event.tempInKelvin >>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>> 273.15)}; >>>>>>>>>>>>>>>>>>> }; >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Will this be a good addition to pipeline-elements? >>>>>>>> What do >>>>>>>>>> you >>>>>>>>>>>>>> think? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Grainier. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>> >>> >>> >>> > >
