Hi Dominik, Awesome. This is really cool & exactly what I was looking for. I'll use this with my implementation, and keep this thread posted with the updates. ps: let us know when you add output strategy as well.
Thanks, Grainier. On Tue, 19 May 2020 at 03:05, Dominik Riemer <[email protected]> wrote: > Hi Grainier, > > I just pushed an initial version of the static property to add javascript > code. > It's a simple codeblock with linting enabled and a (currently very simple) > autocomplete feature that suggests existing properties from the input event > (currently, only if you press ctrl-space directly after typing "event" > followed by a dot, I'll fix that soon ;-) > > You can find an example how to use it here: > > https://github.com/apache/incubator-streampipes-examples/blob/dev/streampipes-pipeline-elements-examples-processors-jvm/src/main/java/org/apache/streampipes/pe/examples/jvm/staticproperty/CodeInputExampleController.java > > Feedback and ideas for improvement are welcome! > > Dominik > > > On 2020/05/17 12:36:05, Grainier Perera <[email protected]> > wrote: > > Hi Dominik, > > > > Thank you. I'll work on `Implement JS evaluator` [1]. > > > > [1] https://issues.apache.org/jira/browse/STREAMPIPES-135 > > > > Grainier. > > > > On Sun, 17 May 2020 at 18:02, Dominik Riemer <[email protected]> wrote: > > > > > Hi Grainier, > > > > > > great! I created an umbrella issue for that: > > > https://issues.apache.org/jira/browse/STREAMPIPES-132 > > > > > > I'll now start to implement the new static property for entering code. > > > > > > Dominik > > > > > > On 2020/05/15 01:29:44, Grainier Perera <[email protected]> > > > wrote: > > > > Hi, > > > > > > > > I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said, > that'll > > > be > > > > re-usable in the future processors as well. So let's proceed with > that. > > > > > > > > Regards, > > > > Grainier > > > > > > > > On Fri, 15 May 2020 at 01:37, Dominik Riemer <[email protected]> > wrote: > > > > > > > > > Hi, > > > > > cool, that sounds very good! > > > > > > > > > > Concerning the output, I think there are two ways to achieve this: > > > > > - A collection static property in conjunction with a runtime > resolvable > > > > > output strategy that creates the output schema based on the input > from > > > the > > > > > collection > > > > > - Or we create a new output strategy, e.g., let's call it > > > > > UserDefinedOutputStrategy, which would render a UI component to let > > > users > > > > > add/remove/change event properties from the input schema. I also > like > > > the > > > > > idea of preselecting all input properties. > > > > > > > > > > I personally tend towards the second option as such a strategy > might be > > > > > useful for other pipeline elements as well and it would ease the > model > > > > > definition. We could also extend this component in the future, > e.g., by > > > > > letting users enrich metadata such as measurement units directly > in a > > > > > processor. > > > > > > > > > > But I'm also open to the first option and we can also just start > > > > > implementing to see what works best. > > > > > > > > > > I'd also be happy to help with the implementation, e.g., I could > work > > > on > > > > > the new static property for code input or the output strategy! > > > > > > > > > > Dominik > > > > > > > > > > On 2020/05/14 11:59:36, Grainier Perera <[email protected] > > > > > > > wrote: > > > > > > Hi, > > > > > > > > > > > > +1 to the idea of having a collection property containing the > fields > > > of > > > > > the > > > > > > output properties. Also, If we can show the user with available > > > fields > > > > > with > > > > > > metadata (of input event) populated in that same collection > property, > > > > > then > > > > > > the user can add/remove/updates fields easily. Let's say they > just > > > have > > > > > to > > > > > > return a map containing all the fields they defined in output > > > properties > > > > > > (i.e return {"existingId": 1, "existingTempInKelvin": 301, > > > > > > "newTempInCelsius": 28 } ). > > > > > > > > > > > > [image: sample.png] > > > > > > > > > > > > Regards, > > > > > > Grainier. > > > > > > > > > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <[email protected]> > > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > sounds reasonable. > > > > > > > > > > > > > > I share the thoughts of Philipp regarding the output schema. > But > > > as you > > > > > > > said Grainier, we can also improve :) > > > > > > > > > > > > > > btw: I really like the idea to also allow the more technical > user > > > to > > > > > > > easily integrate their function snippets without having > > > > > > > to care about the actual execution. This gets a more FaaS like > > > feeling > > > > > for > > > > > > > streaming. > > > > > > > > > > > > > > Patrick > > > > > > > > > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder < > > > [email protected]>: > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > it would be very useful to have such a processor. > > > > > > > > > > > > > > > > I like the ideas for 1 & 2 and I think we can do it this way. > > > > > > > > > > > > > > > > Actually I think that the definition of the output scheme > could > > > be > > > > > quite > > > > > > > tricky. > > > > > > > > Because users can do anything with the event. > > > > > > > > > > > > > > > > How about we provide an option property? > > > > > > > > Users can select whether they want to define the whole event > or > > > just > > > > > > > append properties. > > > > > > > > > > > > > > > > For the configurations of the properties we could do > something > > > > > similar > > > > > > > then we have in the PLC4xS7Adapter. > > > > > > > > A collection property containing the fields of the output > > > properties. > > > > > > > > Therefore, we have to decide what information we have to > > > provide. I > > > > > > > think if we would support all options it becomes very cluttered > > > for the > > > > > > > user. > > > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > > > Philipp > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera < > > > > > [email protected]> > > > > > > > wrote: > > > > > > > >> > > > > > > > >> Hi Dominik, > > > > > > > >> > > > > > > > >> I agree. We have to do some enhancements make this processor > > > user > > > > > > > friendly. > > > > > > > >> Like you said it's nice to have; > > > > > > > >> 1. A new static property that supports code highlighting & > > > syntax > > > > > > > >> checking. We can simply do such client-side validations > using > > > > > > > highlight.js > > > > > > > >> (BSD) [1], jshint (MIT) [2], etc... > > > > > > > >> 2. A mechanism to show the user with available fields > they can > > > > > use to > > > > > > > >> write code (similar to what you've done with usable > templates in > > > > > > > >> EmailPublisher using HtmlInputParameter). > > > > > > > >> 3. A mechanism to map output schema. > > > > > > > >> > > > > > > > >> I think 1 & 2 go together and for the 3rd requirement given > > > that we > > > > > are > > > > > > > >> targetting a more technical user group for this processor, > we > > > can > > > > > let > > > > > > > them > > > > > > > >> define the output. Otherwise, it would be quite difficult to > > > derive > > > > > > > return > > > > > > > >> type from the JS. Moreover, I'm planning to use Java's > builtin > > > > > > > ScriptEngine > > > > > > > >> to eval. Of course, there'll be limitations. But, we can > always > > > > > improve > > > > > > > >> them :) > > > > > > > >> > > > > > > > >> [1] https://highlightjs.org/ > > > > > > > >> [2] https://jshint.com/ > > > > > > > >> > > > > > > > >> Regards, > > > > > > > >> Grainier. > > > > > > > >> > > > > > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer < > [email protected] > > > > > > > > > wrote: > > > > > > > >> > > > > > > > >>> Hi Grainier, > > > > > > > >>> > > > > > > > >>> in my opinion, this would definitely be a very interesting > > > > > addition! We > > > > > > > >>> already briefly discussed such a feature in the past, > before we > > > > > went > > > > > > > to the > > > > > > > >>> ASF. I’d say that a JS evaluator would be definitely > useful to > > > a > > > > > > > slightly > > > > > > > >>> more technical target user group, but would provide great > > > > > flexibility > > > > > > > for, > > > > > > > >>> e.g., data harmonization tasks. So let’s discuss what is > > > needed to > > > > > > > >>> implement this! I’d guess that we (maybe) need to add some > > > > > > > enhancements to > > > > > > > >>> the core to make this processor good from a usability > > > perspective. > > > > > > > >>> > > > > > > > >>> From a conceptual point of view, I guess the JS processor > would > > > > > have no > > > > > > > >>> specific input requirements and a single static property > that > > > > > allows > > > > > > > users > > > > > > > >>> to enter the code (maybe it would be cool to add a new > static > > > > > property > > > > > > > here > > > > > > > >>> that supports code highlighting or even syntax checking). > The > > > only > > > > > open > > > > > > > >>> issue I see is the output schema – we would somehow need to > > > > > extract the > > > > > > > >>> output from the JS function. This could probably partially > be > > > done > > > > > > > using > > > > > > > >>> the CustomTransformOutput (see > > > > > > > >>> > > > > > > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/), > > > > > > > >>> but we would somehow need to derive type information from > the > > > JS > > > > > > > function > > > > > > > >>> or introduce a feature to let users define/refine the > output > > > > > manually > > > > > > > >>> (e.g., to add semantic metadata). > > > > > > > >>> > > > > > > > >>> What do you think would be best? And would you evaluate > the JS > > > > > code in > > > > > > > >>> Java or somehow else? > > > > > > > >>> > > > > > > > >>> Also, as I’ve just started to improve the SDK (e.g., > supporting > > > > > > > optional > > > > > > > >>> static properties and an easier way to define and extract > model > > > > > > > >>> parameters), we can collect your requirements for the JS > > > processor > > > > > and > > > > > > > >>> extend the SDK if needed, just say what you would like to > have > > > 😉 > > > > > > > >>> > > > > > > > >>> Dominik > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> From: Grainier Perera <[email protected]> > > > > > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM > > > > > > > >>> To: [email protected] > > > > > > > >>> Subject: Data processor to evaluate JavaScript > > > > > > > >>> > > > > > > > >>> Hi all, > > > > > > > >>> > > > > > > > >>> I'm planning to implement a JavaScript eval data > processor. As > > > the > > > > > name > > > > > > > >>> suggests, users will be able to write some JavaScript code > > > which > > > > > takes > > > > > > > in > > > > > > > >>> an event (as a map), do some processing on the event, and > > > return a > > > > > new > > > > > > > map > > > > > > > >>> which then gets converted to an sp-event. > > > > > > > >>> > > > > > > > >>> example js: > > > > > > > >>> function process(event) { > > > > > > > >>> // do processing here. > > > > > > > >>> // return processed event. > > > > > > > >>> return {id: http://event.id, tempInCelsius: > > > (event.tempInKelvin > > > > > - > > > > > > > >>> 273.15)}; > > > > > > > >>> }; > > > > > > > >>> > > > > > > > >>> Will this be a good addition to pipeline-elements? What do > you > > > > > think? > > > > > > > >>> > > > > > > > >>> Regards, > > > > > > > >>> Grainier. > > > > > > > >>> > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
