Hi, sounds reasonable.
I share the thoughts of Philipp regarding the output schema. But as you said Grainier, we can also improve :) btw: I really like the idea to also allow the more technical user to easily integrate their function snippets without having to care about the actual execution. This gets a more FaaS like feeling for streaming. Patrick > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <[email protected]>: > > Hi, > > it would be very useful to have such a processor. > > I like the ideas for 1 & 2 and I think we can do it this way. > > Actually I think that the definition of the output scheme could be quite > tricky. > Because users can do anything with the event. > > How about we provide an option property? > Users can select whether they want to define the whole event or just append > properties. > > For the configurations of the properties we could do something similar then > we have in the PLC4xS7Adapter. > A collection property containing the fields of the output properties. > Therefore, we have to decide what information we have to provide. I think if > we would support all options it becomes very cluttered for the user. > > What do you think? > > Philipp > > > >> On 14. May 2020, at 06:55, Grainier Perera <[email protected]> wrote: >> >> Hi Dominik, >> >> I agree. We have to do some enhancements make this processor user friendly. >> Like you said it's nice to have; >> 1. A new static property that supports code highlighting & syntax >> checking. We can simply do such client-side validations using highlight.js >> (BSD) [1], jshint (MIT) [2], etc... >> 2. A mechanism to show the user with available fields they can use to >> write code (similar to what you've done with usable templates in >> EmailPublisher using HtmlInputParameter). >> 3. A mechanism to map output schema. >> >> I think 1 & 2 go together and for the 3rd requirement given that we are >> targetting a more technical user group for this processor, we can let them >> define the output. Otherwise, it would be quite difficult to derive return >> type from the JS. Moreover, I'm planning to use Java's builtin ScriptEngine >> to eval. Of course, there'll be limitations. But, we can always improve >> them :) >> >> [1] https://highlightjs.org/ >> [2] https://jshint.com/ >> >> Regards, >> Grainier. >> >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <[email protected]> wrote: >> >>> Hi Grainier, >>> >>> in my opinion, this would definitely be a very interesting addition! We >>> already briefly discussed such a feature in the past, before we went to the >>> ASF. I’d say that a JS evaluator would be definitely useful to a slightly >>> more technical target user group, but would provide great flexibility for, >>> e.g., data harmonization tasks. So let’s discuss what is needed to >>> implement this! I’d guess that we (maybe) need to add some enhancements to >>> the core to make this processor good from a usability perspective. >>> >>> From a conceptual point of view, I guess the JS processor would have no >>> specific input requirements and a single static property that allows users >>> to enter the code (maybe it would be cool to add a new static property here >>> that supports code highlighting or even syntax checking). The only open >>> issue I see is the output schema – we would somehow need to extract the >>> output from the JS function. This could probably partially be done using >>> the CustomTransformOutput (see >>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/), >>> but we would somehow need to derive type information from the JS function >>> or introduce a feature to let users define/refine the output manually >>> (e.g., to add semantic metadata). >>> >>> What do you think would be best? And would you evaluate the JS code in >>> Java or somehow else? >>> >>> Also, as I’ve just started to improve the SDK (e.g., supporting optional >>> static properties and an easier way to define and extract model >>> parameters), we can collect your requirements for the JS processor and >>> extend the SDK if needed, just say what you would like to have 😉 >>> >>> Dominik >>> >>> >>> From: Grainier Perera <[email protected]> >>> Sent: Wednesday, May 13, 2020 2:41 PM >>> To: [email protected] >>> Subject: Data processor to evaluate JavaScript >>> >>> Hi all, >>> >>> I'm planning to implement a JavaScript eval data processor. As the name >>> suggests, users will be able to write some JavaScript code which takes in >>> an event (as a map), do some processing on the event, and return a new map >>> which then gets converted to an sp-event. >>> >>> example js: >>> function process(event) { >>> // do processing here. >>> // return processed event. >>> return {id: http://event.id, tempInCelsius: (event.tempInKelvin - >>> 273.15)}; >>> }; >>> >>> Will this be a good addition to pipeline-elements? What do you think? >>> >>> Regards, >>> Grainier. >>> >>> > >
