Hi, cool, that sounds very good! Concerning the output, I think there are two ways to achieve this: - A collection static property in conjunction with a runtime resolvable output strategy that creates the output schema based on the input from the collection - Or we create a new output strategy, e.g., let's call it UserDefinedOutputStrategy, which would render a UI component to let users add/remove/change event properties from the input schema. I also like the idea of preselecting all input properties.
I personally tend towards the second option as such a strategy might be useful for other pipeline elements as well and it would ease the model definition. We could also extend this component in the future, e.g., by letting users enrich metadata such as measurement units directly in a processor. But I'm also open to the first option and we can also just start implementing to see what works best. I'd also be happy to help with the implementation, e.g., I could work on the new static property for code input or the output strategy! Dominik On 2020/05/14 11:59:36, Grainier Perera <[email protected]> wrote: > Hi, > > +1 to the idea of having a collection property containing the fields of the > output properties. Also, If we can show the user with available fields with > metadata (of input event) populated in that same collection property, then > the user can add/remove/updates fields easily. Let's say they just have to > return a map containing all the fields they defined in output properties > (i.e return {"existingId": 1, "existingTempInKelvin": 301, > "newTempInCelsius": 28 } ). > > [image: sample.png] > > Regards, > Grainier. > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <[email protected]> wrote: > > > Hi, > > > > sounds reasonable. > > > > I share the thoughts of Philipp regarding the output schema. But as you > > said Grainier, we can also improve :) > > > > btw: I really like the idea to also allow the more technical user to > > easily integrate their function snippets without having > > to care about the actual execution. This gets a more FaaS like feeling for > > streaming. > > > > Patrick > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <[email protected]>: > > > > > > Hi, > > > > > > it would be very useful to have such a processor. > > > > > > I like the ideas for 1 & 2 and I think we can do it this way. > > > > > > Actually I think that the definition of the output scheme could be quite > > tricky. > > > Because users can do anything with the event. > > > > > > How about we provide an option property? > > > Users can select whether they want to define the whole event or just > > append properties. > > > > > > For the configurations of the properties we could do something similar > > then we have in the PLC4xS7Adapter. > > > A collection property containing the fields of the output properties. > > > Therefore, we have to decide what information we have to provide. I > > think if we would support all options it becomes very cluttered for the > > user. > > > > > > What do you think? > > > > > > Philipp > > > > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera <[email protected]> > > wrote: > > >> > > >> Hi Dominik, > > >> > > >> I agree. We have to do some enhancements make this processor user > > friendly. > > >> Like you said it's nice to have; > > >> 1. A new static property that supports code highlighting & syntax > > >> checking. We can simply do such client-side validations using > > highlight.js > > >> (BSD) [1], jshint (MIT) [2], etc... > > >> 2. A mechanism to show the user with available fields they can use to > > >> write code (similar to what you've done with usable templates in > > >> EmailPublisher using HtmlInputParameter). > > >> 3. A mechanism to map output schema. > > >> > > >> I think 1 & 2 go together and for the 3rd requirement given that we are > > >> targetting a more technical user group for this processor, we can let > > them > > >> define the output. Otherwise, it would be quite difficult to derive > > return > > >> type from the JS. Moreover, I'm planning to use Java's builtin > > ScriptEngine > > >> to eval. Of course, there'll be limitations. But, we can always improve > > >> them :) > > >> > > >> [1] https://highlightjs.org/ > > >> [2] https://jshint.com/ > > >> > > >> Regards, > > >> Grainier. > > >> > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <[email protected]> wrote: > > >> > > >>> Hi Grainier, > > >>> > > >>> in my opinion, this would definitely be a very interesting addition! We > > >>> already briefly discussed such a feature in the past, before we went > > to the > > >>> ASF. I’d say that a JS evaluator would be definitely useful to a > > slightly > > >>> more technical target user group, but would provide great flexibility > > for, > > >>> e.g., data harmonization tasks. So let’s discuss what is needed to > > >>> implement this! I’d guess that we (maybe) need to add some > > enhancements to > > >>> the core to make this processor good from a usability perspective. > > >>> > > >>> From a conceptual point of view, I guess the JS processor would have no > > >>> specific input requirements and a single static property that allows > > users > > >>> to enter the code (maybe it would be cool to add a new static property > > here > > >>> that supports code highlighting or even syntax checking). The only open > > >>> issue I see is the output schema – we would somehow need to extract the > > >>> output from the JS function. This could probably partially be done > > using > > >>> the CustomTransformOutput (see > > >>> http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/), > > >>> but we would somehow need to derive type information from the JS > > function > > >>> or introduce a feature to let users define/refine the output manually > > >>> (e.g., to add semantic metadata). > > >>> > > >>> What do you think would be best? And would you evaluate the JS code in > > >>> Java or somehow else? > > >>> > > >>> Also, as I’ve just started to improve the SDK (e.g., supporting > > optional > > >>> static properties and an easier way to define and extract model > > >>> parameters), we can collect your requirements for the JS processor and > > >>> extend the SDK if needed, just say what you would like to have 😉 > > >>> > > >>> Dominik > > >>> > > >>> > > >>> From: Grainier Perera <[email protected]> > > >>> Sent: Wednesday, May 13, 2020 2:41 PM > > >>> To: [email protected] > > >>> Subject: Data processor to evaluate JavaScript > > >>> > > >>> Hi all, > > >>> > > >>> I'm planning to implement a JavaScript eval data processor. As the name > > >>> suggests, users will be able to write some JavaScript code which takes > > in > > >>> an event (as a map), do some processing on the event, and return a new > > map > > >>> which then gets converted to an sp-event. > > >>> > > >>> example js: > > >>> function process(event) { > > >>> // do processing here. > > >>> // return processed event. > > >>> return {id: http://event.id, tempInCelsius: (event.tempInKelvin - > > >>> 273.15)}; > > >>> }; > > >>> > > >>> Will this be a good addition to pipeline-elements? What do you think? > > >>> > > >>> Regards, > > >>> Grainier. > > >>> > > >>> > > > > > > > > > > >
