Hi, I'm +1 to the idea of `UserDefinedOutputStrategy`. As you said, that'll be re-usable in the future processors as well. So let's proceed with that.
Regards, Grainier On Fri, 15 May 2020 at 01:37, Dominik Riemer <[email protected]> wrote: > Hi, > cool, that sounds very good! > > Concerning the output, I think there are two ways to achieve this: > - A collection static property in conjunction with a runtime resolvable > output strategy that creates the output schema based on the input from the > collection > - Or we create a new output strategy, e.g., let's call it > UserDefinedOutputStrategy, which would render a UI component to let users > add/remove/change event properties from the input schema. I also like the > idea of preselecting all input properties. > > I personally tend towards the second option as such a strategy might be > useful for other pipeline elements as well and it would ease the model > definition. We could also extend this component in the future, e.g., by > letting users enrich metadata such as measurement units directly in a > processor. > > But I'm also open to the first option and we can also just start > implementing to see what works best. > > I'd also be happy to help with the implementation, e.g., I could work on > the new static property for code input or the output strategy! > > Dominik > > On 2020/05/14 11:59:36, Grainier Perera <[email protected]> > wrote: > > Hi, > > > > +1 to the idea of having a collection property containing the fields of > the > > output properties. Also, If we can show the user with available fields > with > > metadata (of input event) populated in that same collection property, > then > > the user can add/remove/updates fields easily. Let's say they just have > to > > return a map containing all the fields they defined in output properties > > (i.e return {"existingId": 1, "existingTempInKelvin": 301, > > "newTempInCelsius": 28 } ). > > > > [image: sample.png] > > > > Regards, > > Grainier. > > > > On Thu, 14 May 2020 at 12:33, Patrick Wiener <[email protected]> wrote: > > > > > Hi, > > > > > > sounds reasonable. > > > > > > I share the thoughts of Philipp regarding the output schema. But as you > > > said Grainier, we can also improve :) > > > > > > btw: I really like the idea to also allow the more technical user to > > > easily integrate their function snippets without having > > > to care about the actual execution. This gets a more FaaS like feeling > for > > > streaming. > > > > > > Patrick > > > > > > > Am 14.05.2020 um 08:26 schrieb Philipp Zehnder <[email protected]>: > > > > > > > > Hi, > > > > > > > > it would be very useful to have such a processor. > > > > > > > > I like the ideas for 1 & 2 and I think we can do it this way. > > > > > > > > Actually I think that the definition of the output scheme could be > quite > > > tricky. > > > > Because users can do anything with the event. > > > > > > > > How about we provide an option property? > > > > Users can select whether they want to define the whole event or just > > > append properties. > > > > > > > > For the configurations of the properties we could do something > similar > > > then we have in the PLC4xS7Adapter. > > > > A collection property containing the fields of the output properties. > > > > Therefore, we have to decide what information we have to provide. I > > > think if we would support all options it becomes very cluttered for the > > > user. > > > > > > > > What do you think? > > > > > > > > Philipp > > > > > > > > > > > > > > > >> On 14. May 2020, at 06:55, Grainier Perera < > [email protected]> > > > wrote: > > > >> > > > >> Hi Dominik, > > > >> > > > >> I agree. We have to do some enhancements make this processor user > > > friendly. > > > >> Like you said it's nice to have; > > > >> 1. A new static property that supports code highlighting & syntax > > > >> checking. We can simply do such client-side validations using > > > highlight.js > > > >> (BSD) [1], jshint (MIT) [2], etc... > > > >> 2. A mechanism to show the user with available fields they can > use to > > > >> write code (similar to what you've done with usable templates in > > > >> EmailPublisher using HtmlInputParameter). > > > >> 3. A mechanism to map output schema. > > > >> > > > >> I think 1 & 2 go together and for the 3rd requirement given that we > are > > > >> targetting a more technical user group for this processor, we can > let > > > them > > > >> define the output. Otherwise, it would be quite difficult to derive > > > return > > > >> type from the JS. Moreover, I'm planning to use Java's builtin > > > ScriptEngine > > > >> to eval. Of course, there'll be limitations. But, we can always > improve > > > >> them :) > > > >> > > > >> [1] https://highlightjs.org/ > > > >> [2] https://jshint.com/ > > > >> > > > >> Regards, > > > >> Grainier. > > > >> > > > >> On Thu, 14 May 2020 at 00:53, Dominik Riemer <[email protected]> > wrote: > > > >> > > > >>> Hi Grainier, > > > >>> > > > >>> in my opinion, this would definitely be a very interesting > addition! We > > > >>> already briefly discussed such a feature in the past, before we > went > > > to the > > > >>> ASF. I’d say that a JS evaluator would be definitely useful to a > > > slightly > > > >>> more technical target user group, but would provide great > flexibility > > > for, > > > >>> e.g., data harmonization tasks. So let’s discuss what is needed to > > > >>> implement this! I’d guess that we (maybe) need to add some > > > enhancements to > > > >>> the core to make this processor good from a usability perspective. > > > >>> > > > >>> From a conceptual point of view, I guess the JS processor would > have no > > > >>> specific input requirements and a single static property that > allows > > > users > > > >>> to enter the code (maybe it would be cool to add a new static > property > > > here > > > >>> that supports code highlighting or even syntax checking). The only > open > > > >>> issue I see is the output schema – we would somehow need to > extract the > > > >>> output from the JS function. This could probably partially be done > > > using > > > >>> the CustomTransformOutput (see > > > >>> > http://streampipes.apache.org/docs/docs/dev-guide-output-strategies/), > > > >>> but we would somehow need to derive type information from the JS > > > function > > > >>> or introduce a feature to let users define/refine the output > manually > > > >>> (e.g., to add semantic metadata). > > > >>> > > > >>> What do you think would be best? And would you evaluate the JS > code in > > > >>> Java or somehow else? > > > >>> > > > >>> Also, as I’ve just started to improve the SDK (e.g., supporting > > > optional > > > >>> static properties and an easier way to define and extract model > > > >>> parameters), we can collect your requirements for the JS processor > and > > > >>> extend the SDK if needed, just say what you would like to have 😉 > > > >>> > > > >>> Dominik > > > >>> > > > >>> > > > >>> From: Grainier Perera <[email protected]> > > > >>> Sent: Wednesday, May 13, 2020 2:41 PM > > > >>> To: [email protected] > > > >>> Subject: Data processor to evaluate JavaScript > > > >>> > > > >>> Hi all, > > > >>> > > > >>> I'm planning to implement a JavaScript eval data processor. As the > name > > > >>> suggests, users will be able to write some JavaScript code which > takes > > > in > > > >>> an event (as a map), do some processing on the event, and return a > new > > > map > > > >>> which then gets converted to an sp-event. > > > >>> > > > >>> example js: > > > >>> function process(event) { > > > >>> // do processing here. > > > >>> // return processed event. > > > >>> return {id: http://event.id, tempInCelsius: (event.tempInKelvin > - > > > >>> 273.15)}; > > > >>> }; > > > >>> > > > >>> Will this be a good addition to pipeline-elements? What do you > think? > > > >>> > > > >>> Regards, > > > >>> Grainier. > > > >>> > > > >>> > > > > > > > > > > > > > > > > >
