Re: [Dev] Summarization of JSON data in DAS

[Lahiru Madushanka]

Hi Grainier,

Thanks for the reply. I had a discussion with Nirmal and the final solution
was to use Siddhi Map extension as you also suggesting.
However we change our approach later to push avg data directly from the
device agent, than computing them in DAS. (EDGE analytics)

Anyway thanks for the reply, we can use the suggested approach in future
cases.

Regards,
Lahiru


On Mon, Aug 21, 2017 at 8:20 AM, Grainier Perera  wrote:

> Hi Lahiru,
>
> You can achieve that by using JSON custom mapping [1] + Siddhi MAP
> extension + IfThenElse (optional to check the availability of attributes).
> Basically what you have to do is, have a;
>
> JSON structure like this;
>
>>
>> {
>>"userData": {
>>"timestamp": 19900813115534,
>>
>> *"dataMap": {*
>>
>> *  "id": 1, // these will be your arbitary data
>> "name": grainier**   }*
>>}
>>}
>> }
>
>
> Input mapping like this;
>
>> 
>
>    // other mappings
>
>
>>
>>
>> *
>>   ***
>> 
>
>
> Stream definition like this;
>
>> {
>>   "streamId": "org.wso2.event.user.stream:1.0.0",
>>   "name": "org.wso2.event.user.stream",
>>   "version": "1.0.0",
>>   "nickName": "",
>>   "description": "",
>>   "metaData": [],
>>   "correlationData": [],
>>   "payloadData": [
>> {
>>   "name": "timestamp",
>>   "type": "LONG"
>> },
>>
>>
>>
>> *{  "name": "userData",  "type": "STRING"**}*
>>   ]
>> }
>
>
> Finally within the execution plan;
>
>> @Import('org.wso2.event.user.stream:1.0.0')
>> define stream dataIn (timestamp long, *userData** string*);
>>
>
>
> from dataIn
>> select *map:createFromJSON(**userData) as userData**Map*
>> insert into tempStream;
>
>
>
> from tempStream
>>
>> *select map:get(userDataMap, "id") as id, map:get(userDataMap, "name") as
>> name*insert into tempStream2;
>
>
>
> -- now you can use those arbitary fields here...
>
>
>
> Hope that helped...
>
> [1] https://docs.wso2.com/display/CEP420/Input+Mapping+
> Types#InputMappingTypes-JSONinputmappingJSONInputMapping
> [2] https://docs.wso2.com/display/CEP420/Map+Extension
>
> Regards,
> Grainier.
>
> On Sat, Jul 8, 2017 at 11:34 AM, Lahiru Madushanka 
> wrote:
>
>> Hi Nirmal,
>>
>> Documentation says arbitrary data support can be used with wso2event
>> input format. But in my case its "JSON".
>> Custom event receiver will be an option. But is there a way I can do this
>> without writing a custom event receiver ?
>>
>> Thanks for the help.
>>
>> Regards,
>> Lahiru
>>
>> On Sat, Jul 8, 2017 at 10:02 AM, Nirmal Fernando  wrote:
>>
>>> Check on arbitrary data support https://docs.wso2.com/
>>> display/DAS310/Input+Mapping+Types
>>>
>>> On Sat, Jul 8, 2017 at 7:48 AM, Lahiru Madushanka 
>>> wrote:
>>>
 Hi all,

 I have a requirement where data values published to DAS are not
 predefined (dynamic)
 ( Ex :- cpu usage of a given laptop ) So I push them as a JSON string
 Ex:-  "cpuinfo":{"corePercentages": [4.1, 3.1, 5.2, 7.1], "numOfCores":
 4}
 This JSON string will be changed with no of cores in the PC which
 pushes the data.

 Is there a way I can write a summarization query in siddhiql to take
 average of average corePercentages for a time interval. (first take avg of
 percentage values and then average it over time )

 Thanks
 Lahiru

 ___
 Dev mailing list
 Dev@wso2.org
 http://wso2.org/cgi-bin/mailman/listinfo/dev


>>>
>>>
>>> --
>>>
>>> Thanks & regards,
>>> Nirmal
>>>
>>> Technical Lead, WSO2 Inc.
>>> Mobile: +94715779733 <+94%2071%20577%209733>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> Grainier Perera
> Senior Software Engineer
> Mobile : +94716122384 <+94%2071%20612%202384>
> WSO2 Inc. | http://wso2.com
> lean.enterprise.middleware
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Summarization of JSON data in DAS

[Grainier Perera]

Hi Lahiru,

You can achieve that by using JSON custom mapping [1] + Siddhi MAP
extension + IfThenElse (optional to check the availability of attributes).
Basically what you have to do is, have a;

JSON structure like this;

>
> {
>"userData": {
>"timestamp": 19900813115534,
>
> *"dataMap": {*
>
> *  "id": 1, // these will be your arbitary data
> "name": grainier**   }*
>}
>}
> }


Input mapping like this;

> 

   // other mappings


>
>
> *
>   ***
> 


Stream definition like this;

> {
>   "streamId": "org.wso2.event.user.stream:1.0.0",
>   "name": "org.wso2.event.user.stream",
>   "version": "1.0.0",
>   "nickName": "",
>   "description": "",
>   "metaData": [],
>   "correlationData": [],
>   "payloadData": [
> {
>   "name": "timestamp",
>   "type": "LONG"
> },
>
>
>
> *{  "name": "userData",  "type": "STRING"**}*
>   ]
> }


Finally within the execution plan;

> @Import('org.wso2.event.user.stream:1.0.0')
> define stream dataIn (timestamp long, *userData** string*);
>


from dataIn
> select *map:createFromJSON(**userData) as userData**Map*
> insert into tempStream;



from tempStream
>
> *select map:get(userDataMap, "id") as id, map:get(userDataMap, "name") as
> name*insert into tempStream2;



-- now you can use those arbitary fields here...



Hope that helped...

[1]
https://docs.wso2.com/display/CEP420/Input+Mapping+Types#InputMappingTypes-JSONinputmappingJSONInputMapping
[2] https://docs.wso2.com/display/CEP420/Map+Extension

Regards,
Grainier.

On Sat, Jul 8, 2017 at 11:34 AM, Lahiru Madushanka 
wrote:

> Hi Nirmal,
>
> Documentation says arbitrary data support can be used with wso2event input
> format. But in my case its "JSON".
> Custom event receiver will be an option. But is there a way I can do this
> without writing a custom event receiver ?
>
> Thanks for the help.
>
> Regards,
> Lahiru
>
> On Sat, Jul 8, 2017 at 10:02 AM, Nirmal Fernando  wrote:
>
>> Check on arbitrary data support https://docs.wso2.com/
>> display/DAS310/Input+Mapping+Types
>>
>> On Sat, Jul 8, 2017 at 7:48 AM, Lahiru Madushanka 
>> wrote:
>>
>>> Hi all,
>>>
>>> I have a requirement where data values published to DAS are not
>>> predefined (dynamic)
>>> ( Ex :- cpu usage of a given laptop ) So I push them as a JSON string
>>> Ex:-  "cpuinfo":{"corePercentages": [4.1, 3.1, 5.2, 7.1], "numOfCores":
>>> 4}
>>> This JSON string will be changed with no of cores in the PC which pushes
>>> the data.
>>>
>>> Is there a way I can write a summarization query in siddhiql to take
>>> average of average corePercentages for a time interval. (first take avg of
>>> percentage values and then average it over time )
>>>
>>> Thanks
>>> Lahiru
>>>
>>> ___
>>> Dev mailing list
>>> Dev@wso2.org
>>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>>
>>>
>>
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Technical Lead, WSO2 Inc.
>> Mobile: +94715779733 <+94%2071%20577%209733>
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 
Grainier Perera
Senior Software Engineer
Mobile : +94716122384
WSO2 Inc. | http://wso2.com
lean.enterprise.middleware
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Summarization of JSON data in DAS

[Lahiru Madushanka]

Hi Nirmal,

Documentation says arbitrary data support can be used with wso2event input
format. But in my case its "JSON".
Custom event receiver will be an option. But is there a way I can do this
without writing a custom event receiver ?

Thanks for the help.

Regards,
Lahiru

On Sat, Jul 8, 2017 at 10:02 AM, Nirmal Fernando  wrote:

> Check on arbitrary data support https://docs.wso2.com/
> display/DAS310/Input+Mapping+Types
>
> On Sat, Jul 8, 2017 at 7:48 AM, Lahiru Madushanka 
> wrote:
>
>> Hi all,
>>
>> I have a requirement where data values published to DAS are not
>> predefined (dynamic)
>> ( Ex :- cpu usage of a given laptop ) So I push them as a JSON string
>> Ex:-  "cpuinfo":{"corePercentages": [4.1, 3.1, 5.2, 7.1], "numOfCores": 4}
>> This JSON string will be changed with no of cores in the PC which pushes
>> the data.
>>
>> Is there a way I can write a summarization query in siddhiql to take
>> average of average corePercentages for a time interval. (first take avg of
>> percentage values and then average it over time )
>>
>> Thanks
>> Lahiru
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
>
> Thanks & regards,
> Nirmal
>
> Technical Lead, WSO2 Inc.
> Mobile: +94715779733 <+94%2071%20577%209733>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Summarization of JSON data in DAS

[Nirmal Fernando]

Well the question is about; taking averages; yes it's certainly possible
with Siddhi. https://docs.wso2.com/display/DAS310/Siddhi+Query+Language

On Sat, Jul 8, 2017 at 10:02 AM, Nirmal Fernando  wrote:

> Check on arbitrary data support https://docs.wso2.com/
> display/DAS310/Input+Mapping+Types
>
> On Sat, Jul 8, 2017 at 7:48 AM, Lahiru Madushanka 
> wrote:
>
>> Hi all,
>>
>> I have a requirement where data values published to DAS are not
>> predefined (dynamic)
>> ( Ex :- cpu usage of a given laptop ) So I push them as a JSON string
>> Ex:-  "cpuinfo":{"corePercentages": [4.1, 3.1, 5.2, 7.1], "numOfCores": 4}
>> This JSON string will be changed with no of cores in the PC which pushes
>> the data.
>>
>> Is there a way I can write a summarization query in siddhiql to take
>> average of average corePercentages for a time interval. (first take avg of
>> percentage values and then average it over time )
>>
>> Thanks
>> Lahiru
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
>
> Thanks & regards,
> Nirmal
>
> Technical Lead, WSO2 Inc.
> Mobile: +94715779733 <+94%2071%20577%209733>
> Blog: http://nirmalfdo.blogspot.com/
>
>
>


-- 

Thanks & regards,
Nirmal

Technical Lead, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

Re: [Dev] Summarization of JSON data in DAS

[Nirmal Fernando]

Check on arbitrary data support
https://docs.wso2.com/display/DAS310/Input+Mapping+Types

On Sat, Jul 8, 2017 at 7:48 AM, Lahiru Madushanka 
wrote:

> Hi all,
>
> I have a requirement where data values published to DAS are not predefined
> (dynamic)
> ( Ex :- cpu usage of a given laptop ) So I push them as a JSON string
> Ex:-  "cpuinfo":{"corePercentages": [4.1, 3.1, 5.2, 7.1], "numOfCores": 4}
> This JSON string will be changed with no of cores in the PC which pushes
> the data.
>
> Is there a way I can write a summarization query in siddhiql to take
> average of average corePercentages for a time interval. (first take avg of
> percentage values and then average it over time )
>
> Thanks
> Lahiru
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 

Thanks & regards,
Nirmal

Technical Lead, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] Summarization of JSON data in DAS

[Lahiru Madushanka]

Hi all,

I have a requirement where data values published to DAS are not predefined
(dynamic)
( Ex :- cpu usage of a given laptop ) So I push them as a JSON string
Ex:-  "cpuinfo":{"corePercentages": [4.1, 3.1, 5.2, 7.1], "numOfCores": 4}
This JSON string will be changed with no of cores in the PC which pushes
the data.

Is there a way I can write a summarization query in siddhiql to take
average of average corePercentages for a time interval. (first take avg of
percentage values and then average it over time )

Thanks
Lahiru
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev