Re: [Architecture] IS-Analytics performance

, "Iranga Muthuthanthri" wrote: > > > On Mon, Jun 27, 2016 at 4:35 PM, Mohanadarshan Vivekanandalingam < > mo...@wso2.com> wrote: > >> >> >> On Mon, Jun 27, 2016 at 4:30 PM, Seshika Fernando >> wrote: >> >>> Hey saith, >>&

Re: [Architecture] IS-Analytics performance

Hey saith, This is great. So when you removed the duplicate windows, there were no OOM issues? Seshi On 24 Jun 2016 14:28, "Sajith Ravindra" wrote: > Hi Malith, > > Thanks for the explanation. >> >> I would expect some variation in the throughput. The aim should be >> minimize the variation in

Re: [Architecture] [CEP] Siddhi Extension for calculate percentile values

0% and 95%). What is the best way to >>> handle case? Can we have an implementation where we can pass an array >>> (containing percentiles) and get the function to return an array of >>> percentile values? Or would it make more sense to call the function which >>> acc

Re: [Architecture] [CEP] Siddhi Extension for calculate percentile values

turn the 97th percentile value of last 100 events of > temperature variable. > > Likewise, we can use other windows such as timeBatch, cron etc as for the > requirement. Would that approach be OK with this? > > Thanks, > Ashen > > On Thu, Jun 9, 2016 at 5:18 AM, Seshika Fernando wro

Re: [Architecture] [CEP] Siddhi Extension for calculate percentile values

Ashen, Is there no way that we can provide a window for temperature field? Practically, we may want to know the 97th percentile of the last n events. In that case how would I use this extension? It might be a good idea to allow user to provide an interval based on length and/or time (like we do for

Re: [Architecture] [Dev] WSO2 IS Analytics v1.0.0-ALPHA Released

Hi guys, If we did any changes to the architecture of Authentication analytics, can we update the documentation in [1]. Damith/Mohan: Can one of you take ownership of this? Thanks, Seshi 1. https://docs.google.com/a/wso2.com/document/d/1h47OLzBiHYy-kk12VkmRnoetbG37Dx4LcHa3CnTzDTE/edit?usp=shari

Re: [Architecture] [CEP] Extending the Regression Function to support time window

ports both time and length, a name such as regressTimeLength > would be appropriate IMO. Please give your suggestions. > > Regards, > Charini > > On Sun, Jun 5, 2016 at 11:28 AM, Seshika Fernando > wrote: > >> Hi, >> The length ceiling is necessary along with the duration para

Re: [Architecture] [CEP] Extending the Regression Function to support time window

Hi, The length ceiling is necessary along with the duration parameter. The reason the batch size was originally implemented was to optimize performance when large datasets are considered for regression. We need to be able to give an upper bound. So for example in this case, if user uses a large dur

Re: [Architecture] [Siddhi] Making Disruptor configurable

Hi Suho, Looks good. If (bufferSize='') is optional, what is the default bufferSize that will be taken if I just add @plan:async ? seshi On Sun, May 15, 2016 at 3:58 PM, Sriskandarajah Suhothayan wrote: > > Hi > > We have made Disruptor as optional for Siddhi[1], currently its always > en

Re: [Architecture] Moving Custom Components to a common place

Damith, Just for the record, as discussed yesterday, these will be added to analytics.commons repo where we will keep all components that can be used commonly by "Analytics for X" efforts. @Miyuru: Can you add the other components we discussed to be added to this? N.B. analytics.commons - not to

Re: [Architecture] [Analytics] Improvements in search APIs in AnalyticsDataService

Hi Gimantha, This is a very useful functionality. Thanks for adding this. I have 1 question. Can you tell me the difference of result between the following 2 combinations. "sortType" : "DESC" "reversed" : false and "sortType" : "ASC" "reversed" : true My query is, that if you c

Re: [Architecture] [Analytics][APIM] - Implement Geo location graph in Analytics

Hi Pulasthi, Can we use this for our IP->Region conversion? seshi On Mon, Mar 21, 2016 at 11:06 AM, Tharindu Dharmarathna wrote: > Hi Janaka and All, > > Here is the performance test result which we have done . > > No of Different IP's : 2000 > > Time Taken to Retrieve and Run the spark script

Re: [Architecture] Request Pattern Change Detection (Analytics for APIM)

particular sequence most of the time, but if >> there're multiple applications, API access patterns should be very >> different practically. >> >> On Tue, Feb 16, 2016 at 2:18 PM, Seshika Fernando >> wrote: >> >>> Yep. if 2 is the case, still we do

Re: [Architecture] Request Pattern Change Detection (Analytics for APIM)

ven if [2] is the > case. Under the assumption that there will be a lot more genuine users than > fraudulent users? > > > On Tue, Feb 16, 2016 at 11:19 AM, Seshika Fernando > wrote: > >> Hi Fazlan, >> >> Could you explain the thinking behind assigning userI

Re: [Architecture] Request Pattern Change Detection (Analytics for APIM)

Hi Fazlan, Could you explain the thinking behind assigning userID also as one of the classifiers for the state? The reason I'm asking is that users can learn from each others patterns as well. i.e. if a bunch of users are using the same (or similar) set of apis, they will all follow similar reques

Re: [Architecture] Packaging analytics artifacts into a p2 repo

+1 On Wed, Feb 10, 2016 at 4:51 AM, Srinath Perera wrote: > Thanks!! Seshika, we should try this with Fraud artifacts. > > --Srinath > > On Tue, Feb 9, 2016 at 11:14 AM, Chanika Geeganage > wrote: > >> Hi, >> >> I have done a POC to evaluate whether it is possible to package analytics >> artifa

Re: [Architecture] Security Analytics

ere is a fraud detected due to several API Calls > can we find out out from which API calls and calling which backend service? > > On Tue, Feb 2, 2016 at 11:30 AM, Seshika Fernando > wrote: > >> @Lahiru: +1. I'm hoping to analyze the data that we can get from IS so

Re: [Architecture] Security Analytics

, Feb 1, 2016 at 11:30 PM, Lahiru Sandaruwan > wrote: > >> Hi Seshi, >> >> I think we can consider Authorization stats also. Since WSO2 IS has a >> good implementation of XACML spec, we can collect stats on, the requests >> allowed, denied, with whic

Re: [Architecture] Notebook Support Use cases for DAS

Hi Srinath, The 'notebooks' that you talk about: are they similar to a sort of staging DAS configuration where we test out/ try out things, and once we are happy we deploy that configuration to the respective pipelines (like Realtime). Cannot access doc. Can you share a 'view only' link? seshi

Re: [Architecture] [ML] Anomaly Detection Feature for WSO2 ML

Welldone Ashen. The documentation looks good too. Will review later. seshi On Thu, Nov 19, 2015 at 10:35 AM, Ashen Weerathunga wrote: > Hi all, > > This feature was implemented on ML and released with WSO2 Machine Learner > 1.1.0 - Milestone 1 >

Re: [Architecture] [CEP] Event Simulator Feature

Hi Ramindu, The existing event simulator feature is extremely useful. IMO what would add to its usability is the following... 1. ability to provide a delay between events 2. ability for user to specify how many events to be sent (currently all events in the file will be sent) 3. ability to pause a

Re: [Architecture] Connector: SalesforceBulk

Nashry, Please try this out when its done. seshi On Thu, Sep 3, 2015 at 12:02 PM, Malaka Silva wrote: > Great looks good > > On Thu, Sep 3, 2015 at 11:19 AM, Vinitha Rajenthiran < > vinitha.ra...@gmail.com> wrote: > >> Hi Malaka, >> >> Yes, we have covered all the methods. >> >> However we have

Re: [Architecture] [DAS] Changing the name of Message Console

My vote is for "data explorer" for message console and to keep "spark console" for spark console. seshi On Mon, Jul 13, 2015 at 10:36 AM, Anjana Fernando wrote: > Hi, > > +1 for "Data Explorer" for message console. The name "Spark Console" is > fine the way it is now. > > Cheers, > Anjana. > >

Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library

ex.html > > Thanks, > > On Thu, Mar 26, 2015 at 7:34 PM, Seshika Fernando > wrote: > >> Hi Tharindu, >> >> Can I use aggregate functions with Tables as well? >> >> seshi >> >> >> On Thu, Mar 26, 2015 at 5:44 PM, Seshika Fernando >>

Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library

Hi Tharindu, Can I use aggregate functions with Tables as well? seshi On Thu, Mar 26, 2015 at 5:44 PM, Seshika Fernando wrote: > Oh this is good news indeed. Will use it. > > On Thu, Mar 26, 2015 at 5:31 PM, Tharindu Munasinghe > wrote: > >> Time domain is already there

Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library

ontinuous variable you can't s use it on Bar charts( which > need a discrete domain for X-axis ). For other types of charts like area > ,line and scatter you can use it. > > > [1] http://dunithd.github.io/igviz/samples/discreteTime/index.html > > On Thu, Mar 26, 2015 at 3:57 P

Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library

gt;> Thanks, >> >> On Thu, Mar 26, 2015 at 2:18 PM, Tharindu Munasinghe >> wrote: >> >>> igviz.js uses the aggregated functions that are supported by vega.js . >>> min,max,sum,avg are the functions that I have tested and they are working >>> . I

Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library

What are the aggregate functions that I can use? I see 'sum' is available. Does this support Count too? Please let me know what are the functions that are currently supported. Thanks On Thu, Mar 26, 2015 at 1:36 PM, Seshika Fernando wrote: > Noted and Thanks. Will try this out. &g

Re: [Architecture] [IGVIZ.JS] : D3.js Based Interactive Generic Visualization Library

Noted and Thanks. Will try this out. On Thu, Mar 26, 2015 at 1:34 PM, Tharindu Munasinghe wrote: > Hi Seshika, > I have updated the igviz.js so that now aggregated functions can > be used with bar charts. if you don't use aggregate functions , bar chart > will only show the last Y value

[Architecture] [CEP] Anomaly Detection using Delta Anomaly Detection

Hi all, Inspired by Ted Dunning's talk on Anomaly Detection in [1], we are hoping to implement this in Siddhi. Basically, this is how Delta Anomaly Detection works. a. Use a predictor based on percentiles which predicts the next event based on historical data. b. Obtain the delta between the pre

Re: [Architecture] [CEP] Fraud Scoring

ika, > > Are you planning to test this based on some real world scenarios and data? > ( ex: stock prices) Therefore to get and idea how accurate we can be. > > Thanks, > > On Tue, Nov 18, 2014 at 10:59 AM, Seshika Fernando > wrote: > >> Hi all, >> >>

[Architecture] [CEP] Fraud Scoring

Hi all, Following the implementation of Fraud Rules, and Markov Chain capability in order to do outlier detection in CEP, we are hoping to implement Fraud Scoring capability. Fraud Scoring is a mechanism to evaluate multiple features of a transaction (eg:- geolocation, ip address, billing/shippin

[Architecture] Implementing Markov Chains in CEP

Hi all, A Markov Chain is a memoryless, random process which undergoes transitions from one state to another on a defined state space. Markov chains are quite useful in outlier detection mechanisms and we are currently in the process of implementing Outlier detection using Markov Chains in CEP. [1

Re: [Architecture] Implementing Frequency calculation algorithm in CEP

t; Thanks > > On Fri, Oct 17, 2014 at 5:10 PM, Seshika Fernando > wrote: > >> When considering the usecases for this, getting the frequency during a >> particular time period will be very useful. (eg:- trading frequency of a >> different stocks/products during the l

Re: [Architecture] Implementing Frequency calculation algorithm in CEP

When considering the usecases for this, getting the frequency during a particular time period will be very useful. (eg:- trading frequency of a different stocks/products during the last 5 hours) Does your custom transformer support this? Or does it always take all events for the frequency calculati

Re: [Architecture] Extensions for Siddhi - Math operations

Excellent! Some of these will be very useful for the Fraud Detection, and Forecasting algorithms. On Wed, Oct 15, 2014 at 5:34 PM, Lahiru Cooray wrote: > Hi, > > We have implemented below Math operations (all the Math operations available > in Hive) as extensions to Siddhi engine. > > round(DOUB

[Architecture] Fraud Detection Toolboxes

Hi all, I've started to work on $subject. Fraud Detection is used in many spaces. Some of the most popular ones are credit card fraud, anti money laundering, organizational/operational fraud etc; I'm planning to create toolboxes (for some of these spaces) in CEP, which will work with a defined s

Re: [Architecture] Using Siddhi Event processor to implement/evaluate some clustering algorithms

Hi Lahiru, As Srinath has mentioned as well, frequency counting algorithms are very useful in financial market scenarios as well (especially fraud detection and surveillance). Thanks for doing this and I will take a look too. seshika On Fri, Jun 27, 2014 at 10:52 AM, Mohanadarshan Vivekanandali

Re: [Architecture] Connector:GooglePrediction

Hi Rasika, Do you know what is used to train the models? (regression analysis/ neural networks or something else?) Seshika On Tue, Jun 3, 2014 at 10:06 AM, Rasika Hettige wrote: > Hi Malaka, > > The final list of methods that will implement under GooglePrediction > connector is as follows. >

[Architecture] Siddhi Time Series Extension - Upper limit and Batch processing

Hi all, There was a issue with the time-series extension where all events keep getting added to the extension and all subsequent regressions consider ALL previous events for the regression calculation. This obviously leads to 'out of memory' issues and the solution was to implement an upper limit

Re: [Architecture] Siddhi Time Seriers Extension - Performance

Hi all, Currently we are performing regression incrementally and we do not have an upperbound (or window size) which limits the number of events that are processed in the regression function. Therefore, we run into memory issues when running performance tests. In order to do a proper performance t

Re: [Architecture] Siddhi Time Seriers Extension - Performance

gt; >> Suho >> >> >> On Mon, Apr 14, 2014 at 10:55 AM, Seshika Fernando wrote: >> >>> Hi, >>> >>> This is not over the network. Events are just a couple of data streams >>> joined together (eg:- StockPriceStream, TwitterWordCountStrea

[Architecture] Siddhi Time Series Extension - Next Steps

Hi, After researching on how to handle seasonality in regression, I have the following findings. 1. Can use dummy variables to capture seasonality. The user needs to add dummy variables to the input stream to capture and quantify seasonality in the regression equation. Therefore, this does not ha

Re: [Architecture] Siddhi Time Seriers Extension - Performance

Apr 12, 2014 at 10:01 AM, Sanjiva Weerawarana >>> wrote: >>> >>>> Wow those are big #s ... very impressive! >>>> >>>> What hardware? Is this over the network or in-memory driving Siddhi? >>>> What type of events? Whats the exact que

[Architecture] Siddhi Extensions for Time Series Linear Regression

Waruna and I, have developed two simple siddhi extensions to calculate time series linear regression and outliers. Basically, if a Y event series can be predicted by an X event series, we develop a Y = mx + c , type of equation which quantifies the relationship between X and Y. The actual regressi