Let's chat about this f2f. I guess we need an upper limit for sampling as well.
Thanks Srinath On Mon, Aug 31, 2015 at 9:35 AM, Anjana Fernando <anj...@wso2.com> wrote: > Hi Srinath, > > I see. Then we have to think how to do the sampling efficiently, even in > the backend, we don't want millions of data to be read for the sampling > process. > > Cheers, > Anjana. > > On Mon, Aug 31, 2015 at 7:33 AM, Srinath Perera <srin...@wso2.com> wrote: > >> Hi Anjana, >> >> Even for a bar graph, it is OK to randomly sample if it is bigger than >> some size (e.g. 10000) with a warning that data has been sampled. That is >> the best we can do, and it is much better than going OOM. >> >> Thanks >> Srinath >> >> On Sat, Aug 29, 2015 at 10:31 AM, Anjana Fernando <anj...@wso2.com> >> wrote: >> >>> Hi, >>> >>> On Fri, Aug 28, 2015 at 9:18 PM, Srinath Perera <srin...@wso2.com> >>> wrote: >>> >>>> Correct approch (for Viz) would be to sample it down rather than taking >>>> first 1000 ( but not for DAL data retrival case). Any chance we can do >>>> that? >>>> >>> >>> That also, should be done case by case for graph types I guess, for >>> example, it doesn't make sense to sample data for a bar graphs, like some >>> bars won't be visible! .. >>> >>> Cheers, >>> Anjana. >>> >>> >>>> >>>> On Fri, Aug 28, 2015 at 7:12 PM, Dunith Dhanushka <dun...@wso2.com> >>>> wrote: >>>> >>>>> Hi Anjana, >>>>> >>>>> I'll work on the above change. >>>>> >>>>> Regards, >>>>> Dunith >>>>> >>>>> On Fri, Aug 28, 2015 at 8:22 AM, Anjana Fernando <anj...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Srinath, >>>>>> >>>>>> I was checking on this, and it seems, the issue is, if you've a large >>>>>> result set, and for the gadget, if you select this large result set as >>>>>> its >>>>>> source data, it will go and read everything. For example, if you use the >>>>>> smart home sample and just put all the raw records as the X axis of a bar >>>>>> chart, it will try to read all thousands of data records and try to >>>>>> render >>>>>> it. But this operation takes a bit of time, to do the JSON serialization >>>>>> and send it to the front end. And what makes this far more worse is, the >>>>>> dashboard automatically refreshes the data, and this happens in an >>>>>> asynchronous manner, and it doesn't wait for the earlier cycle to be >>>>>> finished. So these requests effectively builds up and eats up the CPU in >>>>>> the backend and can end up going OOM. >>>>>> >>>>>> So I guess, first, we have to fix the refreshing of the gadgets >>>>>> somehow, so it will wait for the earlier request to finish, before making >>>>>> new ones. Then after that, we should also provide a sensible upper limit >>>>>> to >>>>>> the data records queries, so people won't mistakenly query millions of >>>>>> records from a raw data table. We have to do this carefully, because for >>>>>> a >>>>>> thing like a scatter plot, we may need to read thousands of records to >>>>>> render the image. So for things like that, it maybe easier for it to be >>>>>> rendered in the backend and sent to the front end. And also, Maninda also >>>>>> pointed out, since we have a natural refresh interval for a dashboard, we >>>>>> should also cache the data in the end backend, with a cache validity >>>>>> period >>>>>> similar to the dashboard interval, so we don't have to read data from the >>>>>> physical databases all the time. >>>>>> >>>>>> So as an immediate fix, shall we try to change the dashboard refresh >>>>>> logic and also put a upper limit to the backend data retrieval to about >>>>>> 1000 (by changing "LAST_INDEX" variable from -1 to 100 in analytics.jag >>>>>> at >>>>>> carbon-dashboards). @Dunith, will you be able to do these changes. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> >>>>>> On Fri, Aug 28, 2015 at 3:07 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Srinath, >>>>>>> >>>>>>> Yeah, I also could reproduce the issue. Will check it out. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Fri, Aug 28, 2015 at 2:38 PM, Srinath Perera <srin...@wso2.com> >>>>>>> wrote: >>>>>>> >>>>>>>> This was a batch chart. >>>>>>>> >>>>>>>> On Fri, Aug 28, 2015 at 2:32 PM, Damith Wickramasinghe < >>>>>>>> dami...@wso2.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> AFAIK That message "Data is not available for plotting. The chart >>>>>>>>> will be loaded once the dashboard receives events" normally comes for >>>>>>>>> real >>>>>>>>> time flow gadgets. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Damith. >>>>>>>>> >>>>>>>>> On Fri, Aug 28, 2015 at 2:22 PM, Srinath Perera <srin...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes, I could see the preview ok. >>>>>>>>>> >>>>>>>>>> --Srinath >>>>>>>>>> >>>>>>>>>> On Fri, Aug 28, 2015 at 2:19 PM, Anjana Fernando <anj...@wso2.com >>>>>>>>>> > wrote: >>>>>>>>>> >>>>>>>>>>> Hi Srinath, >>>>>>>>>>> >>>>>>>>>>> Did you have data in that table before you created the gadget? .. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Anjana. >>>>>>>>>>> >>>>>>>>>>> On Fri, Aug 28, 2015 at 2:16 PM, Srinath Perera < >>>>>>>>>>> srin...@wso2.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> It is fixed with new pack. >>>>>>>>>>>> >>>>>>>>>>>> However, when I create a Gadget ( based on beatch data) and >>>>>>>>>>>> add teh Gadget, it says "Data is not available for plotting. The >>>>>>>>>>>> chart will >>>>>>>>>>>> be loaded once the dashboard receives events". >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Srinath >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 28, 2015 at 8:58 AM, Srinath Perera < >>>>>>>>>>>> srin...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Yesterday packs had the same problem >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Aug 27, 2015 at 4:20 PM, Maninda Edirisooriya < >>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> This was fixed in https://wso2.org/jira/browse/DAS-84 on >>>>>>>>>>>>>> yesterday by Dunith. Now the fix should be there in latest >>>>>>>>>>>>>> jenkins built >>>>>>>>>>>>>> packs. >>>>>>>>>>>>>> Thanks. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Maninda Edirisooriya* >>>>>>>>>>>>>> Senior Software Engineer >>>>>>>>>>>>>> >>>>>>>>>>>>>> *WSO2, Inc.*lean.enterprise.middleware. >>>>>>>>>>>>>> >>>>>>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>>>>>> *E-mail* : mani...@wso2.com >>>>>>>>>>>>>> *Skype* : @manindae >>>>>>>>>>>>>> *Twitter* : @maninda >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Aug 27, 2015 at 4:16 PM, Srinath Perera < >>>>>>>>>>>>>> srin...@wso2.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> is this fixed? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Aug 25, 2015 at 4:41 PM, Maninda Edirisooriya < >>>>>>>>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Aug 25, 2015 at 3:28 PM, Srinath Perera < >>>>>>>>>>>>>>>> srin...@wso2.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Using yesterday's pack >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 1. Can we make creating a receiver part of the creating/ >>>>>>>>>>>>>>>>> editing new streams flow? e.g. by asking what transports to >>>>>>>>>>>>>>>>> expose and >>>>>>>>>>>>>>>>> automatically creating it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2. In the gadget generation wizard, x, y axis drop downs >>>>>>>>>>>>>>>>> are now not be populated. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Yes this issue was observed in latest packs. I am looking >>>>>>>>>>>>>>>> at it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [image: Inline image 1] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 3. When you go into Gadget design view, there is no button >>>>>>>>>>>>>>>>> to come back. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [image: Inline image 2] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks >>>>>>>>>>>>>>>>> Srinath >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>>>>>> Dev@wso2.org >>>>>>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> ============================ >>>>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com >>>>>>>>>>>>>>> twitter:@srinath_perera >>>>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> ============================ >>>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> ============================ >>>>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>>>> Phone: 0772360902 >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Anjana Fernando* >>>>>>>>>>> Senior Technical Lead >>>>>>>>>>> WSO2 Inc. | http://wso2.com >>>>>>>>>>> lean . enterprise . middleware >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> ============================ >>>>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>>>> Phone: 0772360902 >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> Dev@wso2.org >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Software Engineer >>>>>>>>> WSO2 Inc.; http://wso2.com >>>>>>>>> <http://www.google.com/url?q=http%3A%2F%2Fwso2.com&sa=D&sntz=1&usg=AFQjCNEZvyc0uMD1HhBaEGCBxs6e9fBObg> >>>>>>>>> lean.enterprise.middleware >>>>>>>>> >>>>>>>>> mobile: *+94728671315 <%2B94728671315>* >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> ============================ >>>>>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>>>>> Site: http://people.apache.org/~hemapani/ >>>>>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>>>>> Phone: 0772360902 >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Senior Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Dunith Dhanushka, >>>>> Senior Software Engineer >>>>> WSO2 Inc, >>>>> >>>>> Mobile - +94 71 8615744 >>>>> Blog - dunithd.wordpress.com <http://blog.dunith.com> >>>>> Twitter - @dunithd <http://twitter.com/dunithd> >>>>> >>>> >>>> >>>> >>>> -- >>>> ============================ >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>> >>> >>> >>> -- >>> *Anjana Fernando* >>> Senior Technical Lead >>> WSO2 Inc. | http://wso2.com >>> lean . enterprise . middleware >>> >> >> >> >> -- >> ============================ >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > *Anjana Fernando* > Senior Technical Lead > WSO2 Inc. | http://wso2.com > lean . enterprise . middleware > -- ============================ Blog: http://srinathsview.blogspot.com twitter:@srinath_perera Site: http://people.apache.org/~hemapani/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
