Hi, On Fri, Jun 10, 2016 at 10:33 PM, Hasitha Hiranya <hasit...@wso2.com> wrote:
> Hi, > > When spark script is running, we want to know if it is executing fine. Ops > might be interested in monitoring this. If execution is stopped or broken > they might need to fire alerts. > Spark analytics tasks are executed by Spark workers. In a clustered mode, Spark worker is not in a carbon environment, it's a separate spawned JVM. There are some dedicated Web UI interfaces to monitor Spark Applications [1] [2], but I am not sure about inbuilt alerting support in Spark. However, if Spark throws an error while executing a certain query, we may be able to capture that from our side(carbon JVM) I think. [1] http://spark.apache.org/docs/latest/monitoring.html [2] https://docs.wso2.com/display/CLUSTER44x/Minimum+High+Availability+Deployment+-+DAS+3.0.1 > With these MBeans can we do that? > > Thanks > > > > On Thu, Jun 9, 2016 at 9:38 PM, Nirmal Fernando <nir...@wso2.com> wrote: > >> Hi All, >> >> I actually used Carbon metrics and added few Mbeans to get the remaining >> queue size etc. from the data publisher agent level and DAS level as part >> of per tests we're doing for API-M Analytics. >> >> I think we could build upon these too. Please see the following diffs. >> >> Agent: >> https://github.com/nirmal070125/carbon-analytics-common/commit/1375677fd2a39303e8cbaaea33c6a6a0cdd672dc >> >> DAS: >> https://github.com/nirmal070125/carbon-analytics-common/commit/0d26ba4a6ac732b161584905ff3ecf7ae20fa53e >> >> On Fri, Jun 10, 2016 at 9:49 AM, Nandika Jayawardana <nand...@wso2.com> >> wrote: >> >>> Hi Gihan, >>> >>> Do you guys have any plans to expose some MBeans for the DAS publisher. >>> Consider a scenario where a user implements a stand-alone DAS publisher to >>> publish a large data set ( Eg : archived log files ). In such a scenario, >>> we need a way to monitor the progress and to know whether the client >>> finished publishing the dataset to DAS. So having some MBeans would be >>> really useful for publisher as well. >>> >>> Regards >>> Nandika >>> >>> On Thu, Jun 9, 2016 at 7:10 PM, Isuru Perera <isu...@wso2.com> wrote: >>> >>>> Hi, >>>> >>>> On Thu, Jun 9, 2016 at 5:30 PM, Gihan Anuruddha <gi...@wso2.com> wrote: >>>> >>>>> Initially we thought about that. But then we decided to go with >>>>> normal Mbean expose due following reasons. >>>>> 1. To minimize depend on external repositories. >>>>> >>>> carbon-metrics is recommended for exposing these kind of values. DAS >>>> already has carbon-metrics, so I don't think this is an issue. >>>> >>>>> 2. To list the analytics subdomain under org.wso2.carbon domain. >>>>> >>>> You can use a meaningful names metrics. You can use a common prefix >>>> analytics metrics. >>>> >>>>> 3. I couldn't find a way to expose operations. >>>>> >>>> Instead of exposing operations, shall we try to use Gauges with some >>>> formatted name with your parameters. For example, you can use following >>>> names for getRemainingBufferCapacityInBytes(int tenantId) and >>>> int getCurrentQueueSize(int tenantId). >>>> >>>> org.wso2.carbon.analytics.<tenant-id>.disruptor.remaining-buffer-size >>>> org.wso2.carbon.analytics.<tenant-id>.disruptor.current-queue-size >>>> >>>> This will add more metrics, as there are gauges for every tenant. But I >>>> think that should be fine. If you use Metrics, you can also get historical >>>> data, which is very useful if you want to debug an issue happened in the >>>> past (may be few mins ago). >>>> >>>> Even the metrics library is available these kinds of requirements. >>>> Therefore I think we must use it and try to avoid implementing separate >>>> MBeans for any requirements that can be done with Metrics. >>>> >>>> Thanks! >>>> >>>> >>>> >>>>> Regards, >>>>> Gihan >>>>> >>>>> On Thu, Jun 9, 2016 at 5:09 PM, Isuru Perera <isu...@wso2.com> wrote: >>>>> >>>>>> Any reason for not using Carbon Metrics? >>>>>> >>>>>> On Thu, Jun 9, 2016 at 5:00 PM, Gihan Anuruddha <gi...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> We have added a couple of MBeans for DAS to expose some debug level >>>>>>> information. These Mbeans will list under org.wso2.carbon.analytics >>>>>>> subdomain. >>>>>>> >>>>>>> EVENT_COUNTER [int getCurrentCount()]- This will count all the >>>>>>> events that received to DAS regardless of steam or tenant. We want to >>>>>>> add >>>>>>> this per tenant per stream. But in order to do that we need to add >>>>>>> a Map to event persistence path and that might add extra delay to the >>>>>>> event >>>>>>> saving critical path. Because of that we thought of adding only a >>>>>>> counter (AtomicLong). >>>>>>> >>>>>>> RECEIVER_REMAINING_QUEUE_BUFFER_SIZE_IN_BYTES >>>>>>> [ >>>>>>> long getRemainingBufferCapacityInBytes(int tenantId), >>>>>>> int getCurrentQueueSize(int tenantId)] >>>>>>> - You can use this Mbean to get remaining buffer size in >>>>>>> disruptor and current queue size. >>>>>>> >>>>>>> LAST_PROCESSED_TIMESTAMP[long getLastProcessedTimestamp(int >>>>>>> tenantId, String id, boolean primary)] >>>>>>> - This will return last saved incremental processed timestamp for >>>>>>> given spark table. >>>>>>> >>>>>>> ANALYTICS_SCRIPT_LAST_EXECUTION_START_TIME[long >>>>>>> getScriptLastExecutionStartTime(int tenantId, String scriptName)] >>>>>>> - This will return latest execution start time of the given Spark >>>>>>> script. >>>>>>> >>>>>>> Please let me know any other information that you think better >>>>>>> expose through Mbeans. >>>>>>> >>>>>>> Regards, >>>>>>> Gihan >>>>>>> >>>>>>> -- >>>>>>> W.G. Gihan Anuruddha >>>>>>> Senior Software Engineer | WSO2, Inc. >>>>>>> M: +94772272595 >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Isuru Perera >>>>>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/ >>>>>> Lean . Enterprise . Middleware >>>>>> >>>>>> about.me/chrishantha >>>>>> Contact: +IsuruPereraWSO2 >>>>>> <https://www.google.com/+IsuruPereraWSO2/about> >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> W.G. Gihan Anuruddha >>>>> Senior Software Engineer | WSO2, Inc. >>>>> M: +94772272595 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> >>>> -- >>>> Isuru Perera >>>> Associate Technical Lead | WSO2, Inc. | http://wso2.com/ >>>> Lean . Enterprise . Middleware >>>> >>>> about.me/chrishantha >>>> Contact: +IsuruPereraWSO2 >>>> <https://www.google.com/+IsuruPereraWSO2/about> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Nandika Jayawardana >>> WSO2 Inc ; http://wso2.com >>> lean.enterprise.middleware >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > *Hasitha Abeykoon* > Senior Software Engineer; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Thanks & Regards, Inosh Goonewardena Associate Technical Lead- WSO2 Inc. Mobile: +94779966317
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
