Hi Inosh, If epoach time is received to the server I don't understand the limitation? Is it because Siddhi time window doesn't operate in UTC time? Cheers~
On Tue, Jun 14, 2016 at 7:00 AM, Inosh Goonewardena <in...@wso2.com> wrote: > Hi Sajith, > > > On Tue, Jun 14, 2016 at 12:21 PM, Sajith Ravindra <saji...@wso2.com> > wrote: > >> Hi Inosh, >> >> In order to do this we might probably need to update the data publishers >> to send their time zones as meta data to do the time zone conversion. >> > > We receive epoch time in event timestamp field from data publishers. So > publisher time zone is not required here. > > However, we have decided to not pursue with UTC time unit approach due to > the time window boundary issue explained in above. > > >> >> In cases like daylight saving changes this information will be required >> to correctly map the incoming time to UTC >> >> Also, isn't it an option to make data agents to publish their data in UTC? >> >> On Jun 13, 2016, at 10:20 PM, Inosh Goonewardena <in...@wso2.com> wrote: >> >> Hi Ruwan, >> >> >> On Mon, Jun 13, 2016 at 7:13 PM, Ruwan Abeykoon <ruw...@wso2.com> wrote: >> >>> Hi All, >>> +1 on storing time in UTC. >>> >>> However this will cause some minor (but annoying) discrepancies to occur >>> when we do windowing on "per-hour" basis, when the time zone difference >>> between UTC and the server is not full hour. e.g. +5:30 UTC. >>> Is there any solution for that? >>> >> >> Yes. This could be a problem. For example, UTC time window [00.00 - >> 01.00] is equals to [05.30 - 06.30] IST. And also if a timezone conversion >> happen in a dashboard level, this again could affect graphs too. So need to >> check those areas before moving ahead. >> >> >>> >>> Cheers, >>> Ruwan >>> >>> On Mon, Jun 13, 2016 at 6:10 PM, Inosh Goonewardena <in...@wso2.com> >>> wrote: >>> >>>> Hi, >>>> >>>> In analytics use cases we use a couple of extensions to extract >>>> attributes (year, month, day, hour, etc.) from the event timestamp. In >>>> product anaytics implementations, we mostly use time:extract() Siddhi >>>> extension [1] in real time scenarios and DateTimeUDF in batch scenarios >>>> [2]. However, both those extensions give output time units based on the >>>> local time zone of the server on which DAS is running. >>>> >>>> For example, if the DAS is running in a server which has the timezone >>>> set as IST, for timestamp 1461135538669 (2016/04/20 06:58:58 GMT) output >>>> time units are resolved as 2016, 04, 20, 12, 28. >>>> >>>> In most of the scenarios, analytics is implemented in per <time unit> >>>> basis, i.e., we maintain summary tables for per minute, per hour, per day, >>>> per month. These summary tables has columns for year, month, date, hour, >>>> etc. Since aforementioned extensions are giving time units based on local >>>> timezone what we store in there are are local time units. >>>> >>>> IMO, we should store UTC time units instead, since it is better to >>>> maintain time units uniformly without depending on the time zone of the >>>> server DAS is running. We have also found that this inconsistency is >>>> capable of producing issues in incremental data processing. >>>> >>>> Shall we extend our analytics extensions to support UTC time units >>>> throughout? >>>> >>>> [1] >>>> https://github.com/wso2/siddhi/blob/master/modules/siddhi-extensions/time/src/main/java/org/wso2/siddhi/extension/time/ExtractAttributesFunctionExtension.java >>>> [2] >>>> https://github.com/wso2/shared-analytics/blob/master/components/spark-udf/org.wso2.carbon.analytics.shared.spark.common.udf/src/main/java/org/wso2/carbon/analytics/shared/common/udf/DateTimeUDF.java >>>> >>>> -- >>>> Thanks & Regards, >>>> >>>> Inosh Goonewardena >>>> Associate Technical Lead- WSO2 Inc. >>>> Mobile: +94779966317 >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> >>> *Ruwan Abeykoon* >>> *Associate Director/Architect**,* >>> *WSO2, Inc. http://wso2.com <http://wso2.com/> * >>> *lean.enterprise.middleware.* >>> >>> email: ruw...@wso2.com >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> Thanks & Regards, >> >> Inosh Goonewardena >> Associate Technical Lead- WSO2 Inc. >> Mobile: +94779966317 >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Thanks & Regards, > > Inosh Goonewardena > Associate Technical Lead- WSO2 Inc. > Mobile: +94779966317 > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Dulitha Wijewantha (Chan) Software Engineer - Mobile Development WSO2 Inc Lean.Enterprise.Middleware * ~Email duli...@wso2.com <duli...@wso2mobile.com>* * ~Mobile +94712112165* * ~Website dulitha.me <http://dulitha.me>* * ~Twitter @dulitharw <https://twitter.com/dulitharw>* *~Github @dulichan <https://github.com/dulichan>* *~SO @chan <http://stackoverflow.com/users/813471/chan>*
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
