Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas

2018-11-15 Thread Andrew Otto
OH I’m sorry! There is a Phab task, and this is fixed. https://phabricator.wikimedia.org/T209407 Very sorry, I should have updated and linked the task. On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc wrote: > Is this issue still ongoing? Is there a corresponding Phabricator task to > follow? > >

Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas

2018-11-15 Thread Tilman Bayer
Does "fixed" mean that the missing data already been backfilled? I'm seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12. On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto wrote: > OH I’m sorry! There

Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas

2018-11-15 Thread Gilles Dubuc
Is this issue still ongoing? Is there a corresponding Phabricator task to follow? On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto wrote: > Hi all, > > Yesterday we upgraded the Hadoop cluster to a newer version. It seems > that along the way the job that imports EventLogging data into Hive has >

Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas

2018-11-15 Thread Andrew Otto
> Does "fixed" mean that the missing data already been backfilled? I’m seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the timespans between around 6am-16pm on November 13, and 7am-10am on November 12. Hm. Fixed means the data has been refined to Hive. I didn’t check on

Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas

2018-11-15 Thread Nuria Ruiz
Hello, Not all data sources are populated at the same time, the data on Druid is ingested twice, once per hour and once daily looking 4 days back. Data should appear once daily job runs for the "holes" missing. Thanks, Nuria On Thu, Nov 15, 2018 at 7:49 AM Andrew Otto wrote: > > Does "fixed"

Re: [Analytics] EventLogging Hive Refine currently stalled for some Schemas

2018-11-15 Thread Marcel Ruiz Forns
> > Not all data sources are populated at the same time, the data on Druid is > ingested twice, once per hour and once daily looking 4 days back. Data > should appear once daily job runs for the "holes" missing. +1 The EL2Druid daily loading job will cover up the holes for the 12th and 13th in 1