> > Not all data sources are populated at the same time, the data on Druid is > ingested twice, once per hour and once daily looking 4 days back. Data > should appear once daily job runs for the "holes" missing.
+1 The EL2Druid daily loading job will cover up the holes for the 12th and 13th in 1 or 2 days. On Thu, Nov 15, 2018 at 5:03 PM Nuria Ruiz <nu...@wikimedia.org> wrote: > Hello, > > Not all data sources are populated at the same time, the data on Druid is > ingested twice, once per hour and once daily looking 4 days back. Data > should appear once daily job runs for the "holes" missing. > > Thanks, > > Nuria > > On Thu, Nov 15, 2018 at 7:49 AM Andrew Otto <o...@wikimedia.org> wrote: > >> > Does "fixed" mean that the missing data already been backfilled? I’m >> seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the >> timespans between around 6am-16pm on November 13, and 7am-10am on November >> 12. >> >> Hm. Fixed means the data has been refined to Hive. I didn’t check on >> Druid EL data. Marcel, can you check up on that? This sounds related, we >> might need to figure out how to re run the Druid ingestion if/when base >> data partitions change. >> >> >> >> On Thu, Nov 15, 2018 at 10:17 AM Tilman Bayer <tba...@wikimedia.org> >> wrote: >> >>> Hi Andrew, >>> >>> PS, offtopic: reminder that data-analy...@wikimedia.org has been >>> deprecated/retired a while ago; our team can be reached at >>> product-analyt...@wikimedia.org (CCed) >>> >>> On Thu, Nov 15, 2018 at 7:13 AM Tilman Bayer <tba...@wikimedia.org> >>> wrote: >>> >>>> Does "fixed" mean that the missing data already been backfilled? I'm >>>> seeing gaps (zero events) in Turnilo for Druid-ingested EL data, for the >>>> timespans between around 6am-16pm on November 13, and 7am-10am on November >>>> 12. >>>> >>>> On Thu, Nov 15, 2018 at 6:51 AM Andrew Otto <o...@wikimedia.org> wrote: >>>> >>>>> OH I’m sorry! There is a Phab task, and this is fixed. >>>>> https://phabricator.wikimedia.org/T209407 >>>>> >>>>> Very sorry, I should have updated and linked the task. >>>>> >>>>> On Thu, Nov 15, 2018 at 9:50 AM Gilles Dubuc <gil...@wikimedia.org> >>>>> wrote: >>>>> >>>>>> Is this issue still ongoing? Is there a corresponding Phabricator >>>>>> task to follow? >>>>>> >>>>>> On Tue, Nov 13, 2018 at 6:27 PM Andrew Otto <o...@wikimedia.org> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Yesterday we upgraded the Hadoop cluster to a newer version. It >>>>>>> seems that along the way the job that imports EventLogging data into >>>>>>> Hive >>>>>>> has started failing for some EventLogging datasets. I’m still >>>>>>> investigating, but it seems that tables (I believe ones that need to >>>>>>> have >>>>>>> their schema modified during refinement) are failing refinement. The >>>>>>> list >>>>>>> of affected tables at this moment is large, and will likely only grow as >>>>>>> time goes on. >>>>>>> >>>>>>> We’re looking into this and will try to have things fixed ASAP. For >>>>>>> now you might expect Hive EventLogging data to be delayed. >>>>>>> >>>>>>> Sorry for the trouble, and we’ll keep you updated! >>>>>>> -Andrew Otto >>>>>>> Systems Engineer, WMF >>>>>>> >>>>>>> >>>>>>> >>>>>>> The list of currently affected tables is: >>>>>>> >>>>>>> AdvancedSearchRequest >>>>>>> CentralAuth >>>>>>> CentralNoticeBannerHistory >>>>>>> CentralNoticeImpression >>>>>>> CentralNoticeTiming >>>>>>> ChangesListFilterGrouping >>>>>>> ChangesListFilters >>>>>>> CitationUsage >>>>>>> CitationUsagePageLoad >>>>>>> ContentTranslation >>>>>>> ContentTranslationAbuseFilter >>>>>>> ContentTranslationCTA >>>>>>> ContentTranslationError >>>>>>> ContentTranslationSuggestion >>>>>>> CpuBenchmark >>>>>>> EchoInteraction >>>>>>> EchoMail >>>>>>> EditAttemptStep >>>>>>> EditConflict >>>>>>> EditorActivation >>>>>>> EUCCVisit >>>>>>> EventError >>>>>>> FlowReplies >>>>>>> GettingStartedRedirectImpression >>>>>>> GuidedTourButtonClick >>>>>>> GuidedTourExited >>>>>>> GuidedTourExternalLinkActivation >>>>>>> GuidedTourGuiderHidden >>>>>>> GuidedTourGuiderImpression >>>>>>> LandingPageImpression >>>>>>> MediaViewer >>>>>>> MediaWikiPingback >>>>>>> MobileAppCategorizationAttempts >>>>>>> MobileAppUploadAttempts >>>>>>> MobileWebMainMenuClickTracking >>>>>>> MobileWebSearch >>>>>>> MobileWebUIClickTracking >>>>>>> MobileWikiAppAppearanceSettings >>>>>>> MobileWikiAppArticleSuggestions >>>>>>> MobileWikiAppCreateAccount >>>>>>> MobileWikiAppDailyStats >>>>>>> MobileWikiAppEdit >>>>>>> MobileWikiAppFeed >>>>>>> MobileWikiAppFeedConfigure >>>>>>> MobileWikiAppFindInPage >>>>>>> MobileWikiAppInstallReferrer >>>>>>> MobileWikiAppIntents >>>>>>> MobileWikiAppiOSFeed >>>>>>> MobileWikiAppiOSLoginAction >>>>>>> MobileWikiAppiOSReadingLists >>>>>>> MobileWikiAppiOSSearch >>>>>>> MobileWikiAppiOSSessions >>>>>>> MobileWikiAppiOSSettingAction >>>>>>> MobileWikiAppiOSUserHistory >>>>>>> MobileWikiAppLangSelect >>>>>>> MobileWikiAppLanguageSearching >>>>>>> MobileWikiAppLanguageSettings >>>>>>> MobileWikiAppLinkPreview >>>>>>> MobileWikiAppLogin >>>>>>> MobileWikiAppMediaGallery >>>>>>> MobileWikiAppNavMenu >>>>>>> MobileWikiAppNotificationInteraction >>>>>>> MobileWikiAppNotificationPreferences >>>>>>> MobileWikiAppOfflineLibrary >>>>>>> MobileWikiAppOnboarding >>>>>>> MobileWikiAppOnThisDay >>>>>>> MobileWikiAppPageScroll >>>>>>> MobileWikiAppProtectedEditAttempt >>>>>>> MobileWikiAppRandomizer >>>>>>> MobileWikiAppReadingLists >>>>>>> MobileWikiAppSavedPages >>>>>>> MobileWikiAppSearch >>>>>>> MobileWikiAppSessions >>>>>>> MobileWikiAppShareAFact >>>>>>> MobileWikiAppStuffHappens >>>>>>> MobileWikiAppTabs >>>>>>> MobileWikiAppToCInteraction >>>>>>> MobileWikiAppWidgets >>>>>>> MobileWikiAppWiktionaryPopup >>>>>>> MultimediaViewerAttribution >>>>>>> MultimediaViewerDimensions >>>>>>> MultimediaViewerDuration >>>>>>> MultimediaViewerNetworkPerformance >>>>>>> NavigationTiming >>>>>>> PageIssues >>>>>>> Popups >>>>>>> PrefUpdate >>>>>>> Print >>>>>>> QuickSurveyInitiation >>>>>>> QuickSurveysResponses >>>>>>> ReadingDepth >>>>>>> ResourceTiming >>>>>>> SaveTiming >>>>>>> SearchSatisfaction >>>>>>> SearchSatisfactionErrors >>>>>>> ServerSideAccountCreation >>>>>>> TemplateWizard >>>>>>> TestSearchSatisfaction2 >>>>>>> TranslationRecommendationAPIRequests >>>>>>> TranslationRecommendationUIRequests >>>>>>> TranslationRecommendationUserAction >>>>>>> TwoColConflictConflict >>>>>>> UniversalLanguageSelector >>>>>>> UploadWizardErrorFlowEvent >>>>>>> UploadWizardExceptionFlowEvent >>>>>>> UploadWizardFlowEvent >>>>>>> UploadWizardStep >>>>>>> UploadWizardTutorialActions >>>>>>> UploadWizardUploadFlowEvent >>>>>>> VirtualPageView >>>>>>> WikidataCompletionSearchClicks >>>>>>> WikimediaBlogVisit >>>>>>> WikipediaPortal >>>>>>> WikipediaZeroUsage >>>>>>> WMDEBannerEvents >>>>>>> WMDEBannerSizeIssue >>>>>>> _______________________________________________ >>>>>>> Analytics mailing list >>>>>>> Analytics@lists.wikimedia.org >>>>>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>>>>>> >>>>>> _______________________________________________ >>>>>> Analytics mailing list >>>>>> Analytics@lists.wikimedia.org >>>>>> https://lists.wikimedia.org/mailman/listinfo/analytics >>>>>> >>>>> >>>> >>>> -- >>>> Tilman Bayer >>>> Senior Analyst >>>> Wikimedia Foundation >>>> IRC (Freenode): HaeB >>>> >>> >>> >>> -- >>> Tilman Bayer >>> Senior Analyst >>> Wikimedia Foundation >>> IRC (Freenode): HaeB >>> >> _______________________________________________ >> Analytics mailing list >> Analytics@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> > _______________________________________________ > Analytics mailing list > Analytics@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/analytics > -- *Marcel Ruiz Forns* Analytics Developer Wikimedia Foundation
_______________________________________________ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
