[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684077#comment-16684077 ] Hadoop QA commented on OOZIE-3381: -- PreCommit-OOZIE-Build started > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684221#comment-16684221 ] Hadoop QA commented on OOZIE-3381: -- Testing JIRA OOZIE-3381 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:red}-1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any star imports .{color:green}+1{color} the patch does not introduce any line longer than 132 .{color:red}-1{color} the patch does not add/modify any testcase {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} Javadoc generation succeeded with the patch .{color:green}+1{color} the patch does not seem to introduce new Javadoc warning(s) .{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s) {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1{color} There are no new bugs found in total. . {color:green}+1{color} There are no new bugs found in [sharelib/hive2]. . {color:green}+1{color} There are no new bugs found in [sharelib/spark]. . {color:green}+1{color} There are no new bugs found in [sharelib/oozie]. . {color:green}+1{color} There are no new bugs found in [sharelib/pig]. . {color:green}+1{color} There are no new bugs found in [sharelib/streaming]. . {color:green}+1{color} There are no new bugs found in [sharelib/hive]. . {color:green}+1{color} There are no new bugs found in [sharelib/distcp]. . {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. . {color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. . {color:green}+1{color} There are no new bugs found in [sharelib/git]. . {color:green}+1{color} There are no new bugs found in [client]. . {color:green}+1{color} There are no new bugs found in [docs]. . {color:green}+1{color} There are no new bugs found in [tools]. . {color:green}+1{color} There are no new bugs found in [fluent-job/fluent-job-api]. . {color:green}+1{color} There are no new bugs found in [server]. . {color:green}+1{color} There are no new bugs found in [webapp]. . {color:green}+1{color} There are no new bugs found in [examples]. . {color:green}+1{color} There are no new bugs found in [core]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:red}-1 TESTS{color} .Tests run: 3063 .Tests failed : 1 .Tests in error : 0 .Tests timed out : 0 {color:red}-1{color} [ERROR] There are [1] test failures in [core]. Listing only the first [5] ones testRetryConsoleUrlForked:org.apache.oozie.command.wf.TestWorkflowActionRetryInfoXCommand Check console output for the full list of errors/failures .{color:orange}Tests failed at first run:{color} TestPurgeXCommand#testPurgeBundleWithCoordChildWithWFChild3MoreThanLimit TestZKUUIDService#testFallback .For the complete list of flaky tests, see TEST-SUMMARY-FULL files. {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:red}*-1 Overall result, please check the reported -1(s)*{color} {color:red}. There is at least one warning, please check{color} The full output of the test-patch run is available at . https://builds.apache.org/job/PreCommit-OOZIE-Build/915/ > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16686507#comment-16686507 ] Andras Piros commented on OOZIE-3381: - Test failure is unrelated and caused by OOZIE-3384. [~kmarton] can you please review the patch? Thanks! > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687849#comment-16687849 ] Julia Kinga Marton commented on OOZIE-3381: --- [~andras.piros], can you please upload the patch to the Review Board? > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16687910#comment-16687910 ] Andras Piros commented on OOZIE-3381: - [~kmarton] sure thing, [here u go|https://reviews.apache.org/r/69348/]. > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16688195#comment-16688195 ] Julia Kinga Marton commented on OOZIE-3381: --- Thank you [~andras.piros]! I left some comments in the review. Mostly nice to have ones. > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693044#comment-16693044 ] Andras Piros commented on OOZIE-3381: - Thanks for the review [~kmarton]! Updated patch based on review comments. > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693058#comment-16693058 ] Hadoop QA commented on OOZIE-3381: -- PreCommit-OOZIE-Build started > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16693172#comment-16693172 ] Hadoop QA commented on OOZIE-3381: -- Testing JIRA OOZIE-3381 Cleaning local git workspace {color:green}+1 PATCH_APPLIES{color} {color:green}+1 CLEAN{color} {color:green}+1 RAW_PATCH_ANALYSIS{color} .{color:green}+1{color} the patch does not introduce any @author tags .{color:green}+1{color} the patch does not introduce any tabs .{color:green}+1{color} the patch does not introduce any trailing spaces .{color:green}+1{color} the patch does not introduce any star imports .{color:green}+1{color} the patch does not introduce any line longer than 132 .{color:green}+1{color} the patch adds/modifies 2 testcase(s) {color:green}+1 RAT{color} .{color:green}+1{color} the patch does not seem to introduce new RAT warnings {color:green}+1 JAVADOC{color} .{color:green}+1{color} Javadoc generation succeeded with the patch .{color:green}+1{color} the patch does not seem to introduce new Javadoc warning(s) .{color:orange}WARNING{color}: the current HEAD has 100 Javadoc warning(s) {color:green}+1 COMPILE{color} .{color:green}+1{color} HEAD compiles .{color:green}+1{color} patch compiles .{color:green}+1{color} the patch does not seem to introduce new javac warnings {color:green}+1{color} There are no new bugs found in total. . {color:green}+1{color} There are no new bugs found in [sharelib/hive2]. . {color:green}+1{color} There are no new bugs found in [sharelib/spark]. . {color:green}+1{color} There are no new bugs found in [sharelib/oozie]. . {color:green}+1{color} There are no new bugs found in [sharelib/pig]. . {color:green}+1{color} There are no new bugs found in [sharelib/streaming]. . {color:green}+1{color} There are no new bugs found in [sharelib/hive]. . {color:green}+1{color} There are no new bugs found in [sharelib/distcp]. . {color:green}+1{color} There are no new bugs found in [sharelib/hcatalog]. . {color:green}+1{color} There are no new bugs found in [sharelib/sqoop]. . {color:green}+1{color} There are no new bugs found in [sharelib/git]. . {color:green}+1{color} There are no new bugs found in [client]. . {color:green}+1{color} There are no new bugs found in [docs]. . {color:green}+1{color} There are no new bugs found in [tools]. . {color:green}+1{color} There are no new bugs found in [fluent-job/fluent-job-api]. . {color:green}+1{color} There are no new bugs found in [server]. . {color:green}+1{color} There are no new bugs found in [webapp]. . {color:green}+1{color} There are no new bugs found in [examples]. . {color:green}+1{color} There are no new bugs found in [core]. {color:green}+1 BACKWARDS_COMPATIBILITY{color} .{color:green}+1{color} the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations .{color:green}+1{color} the patch does not modify JPA files {color:green}+1 TESTS{color} .Tests run: 3075 .{color:orange}Tests failed at first run:{color} TestPurgeXCommand#testPurgeCoordWithWFChild2 .For the complete list of flaky tests, see TEST-SUMMARY-FULL files. {color:green}+1 DISTRO{color} .{color:green}+1{color} distro tarball builds with the patch {color:green}*+1 Overall result, good!, no -1s*{color} {color:red}. There is at least one warning, please check{color} The full output of the test-patch run is available at . https://builds.apache.org/job/PreCommit-OOZIE-Build/918/ > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695758#comment-16695758 ] Julia Kinga Marton commented on OOZIE-3381: --- Thank you [~andras.piros] for the patch! +1 > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3381) [coordinator] Enhance logging of CoordElFunctions
[ https://issues.apache.org/jira/browse/OOZIE-3381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695761#comment-16695761 ] Julia Kinga Marton commented on OOZIE-3381: --- committed to master. > [coordinator] Enhance logging of CoordElFunctions > - > > Key: OOZIE-3381 > URL: https://issues.apache.org/jira/browse/OOZIE-3381 > Project: Oozie > Issue Type: Task > Components: coordinator >Affects Versions: 5.1.0 >Reporter: Andras Piros >Assignee: Andras Piros >Priority: Major > Attachments: OOZIE-3381.001.patch, OOZIE-3381.002.patch > > > When using [{{$\{coord:latest\(n\)}}} coordinator EL > function|https://oozie.apache.org/docs/5.0.0/CoordinatorFunctionalSpec.html#a6.6.6._coord:latestint_n_EL_Function_for_Synchronous_Datasets] > inside an input dataset dependency, it's often the case that more > information is needed how many HDFS URIs are being checked for each > {{}}. > Right now we don't have this information. While debugging and fine tuning > parameters like {{dataset frequency}}, {{initial-instance}}, and {{data-in}} > {{instance}}, it would be very useful to know how many HDFS roundtrips are > issues by the current settings {{CoordELFunctions#coord_latestRange_sync()}} > and {{CoordELFunctions#coord_futureRange_sync()}} having called > {{DFSClient#exists()}}. We need appropriate logging there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)