[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850658#comment-16850658 ] Andras Salamon commented on OOZIE-3393: --- Thanks for the fix [~zuston], +1, committed the amend patch to master. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Fix For: 5.2.0 > > Attachments: amend-oozie-3393-1.patch, oozie-3393-1.patch, > oozie-3393-2.patch, oozie-3393-3.patch, oozie-3393-4.patch, > oozie-3393-5.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849753#comment-16849753
]
Hadoop QA commented on OOZIE-3393:
--
Testing JIRA OOZIE-3393
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [5] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:red}-1{color} There are [5] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [tools].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3170
.{color:orange}Tests failed at first run:{color}
TestCoordActionsKillXCommand#testActionKillCommandActionNumbers
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1126/
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: amend-oozie-3393-1.patch, oozie-3393-1.patch,
> oozie-3393-2.patch, oozie-3393-3.patch, oozie-3393-4.patch,
>
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849653#comment-16849653 ] Hadoop QA commented on OOZIE-3393: -- PreCommit-OOZIE-Build started > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Fix For: 5.2.0 > > Attachments: amend-oozie-3393-1.patch, oozie-3393-1.patch, > oozie-3393-2.patch, oozie-3393-3.patch, oozie-3393-4.patch, > oozie-3393-5.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849416#comment-16849416
]
Junfan Zhang commented on OOZIE-3393:
-
Sorry, i write a bug. Because {{GET_COORD_JOBS_OLDER_FOR_MATERIALIZATION}} only
get the id to bean, the metric of mat_delayed_size is always zero.
Now i fix it. Please review the patch [~asalamon74]
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Fix For: 5.2.0
>
> Attachments: amend-oozie-3393-1.patch, oozie-3393-1.patch,
> oozie-3393-2.patch, oozie-3393-3.patch, oozie-3393-4.patch,
> oozie-3393-5.patch, oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832410#comment-16832410 ] Andras Salamon commented on OOZIE-3393: --- Thanks for the contribution [~zuston], +1, committed to master. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-3393-5.patch, > oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832403#comment-16832403 ] Junfan Zhang commented on OOZIE-3393: - [~asalamon74] oozie-3393-5.patch is ok. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-3393-5.patch, > oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832241#comment-16832241
]
Hadoop QA commented on OOZIE-3393:
--
Testing JIRA OOZIE-3393
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [4] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:red}-1{color} There are [4] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3170
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1102/
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-3393-3.patch, oozie-3393-4.patch, oozie-3393-5.patch,
> oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832163#comment-16832163 ] Hadoop QA commented on OOZIE-3393: -- PreCommit-OOZIE-Build started > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-3393-5.patch, > oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831506#comment-16831506
]
Andras Salamon commented on OOZIE-3393:
---
[~zuston] Yes, this test failure really looks unrelated.
Just one last request: Could could please add assert message to the
{{assertNotNull}} calls?
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830078#comment-16830078 ] Junfan Zhang commented on OOZIE-3393: - Could you please review it [~asalamon74], thanks. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820955#comment-16820955 ] Junfan Zhang commented on OOZIE-3393: - [~asalamon74] This test case looks unrelated. :( > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820947#comment-16820947
]
Hadoop QA commented on OOZIE-3393:
--
Testing JIRA OOZIE-3393
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [4] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:red}-1{color} There are [4] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3170
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeableBundleUnpurgeableCoordinatorUnpurgebleWorkflowPurgeableSubWorkflow
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1094/
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log
>
>
> We
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820848#comment-16820848 ] Hadoop QA commented on OOZIE-3393: -- PreCommit-OOZIE-Build started > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820774#comment-16820774
]
Hadoop QA commented on OOZIE-3393:
--
Testing JIRA OOZIE-3393
Cleaning local git workspace
{color:red}-1{color} Patch failed to apply to head of branch
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-3393-3.patch, oozie-3393-4.patch, oozie-3393-4.patch,
> oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820768#comment-16820768 ] Hadoop QA commented on OOZIE-3393: -- PreCommit-OOZIE-Build started > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-3393-4.patch, > oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820425#comment-16820425
]
Hadoop QA commented on OOZIE-3393:
--
Testing JIRA OOZIE-3393
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:red}-1{color} the patch contains 1 star import(s)
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [4] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:red}-1{color} There are [4] new bugs found below threshold in
[core] that must be fixed.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3170
.{color:orange}Tests failed at first run:{color}
TestPurgeXCommand#testPurgeableWorkflowPurgeableSubWorkflowPurgeableSubSubWorkflow
TestWorkflowRetries>TestWorkflow#testWorkflowWithStartAndEndCompletesSuccessfully
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1092/
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-3393-3.patch,
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820328#comment-16820328 ] Hadoop QA commented on OOZIE-3393: -- PreCommit-OOZIE-Build started > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820324#comment-16820324
]
Junfan Zhang commented on OOZIE-3393:
-
[~asalamon74] Sorry, after recheck the test case, i have uploaded patch-4. In
order to use the {{MetricsInstrumentationService.getMetricRegistry}} in the
{{TestCoordMaterializeTriggerService.testCoordMaterializeTriggerService4}}, I
have to change the scope of the {{getMetricRegistry}} method to public.
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-3393-3.patch, oozie-3393-4.patch, oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820083#comment-16820083 ] Andras Salamon commented on OOZIE-3393: --- [~zuston] For me, the test error looks related, your new test case fails. Could you please check it. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820067#comment-16820067 ] Junfan Zhang commented on OOZIE-3393: - Test error looks unrelated. [~asalamon74] > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817854#comment-16817854
]
Hadoop QA commented on OOZIE-3393:
--
Testing JIRA OOZIE-3393
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [7] new bugs found below threshold in total that
must be fixed.
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [tools].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:red}-1{color} There are [7] new bugs found below threshold in
[core] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
core/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At BulkJPAExecutor.java:[line 207]: This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection
.At BulkJPAExecutor.java:[line 177]: At BulkJPAExecutor.java:[line 176]
.At BulkJPAExecutor.java:[line 206]: At BulkJPAExecutor.java:[line 200]
.This use of
javax/persistence/EntityManager.createQuery(Ljava/lang/String;)Ljavax/persistence/Query;
can be vulnerable to SQL/JPQL injection: At BulkJPAExecutor.java:[line 207]
.At BulkJPAExecutor.java:[line 112]: At BulkJPAExecutor.java:[line 128]
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 3170
.Tests failed : 0
.Tests in error : 1
.Tests timed out : 0
{color:red}-1{color} [ERROR] There are [1] test errors in [core]. Listing only
the first [5] ones
testCoordMaterializeTriggerService4:org.apache.oozie.service.TestCoordMaterializeTriggerService
Check console output for the full list of errors/failures
.{color:orange}Tests failed at first run:{color}
TestWorkflowRetries>TestWorkflow#testWorkflowWithStartAndEndCompletesSuccessfully
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1090/
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL:
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817664#comment-16817664 ] Hadoop QA commented on OOZIE-3393: -- PreCommit-OOZIE-Build started > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16817643#comment-16817643 ] Junfan Zhang commented on OOZIE-3393: - Thanks [~andras.piros]. Patch 003 has uploaded to review board, the link is [https://reviews.apache.org/r/69469]. I am looking forward to your comment. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement > Components: coordinator >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch, oozie-3393-2.patch, > oozie-3393-3.patch, oozie-instrumentation-1.log > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16711540#comment-16711540
]
Andras Piros commented on OOZIE-3393:
-
[~zuston] you're welcome. Patch 002 looks good to me thus far. One remaining
point being unit testing.
I'd create new test cases in {{TestCoordMaterializeTriggerService}} with
following semantics:
* setting things up and waiting till a task gets into
{{CoordinatorJob.Status.RUNNING}} [like
this|https://github.com/apache/oozie/blob/master/core/src/test/java/org/apache/oozie/service/TestCoordMaterializeTriggerService.java#L77-L84]
* asserting that the service was instrumented by accessing
{{Services.get().get(InstrumentationService).get().getVariables()}}
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
> Components: coordinator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710260#comment-16710260
]
Junfan Zhang commented on OOZIE-3393:
-
Thanks [~andras.piros]. I have submitted the {{oozie-3379-2.patch}} and
{{oozie-instrumentation.log}} snippets. In addition, I have uploaded the patch
in review board.
But I don't know how to add unit tests, because the
{{TestMetricsInstrumentation}} and {{TestInstrumentation}} classes are all
general method, not to test specific metrics. If you know, please tell me.
Thanks :D
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch, oozie-3393-2.patch,
> oozie-instrumentation-1.log
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[
https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702003#comment-16702003
]
Andras Piros commented on OOZIE-3393:
-
Thanks [~zuston]. Overall direction is OK. Please add some unit tests, and
sample {{oozie-instrumentation.log}} snippets.
> Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
> --
>
> Key: OOZIE-3393
> URL: https://issues.apache.org/jira/browse/OOZIE-3393
> Project: Oozie
> Issue Type: Improvement
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Attachments: oozie-3393-1.patch
>
>
> We have multiple Oozie scheduled clusters. The problem that is currently
> encountered is that some materialize queues are too late to process during
> the peak hours of the scheduled tasks. We need to get the metrics of the
> delay queue for alarm monitoring and adjust the configuration to optimize.
> Currently we have added metrics for queue size and latency in Oozie
> CoordMaterializeTriggerService class, in order to reflect the general trend
> of current scheduling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701775#comment-16701775 ] Junfan Zhang commented on OOZIE-3393: - Thanks [~andras.piros]. The review board link is [https://reviews.apache.org/r/69469]. > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (OOZIE-3393) Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService
[ https://issues.apache.org/jira/browse/OOZIE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701763#comment-16701763 ] Andras Piros commented on OOZIE-3393: - Thanks for the contribution [~zuston]! Can you please upload the patch to ReviewBoard? Thanks! > Add Oozie instrumentation delayed metric in CoordMaterializeTriggerService > -- > > Key: OOZIE-3393 > URL: https://issues.apache.org/jira/browse/OOZIE-3393 > Project: Oozie > Issue Type: Improvement >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Attachments: oozie-3393-1.patch > > > We have multiple Oozie scheduled clusters. The problem that is currently > encountered is that some materialize queues are too late to process during > the peak hours of the scheduled tasks. We need to get the metrics of the > delay queue for alarm monitoring and adjust the configuration to optimize. > Currently we have added metrics for queue size and latency in Oozie > CoordMaterializeTriggerService class, in order to reflect the general trend > of current scheduling. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
