[jira] [Comment Edited] (SLING-9389) Distribution Event Packages should contain queue item creation time
[ https://issues.apache.org/jira/browse/SLING-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125354#comment-17125354 ] Timothee Maret edited comment on SLING-9389 at 6/3/20, 10:01 PM: - bq. The idea is to capture a metric (using Sling Metrics) exposing the end to end distribution behavior (AEM to BP in this context). [~harshchiki] I think we should do exactly what your requirement describes: add Sling metrics in the implementations to monitor end to end latency. I think that we could achieve that without extending the event API. It's possible to dissociate metrics by agent, use case, etc. See for instance the [Bookeeper|https://github.com/apache/sling-org-apache-sling-distribution-journal/blob/4190c2befc6da8277e12ba019b8c41087cb217e1/src/main/java/org/apache/sling/distribution/journal/impl/subscriber/BookKeeper.java#L114-L115] which emits metrics by subscriber agent. bq. the information would not come down to the consumers of SCD apparently. It would come via the Sling metrics rather than the SCD API. Would that be satisfactory for your use case ? was (Author: marett): bq. The idea is to capture a metric (using Sling Metrics) exposing the end to end distribution behavior (AEM to BP in this context). [~harshchiki] I think we should do exactly what your requirement describes: add Sling metrics in the implementations to monitor end to end latency. I think that we could achieve that without extending the event API. It's possible to dissociate metrics by agent, use case, etc. See for instance the [Bookeeper|https://github.com/apache/sling-org-apache-sling-distribution-journal/blob/4190c2befc6da8277e12ba019b8c41087cb217e1/src/main/java/org/apache/sling/distribution/journal/impl/subscriber/BookKeeper.java#L114-L115] which emits metrics by subscriber agent. bq. the information would not come down to the consumers of SCD apparently. It would come via the Sling metrics rather than the SCD API. Would that be satisfactory for your use case ? > Distribution Event Packages should contain queue item creation time > --- > > Key: SLING-9389 > URL: https://issues.apache.org/jira/browse/SLING-9389 > Project: Sling > Issue Type: Improvement > Components: Content Distribution >Reporter: Harsh Chiki >Assignee: Timothee Maret >Priority: Major > Attachments: image-2020-04-30-10-28-58-011.png, scdapi.patch, > scdcore.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Currently the Distribution Event package contains the following details: > * Distribution Component Name > * Distribution Component Kind > * Distribution Type > * Distribution Paths > > Improvement aims at adding the queue item creation time, essentially when the > the item was creation for the first time, and enqueue into the queue. The > value does not change over retries (on failure). > > The purpose to get this detail is to be able to capture metrics at the > consumer level. The consumers could have an event handler, which can capture > the duration which turns out to be (NOW MINUS queue item creation time thrown > in the distribution event package); NOW being the current time in the event > handler (consumer). > > \cc: [~shgu...@adobe.com], [~ashishc] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (SLING-9389) Distribution Event Packages should contain queue item creation time
[ https://issues.apache.org/jira/browse/SLING-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096378#comment-17096378 ] Harsh Chiki edited comment on SLING-9389 at 5/27/20, 5:42 AM: -- Hi [~marett], [~ashishc], I have raised the PRs: * [-https://github.com/apache/sling-org-apache-sling-distribution-api/pull/2-] * [https://github.com/apache/sling-org-apache-sling-distribution-api/pull/7] * [https://github.com/apache/sling-org-apache-sling-distribution-core/pull/41] Kindly review and share your feedback. Thanks _edit_: Updated PR links above. \cc: [~shgu...@adobe.com] was (Author: harshchiki): Hi [~marett], [~ashishc], I have raised the PRs: * [https://github.com/apache/sling-org-apache-sling-distribution-api/pull/2] * [https://github.com/apache/sling-org-apache-sling-distribution-core/pull/41] Kindly review and share your feedback. Thanks \cc: [~shgu...@adobe.com] > Distribution Event Packages should contain queue item creation time > --- > > Key: SLING-9389 > URL: https://issues.apache.org/jira/browse/SLING-9389 > Project: Sling > Issue Type: Improvement > Components: Content Distribution >Reporter: Harsh Chiki >Assignee: Timothee Maret >Priority: Major > Attachments: image-2020-04-30-10-28-58-011.png, scdapi.patch, > scdcore.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently the Distribution Event package contains the following details: > * Distribution Component Name > * Distribution Component Kind > * Distribution Type > * Distribution Paths > > Improvement aims at adding the queue item creation time, essentially when the > the item was creation for the first time, and enqueue into the queue. The > value does not change over retries (on failure). > > The purpose to get this detail is to be able to capture metrics at the > consumer level. The consumers could have an event handler, which can capture > the duration which turns out to be (NOW MINUS queue item creation time thrown > in the distribution event package); NOW being the current time in the event > handler (consumer). > > \cc: [~shgu...@adobe.com], [~ashishc] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (SLING-9389) Distribution Event Packages should contain queue item creation time
[ https://issues.apache.org/jira/browse/SLING-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106286#comment-17106286 ] Timothee Maret edited comment on SLING-9389 at 5/13/20, 1:33 PM: - [~harshchiki] thanks I looked at it. Is it correct that your requirement is to measure the end to end distribution latency for each package on the source service (author) ? If so, could you also clarify if it's a requirement for you to have the information in the event, or if you could not have the information exposed as a metric. was (Author: marett): [~harshchiki] thanks I looked at it. Is it correct that your requirement is to measure the end to end distribution latency for each package on the source service (author) ? > Distribution Event Packages should contain queue item creation time > --- > > Key: SLING-9389 > URL: https://issues.apache.org/jira/browse/SLING-9389 > Project: Sling > Issue Type: Improvement > Components: Content Distribution >Reporter: Harsh Chiki >Assignee: Timothee Maret >Priority: Major > Attachments: image-2020-04-30-10-28-58-011.png, scdapi.patch, > scdcore.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Currently the Distribution Event package contains the following details: > * Distribution Component Name > * Distribution Component Kind > * Distribution Type > * Distribution Paths > > Improvement aims at adding the queue item creation time, essentially when the > the item was creation for the first time, and enqueue into the queue. The > value does not change over retries (on failure). > > The purpose to get this detail is to be able to capture metrics at the > consumer level. The consumers could have an event handler, which can capture > the duration which turns out to be (NOW MINUS queue item creation time thrown > in the distribution event package); NOW being the current time in the event > handler (consumer). > > \cc: [~shgu...@adobe.com], [~ashishc] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (SLING-9389) Distribution Event Packages should contain queue item creation time
[
https://issues.apache.org/jira/browse/SLING-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093799#comment-17093799
]
Harsh Chiki edited comment on SLING-9389 at 4/30/20, 4:59 AM:
--
Hi [~marett],
Please consider this a gentle reminder.
I worked out some basic changes to capture the queue item creation time, to be
put in the {{AGENT_PACKAGE_DISTRIBUTED}} and {{AGENT_PACKAGE_DROPPED}}. From
other paths, have sent out null. PFA SCD API and Core patches which of course
are very rough at the moment.
May I request your views, in general from this requirement pov, essentially:
* Whether we should add a new parameter in this internal API and internal
implementation, or a new method to capture time stamp as well. (Prior one is
done in the patch shared)
* Is it fine to send null in the other event topic cases?
* For scheduled import export cases.
* In our local experiments, we identified the queue item creation time stamp
is what one would need from an enqueue (first time creation, not on retries)
time stamp pov, so we intend to use
[https://github.com/apache/sling-org-apache-sling-distribution-core/blob/429f6bc225970c34e948d0ad41c38fe4c0ef9ef9/src/main/java/org/apache/sling/distribution/queue/impl/resource/ResourceQueueUtils.java#L219]
** Figured that
[https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/QueueingDistributionPackageProcessor.java#L119]
which is later used for getting {{globalStartTime}} here -
[https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/SimpleDistributionAgentQueueProcessor.java#L115.|https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/SimpleDistributionAgentQueueProcessor.java#L115]
** But, relied on the queue item creation time - generated in
`ResourceQueueUtils` linked above (in this comment)
FTR, the event thrown looks like (with the attached patches) - the newly added
`distribution.enqueue.timestamp`
!image-2020-04-30-10-28-58-011.png!
Kindly have a look, and advise.
Thanks!
\cc: [~ashishc], [~shgu...@adobe.com]
was (Author: harshchiki):
Hi [~marett],
Please consider this a gentle reminder.
I worked out some basic changes to capture the queue item creation time, to be
put in the {{AGENT_PACKAGE_DISTRIBUTED}} and {{AGENT_PACKAGE_DROPPED}}. From
other paths, have sent out null. PFA SCD API and Core patches which of course
are very rough at the moment.
May I request your views, in general from this requirement pov, essentially:
* Whether we should add a new parameter in this internal API and internal
implementation, or a new method to capture time stamp as well. (Prior one is
done in the patch shared)
* Is it fine to send null in the other event topic cases?
* For scheduled import export cases.
* In our local experiments, we identified the queue item creation time stamp
is what one would need from an enqueue (first time creation, not on retries)
time stamp pov, so we intend to use
[https://github.com/apache/sling-org-apache-sling-distribution-core/blob/429f6bc225970c34e948d0ad41c38fe4c0ef9ef9/src/main/java/org/apache/sling/distribution/queue/impl/resource/ResourceQueueUtils.java#L219]
** Figured that
[https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/QueueingDistributionPackageProcessor.java#L119]
which is later used for getting {{globalStartTime}} here -
[https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/SimpleDistributionAgentQueueProcessor.java#L115.|https://github.com/apache/sling-org-apache-sling-distribution-core/blob/master/src/main/java/org/apache/sling/distribution/agent/impl/SimpleDistributionAgentQueueProcessor.java#L115]
** But, relied on the queue item creation time - generated in
`ResourceQueueUtils` linked above (in this comment)
Kindly have a look, and advise.
Thanks!
\cc: [~ashishc], [~shgu...@adobe.com]
> Distribution Event Packages should contain queue item creation time
> ---
>
> Key: SLING-9389
> URL: https://issues.apache.org/jira/browse/SLING-9389
> Project: Sling
> Issue Type: Improvement
> Components: Content Distribution
>Reporter: Harsh Chiki
>Assignee: Timothee Maret
>Priority: Major
> Attachments: image-2020-04-30-10-28-58-011.png, scdapi.patch,
> scdcore.patch
>
>
> Currently the Dsitribution Event package contains the following details:
> * Distribution Component Name
> *
[jira] [Comment Edited] (SLING-9389) Distribution Event Packages should contain queue item creation time
[ https://issues.apache.org/jira/browse/SLING-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091219#comment-17091219 ] Harsh Chiki edited comment on SLING-9389 at 4/24/20, 5:54 AM: -- Hi [~marett], atm (and so for the scope of this improvement) we need it on the source service (author). This is in conjunction to the AEM on BP publish cloud service, where the purpose is to effectively capture the round trip time of the distribution from AEM (author - source service) to BP (destination service). was (Author: harshchiki): Hi [~marett], atm we need it on the source service (author). This is in conjunction to the AEM on BP publish cloud service, where the purpose is to effectively capture the round trip time of the distribution from AEM (author - source service) to BP (destination service). > Distribution Event Packages should contain queue item creation time > --- > > Key: SLING-9389 > URL: https://issues.apache.org/jira/browse/SLING-9389 > Project: Sling > Issue Type: Improvement > Components: Content Distribution >Reporter: Harsh Chiki >Assignee: Timothee Maret >Priority: Major > > Currently the Dsitribution Event package contains the following details: > * Distribution Component Name > * Distribution Component Kind > * Distribution Type > * Distribution Paths > > Improvement aims at adding the queue item creation time, essentially when the > the item was creation for the first time, and enqueue into the queue. The > value does not change over retries (on failure). > > The purpose to get this detail is to be able to capture metrics at the > consumer level. The consumers could have an event handler, which can capture > the duration which turns out to be (NOW MINUS queue item creation time thrown > in the distribution event package); NOW being the current time in the event > handler (consumer). > > \cc: [~shgu...@adobe.com], [~ashishc] -- This message was sent by Atlassian Jira (v8.3.4#803005)
