[
https://issues.apache.org/jira/browse/ATLAS-4787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szymon Orzechowski updated ATLAS-4787:
--------------------------------------
Description:
As of now, current code of MetricsService.purgeMetricsStats():
# retrieves all metric stats (getAllMetricsStats(true))
# filters out metrics to delete
# iteraters over each entry (deleteMetricsStatByCollectionTime)
## queries metric stat by timestamp
## deletes metric stat by guid
This however is an n+1 error because at step 1. we already have the necessary
guids. At step 3. Atlas could either do a batch delete or delete by guid right
off the bat.
Deleting by collectionTime also causes an issue if atlas.cluster.name was
changed after some time, because it would try to look for obsolete metric stat
with new cluster name, which results with
{code:java}
org.apache.atlas.exception.AtlasBaseException: Instance __AtlasMetricsStat with
unique attribute {metricsId=atlas_metrics_1671346802778@newclustername} does
not exist
{code}
and rollbacks the entire transaction.
was:
As of now, current code of MetricsService.purgeMetricsStats() does:
# Retrieve all metric stats (getAllMetricsStats(true))
# filters out metrics to delete
# iteraters over each entry (deleteMetricsStatByCollectionTime)
## queries metric stat by timestamp
## deletes metric stat by guid
This however is an n+1 error because at step 1. we already have the necessary
guids. At step 3. Atlas could either do a batch delete or delete by guid right
off the bat.
Deleting by collectionTime also causes an issue if atlas.cluster.name was
changed after some time, because it would try to look for obsolete metric stat
with new cluster name, which results with
{code:java}
org.apache.atlas.exception.AtlasBaseException: Instance __AtlasMetricsStat with
unique attribute {metricsId=atlas_metrics_1671346802778@newclustername} does
not exist
{code}
and rollbacks the entire transaction.
> MetricsService.purgeMetricsStats() does unnecessary queries to graph backend
> ----------------------------------------------------------------------------
>
> Key: ATLAS-4787
> URL: https://issues.apache.org/jira/browse/ATLAS-4787
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
> Affects Versions: 3.0.0
> Reporter: Szymon Orzechowski
> Priority: Major
>
> As of now, current code of MetricsService.purgeMetricsStats():
> # retrieves all metric stats (getAllMetricsStats(true))
> # filters out metrics to delete
> # iteraters over each entry (deleteMetricsStatByCollectionTime)
> ## queries metric stat by timestamp
> ## deletes metric stat by guid
> This however is an n+1 error because at step 1. we already have the necessary
> guids. At step 3. Atlas could either do a batch delete or delete by guid
> right off the bat.
>
> Deleting by collectionTime also causes an issue if atlas.cluster.name was
> changed after some time, because it would try to look for obsolete metric
> stat with new cluster name, which results with
> {code:java}
> org.apache.atlas.exception.AtlasBaseException: Instance __AtlasMetricsStat
> with unique attribute {metricsId=atlas_metrics_1671346802778@newclustername}
> does not exist
> {code}
> and rollbacks the entire transaction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
