[
https://issues.apache.org/jira/browse/ATLAS-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chaitali borole reassigned ATLAS-4984:
--------------------------------------
Assignee: chaitali borole
> Option to ignore spark_process attributes details and sparkPlanDescription
> ---------------------------------------------------------------------------
>
> Key: ATLAS-4984
> URL: https://issues.apache.org/jira/browse/ATLAS-4984
> Project: Atlas
> Issue Type: Improvement
> Components: spark-integration
> Reporter: Madhan Neethiraj
> Assignee: chaitali borole
> Priority: Major
>
> {{spark_process}} entity attributes {{details}} and {{sparkPlanDescription}}
> are populated with query plan details, which can contain a large amount of
> text, often in megabytes. In environments that don't require these
> attributes, it will help to ignore these attributes while processing
> create/update of {{spark_process}} entities. I suggest Atlas server
> notification processor to support following configurations:
> -
> {{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.sparkPlanDescription}}
> -
> {{atlas.notification.consumer.preprocess.spark_process.ignore-attribute.details}}
> A deployment can choose to ignore these attributes using above configurations
> in Atlas server.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
