UmeshPatil-1 opened a new pull request, #490: URL: https://github.com/apache/atlas/pull/490
**ATLAS-4988** **What changes were proposed in this pull request?** Background: In Apache Atlas, deleting a Business Metadata (BM) definition requires a pre-check to ensure no existing entities are assigned to those attributes. For attributes with a data type of Array (multi-valued), the deletion process was taking an exceptionally long time (e.g., over 5 minutes for -38k entities), even when the BM was not assigned to any entity. **Root Cause:** 1. Missing Solr Index: Attributes of type Array are generally not indexed in Solr. 2. Inefficient Search: The old logic forced a Solr-based search (entityDiscoveryService) for all attributes. When an attribute is not indexed, Solr performs a full collection scan, leading to severe performance degradation. 3. Data Payload: The search request was fetching full attribute values (setAttributes), adding unnecessary I/O overhead for a simple existence check. **Changes Proposed:** 1. Hybrid Validation Strategy: Introduced a conditional check to determine if an attribute is indexable before deciding the search path. 2. Optimized Indexed Search: For indexable attributes, the Solr search was optimized by using the NOT_NULL operator and clearing requested attributes (Collections.emptySet()) to return only the GUID, significantly speeding up the response. 3. Direct Graph Fallback: Introduced isBusinessAttributePresentInGraph() for non-indexable attributes. This method queries the Graph database (JanusGraph) directly. 4. Targeted Lookups: The graph fallback uses Constants.ENTITY_TYPE_PROPERTY_KEY to narrow the scope to only relevant entity types, avoiding full system scans. 5. Integrity Maintenance: Ensured that even unindexed attributes are checked before deletion to prevent "dangling" metadata, while maintaining sub-second performance. **Impact**: Performance Gain: Deletion time for Business Metadata with Array attributes reduced from -341 seconds to -1.5 second. Reliability: Maintains strict data integrity by ensuring no "in-use" Business Metadata can be deleted. Scalability: The fix ensures that as the number of entities grows, the deletion of metadata remains performant. **How was this patch tested?** Maven Build: Build Successful. **Manual Testing:** 1. Creation & Deletion (No Assignment): Created Business Metadata with String and Array<String> types. Verified deletion is nearly instantaneous (-0.8s - 1.5s). 2. Deletion Blocked (With Assignment): * Assigned a Business Metadata attribute to a hive_table entity. 3. Attempted to delete the BM definition. 4. Verified the system correctly throws ATLAS-409-00-002: Given type has references. 5. Large Dataset Validation: Tested against a repository containing -38,000 hive_table entities to confirm the Solr timeout and latency issue is resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
