[
https://issues.apache.org/jira/browse/ATLAS-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Disha Talreja updated ATLAS-4922:
---------------------------------
Summary: Atlas Async Import Functionality using Kafka (was: Atlas Parallel
Import Functionality using Kafka)
> Atlas Async Import Functionality using Kafka
> --------------------------------------------
>
> Key: ATLAS-4922
> URL: https://issues.apache.org/jira/browse/ATLAS-4922
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
> Priority: Major
>
> Develop an import process flow in Apache Atlas that leverages Kafka to manage
> parallel import requests. This solution will use a dedicated Kafka topic
> (ATLAS_IMPORT) to ensure import requests are processed efficiently and
> without conflicts. The ImportService will be enhanced to consume and process
> each message from the topic, enabling seamless behind-the-scenes import
> execution.
> Users will be able to submit new import requests after the staging process,
> which prepares the file stream for Kafka, is complete. This approach supports
> concurrent imports and enhances the handling of import operations in Apache
> Atlas.
> Additionally, this feature will introduce two new API endpoints for tracking
> the current status of imports. These endpoints will allow users to check both
> the overall progress and detailed status of each import request at any stage
> of the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using
> the unique request ID. Users will also gain access to request metadata,
> providing insights into the status and progress of their imports.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
