Disha Talreja created ATLAS-4922:
------------------------------------
Summary: Atlas Parallel Import Functionality using Kafka
Key: ATLAS-4922
URL: https://issues.apache.org/jira/browse/ATLAS-4922
Project: Atlas
Issue Type: New Feature
Components: atlas-core
Reporter: Disha Talreja
Assignee: Disha Talreja
Develop an import process flow in Apache Atlas that leverages Kafka to manage
parallel import requests. This solution will use a dedicated Kafka topic
(ATLAS_IMPORT) to ensure import requests are processed efficiently and without
conflicts. The ImportService will be enhanced to consume and process each
message from the topic, enabling seamless behind-the-scenes import execution.
Users will be able to submit new import requests after the staging process,
which prepares the file stream for Kafka, is complete. This approach supports
concurrent imports and enhances the handling of import operations in Apache
Atlas.
Additionally, this feature will introduce two new API endpoints for tracking
the current status of imports. These endpoints will allow users to check both
the overall progress and detailed status of each import request at any stage of
the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using the
unique request ID. Users will also gain access to request metadata, providing
insights into the status and progress of their imports.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
