[
https://issues.apache.org/jira/browse/ATLAS-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17953563#comment-17953563
]
ASF subversion and git services commented on ATLAS-4922:
--------------------------------------------------------
Commit d776a87da8d0d14a3816c6d7e7cceccab76dabea in atlas's branch
refs/heads/atlas-2.5 from Disha Talreja
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=d776a87da ]
ATLAS-4922: Atlas Async Import using Kafka (#307)
Co-authored-by: jackhalfalltrades <[email protected]>
(cherry picked from commit f3d302a8025289e7c71e9cae88a018005a6d9b72)
> Atlas Async Import using Kafka
> ------------------------------
>
> Key: ATLAS-4922
> URL: https://issues.apache.org/jira/browse/ATLAS-4922
> Project: Atlas
> Issue Type: New Feature
> Components: atlas-core
> Reporter: Disha Talreja
> Assignee: Disha Talreja
> Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> Develop an import process flow in Apache Atlas that leverages Kafka to manage
> parallel import requests. This solution will use a dedicated Kafka topic
> (ATLAS_IMPORT) to ensure import requests are processed efficiently and
> without conflicts. The ImportService will be enhanced to consume and process
> each message from the topic, enabling seamless behind-the-scenes import
> execution.
> Users will be able to submit new import requests after the staging process,
> which prepares the file stream for Kafka, is complete. This approach supports
> concurrent imports and enhances the handling of import operations in Apache
> Atlas.
> Additionally, this feature will introduce two new API endpoints for tracking
> the current status of imports. These endpoints will allow users to check both
> the overall progress and detailed status of each import request at any stage
> of the import lifecycle (e.g., "PROCESSING," "PROCESSED," "COMPLETE") using
> the unique request ID. Users will also gain access to request metadata,
> providing insights into the status and progress of their imports.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
