[
https://issues.apache.org/jira/browse/ATLAS-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Mestry updated ATLAS-3641:
-----------------------------------
Attachment: ATLAS-3641-Import-Service-Support-zipDirect-format-o.patch
> Import Service: Support New 'zipDirect' Format for Import
> ---------------------------------------------------------
>
> Key: ATLAS-3641
> URL: https://issues.apache.org/jira/browse/ATLAS-3641
> Project: Atlas
> Issue Type: Improvement
> Components: atlas-core
> Affects Versions: trunk
> Reporter: Ashutosh Mestry
> Assignee: Ashutosh Mestry
> Priority: Major
> Fix For: trunk
>
> Attachments:
> ATLAS-3641-Import-Service-Support-zipDirect-format-o.patch
>
>
> *Background*
> Existing exported ZIP formats stores the entities and the order of import in
> 2 separate files. During import it is thus necessary for both these (entities
> and order) to be in memory to be able to correctly import.
> This causes large import files not function well. Also it puts a lot of
> pressure on memory.
> Existing format also uses 1 zip entry per entity. This does not effectively
> compress the files.
> The new format addresses these short comings.
> *Solution*
> Create a format that has:
> * Entities are stored in the order such that they are correctly imported.
> * Lends itself to be able to read as a stream. This will reduce memory
> constraints.
> * Store entities as 1 zip entry. This will improve compression rate.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
