----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56959/ -----------------------------------------------------------
(Updated May 30, 2017, 7:50 p.m.) Review request for atlas and Madhan Neethiraj. Changes ------- Updates: - Addressed review comments. - Renamed unit test suite to reflect functionality. Bugs: ATLAS-1825 https://issues.apache.org/jira/browse/ATLAS-1825 Repository: atlas Description ------- Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios and solution approach. Impact ====== * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This holds a map of entity-type to attribute and the tranforms to be applied. * New class _AtlasImportTransformBuiler_ reads the requests specified in the query string of the import API and creates a _AtlasImportTransfor_ structure. * _AtlasImportTransformBuiler_ has new classes that support transformations. They are: * _replace_ Implements find and replace on attribute values. * _lowercase_ Converts attribute value to lower case. * Additional classes that unit test these classes have been added. Implementation Approach ======================= * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the transform every entity that is fetched from the source. * _Import API_ now accepts multi-part input. This allows for elegantly specifying import parameters as well as the binary input from the file. CURL ==== *Setup* Create file with these contents call it _importTransform.json_: ```javascript { "options": { "transforms": { "hive_db": { "qualifiedName": [ "replace:@cl1:@cl2" ] }, "hive_table": { "qualifiedName": [ "lowercase", "replace:@cl1:@cl2" ] } } } } ``` ``` curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import" ``` Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not required anymore. Notice the output from the API, it has the _AtlasRequest_ that was passed as input. Documentation ============= Need to update TWIKI documents. Diffs (updated) ----- intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java 7530196d pom.xml 4360604f repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java 9ec15e00 repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java PRE-CREATION repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java 87c9f0eb repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java 08e9ee81 repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersJSONTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java PRE-CREATION repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java 8b042d7a webapp/pom.xml 4132912d webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java 01f9a860 Diff: https://reviews.apache.org/r/56959/diff/8/ Changes: https://reviews.apache.org/r/56959/diff/7-8/ Testing ------- **Unit Test** * Added tests for *AtlasImportTransformBuilder* and contained classes. * Updated *ZipSourceTest* * Added tests to *ImportServiceTest* to exercise new option. **Functional Test** * Small size data export & import. * Medium size data export & import. **Performance Tests** Not exhaustive. Simple stop watch tests to compare against baseline. Thanks, Ashutosh Mestry