----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/56959/#review176328 -----------------------------------------------------------
repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java Lines 29 (patched) <https://reviews.apache.org/r/56959/#comment249730> possible NPE, add null checks for entityWithExtInfo repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java Lines 67 (patched) <https://reviews.apache.org/r/56959/#comment249725> this line could be part of a new else block above. It would be good to pass the "key" string in the error message to know the invalid transformer name in log. repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java Line 31 (original), 31 (patched) <https://reviews.apache.org/r/56959/#comment249726> revert these changes, it is already committed. - Sarath Subramanian On May 25, 2017, 2:32 p.m., Ashutosh Mestry wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/56959/ > ----------------------------------------------------------- > > (Updated May 25, 2017, 2:32 p.m.) > > > Review request for atlas and Madhan Neethiraj. > > > Bugs: ATLAS-1825 > https://issues.apache.org/jira/browse/ATLAS-1825 > > > Repository: atlas > > > Description > ------- > > Please refer to > [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825) for scenarios > and solution approach. > > Impact > ====== > * New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This > holds a map of entity-type to attribute and the tranforms to be applied. > * New class _AtlasImportTransformBuiler_ reads the requests specified in the > query string of the import API and creates a _AtlasImportTransfor_ structure. > * _AtlasImportTransformBuiler_ has new classes that support transformations. > They are: > * _replace_ Implements find and replace on attribute values. > * _lowercase_ Converts attribute value to lower case. > * Additional classes that unit test these classes have been added. > > Implementation Approach > ======================= > * _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the > transform every entity that is fetched from the source. > * _Import API_ now accepts multi-part input. This allows for elegantly > specifying import parameters as well as the binary input from the file. > > CURL > ==== > > *Setup* > Create file with these contents call it _importTransform.json_: > ```javascript > { "options": { > "transforms": { > "hive_db": { > "qualifiedName": [ > "replace:@cl1:@cl2" > ] > }, > "hive_table": { > "qualifiedName": [ > "lowercase", > "replace:@cl1:@cl2" > ] > } > } > } > } > ``` > > ``` > curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H > "Cache-Control: no-cache" -F request=@../docs/importTransform.json -F > data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import" > ``` > > Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is > not required anymore. > > Notice the output from the API, it has the _AtlasRequest_ that was passed as > input. > > Documentation > ============= > Need to update TWIKI documents. > > > Diffs > ----- > > intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java > 7530196d > pom.xml a93b8ad9 > > repository/src/main/java/org/apache/atlas/repository/impexp/EntityAttributeTransformer.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java > 9ec15e00 > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformers.java > PRE-CREATION > > repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java > PRE-CREATION > repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java > 87c9f0eb > > repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/EntityAttributeTransformerTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java > 08e9ee81 > > repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformersTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/TransformEvaluatorTest.java > PRE-CREATION > > repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java > 8b042d7a > > repository/src/test/java/org/apache/atlas/services/EntityDiscoveryServiceTest.java > d4900838 > webapp/pom.xml 4132912d > webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java > 01f9a860 > > > Diff: https://reviews.apache.org/r/56959/diff/7/ > > > Testing > ------- > > **Unit Test** > * Added tests for *AtlasImportTransformBuilder* and contained classes. > * Updated *ZipSourceTest* > * Added tests to *ImportServiceTest* to exercise new option. > > **Functional Test** > * Small size data export & import. > * Medium size data export & import. > > **Performance Tests** > Not exhaustive. Simple stop watch tests to compare against baseline. > > > Thanks, > > Ashutosh Mestry > >