-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56959/
-----------------------------------------------------------
(Updated June 2, 2017, 4:30 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates include:
- Addressed review comments.
- Modified JSON format for transforms.
Bugs: ATLAS-1825
https://issues.apache.org/jira/browse/ATLAS-1825
Repository: atlas
Description
-------
Please refer to [ATLAS-1825](https://issues.apache.org/jira/browse/ATLAS-1825)
for scenarios and solution approach.
Impact
======
* New class _AtlasImportTransform_ is now part of _AtlasImportRequest_. This
holds a map of entity-type to attribute and the tranforms to be applied.
* New class _AtlasImportTransformBuiler_ reads the requests specified in the
query string of the import API and creates a _AtlasImportTransfor_ structure.
* _AtlasImportTransformBuiler_ has new classes that support transformations.
They are:
* _replace_ Implements find and replace on attribute values.
* _lowercase_ Converts attribute value to lower case.
* Additional classes that unit test these classes have been added.
Implementation Approach
=======================
* _ZipSource_ now accepts _AtlasImportTransform_ object. It applies the
transform every entity that is fetched from the source.
* _Import API_ now accepts multi-part input. This allows for elegantly
specifying import parameters as well as the binary input from the file.
CURL
====
*Setup*
Create file with these contents call it _importTransform.json_:
```javascript
{ "options": {
"transforms": {
"hive_db": {
"qualifiedName": [
"replace:@cl1:@cl2"
]
},
"hive_table": {
"qualifiedName": [
"lowercase",
"replace:@cl1:@cl2"
]
}
}
}
}
```
```
curl -g -X POST -u admin:admin -H "Content-Type: multipart/form-data" -H
"Cache-Control: no-cache" -F request=@../docs/importTransform.json -F
data=@../docs/Stocks-2.zip "http://localhost:21000/api/atlas/admin/import";
```
Note: _-g_ needs to be added to CURL calls to prevent gobble check. This is not
required anymore.
Notice the output from the API, it has the _AtlasRequest_ that was passed as
input.
Documentation
=============
Need to update TWIKI documents.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/model/impexp/AtlasImportRequest.java
7530196d
pom.xml 4360604f
repository/src/main/java/org/apache/atlas/repository/impexp/ImportService.java
9ec15e00
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransformer.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ImportTransforms.java
PRE-CREATION
repository/src/main/java/org/apache/atlas/repository/impexp/ZipSource.java
87c9f0eb
repository/src/test/java/org/apache/atlas/repository/impexp/AtlasImportRequestTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceReportingTest.java
0aaaa704
repository/src/test/java/org/apache/atlas/repository/impexp/ImportServiceTest.java
08e9ee81
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerJSONTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformerTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ImportTransformsTest.java
PRE-CREATION
repository/src/test/java/org/apache/atlas/repository/impexp/ZipSourceTest.java
8b042d7a
webapp/pom.xml 4132912d
webapp/src/main/java/org/apache/atlas/web/resources/AdminResource.java
01f9a860
Diff: https://reviews.apache.org/r/56959/diff/9/
Changes: https://reviews.apache.org/r/56959/diff/8-9/
Testing
-------
**Unit Test**
* Added tests for *AtlasImportTransformBuilder* and contained classes.
* Updated *ZipSourceTest*
* Added tests to *ImportServiceTest* to exercise new option.
**Functional Test**
* Small size data export & import.
* Medium size data export & import.
**Performance Tests**
Not exhaustive. Simple stop watch tests to compare against baseline.
Thanks,
Ashutosh Mestry
