Taybou commented on a change in pull request #94: UNOMI-240 Document profile import/export URL: https://github.com/apache/unomi/pull/94#discussion_r318565822
########## File path: manual/src/main/asciidoc/profile-import-export.adoc ########## @@ -0,0 +1,239 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +The profile import and export feature in Apache Unomi is based on configurations and consumes or produces CSV files that +contain the profiles to import and export. + +=== Importing profiles + +Only `ftp`, `sftp`, `ftps` and ``file` are supported in the source path. For example: + + file:///tmp/?fileName=profiles.csv&move=.done&consumer.delay=25s + +Where: + +- `fileName` Can be a pattern, for example `include=.*.csv` instead of `fileName=...` to consume all CSV files. +By default the processed files are moved to `.camel` folder you can change it using the `move` option. +- `consumer.delay` Is the frequency of polling in milliseconds. For example, 20000 milliseconds is 20 seconds. This +frequency can also be 20s. Other possible format are: 2h30m10s = 2 hours and 30 minutes and 10 seconds. + +See http://camel.apache.org/ftp.html and http://camel.apache.org/file2.html to build more complex source path. Also be +careful with FTP configuration as most servers won't accept clear text FTP anymore and you should use SFTP or FTPS +instead, but they are a little more difficult to configure properly. It is recommended to test the connection with an +FTP client first before setting up these source paths to make sure everything is working properly. Also on FTP +connections most servers require PASSIVE mode so you can specify that in the path using the `passiveMode=true` parameter. + +Here are some examples of FTPS and SFTP source paths: + + sftp://USER@HOST/PATH?password=PASSWORD&include=.*.csv + ftps://USER@HOST?password=PASSWORD&fileName=profiles.csv&passiveMode=true + +Where: + +- `USER` is the user name of the SFTP/FTPS user account to login with +- `PASSWORD` is the password for the user account +- `HOST` is the host name (or IP address) of the host server that provides the SFTP / FTPS server +- `PATH` is a path to a directory inside the user's account where the file will be retrieved. + +==== Import API + +Apache Unomi provides REST endpoints to manage import configurations: + + GET /cxs/importConfiguration + GET /cxs/importConfiguration/{configId} + POST /cxs/importConfiguration + DELETE /cxs/importConfiguration/{configId} + +This is how a oneshot import configuration looks like: + + { + "itemId": "importConfigId", + "itemType": "importConfig", + "name": "Import Config Sample", + "description": "Sample description", + "configType": "oneshot", //Config type can be 'oneshot' or 'recurrent' + "properties": { + "mapping": { + "email": 0, //<Apache Unomi Property Id> : <Column Index In the CSV> + "firstName": 2, + ... + } + }, + "columnSeparator": ",", //Character used to separate columns + "lineSeparator": "\\n", //Character used to separate lines (\n or \r) + "multiValueSeparator": ";", //Character used to separate values for multivalued columns + "multiValueDelimiter": "[]", //Character used to wrap values for multivalued columns + "status": "SUCCESS", //Status of last execution + "executions": [ //(RETURN) Last executions by default only last 5 are returned + ... + ], + "mergingProperty": "email", //Apache Unomi Property Id used to check duplicates + "overwriteExistingProfiles": true, //Overwrite profiles that have duplicates + "propertiesToOverwrite": "firstName, lastName, ...", //If last is set to true, which property to overwrite, 'null' means overwrite all + "hasHeader": true, //CSV file to import contains a header line + "hasDeleteColumn": false //CSV file to import doesn't contain a TO DELETE column (if it contains, will be the last column) + } + +A recurrent import configuration is similar to the previous one with some specific information to add to the JSON like: + + { + ... + "configType": "recurrent", + "properties": { + "source": "ftp://USER@SERVER[:PORT]/PATH?password=xxx&fileName=profiles.csv&move=.done&consumer.delay=20000";, + // Only 'ftp', 'sftp', 'ftps' and 'file' are supported in the 'source' path + // eg. file:///tmp/?fileName=profiles.csv&move=.done&consumer.delay=25s + // 'fileName' can be a pattern eg 'include=.*.csv' instead of 'fileName=...' to consume all CSV files + // By default the processed files are moved to '.camel' folder you can change it using the 'move' option + // 'consumer.delay' is the frequency of polling. '20000' (in milliseconds) means 20 seconds. Can be also '20s' + // Other possible format are: '2h30m10s' = 2 hours and 30 minutes and 10 seconds + "mapping": { + ... + } + }, + ... + "active": true, //If true the polling will start according to the 'source' configured above + ... + } + + +=== Exporting profiles + +Only `ftp`, `sftp`, `ftps` and `file are supported in the source path. For example: + + file:///tmp/?fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append) + sftp://USER@HOST/PATH?password=PASSWORD&binary=true&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append + ftps://USER@HOST?password=PASSWORD&binary=true&fileName=profiles-export-${date:now:yyyyMMddHHmm}.csv&fileExist=Append&passiveMode=true + +As you can see in the examples above, you can inject variables in the produced file name `${date:now:yyyyMMddHHmm}` is +the current date formatted with the pattern `yyyyMMddHHmm`. `fileExist` option put as `Append` will tell the file writer Review comment: ```suggestion the current date formatted with the pattern `yyyyMMddHHmm`. `fileExist` option placed as `Append` will tell the file writer ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
