DMF-1343 : Improvement and documentation
Project: http://git-wip-us.apache.org/repos/asf/incubator-unomi/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-unomi/commit/3c602041 Tree: http://git-wip-us.apache.org/repos/asf/incubator-unomi/tree/3c602041 Diff: http://git-wip-us.apache.org/repos/asf/incubator-unomi/diff/3c602041 Branch: refs/heads/feature-DMF-1343 Commit: 3c602041a7402dddb4ab119a7c62984149db65c1 Parents: 2f975d3 Author: Abdelkader Midani <amid...@apache.org> Authored: Wed May 24 17:26:14 2017 +0200 Committer: Abdelkader Midani <amid...@apache.org> Committed: Tue Jun 6 17:35:46 2017 +0200 ---------------------------------------------------------------------- extensions/router/README.md | 113 +++++++++++++++++++ .../ProfileImportConfigUpdateRouteBuilder.java | 2 +- .../resources/OSGI-INF/blueprint/blueprint.xml | 6 +- .../main/resources/org.apache.unomi.router.cfg | 4 +- .../ImportConfigurationServiceEndPoint.java | 37 +++--- .../resources/OSGI-INF/blueprint/blueprint.xml | 14 ++- 6 files changed, 154 insertions(+), 22 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/README.md ---------------------------------------------------------------------- diff --git a/extensions/router/README.md b/extensions/router/README.md new file mode 100644 index 0000000..8fa1994 --- /dev/null +++ b/extensions/router/README.md @@ -0,0 +1,113 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one or more + ~ contributor license agreements. See the NOTICE file distributed with + ~ this work for additional information regarding copyright ownership. + ~ The ASF licenses this file to You under the Apache License, Version 2.0 + ~ (the "License"); you may not use this file except in compliance with + ~ the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, software + ~ distributed under the License is distributed on an "AS IS" BASIS, + ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + ~ See the License for the specific language governing permissions and + ~ limitations under the License. + --> + +Unomi Router +========================== + +## Getting started +Unomi Router Extension a Karaf Feature that provide an Enterprise Application Integration tool. +It is optional so you must configure it and install it in Karaf, and can be used for Machine - Machine or Human - Machine integration with Unomi. +Mainly Unomi Router Extension aim to make it easy to import third party applications/platforms profiles into Unomi. +This extension is implemented using Apache Camel routes and is using Apache Kafka to buffer import process and make it failsafe. + +## Getting started +1. Configure your Unomi Router: + In the `etc/org.apache.unomi.sfdc.cfg` file, you might want to update the following settings: + Kafka settings + >`#Kafka settings` + + >`kafka.host=localhost` + + >`kafka.port=9092` + + >`kafka.import.topic=camel-deposit` + + >`kafka.import.groupId=unomi-import-group` + + Kafka host and port with the topic name and the groupId ti which the topic is assigned + + >`#Import One Shot upload directory` + + >`import.oneshot.uploadDir=/tmp/unomi_oneshot_import_configs/` + + Path to the folder where unomi should stock file imported for a oneshot processing + + +2. Deploy into Apache Unomi using the following commands from the Apache Karaf shell: + ```sh + $ feature:repo-add mvn:org.apache.unomi/unomi-router-karaf-feature/${version}/xml/features + $ feature:install unomi-router-karaf-feature + ``` + +3. Send your import configuration: + + An import configuration is nothing else than a simple JSON to describe how you want to import your data (Profiles). + To create/update an import configuration + + `POST /cxs/importConfiguration` + ```json + { + "itemId": "f57f1f86-97bf-4ba0-b4e4-7d5e77e7c0bd", + "itemType": "importConfig", + "scope": "integration", + "name": "Test Recurrent", + "description": "Just test recurrent import", + "configType": "recurrent", + "properties": { + "source": "{file/ftp}://{path}?fileName={file-name}.csv&move=.done&consumer.delay=20000", + "mapping": { + "firstName": 0, + "lastName": 1, + ... + } + }, + "mergingProperty": "email", + "overwriteExistingProfiles": true, + "propertiesToOverwrite": ["firstName", "lastName"], + "active": true + } + ``` + + Omit the `itemId` when creating new entry, `configType` can be '**recurrent**' for file/ftp/network path polling or '**oneshot**' for one time import. + + The `properties.source` attribute is an Apache Camel endpoint uri (See http://camel.apache.org/uris.html for more details). Unomi Router is designed to use **File** and **FTP** Camel components. + + The attribute `properties.mapping` is a Map of: + * Key: Profile property id in Unomi + * Value: Index of the column in the imported file to copy the in the previous property. + + The attribute `mergingProperty` is the profile property id in Unomi to use to check for duplication. + + The attribute `propertiesToOverwrite` is a list of profile properties ids to overwrite, if **null** all properties + will be overwritten. + + The attribute `active` is the flag to activate or deactivate the import configuration. + + Concerning oneshot import configuration using the previously described service will only create the import configuration, to send the file to process + you need to call : + + `POST /cxs/importConfiguration/oneshot` + + `Content-Type : multipart/form-data` + + First multipart with the name '**importConfigId**' is the importConfiguration to use to import the file, second one with the name '**file**' is the file to import. + + + + + + \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java ---------------------------------------------------------------------- diff --git a/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java b/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java index ac71798..40575d5 100644 --- a/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java +++ b/extensions/router/router-core/src/main/java/org/apache/unomi/router/core/route/ProfileImportConfigUpdateRouteBuilder.java @@ -38,7 +38,7 @@ public class ProfileImportConfigUpdateRouteBuilder extends RouteBuilder { logger.info("Preparing REST Configuration for servlet with context path [/importConfigAdmin]"); restConfiguration().component("servlet") .contextPath("/importConfigAdmin") - .enableCORS(true) + .enableCORS(false) .bindingMode(RestBindingMode.json) .dataFormatProperty("prettyPrint", "true"); http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml ---------------------------------------------------------------------- diff --git a/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml b/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml index ff29755..4c36b9e 100644 --- a/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml +++ b/extensions/router/router-core/src/main/resources/OSGI-INF/blueprint/blueprint.xml @@ -79,9 +79,9 @@ <property name="bundleContext" ref="blueprintBundleContext"/> </bean> - <camelContext id="httpEndpoint" xmlns="http://camel.apache.org/schema/blueprint"> - <routeBuilder ref="profileImportConfigUpdateRouteBuilder" /> - </camelContext> + <camel:camelContext id="httpEndpoint" xmlns="http://camel.apache.org/schema/blueprint"> + <camel:routeBuilder ref="profileImportConfigUpdateRouteBuilder" /> + </camel:camelContext> <bean id="profileImportConfigUpdateRouteBuilder" class="org.apache.unomi.router.core.route.ProfileImportConfigUpdateRouteBuilder"> <property name="profileImportCamelContext" ref="camelContext"/> http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg ---------------------------------------------------------------------- diff --git a/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg b/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg index b11d983..ff2c8ef 100644 --- a/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg +++ b/extensions/router/router-core/src/main/resources/org.apache.unomi.router.cfg @@ -14,7 +14,9 @@ # See the License for the specific language governing permissions and # limitations under the License. # -kafka.host=localhost + +#Kafka + settingskafka.host=localhost kafka.port=9092 kafka.import.topic=camel-deposit kafka.import.groupId=unomi-import-group http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java ---------------------------------------------------------------------- diff --git a/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java b/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java index e81930a..42670b2 100644 --- a/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java +++ b/extensions/router/router-rest/src/main/java/org/apache/unomi/router/rest/ImportConfigurationServiceEndPoint.java @@ -52,9 +52,11 @@ import java.util.List; public class ImportConfigurationServiceEndPoint { private static final Logger logger = LoggerFactory.getLogger(ImportConfigurationServiceEndPoint.class.getName()); + private final String CONFIG_TYPE_RECURRENT = "recurrent"; private ImportConfigurationService importConfigurationService; private String uploadDir; + private String serverPort; public ImportConfigurationServiceEndPoint () { logger.info("Initializing import configuration service endpoint..."); @@ -70,6 +72,11 @@ public class ImportConfigurationServiceEndPoint { this.uploadDir = uploadDir; } + @WebMethod(exclude = true) + public void setServerPort(String serverPort) { + this.serverPort = serverPort; + } + /** * Retrieves all the import configurations. * @@ -122,21 +129,23 @@ public class ImportConfigurationServiceEndPoint { @Consumes(MediaType.APPLICATION_JSON) public ImportConfiguration saveImportConfiguration(ImportConfiguration importConfiguration) { ImportConfiguration importConfigSaved = importConfigurationService.save(importConfiguration); - CloseableHttpClient httpClient = HttpClients.createDefault(); - try { - HttpPut httpPut = new HttpPut("http://localhost:8181/importConfigAdmin/"); - StringEntity input = new StringEntity(new ObjectMapper().writeValueAsString(importConfigSaved)); - input.setContentType(MediaType.APPLICATION_JSON); - httpPut.setEntity(input); - - HttpResponse response = httpClient.execute(httpPut); - - if (response.getStatusLine().getStatusCode() != 200) { - throw new RuntimeException("Failed : HTTP error code : " - + response.getStatusLine().getStatusCode()); + if(CONFIG_TYPE_RECURRENT.equals(importConfigSaved.getConfigType())) { + CloseableHttpClient httpClient = HttpClients.createDefault(); + try { + HttpPut httpPut = new HttpPut("http://localhost:"+serverPort+"/importConfigAdmin/"); + StringEntity input = new StringEntity(new ObjectMapper().writeValueAsString(importConfigSaved)); + input.setContentType(MediaType.APPLICATION_JSON); + httpPut.setEntity(input); + + HttpResponse response = httpClient.execute(httpPut); + + if (response.getStatusLine().getStatusCode() != 200) { + throw new RuntimeException("Failed : HTTP error code : " + + response.getStatusLine().getStatusCode()); + } + } catch (IOException e) { + logger.warn("Unable to update Camel route [{}]", importConfiguration.getItemId()); } - } catch (IOException e) { - logger.warn("Unable to update Camel route [{}]", importConfiguration.getItemId()); } return importConfigSaved; } http://git-wip-us.apache.org/repos/asf/incubator-unomi/blob/3c602041/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml ---------------------------------------------------------------------- diff --git a/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml b/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml index a5cf1be..7e3b5d1 100644 --- a/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml +++ b/extensions/router/router-rest/src/main/resources/OSGI-INF/blueprint/blueprint.xml @@ -21,15 +21,22 @@ xmlns:cxf="http://cxf.apache.org/blueprint/core" xmlns:jaxrs="http://cxf.apache.org/blueprint/jaxrs" xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd - http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd - http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd - http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd"> + http://cxf.apache.org/blueprint/jaxrs http://cxf.apache.org/schemas/blueprint/jaxrs.xsd + http://cxf.apache.org/blueprint/core http://cxf.apache.org/schemas/blueprint/core.xsd + http://aries.apache.org/blueprint/xmlns/blueprint-cm/v1.1.0 http://aries.apache.org/schemas/blueprint-cm/blueprint-cm-1.1.0.xsd"> <cm:property-placeholder persistent-id="org.apache.unomi.router" update-strategy="reload"> <cm:default-properties> <cm:property name="import.oneshot.uploadDir" value="/tmp/oneshot_import_configs/"/> </cm:default-properties> </cm:property-placeholder> + <cm:property-placeholder persistent-id="org.apache.unomi.cluster" + update-strategy="reload" placeholder-prefix="${cluster."> + <cm:default-properties> + <cm:property name="contextserver.port" value="8181"/> + </cm:default-properties> + </cm:property-placeholder> + <cxf:bus id="cxsServiceBus"> <cxf:features> <cxf:logging/> @@ -67,6 +74,7 @@ <bean id="importConfigurationServiceEndPoint" class="org.apache.unomi.router.rest.ImportConfigurationServiceEndPoint"> <property name="importConfigurationService" ref="importConfigurationService"/> <property name="uploadDir" value="${import.oneshot.uploadDir}"/> + <property name="serverPort" value="${contextserver.port}"/> </bean> </blueprint>