another round of website improvements
Project: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/commit/3c476284 Tree: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/tree/3c476284 Diff: http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/diff/3c476284 Branch: refs/heads/master Commit: 3c47628474da4e4574d4762c1d4376211104e812 Parents: f8c0a11 Author: Steve Blackmon @steveblackmon <sblack...@apache.org> Authored: Wed Apr 27 15:57:26 2016 -0500 Committer: Steve Blackmon @steveblackmon <sblack...@apache.org> Committed: Wed Apr 27 15:57:26 2016 -0500 ---------------------------------------------------------------------- local/elasticsearch-hdfs/README.md | 33 +---- local/elasticsearch-hdfs/pom.xml | 86 ++----------- .../src/site/markdown/ElasticsearchHdfs.md | 41 +++---- .../src/site/markdown/HdfsElasticsearch.md | 44 +++---- .../src/site/markdown/index.md | 15 ++- local/elasticsearch-reindex/README.md | 77 +----------- local/elasticsearch-reindex/pom.xml | 75 +---------- .../src/site/markdown/index.md | 56 +++------ local/mongo-elasticsearch-sync/README.md | 66 +--------- local/mongo-elasticsearch-sync/pom.xml | 73 +---------- .../src/site/markdown/index.md | 40 ++---- local/pom.xml | 3 + local/twitter-follow-graph/README.md | 95 +------------- local/twitter-follow-graph/pom.xml | 73 ++--------- .../src/site/markdown/index.md | 75 +++-------- local/twitter-history-elasticsearch/README.md | 76 +----------- local/twitter-history-elasticsearch/pom.xml | 89 ++------------ .../src/site/markdown/index.md | 54 +++----- .../twitter-userstream-elasticsearch/README.md | 70 +---------- local/twitter-userstream-elasticsearch/pom.xml | 88 ++----------- .../src/site/markdown/index.md | 48 +++----- pom.xml | 123 +++++++++++++++++++ src/site/markdown/index.md | 23 ---- src/site/site.xml | 26 ++-- 24 files changed, 326 insertions(+), 1123 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-hdfs/README.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/README.md b/local/elasticsearch-hdfs/README.md index b3e2e3f..b6653da 100644 --- a/local/elasticsearch-hdfs/README.md +++ b/local/elasticsearch-hdfs/README.md @@ -2,34 +2,7 @@ Apache Streams (incubating) Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 -------------------------------------------------------------------------------- -elasticsearch-hdfs -============================== - -Requirements: -------------- - - A running ElasticSearch 1.0.0+ instance - -Description: ------------- -Copies documents between elasticsearch and file system using the hdfs persist module. - -Streams: --------- - -[ElasticsearchHdfs](src/site/markdown/ElasticsearchHdfs.md "ElasticsearchHdfs" ) - -[HdfsElasticsearch](src/site/markdown/HdfsElasticsearch.md "HdfsElasticsearch" ) - -Build: ---------- - -`mvn clean package` - -Note that an alternative version of hdfs is packaged, by excluding org.apache.hadoop.hadoop-hdfs when -importing org.apache.streams.streams-persist-hdfs, and specifically depending on a different preferred version. - -Deploy: --------- - -`mvn -Pdocker clean package docker:build` +org.apache.streams:elasticsearch-reindex +======================================== +[README.md](src/site/markdown/index.md "README") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-hdfs/pom.xml ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/pom.xml b/local/elasticsearch-hdfs/pom.xml index ca3f3f8..769da44 100644 --- a/local/elasticsearch-hdfs/pom.xml +++ b/local/elasticsearch-hdfs/pom.xml @@ -34,6 +34,7 @@ <description>Copies documents between elasticsearch and file system using the hdfs persist module.</description> <properties> + <docker.repo>apachestreams</docker.repo> <elasticsearch.version>1.1.0</elasticsearch.version> <lucene.version>4.7.2</lucene.version> <hdfs.version>2.7.0</hdfs.version> @@ -215,37 +216,6 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <finalName>${project.build.finalName}</finalName> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - <exclude>**/logback.xml</exclude> - <exclude>**/log4j.properties</exclude> - </excludes> - </filter> - </filters> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass>org.apache.streams.example.elasticsearch.ElasticsearchReindex</mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> </plugin> <plugin> <groupId>org.jsonschema2pojo</groupId> @@ -290,19 +260,19 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> - <version>2.4</version> + <configuration> + <includes>**/*.json</includes> + <outputDirectory>${project.build.directory}/test-classes</outputDirectory> + <includeGroupIds>org.apache.streams</includeGroupIds> + <includeTypes>test-jar</includeTypes> + </configuration> <executions> <execution> - <id>resource-dependencies</id> + <id>test-resource-dependencies</id> <phase>process-test-resources</phase> <goals> <goal>unpack-dependencies</goal> </goals> - <configuration> - <includeArtifactIds>streams-pojo</includeArtifactIds> - <includes>**/*.json</includes> - <outputDirectory>${project.build.directory}/test-classes</outputDirectory> - </configuration> </execution> </executions> </plugin> @@ -320,43 +290,11 @@ </execution> </executions> </plugin> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + </plugin> </plugins> </build> - <profiles> - <profile> - <id>docker</id> - <build> - <plugins> - <plugin> - <!-- The Docker Maven plugin is used to create docker image with the fat jar --> - <groupId>org.jolokia</groupId> - <artifactId>docker-maven-plugin</artifactId> - <version>0.11.0</version> - <configuration> - <images> - - <image> - <alias>${project.artifactId}</alias> - <name>${project.artifactId}:${project.version}</name> - <build> - <from>dockerfile/java:oracle-java8</from> - <assembly> - <basedir>/</basedir> - <descriptorRef>artifact</descriptorRef> - </assembly> - <!-- Default command for the build image --> - </build> - - </image> - - </images> - </configuration> - - </plugin> - - </plugins> - </build> - </profile> - </profiles> </project> http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md b/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md index f4013e7..e524e2d 100644 --- a/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md +++ b/local/elasticsearch-hdfs/src/site/markdown/ElasticsearchHdfs.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - elasticsearch-hdfs ============================== @@ -23,25 +19,18 @@ Diagram: Example Configuration: ---------------------- - { - "source": { - "host": "localhost", - "port": 27017, - "db": "streams", - "collection": "activities" - }, - "destination": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "index": "destination", - "type": "activity" - } - } - -Run: --------- - -`docker run elasticsearch-hdfs:0.2-incubating-SNAPSHOT-PP.jar java -cp stash-migrate-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.elasticsearch.example.HdfsElasticsearch` +[testBackup.json](testBackup.json "testBackup.json" ) + +Run (Local): +------------ + + java -cp dist/elasticsearch-hdfs-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.elasticsearch.ElasticsearchHdfs + +Run (Docker): +------------- + + docker run elasticsearch-hdfs java -cp elasticsearch-hdfs-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.elasticsearch.example.ElasticsearchHdfs + +[JavaDocs](apidocs/index.html "JavaDocs") + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md b/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md index 8278475..f3778fc 100644 --- a/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md +++ b/local/elasticsearch-hdfs/src/site/markdown/HdfsElasticsearch.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - hdfs-elasticsearch ============================== @@ -23,28 +19,18 @@ Diagram: Example Configuration: ---------------------- - { - "source": { - "scheme": "file", - "host": "localhost", - "user": "cloudera", - "path": "/tmp", - "writerPath": "activity" - }, - "destination": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "index": "activity2", - "type": "activity", - "forceUseConfig": true - } - } - - -Run: --------- - -`docker run elasticsearch-hdfs:0.2-incubating-SNAPSHOT.jar java -cp elasticsearch-hdfs-0.2-incubating-SNAPSHOT.jar -Dconfig.file=file://<location_of_config_file>.json org.apache.streams.elasticsearch.example.HdfsElasticsearch` +[testRestore.json](testRestore.json "testRestore.json" ) + +Run (Local): +------------ + + java -cp dist/elasticsearch-hdfs-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.elasticsearch.HdfsElasticsearch + +Run (Docker): +------------- + + docker run elasticsearch-hdfs java -cp elasticsearch-hdfs-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.elasticsearch.example.HdfsElasticsearch + +[JavaDocs](apidocs/index.html "JavaDocs") + +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-hdfs/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-hdfs/src/site/markdown/index.md b/local/elasticsearch-hdfs/src/site/markdown/index.md index cfd5a30..e54924f 100644 --- a/local/elasticsearch-hdfs/src/site/markdown/index.md +++ b/local/elasticsearch-hdfs/src/site/markdown/index.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - elasticsearch-hdfs ============================== @@ -23,13 +19,16 @@ Streams: Build: --------- -`mvn clean package` + mvn clean install verify Note that an alternative version of hdfs is packaged, by excluding org.apache.hadoop.hadoop-hdfs when importing org.apache.streams.streams-persist-hdfs, and specifically depending on a different preferred version. -Deploy: --------- +Deploy (Docker): +---------------- + + mvn -Pdocker -Ddocker.repo=<your docker host>:<your docker repo> clean package docker:build docker:push -`mvn -Pdocker clean package docker:build` +[JavaDocs](apidocs/index.html "JavaDocs") +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-reindex/README.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-reindex/README.md b/local/elasticsearch-reindex/README.md index 6c2fbcf..b6653da 100644 --- a/local/elasticsearch-reindex/README.md +++ b/local/elasticsearch-reindex/README.md @@ -2,78 +2,7 @@ Apache Streams (incubating) Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 -------------------------------------------------------------------------------- -elasticsearch-reindex -============================== - -Requirements: -------------- - - A running ElasticSearch 1.0.0+ cluster - - Transport client access to cluster - - elasticsearch.version and lucene.version set to match cluster - -Description: ------------- -Copies documents between indexes. - -Specification: ------------------ - -[ElasticsearchReindex.dot](src/main/resources/ElasticsearchReindex.dot "ElasticsearchReindex.dot" ) - -Diagram: ------------------ - - - -Example Configuration: ----------------------- - - { - "source": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "indexes": [ - "activity" - ], - "types": [ - "activity" - ], - "forceUseConfig": true - }, - "destination": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "index": "activity2", - "type": "activity", - "forceUseConfig": true - } - } - -Populate source and destination in configuration with cluster / index / type details. - -Build: ---------- - -`mvn clean package verify` - -Run: --------- - -`java -cp target/elasticsearch-reindex-0.1-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.elasticsearch.ElasticsearchReindex` - -Deploy: --------- -`mvn -Pdocker clean package docker:build` - -`docker tag elasticsearch-reindex:0.2-incubating-SNAPSHOT <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT` - -`docker push <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT` - -`docker run <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT.jar java -cp elasticsearch-reindex-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.elasticsearch.ElasticsearchReindex` +org.apache.streams:elasticsearch-reindex +======================================== +[README.md](src/site/markdown/index.md "README") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-reindex/pom.xml ---------------------------------------------------------------------- diff --git a/local/elasticsearch-reindex/pom.xml b/local/elasticsearch-reindex/pom.xml index 50b20f8..41480e5 100644 --- a/local/elasticsearch-reindex/pom.xml +++ b/local/elasticsearch-reindex/pom.xml @@ -34,6 +34,7 @@ <description>Copies documents between indexes.</description> <properties> + <docker.repo>apachestreams</docker.repo> <elasticsearch.version>1.1.0</elasticsearch.version> <lucene.version>4.7.2</lucene.version> </properties> @@ -179,37 +180,6 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <finalName>${project.build.finalName}</finalName> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - <exclude>**/logback.xml</exclude> - <exclude>**/log4j.properties</exclude> - </excludes> - </filter> - </filters> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass>org.apache.streams.example.elasticsearch.ElasticsearchReindex</mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> </plugin> <plugin> <groupId>org.jsonschema2pojo</groupId> @@ -284,46 +254,11 @@ </execution> </executions> </plugin> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + </plugin> </plugins> </build> - <profiles> - <profile> - <id>docker</id> - <build> - <plugins> - <plugin> - <!-- The Docker Maven plugin is used to create docker image with the fat jar --> - <groupId>org.jolokia</groupId> - <artifactId>docker-maven-plugin</artifactId> - <version>0.11.0</version> - <configuration> - <images> - - <image> - <alias>${project.artifactId}</alias> - <name>${project.artifactId}:${project.version}</name> - <build> - <from>dockerfile/java:oracle-java8</from> - <assembly> - <basedir>/</basedir> - <descriptorRef>artifact</descriptorRef> - </assembly> - <!-- Default command for the build image --> - </build> - - </image> - - </images> - </configuration> - - </plugin> - - </plugins> - </build> - <activation> - <activeByDefault/> - </activation> - </profile> - </profiles> </project> http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/elasticsearch-reindex/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/elasticsearch-reindex/src/site/markdown/index.md b/local/elasticsearch-reindex/src/site/markdown/index.md index 758a3b4..83eb123 100644 --- a/local/elasticsearch-reindex/src/site/markdown/index.md +++ b/local/elasticsearch-reindex/src/site/markdown/index.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - elasticsearch-reindex ============================== @@ -18,7 +14,7 @@ Copies documents into a different index Specification: ----------------- -[ElasticsearchReindex.dot](src/main/resources/ElasticsearchReindex.dot "ElasticsearchReindex.dot" ) +[ElasticsearchReindex.dot](ElasticsearchReindex.dot "ElasticsearchReindex.dot" ) Diagram: ----------------- @@ -28,52 +24,30 @@ Diagram: Example Configuration: ---------------------- - { - "source": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "indexes": [ - "activity" - ], - "types": [ - "activity" - ], - "forceUseConfig": true - }, - "destination": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "index": "activity2", - "type": "activity", - "forceUseConfig": true - } - } +[testReindex.json](testReindex.json "testReindex.json" ) Populate source and destination in configuration with cluster / index / type details. Build: --------- -`mvn clean package verify` + mvn clean package verify + +Run (Local): +------------ -Run: --------- + java -cp dist/elasticsearch-reindex-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.elasticsearch.ElasticsearchReindex -`java -cp target/elasticsearch-reindex-0.1-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.elasticsearch.ElasticsearchReindex` +Deploy (Docker): +---------------- -Deploy: --------- -`mvn -Pdocker clean package docker:build` + mvn -Pdocker -Ddocker.repo=<your docker host>:<your docker repo> docker:build docker:push -`docker tag elasticsearch-reindex:0.2-incubating-SNAPSHOT <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT` +Run (Docker): +------------- -`docker push <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT` + docker run elasticsearch-reindex java -cp elasticsearch-reindex-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.elasticsearch.example.ElasticsearchReindex -`docker run <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT.jar java -cp elasticsearch-reindex-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.elasticsearch.ElasticsearchReindex` +[JavaDocs](apidocs/index.html "JavaDocs") +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/mongo-elasticsearch-sync/README.md ---------------------------------------------------------------------- diff --git a/local/mongo-elasticsearch-sync/README.md b/local/mongo-elasticsearch-sync/README.md index 7b227bf..e74196c 100644 --- a/local/mongo-elasticsearch-sync/README.md +++ b/local/mongo-elasticsearch-sync/README.md @@ -2,67 +2,7 @@ Apache Streams (incubating) Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 -------------------------------------------------------------------------------- -mongo-elasticsearch-sync -============================== - -Requirements: -------------- - - A running MongoDB 2.4+ instance - - A running ElasticSearch 1.0.0+ instance - -Description: ------------- -Copies documents from mongodb to elasticsearch. - -Specification: ------------------ - -[MongoElasticsearchSync.dot](src/main/resources/MongoElasticsearchSync.dot "MongoElasticsearchSync.dot" ) - -Diagram: ------------------ - - - -Example Configuration: ----------------------- - - { - "source": { - "host": "localhost", - "port": 27017, - "db": "streams", - "collection": "activities" - }, - "destination": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "index": "destination", - "type": "activity" - } - } - -Build: ---------- - -`mvn clean package verify` - -Run: --------- - -`java -cp target/mongo-elasticsearch-sync-0.1-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.elasticsearch.MongoElasticsearchSync` - -Deploy: --------- - -`mvn -Pdocker clean package docker:build` - -`docker tag mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT <dockerregistry>:mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT` - -`docker push <dockerregistry>:mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT` - -`docker run <dockerregistry>:mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT java -cp mongo-elasticsearch-sync-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.elasticsearch.MongoElasticsearchSync` +org.apache.streams:mongo-elasticsearch-sync +=========================================== +[README.md](src/site/markdown/index.md "README") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/mongo-elasticsearch-sync/pom.xml ---------------------------------------------------------------------- diff --git a/local/mongo-elasticsearch-sync/pom.xml b/local/mongo-elasticsearch-sync/pom.xml index 137446d..a01c0d4 100644 --- a/local/mongo-elasticsearch-sync/pom.xml +++ b/local/mongo-elasticsearch-sync/pom.xml @@ -34,6 +34,7 @@ <description>Copies documents from mongodb to elasticsearch.</description> <properties> + <docker.repo>apachestreams</docker.repo> <elasticsearch.version>1.1.0</elasticsearch.version> <lucene.version>4.7.2</lucene.version> </properties> @@ -209,35 +210,6 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <finalName>${project.build.finalName}</finalName> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - </excludes> - </filter> - </filters> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass>org.apache.streams.example.elasticsearch.MongoElasticsearchSync</mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> </plugin> <plugin> <groupId>org.jsonschema2pojo</groupId> @@ -312,46 +284,11 @@ </execution> </executions> </plugin> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + </plugin> </plugins> </build> - <profiles> - <profile> - <id>docker</id> - <build> - <plugins> - <plugin> - <!-- The Docker Maven plugin is used to create docker image with the fat jar --> - <groupId>org.jolokia</groupId> - <artifactId>docker-maven-plugin</artifactId> - <version>0.11.0</version> - <configuration> - <images> - - <image> - <alias>${project.artifactId}</alias> - <name>${project.artifactId}:${project.version}</name> - <build> - <from>dockerfile/java:oracle-java8</from> - <assembly> - <basedir>/</basedir> - <descriptorRef>artifact</descriptorRef> - </assembly> - <!-- Default command for the build image --> - </build> - - </image> - - </images> - </configuration> - - </plugin> - - </plugins> - </build> - <activation> - <activeByDefault/> - </activation> - </profile> - </profiles> </project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/mongo-elasticsearch-sync/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/mongo-elasticsearch-sync/src/site/markdown/index.md b/local/mongo-elasticsearch-sync/src/site/markdown/index.md index fa45d85..10bad65 100644 --- a/local/mongo-elasticsearch-sync/src/site/markdown/index.md +++ b/local/mongo-elasticsearch-sync/src/site/markdown/index.md @@ -27,42 +27,28 @@ Diagram: Example Configuration: ---------------------- - { - "source": { - "host": "localhost", - "port": 27017, - "db": "streams", - "collection": "activities" - }, - "destination": { - "hosts": [ - "localhost" - ], - "port": 9300, - "clusterName": "elasticsearch", - "index": "destination", - "type": "activity" - } - } +[testSync.json](testSync.json "testSync.json" ) Build: --------- -`mvn clean package verify` + mvn clean package verify -Run: --------- +Run (Local): +------------ -`java -cp target/mongo-elasticsearch-sync-0.1-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.elasticsearch.MongoElasticsearchSync` + java -cp dist/mongo-elasticsearch-sync-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.elasticsearch.MongoElasticsearchSync -Deploy: --------- +Deploy (Docker): +---------------- -`mvn -Pdocker clean package docker:build` + mvn -Pdocker -Ddocker.repo=<your docker host>:<your docker repo> docker:build docker:push -`docker tag mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT <dockerregistry>:mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT` +Run (Docker): +------------- -`docker push <dockerregistry>:mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT` + docker run mongo-elasticsearch-sync java -cp mongo-elasticsearch-sync-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.elasticsearch.example.MongoElasticsearchSync -`docker run <dockerregistry>:mongo-elasticsearch-sync:0.2-incubating-SNAPSHOT java -cp mongo-elasticsearch-sync-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.elasticsearch.MongoElasticsearchSync` +[JavaDocs](apidocs/index.html "JavaDocs") +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/pom.xml ---------------------------------------------------------------------- diff --git a/local/pom.xml b/local/pom.xml index 405c4b5..07f0511 100644 --- a/local/pom.xml +++ b/local/pom.xml @@ -47,4 +47,7 @@ <module>twitter-userstream-elasticsearch</module> </modules> + <build> + + </build> </project> http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-follow-graph/README.md ---------------------------------------------------------------------- diff --git a/local/twitter-follow-graph/README.md b/local/twitter-follow-graph/README.md index 989fbb7..3e63a53 100644 --- a/local/twitter-follow-graph/README.md +++ b/local/twitter-follow-graph/README.md @@ -2,96 +2,7 @@ Apache Streams (incubating) Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 -------------------------------------------------------------------------------- -twitter-follow-graph -============================== - -Requirements: -------------- - - Authorized Twitter API credentials - - A running Neo4J 1.9.0+ instance - -Description: ------------- -Collects friend or follower connections for a set of twitter users to build a graph database in neo4j. - -Specification: ------------------ - -[TwitterFollowGraph.dot](src/main/resources/TwitterFollowGraph.dot "TwitterFollowGraph.dot" ) - -Diagram: ------------------ - - - -Example Configuration: ----------------------- - - { - "twitter": { - "endpoint": "friends", - "oauth": { - "consumerSecret": "", - "consumerKey": "", - "accessToken": "", - "accessTokenSecret": "" - }, - "info": [ - 42232950 - ] - }, - "graph": { - "vertices": { - "objects": [ - "actor", - "object" - ], - "verbs": [ - "follow" - ], - "objectTypes": [ - "page" - ] - }, - "edges": { - "objects": [ - "actor", - "object" - ], - "verbs": [ - "follow" - ], - "objectTypes": [ - "page" - ] - }, - "protocol": "http", - "hostname": "localhost", - "port": 7474, - "requestMethod": "GET", - "content-type": "application/json", - "type": "neo4j", - "graph": "data" - } - } - -Build: ---------- - -`mvn clean package verify` - -Run: --------- - -`java -cp target/twitter-follow-graph-0.2-incubating-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.graph.TwitterFollowGraph` - -Deploy: --------- -`mvn -Pdocker clean package docker:build` - -`docker tag twitter-follow-graph:0.2-incubating-SNAPSHOT <dockerregistry>:twitter-follow-graph:0.2-incubating-SNAPSHOT` - -`docker push <dockerregistry>:twitter-follow-graph:0.2-incubating-SNAPSHOT` - -`docker run <dockerregistry>:twitter-follow-graph:0.2-incubating-SNAPSHOT java -cp twitter-follow-graph-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.graph.TwitterFollowGraph` +org.apache.streams:twitter-follow-graph +======================================= +[README.md](src/site/markdown/index.md "README") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-follow-graph/pom.xml ---------------------------------------------------------------------- diff --git a/local/twitter-follow-graph/pom.xml b/local/twitter-follow-graph/pom.xml index 5fa4ab1..81f92ff 100644 --- a/local/twitter-follow-graph/pom.xml +++ b/local/twitter-follow-graph/pom.xml @@ -35,6 +35,10 @@ Collects friend or follower connections for a set of twitter users to build a graph database in neo4j. </description> + <properties> + <docker.repo>apachestreams</docker.repo> + </properties> + <dependencies> <dependency> <groupId>com.typesafe</groupId> @@ -160,35 +164,6 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <finalName>${project.build.finalName}</finalName> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - </excludes> - </filter> - </filters> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass>org.apache.streams.example.graph.TwitterFollowingNeo4j</mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> </plugin> <plugin> <groupId>org.jsonschema2pojo</groupId> @@ -263,43 +238,11 @@ </execution> </executions> </plugin> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + </plugin> </plugins> </build> - <profiles> - <profile> - <id>docker</id> - <build> - <plugins> - <plugin> - <!-- The Docker Maven plugin is used to create docker image with the fat jar --> - <groupId>org.jolokia</groupId> - <artifactId>docker-maven-plugin</artifactId> - <version>0.11.0</version> - <configuration> - <images> - - <image> - <alias>${project.artifactId}</alias> - <name>${project.artifactId}:${project.version}</name> - <build> - <from>dockerfile/java:latest</from> - <assembly> - <basedir>/</basedir> - <descriptorRef>artifact</descriptorRef> - </assembly> - <!-- Default command for the build image --> - </build> - - </image> - - </images> - </configuration> - - </plugin> - - </plugins> - </build> - </profile> - </profiles> </project> \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-follow-graph/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/twitter-follow-graph/src/site/markdown/index.md b/local/twitter-follow-graph/src/site/markdown/index.md index b96b551..b8494cd 100644 --- a/local/twitter-follow-graph/src/site/markdown/index.md +++ b/local/twitter-follow-graph/src/site/markdown/index.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - twitter-follow-graph ============================== @@ -27,71 +23,28 @@ Diagram: Example Configuration: ---------------------- - { - "twitter": { - "endpoint": "friends", - "oauth": { - "consumerSecret": "", - "consumerKey": "", - "accessToken": "", - "accessTokenSecret": "" - }, - "info": [ - 42232950 - ] - }, - "graph": { - "vertices": { - "objects": [ - "actor", - "object" - ], - "verbs": [ - "follow" - ], - "objectTypes": [ - "page" - ] - }, - "edges": { - "objects": [ - "actor", - "object" - ], - "verbs": [ - "follow" - ], - "objectTypes": [ - "page" - ] - }, - "protocol": "http", - "hostname": "localhost", - "port": 7474, - "requestMethod": "GET", - "content-type": "application/json", - "type": "neo4j", - "graph": "data" - } - } +[testGraph.json](testGraph.json "testGraph.json" ) Build: --------- -`mvn clean package verify` + mvn clean package verify + +Run (Local): +------------ -Run: --------- + java -cp dist/twitter-follow-graph-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.graph.TwitterFollowGraph -`java -cp target/twitter-follow-graph-0.2-incubating-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.graph.TwitterFollowGraph` +Deploy (Docker): +---------------- -Deploy: --------- -`mvn -Pdocker clean package docker:build` + mvn -Pdocker -Ddocker.repo=<your docker host>:<your docker repo> docker:build docker:push -`docker tag twitter-follow-graph:0.2-incubating-SNAPSHOT <dockerregistry>:twitter-follow-graph:0.2-incubating-SNAPSHOT` +Run (Docker): +------------- -`docker push <dockerregistry>:twitter-follow-graph:0.2-incubating-SNAPSHOT` + docker run twitter-follow-graph java -cp twitter-follow-graph-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.elasticsearch.example.TwitterFollowGraph -`docker run <dockerregistry>:twitter-follow-graph:0.2-incubating-SNAPSHOT java -cp twitter-follow-graph-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.graph.TwitterFollowGraph` +[JavaDocs](apidocs/index.html "JavaDocs") +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-history-elasticsearch/README.md ---------------------------------------------------------------------- diff --git a/local/twitter-history-elasticsearch/README.md b/local/twitter-history-elasticsearch/README.md index 2bfc820..5b0b3ec 100644 --- a/local/twitter-history-elasticsearch/README.md +++ b/local/twitter-history-elasticsearch/README.md @@ -2,77 +2,7 @@ Apache Streams (incubating) Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 -------------------------------------------------------------------------------- -twitter-history-elasticsearch -============================== +org.apache.streams:twitter-history-elasticsearch +================================================ -Requirements: -------------- - - Authorized Twitter API credentials - - A running ElasticSearch 1.0.0+ instance - -Description: ------------- -Retrieves as many posts from a known list of users as twitter API allows. - -Converts them to activities, and writes them in activity format to Elasticsearch. - -Specification: ------------------ - -[TwitterHistoryElasticsearch.dot](src/main/resources/TwitterHistoryElasticsearch.dot "TwitterHistoryElasticsearch.dot" ) - -Diagram: ------------------ - - - -Example Configuration: ----------------------- - - twitter { - â â host = "api.twitter.com" - â â endpoint = "statuses/user_timeline" - â â oauth { - â â â â consumerKey = "" - â â â â consumerSecret = "" - â â â â accessToken = "" - â â â â accessTokenSecret = "" - â â } - â â info = [ - "42232950" - "211620426" - ] - } - elasticsearch { - â â hosts = [ - â â â â localhost - â â ] - â â port = 9300 - â â clusterName = elasticsearch - â â index = userhistory_activity - â â type = activity - } - -In the Twitter section you should place all of your relevant authentication keys and whichever Twitter IDs you want to pull history for. - -Twitter IDs can be converted from screennames at http://www.gettwitterid.com - -Build: ---------- - -`mvn clean package verify` - -Run: --------- - - java -cp target/twitter-history-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=application.conf org.apache.streams.example.twitter.TwitterHistoryElasticsearch - -Deploy: --------- -`mvn -Pdocker clean package docker:build` - -`docker tag twitter-history-elasticsearch:0.2-incubating-SNAPSHOT <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT` - -`docker push <dockerregistry>:twitter-history-elasticsearch:0.2-incubating-SNAPSHOT` - -`docker run <dockerregistry>:twitter-history-elasticsearch:0.2-incubating-SNAPSHOT.jar java -cp twitter-history-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.twitter.TwitterHistoryElasticsearch` +[README.md](src/site/markdown/index.md "README") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-history-elasticsearch/pom.xml ---------------------------------------------------------------------- diff --git a/local/twitter-history-elasticsearch/pom.xml b/local/twitter-history-elasticsearch/pom.xml index c263bfe..2fe4fe2 100644 --- a/local/twitter-history-elasticsearch/pom.xml +++ b/local/twitter-history-elasticsearch/pom.xml @@ -37,6 +37,7 @@ </description> <properties> + <docker.repo>apachestreams</docker.repo> <elasticsearch.version>1.1.0</elasticsearch.version> <lucene.version>4.7.2</lucene.version> </properties> @@ -198,37 +199,6 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <finalName>${project.build.finalName}</finalName> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - <exclude>**/logback.xml</exclude> - <exclude>**/log4j.properties</exclude> - </excludes> - </filter> - </filters> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass>org.apache.streams.example.twitter.TwitterHistoryElasticsearch</mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> </plugin> <plugin> <groupId>org.jsonschema2pojo</groupId> @@ -273,19 +243,19 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> - <version>2.4</version> + <configuration> + <includes>**/*.json</includes> + <outputDirectory>${project.build.directory}/test-classes</outputDirectory> + <includeGroupIds>org.apache.streams</includeGroupIds> + <includeTypes>test-jar</includeTypes> + </configuration> <executions> <execution> - <id>resource-dependencies</id> + <id>test-resource-dependencies</id> <phase>process-test-resources</phase> <goals> <goal>unpack-dependencies</goal> </goals> - <configuration> - <includeArtifactIds>streams-pojo</includeArtifactIds> - <includes>**/*.json</includes> - <outputDirectory>${project.build.directory}/test-classes</outputDirectory> - </configuration> </execution> </executions> </plugin> @@ -303,46 +273,11 @@ </execution> </executions> </plugin> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + </plugin> </plugins> </build> - <profiles> - <profile> - <id>docker</id> - <build> - <plugins> - <plugin> - <!-- The Docker Maven plugin is used to create docker image with the fat jar --> - <groupId>org.jolokia</groupId> - <artifactId>docker-maven-plugin</artifactId> - <version>0.11.0</version> - <configuration> - <images> - - <image> - <alias>${project.artifactId}</alias> - <name>${project.artifactId}:${project.version}</name> - <build> - <from>dockerfile/java:oracle-java8</from> - <assembly> - <basedir>/</basedir> - <descriptorRef>artifact</descriptorRef> - </assembly> - <!-- Default command for the build image --> - </build> - - </image> - - </images> - </configuration> - - </plugin> - - </plugins> - </build> - <activation> - <activeByDefault/> - </activation> - </profile> - </profiles> </project> http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-history-elasticsearch/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/twitter-history-elasticsearch/src/site/markdown/index.md b/local/twitter-history-elasticsearch/src/site/markdown/index.md index 6786bab..c36243b 100644 --- a/local/twitter-history-elasticsearch/src/site/markdown/index.md +++ b/local/twitter-history-elasticsearch/src/site/markdown/index.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - twitter-history-elasticsearch ============================== @@ -19,7 +15,7 @@ Converts them to activities, and writes them in activity format to Elasticsearch Specification: ----------------- -[TwitterHistoryElasticsearch.dot](src/main/resources/TwitterHistoryElasticsearch.dot "TwitterHistoryElasticsearch.dot" ) +[TwitterHistoryElasticsearch.dot](TwitterHistoryElasticsearch.dot "TwitterHistoryElasticsearch.dot" ) Diagram: ----------------- @@ -29,29 +25,7 @@ Diagram: Example Configuration: ---------------------- - twitter { - â â host = "api.twitter.com" - â â endpoint = "statuses/user_timeline" - â â oauth { - â â â â consumerKey = "" - â â â â consumerSecret = "" - â â â â accessToken = "" - â â â â accessTokenSecret = "" - â â } - â â info = [ - "42232950" - "211620426" - ] - } - elasticsearch { - â â hosts = [ - â â â â localhost - â â ] - â â port = 9300 - â â clusterName = elasticsearch - â â index = userhistory_activity - â â type = activity - } +[application.conf](application.conf "application.conf" ) In the Twitter section you should place all of your relevant authentication keys and whichever Twitter IDs you want to pull history for. @@ -60,19 +34,23 @@ Twitter IDs can be converted from screennames at http://www.gettwitterid.com Build: --------- -`mvn clean package verify` + mvn clean package verify + +Run (Local): +------------ + + java -cp dist/twitter-history-elasticsearch-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.twitter.TwitterHistoryElasticsearch -Run: --------- +Deploy (Docker): +---------------- - java -cp target/twitter-history-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=application.conf org.apache.streams.example.twitter.TwitterHistoryElasticsearch + mvn -Pdocker -Ddocker.repo=<your docker host>:<your docker repo> docker:build docker:push -Deploy: --------- -`mvn -Pdocker clean package docker:build` +Run (Docker): +------------- -`docker tag twitter-history-elasticsearch:0.2-incubating-SNAPSHOT <dockerregistry>:elasticsearch-reindex:0.2-incubating-SNAPSHOT` + docker run twitter-history-elasticsearch java -cp twitter-history-elasticsearch-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.example.twitter.TwitterHistoryElasticsearch -`docker push <dockerregistry>:twitter-history-elasticsearch:0.2-incubating-SNAPSHOT` +[JavaDocs](apidocs/index.html "JavaDocs") -`docker run <dockerregistry>:twitter-history-elasticsearch:0.2-incubating-SNAPSHOT.jar java -cp twitter-history-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.twitter.TwitterHistoryElasticsearch` +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-userstream-elasticsearch/README.md ---------------------------------------------------------------------- diff --git a/local/twitter-userstream-elasticsearch/README.md b/local/twitter-userstream-elasticsearch/README.md index e83f32b..a09b345 100644 --- a/local/twitter-userstream-elasticsearch/README.md +++ b/local/twitter-userstream-elasticsearch/README.md @@ -2,71 +2,7 @@ Apache Streams (incubating) Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 -------------------------------------------------------------------------------- -twitter-userstream-elasticsearch -============================== +org.apache.streams:twitter-userstream-elasticsearch +=================================================== -Requirements: -------------- - - Authorized Twitter API credentials - - A running ElasticSearch 1.0.0+ instance - -Description: ------------- -Connects to an active twitter account and stores the userstream as activities in Elasticsearch - -Specification: ------------------ - -[TwitterUserstreamElasticsearch.dot](src/main/resources/TwitterUserstreamElasticsearch.dot "TwitterUserstreamElasticsearch.dot" ) - -Diagram: ------------------ - - - -Example Configuration: ----------------------- - - twitter { - â â endpoint = "userstream" - â â oauth { - â â â â â â consumerKey = "bcg14JThZEGoZ3MZOoT2HnJS7" - â â â â â â consumerSecret = "S4dwxnZni58CIJaoupGnUrO4HRHmbBGOb28W6IqOJBx36LPw2z" - â â â â â â accessToken = "" - â â â â â â accessTokenSecret = "" - } - } - elasticsearch { - â â hosts = [ - â â â â localhost - â â ] - â â port = 9300 - â â clusterName = elasticsearch - â â index = userstream_activity - â â type = activity - batchSize = 1 - } - -The consumerKey and consumerSecret are set for our streams-example application -The accessToken and accessTokenSecret can be obtained by navigating to: - https://api.twitter.com/oauth/authenticate?oauth_token=UIJ0AUxCJatpKDUyFt0OTSEP4asZgqxRwUCT0AMSwc&oauth_callback=http%3A%2F%2Foauth.streamstutorial.w2odata.com%3A8080%2Fsocialauthdemo%2FsocialAuthSuccessAction.do - -Build: ---------- - -`mvn clean package verify` - -Run: --------- - -`java -cp target/twitter-userstream-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.twitter.TwitterUserstreamElasticsearch` - -Deploy: --------- -`mvn -Pdocker clean package docker:build` - -`docker tag twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT <dockerregistry>:twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT` - -`docker push <dockerregistry>:twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT` - -`docker run <dockerregistry>:twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT.jar java -cp twitter-userstream-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.twitter.TwitterUserstreamElasticsearch` +[README.md](src/site/markdown/index.md "README") http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-userstream-elasticsearch/pom.xml ---------------------------------------------------------------------- diff --git a/local/twitter-userstream-elasticsearch/pom.xml b/local/twitter-userstream-elasticsearch/pom.xml index 9cfbdec..e89a224 100644 --- a/local/twitter-userstream-elasticsearch/pom.xml +++ b/local/twitter-userstream-elasticsearch/pom.xml @@ -35,6 +35,7 @@ </description> <properties> + <docker.repo>apachestreams</docker.repo> <elasticsearch.version>1.1.0</elasticsearch.version> <lucene.version>4.7.2</lucene.version> </properties> @@ -145,35 +146,6 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <finalName>${project.build.finalName}</finalName> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - </excludes> - </filter> - </filters> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" /> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass>org.apache.streams.example.twitter.TwitterUserstreamElasticsearch</mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> </plugin> <plugin> <groupId>org.jsonschema2pojo</groupId> @@ -218,19 +190,20 @@ <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-dependency-plugin</artifactId> - <version>2.4</version> + <configuration> + <includes>**/*.json</includes> + <outputDirectory>${project.build.directory}/test-classes</outputDirectory> + <includeGroupIds>org.apache.streams</includeGroupIds> + <includeArtifactIds>streams-pojo</includeArtifactIds> + <includeTypes>test-jar</includeTypes> + </configuration> <executions> <execution> - <id>resource-dependencies</id> + <id>test-resource-dependencies</id> <phase>process-test-resources</phase> <goals> <goal>unpack-dependencies</goal> </goals> - <configuration> - <includeArtifactIds>streams-pojo</includeArtifactIds> - <includes>**/*.json</includes> - <outputDirectory>${project.build.directory}/test-classes</outputDirectory> - </configuration> </execution> </executions> </plugin> @@ -248,46 +221,11 @@ </execution> </executions> </plugin> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + </plugin> </plugins> </build> - <profiles> - <profile> - <id>docker</id> - <build> - <plugins> - <plugin> - <!-- The Docker Maven plugin is used to create docker image with the fat jar --> - <groupId>org.jolokia</groupId> - <artifactId>docker-maven-plugin</artifactId> - <version>0.11.0</version> - <configuration> - <images> - - <image> - <alias>${project.artifactId}</alias> - <name>${project.artifactId}:${project.version}</name> - <build> - <from>dockerfile/java:oracle-java8</from> - <assembly> - <basedir>/</basedir> - <descriptorRef>artifact</descriptorRef> - </assembly> - <!-- Default command for the build image --> - </build> - - </image> - - </images> - </configuration> - - </plugin> - - </plugins> - </build> - <activation> - <activeByDefault/> - </activation> - </profile> - </profiles> </project> http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/local/twitter-userstream-elasticsearch/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/local/twitter-userstream-elasticsearch/src/site/markdown/index.md b/local/twitter-userstream-elasticsearch/src/site/markdown/index.md index c152f6e..baeee26 100644 --- a/local/twitter-userstream-elasticsearch/src/site/markdown/index.md +++ b/local/twitter-userstream-elasticsearch/src/site/markdown/index.md @@ -1,7 +1,3 @@ -Apache Streams (incubating) -Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 --------------------------------------------------------------------------------- - twitter-userstream-elasticsearch ============================== @@ -27,25 +23,7 @@ Diagram: Example Configuration: ---------------------- - twitter { - â â endpoint = "userstream" - â â oauth { - â â â â â â consumerKey = "bcg14JThZEGoZ3MZOoT2HnJS7" - â â â â â â consumerSecret = "S4dwxnZni58CIJaoupGnUrO4HRHmbBGOb28W6IqOJBx36LPw2z" - â â â â â â accessToken = "" - â â â â â â accessTokenSecret = "" - } - } - elasticsearch { - â â hosts = [ - â â â â localhost - â â ] - â â port = 9300 - â â clusterName = elasticsearch - â â index = userstream_activity - â â type = activity - batchSize = 1 - } +[application.conf](application.conf "application.conf" ) The consumerKey and consumerSecret are set for our streams-example application The accessToken and accessTokenSecret can be obtained by navigating to: @@ -55,19 +33,23 @@ The accessToken and accessTokenSecret can be obtained by navigating to: Build: --------- -`mvn clean package verify` + mvn clean package verify + +Run (Local): +------------ -Run: --------- + java -cp dist/twitter-userstream-elasticsearch-jar-with-dependencies.jar -Dconfig.file=file://<location_of_config_file> org.apache.streams.example.twitter.TwitterUserstreamElasticsearch -`java -cp target/twitter-userstream-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=src/main/resources/application.json org.apache.streams.example.twitter.TwitterUserstreamElasticsearch` +Deploy (Docker): +---------------- -Deploy: --------- -`mvn -Pdocker clean package docker:build` + mvn -Pdocker -Ddocker.repo=<your docker host>:<your docker repo> docker:build docker:push + +Run (Docker): +------------- -`docker tag twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT <dockerregistry>:twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT` + docker run twitter-userstream-elasticsearch java -cp twitter-userstream-elasticsearch-jar-with-dependencies.jar -Dconfig.url=http://<location_of_config_file> org.apache.streams.example.twitter.TwitterUserstreamElasticsearch -`docker push <dockerregistry>:twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT` +[JavaDocs](apidocs/index.html "JavaDocs") -`docker run <dockerregistry>:twitter-userstream-elasticsearch:0.2-incubating-SNAPSHOT.jar java -cp twitter-userstream-elasticsearch-0.2-incubating-SNAPSHOT.jar -Dconfig.file=http://<location_of_config_file>.json org.apache.streams.example.twitter.TwitterUserstreamElasticsearch` +###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index 1e385b7..9213ff3 100644 --- a/pom.xml +++ b/pom.xml @@ -165,6 +165,8 @@ https://repository.apache.org/service/local/staging/deploy/maven2 </release.repository.url> + <docker.plugin.version>0.15.0</docker.plugin.version> + </properties> <modules> @@ -262,6 +264,60 @@ <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-shade-plugin</artifactId> + <version>${shade.plugin.version}</version> + <configuration> + <outputDirectory>dist</outputDirectory> + <outputFile>dist/${project.artifactId}-jar-with-dependencies.jar</outputFile> + <filters> + <filter> + <artifact>*:*</artifact> + <excludes> + <exclude>**/META-INF/**</exclude> + </excludes> + </filter> + </filters> + </configuration> + <executions> + <execution> + <phase>package</phase> + <goals> + <goal>shade</goal> + </goals> + <configuration> + <finalName>${project.build.finalName}</finalName> + <filters> + <filter> + <artifact>*:*</artifact> + <excludes> + <exclude>META-INF/*.SF</exclude> + <exclude>META-INF/*.DSA</exclude> + <exclude>META-INF/*.RSA</exclude> + <exclude>**/logback.xml</exclude> + <exclude>**/log4j.properties</exclude> + </excludes> + </filter> + </filters> + <transformers> + <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> + <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> + <resource>reference.conf</resource> + </transformer> + <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> + <resource>application.conf</resource> + </transformer> + <!--<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">--> + <!--<manifestEntries>--> + <!--<Main-Class>akka.Main</Main-Class>--> + <!--</manifestEntries>--> + <!--</transformer>--> + </transformers> + </configuration> + </execution> + </executions> + </plugin> + <plugin> + <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-site-plugin</artifactId> <version>3.4</version> <dependencies> @@ -290,4 +346,71 @@ </plugins> </pluginManagement> </build> + + <profiles> + <profile> + <id>docker</id> + <activation> + <activeByDefault>false</activeByDefault> + <property> + <name>docker.repo</name> + </property> + </activation> + <build> + <plugins> + <plugin> + <groupId>io.fabric8</groupId> + <artifactId>docker-maven-plugin</artifactId> + <version>${docker.plugin.version}</version> + <executions> + <execution> + <id>deploy</id> + <phase>deploy</phase> + <goals> + <goal>build</goal> + <goal>push</goal> + </goals> + </execution> + </executions> + <configuration combine.self="append"> + <watchInterval>500</watchInterval> + <logDate>default</logDate> + <verbose>true</verbose> + <images> + <image> + <name>${docker.repo}/${project.artifactId}</name> + <alias>${project.artifactId}</alias> + <build> + <from>1science/java</from> + <maintainer>sblack...@apache.org</maintainer> + <env> + <LANG>en_US.UTF-8</LANG> + <LC_ALL>en_US.UTF-8</LC_ALL> + </env> + <tags> + <tag>${project.version}</tag> + <tag>latest</tag> + </tags> + <assembly> + <mode>tar</mode> + <basedir>/</basedir> + <inline> + <baseDirectory>/</baseDirectory> + <fileSets> + <fileSet> + <directory>dist</directory> + <outputDirectory>/</outputDirectory> + </fileSet> + </fileSets> + </inline> + </assembly> + </build> + </image> + </images> + </configuration> + </plugin> + </plugins> + </build> + </profile> + </profiles> </project> http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/src/site/markdown/index.md ---------------------------------------------------------------------- diff --git a/src/site/markdown/index.md b/src/site/markdown/index.md deleted file mode 100644 index ac037b5..0000000 --- a/src/site/markdown/index.md +++ /dev/null @@ -1,23 +0,0 @@ -Streams Examples -================ -This repository contains 'ready-to-run' community-supported example applications built on Apache Streams (incubating). - -These examples demonstrate some ways you can use various streams components and runtimes, and patterns for testing and documenting your organization's streams. - -Note -==== - -The Streams Examples project itself (this one) depends on the separate Streams project, which contains the source code and poms for Apache Streams. - -Releases of streams-project are published to the Apache Releases repository, so there is typically no need to check it out manually and build it locally yourself. - -However, you may want to work from a SNAPSHOT branch and/or alter the versions of dependency libraries, please refer to the Tutorial steps at http://streams.incubator.apache.org to build streams-project and streams-examples. - -[Modules](modules.html "Modules") - -Release Notes -============= - -[CHANGELOG.md](CHANGELOG.md "CHANGELOG.md") - -###### Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0 http://git-wip-us.apache.org/repos/asf/incubator-streams-examples/blob/3c476284/src/site/site.xml ---------------------------------------------------------------------- diff --git a/src/site/site.xml b/src/site/site.xml index fef0628..5cde002 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -17,13 +17,21 @@ ~ under the License. --> <project> - <version position="right"/> - <skin> - <groupId>org.apache.maven.skins</groupId> - <artifactId>maven-fluido-skin</artifactId> - <version>1.3.0</version> - </skin> - <body> - <menu ref="reports"/> - </body> + <custom> + <fluidoSkin> + <topBarEnabled>true</topBarEnabled> + <navBarStyle>navbar-inverse</navBarStyle> + <sideBarEnabled>false</sideBarEnabled> + <gitHub> + <projectId>apache/incubator-streams-examples</projectId> + <ribbonOrientation>right</ribbonOrientation> + <ribbonColor>black</ribbonColor> + </gitHub> + <twitter> + <user>ApacheStreams</user> + <showUser>true</showUser> + <showFollowers>true</showFollowers> + </twitter> + </fluidoSkin> + </custom> </project> \ No newline at end of file
