Repository: incubator-senssoft Updated Branches: refs/heads/master 8e255a73d -> 10a0481d6
updates to readme-added context for dependencies, guidance, images, license Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/10a0481d Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/10a0481d Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/10a0481d Branch: refs/heads/master Commit: 10a0481d62d18079481585a472e441b15ac8aff1 Parents: 8e255a7 Author: poorejc <poor...@apache.org> Authored: Sun Dec 30 23:31:16 2018 -0500 Committer: poorejc <poor...@apache.org> Committed: Sun Dec 30 23:31:16 2018 -0500 ---------------------------------------------------------------------- docker/README.md | 135 ++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 102 insertions(+), 33 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/10a0481d/docker/README.md ---------------------------------------------------------------------- diff --git a/docker/README.md b/docker/README.md index 3b7f51d..6b2596f 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,18 +1,33 @@ How to Build SensSoft Docker Containers ======================================= +*Last Tested (on macOS Mojave) 22 DEC 2018* Prerequisites ------------- 1. Install [``Docker``](http://docker.com) on your machine. Requires Docker 1.7 and above. -1. Install docker-compose. Full instructions can be found [``here``](https://docs.docker.com/compose/install/). +1. Install docker-compose. Full instructions can be found [``here``](https://docs.docker.com/compose/install/). + If you install Docker through the [``Desktop bundle``](https://www.docker.com/products/docker-desktop), docker-compose is included. + +Single Node Deployment +---------------------- -1. Create docker-machine instance +The single node deployment steps below will build a single-node logging server on a single +machine. This is suitable for demonstrations and very limited data collections. Please +refer to our [``Kubernetes``](https://github.com/apache/incubator-senssoft/tree/master/kubernetes) guide for deployments that scale to your needs. In some cases, +you may be able to use single node containers within scaling services (e.g., AWS EBS), but this +requires special configuration. Please reach out to us at [our dev list](mailto:d...@senssoft.incubator.apache.org) for recommendations. + +1. Create docker-machine instance. + **Note**: If using Docker Desktop bundle, there is a known bug in + the bundled version of virtualbox that will prevent a successful docker-machine creation. + Before installation, check that virtualbox version is at least 5.2. [``Reinstall virtualbox``](https://www.virtualbox.org/wiki/Downloads), if needed. + ```bash - docker-machine create --virtualbox-memory 3072 --virtualbox-cpu-count 2 senssoft + $ docker-machine create --virtualbox-memory 3072 --virtualbox-cpu-count 2 senssoft ``` - + 1. Before launching the Docker containers, ensure your ``vm_max_map_count`` kernel setting is set to at least 262144. Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions. @@ -29,15 +44,11 @@ Prerequisites $ docker network create esnet ``` -Single Node Deployment ----------------------- - -1. Start Elasticsearch 5.6.3 or 6.2.2. Give Elasticsearch about 1-2 minutes to start before confirming - its state. +1. Start Elasticsearch 5.6.3 or 6.2.2. Give Elasticsearch about 1-2 minutes to start before confirming its state. ```bash $ docker-compose -f docker-compose.single-5.6.3.yml up -d elasticsearch - + or $ docker-compose up -d elasticsearch @@ -45,8 +56,11 @@ Single Node Deployment 1. Confirm state: ```bash + # if senssoft vm is remote $ docker-machine ssh senssoft curl -XGET http://localhost:9200/_cluster/health?pretty - { + # if senssoft virtual machine is running on your local machine, no need for ssh, instead: + $ curl -XGET http://localhost:9200/_cluster/health?pretty + #output should look like this: "cluster_name" : "SensSoft", "status" : "yellow", "timed_out" : false, @@ -62,7 +76,6 @@ Single Node Deployment "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 - } ``` 1. Launch logging server. Give Logstash about 2 minutes to start before confirming @@ -70,8 +83,12 @@ Single Node Deployment ```bash $ docker-compose up -d logstash - $ docker-machine ssh senssoft curl -XGET http://localhost:8100 - ok + # if senssoft vm is remote + $ docker-machine ssh senssoft curl -XGET http://localhost:8100 + # if senssoft virtual machine is running on your local machine, no need for ssh, instead: + $ curl -XGET http://localhost:8100 + #ouput should look like this: + ok ``` 1. Before Kibana can be used, we will need to generate some data. We have already @@ -79,6 +96,7 @@ Single Node Deployment ```bash $ docker-compose up -d site + # for remote users, forwards port to localhost $ ssh docker@$(docker-machine ip senssoft) -L 8080:localhost:8080 ``` @@ -89,6 +107,7 @@ Single Node Deployment ```bash $ docker-compose up -d kibana + # for remote users, forwards port to localhost $ ssh docker@$(docker-machine ip senssoft) -L 5601:localhost:5601 ``` @@ -102,7 +121,7 @@ Single Node Deployment 1. Load example Dashboard and Visualizations under docker/kibana/. Goto: Management -> Saved Objects and select the `Import` button. Import the - `visualizations.json` and `dashboard.json` file. + `visualizations.json`, `Apache SensSoft Usage Dashboard.json`, and `Drill-Down Search.json` files from the "Saved Objects" folder in the kibana directory. ![alt text][management] @@ -111,26 +130,41 @@ Single Node Deployment ![alt text][confirmation] Once that is complete, navigate to the `Dashboard` view in Kibana and click the - `Apache SensSoft Dashboard` object. + `Apache SensSoft Usage Dashboard` object. ![alt text][dashboard] - -1. To Launch Tap and Distill - ```bash - $ docker-compose up -d distill tap - ``` 1. To stop all containers. ```bash $ docker-compose stop ``` + + 1. To kill the senssoft machine. + ```bash + $ docker-machine rm senssoft + ``` + +If running on a single machine, on reboot or restart your senssoft machine is available, but +in a "stopped" state. You'll need to restart the machine, then you'll need to use docker- +compose up commands above to restart containers. + + 1. Restart the senssoft machine. + ```bash + $ docker-machine start senssoft + $ docker-machine ls #confirm state + #output should look like this: + NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS + senssoft - virtualbox Running tcp://192.168.99.100:2376 v18.09.0 + ``` Multi-Node Deployment on a Single Machine ----------------------------------------- -1. Install [``Docker``](http://docker.com) on your machine. Require Docker 1.7 and above. - -1. Install ``docker-compose`` in an virtual environment. Full instructions can be found [``here``](https://docs.docker.com/compose/install/). +<aside class="warning"> + Starting an elasticsearch cluster is not recommended on a single server. This + is just for demonstration purposes only. Please refer to our [Kubernetes] guide to + deploy an Elasticsearch cluster for an enterprise scale logging capability. + </aside> 1. Create docker-machine instance ```bash @@ -166,15 +200,12 @@ Multi-Node Deployment on a Single Machine and balances them across the elasticsearch worker nodes. The elasticsearch worker nodes communicate to each other and the loadbalancer via TCP on port 9300. - <aside class="warning"> - Starting an elasticsearch cluster is not recommended on a single server. This - is just for demonstration purposes only. Please refer to our [Kubernetes] guide to - deploy an Elasticsearch cluster. - </aside> - 1. Confirm cluster state: ```bash $ docker-machine ssh senssoft curl -XGET http://localhost:9200/_cluster/health\?pretty + # if senssoft virtual machine is running on your local machine, no need for ssh, instead: + $ curl -XGET http://localhost:9200/_cluster/health?pretty + #output should look like this: { "cluster_name" : "SensSoft", "status" : "green", @@ -196,6 +227,29 @@ Multi-Node Deployment on a Single Machine Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3. 1. Follow remaining instructions in [Single Node Deployment] starting at #6. + +1. To stop all containers. + ```bash + $ docker-compose stop + ``` + + 1. To kill the "senssoft" machine. + ```bash + $ docker-machine rm senssoft + ``` + +If running on a single machine, on reboot or restart your senssoft machine is available, but +in a "stopped" state. You'll need to restart the machine, then you'll need to use docker- +compose up commands above to restart containers. + + 1. Restart the senssoft machine. + ```bash + $ docker-machine start senssoft + $ docker-machine ls #confirm state + #output should look like this: + NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS + senssoft - virtualbox Running tcp://192.168.99.100:2376 v18.09.0 + ``` Having Issues? -------------- @@ -204,15 +258,30 @@ Having Issues? ```bash $ docker-compose ps ``` -1. Make sure to send us the docker-compose logs to help diagnose your issues please! + +1. If you find containers failing, you may have duplicate or dangling images! + This can happen if you've played around with multiple machines and builds of the containers + on the same machine. Visit [``this excellent how to guide for removing images, containers, + and volumes``](https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes). Remove any duplicate images and rebuild the containers. + +1. Make sure to send us the docker-compose logs to help diagnose your issues please! + ```bash $ docker-compose logs > err.dump ``` + + You can attach logs directly to tickets on our [``Apache Jira board``](https://issues.apache.org/jira/issues/?jql=project+%3D+SENSSOFT+AND+component+%3D+builds) + +1. Still having issues? Reach out to us at at [our dev list](mailto:d...@senssoft.incubator.apache.org). [configure_index]: ./docs/images/configure_index.png "Configure Kibana index" [confirmation]: ./docs/images/confirmation.png "Confirm index pattern conflicts" -[dashboard]: ./docs/images/dashboard.png "Apache Senssoft Dashboard" +[dashboard]: ./docs/images/Apache_SensSoft_Usage_Dashboard.png "Apache Senssoft Usage Dashboard" [management]: ./docs/images/management.png "Kibana management console" -© Copyright 2016 The Charles Stark Draper Laboratory, Inc. All rights reserved. +Licensing +-------------- + +Apache SensSoft is provided under Apache License version 2.0. See LICENSE (at Master) file for more +details. "Software as a Sensor" is Copyright © 2016 The Charles Stark Draper Laboratory, Inc. All rights reserved.
