adoroszlai commented on a change in pull request #68: AMBARI-23456. Add developement guide markdowns & remove some old docs URL: https://github.com/apache/ambari-logsearch/pull/68#discussion_r245392746
########## File path: README.md ########## @@ -2,89 +2,38 @@ [](https://builds.apache.org/view/A/view/Ambari/job/Ambari-LogSearch-master-Commit/)  -Log aggregation, analysis, and visualization for Ambari managed (and any other) services. - Log Search is a sub-project of [Apache Ambari](https://github.com/apache/ambari) -## Development - -Requires JDK 8 (JDK 11 is recommended) and Maven 3.5.x - -### Prerequisites - -- Install [docker](https://docs.docker.com/) -- For Mac OS X use [Docker for Mac](https://docs.docker.com/docker-for-mac/) -- [Docker compose](https://docs.docker.com/compose/) is also required. - -### Build and start Log Search in docker container -```bash -# to see available commands: run start-logsearch without arguments -cd docker -./logsearch-docker build-and-run # build mvn project locally, build docker image, start containers -``` -If you run the script at first time, it will generate you a new `Profile` file or an `.env` file inside docker directory (run twice if both missing and you want to generate Profile and .env as well), in .env file you should set `MAVEN_REPOSITORY_LOCATION` (point to local maven repository location, it uses `~/.m2` by default). These will be used as volumes for the docker container. Profile file holds the environment variables that are used inside the containers, the .env file is used outside of the containers - -Then you can use the `logsearch-docker` script to start the containers (`start` command). -Also you can use docker-compose manually to start/manage the containers. -```bash -docker-compose up -d -# or start all services in one container: -docker-compose -f all.yml up -d -``` -After the logsearch container is started you can enter to it with following commands: -```bash -docker exec -it docker_logsearch_1 bash -# or if you used all.yml for starting the logsearch docker container: -docker exec -it logsearch bash -``` -In case if you started the containers separately and if you would like to access Solr locally with through your external ZooKeeper container, then point `solr` to `localhost` in your `/etc/hosts` file. - -### Run applications from IDE / maven - -- [Start Log Search locally](ambari-logsearch-server/README.md) -- [Start Log Feeder locally](ambari-logsearch-logfeeder/README.md) - -## Package build process +## Introduction -1. Check out the code from GIT repository +Log aggregation, analysis, and visualization for Ambari managed (or any other) services. -2. On the logsearch root folder (ambari/ambari-logsearch), please execute the following make command to build RPM/DPKG: -```bash -make rpm -# or for jdk11 -export LOGSEARCH_JDK_11=true -make rpm -``` - or -```bash -make deb -# or for jdk11 -export LOGSEARCH_JDK_11=true -make deb -``` -3. Generated RPM/DPKG files will be found in ambari-logsearch-assembly/target folder +## Features -## Running Integration Tests +- Parse / aggregate and ship logs +- Send and index logs in Solr +- Store logs (structured or un-sructured format) in Cloud Storage (S3 / GCS / ADLS / WASB) +- Fultext Search in logs (if the logs are shipped to Solr) +- JWT/SSO support +- Support testing the log parsing on the UI -By default integration tests are not a part of the build process, you need to set -Dbackend-tests or -Dselenium-tests (or you can use -Dall-tests to run both). To running the tests you will need docker here as well (right now docker-for-mac and unix are supported by default, for boot2docker you need to pass -Ddocker.host parameter to the build). +## Architecture -```bash -# from ambari-logsearch folder -mvn clean integration-test -Dbackend-tests failsafe:verify -# or run selenium tests with docker for mac, but before that you nedd to start xquartz -open -a XQuartz -# then in an another window you can start ui tests -mvn clean integration-test -Dselenium-tests failsafe:verify -# you can specify story file folde location with -Dbackend.stories.location and -Dui.stories.location (absolute file path) in the commands -``` -Also you can run from the IDE, but make sure all of the ambari logsearch modules are built. +- Log Feeder: agent component on all hosts to monitor and shipping logs. +- Log Search Portal: REST API + UI for rendering logs +- Solr (Optional - default): storage for logs, used by both Log Search Portal and Log Feeder +- ZooKeeper (Optional - default): configuration service for Solr, Log Search and Log Feeder +- HDFS / S3 / GCS / ADLS: storage for logs (write only), used by Log Feeder [cloud mode](docs/cloud_mode.md) + -## Update version (for release or specific builds) +## Contents -```bash -make update-version new-version="2.8.0.0-11" -``` +- [1, Installation](docs/installation.md) Review comment: `1,` -> `1.` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
