Repository: incubator-distributedlog Updated Branches: refs/heads/master 9c6c9c452 -> 0cb775c93
http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-basic/basic-2.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-basic/basic-2.md b/distributedlog-tutorials/distributedlog-basic/basic-2.md index 0459bbf..b28cb9a 100644 --- a/distributedlog-tutorials/distributedlog-basic/basic-2.md +++ b/distributedlog-tutorials/distributedlog-basic/basic-2.md @@ -74,7 +74,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create the stream under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-basic/basic-3.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-basic/basic-3.md b/distributedlog-tutorials/distributedlog-basic/basic-3.md index 56972bf..5ddea90 100644 --- a/distributedlog-tutorials/distributedlog-basic/basic-3.md +++ b/distributedlog-tutorials/distributedlog-basic/basic-3.md @@ -107,7 +107,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create multi streams under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-basic/basic-4.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-basic/basic-4.md b/distributedlog-tutorials/distributedlog-basic/basic-4.md index 684bdeb..e30d628 100644 --- a/distributedlog-tutorials/distributedlog-basic/basic-4.md +++ b/distributedlog-tutorials/distributedlog-basic/basic-4.md @@ -92,7 +92,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create the stream under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-basic/basic-5.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-basic/basic-5.md b/distributedlog-tutorials/distributedlog-basic/basic-5.md index 684f228..ba984ee 100644 --- a/distributedlog-tutorials/distributedlog-basic/basic-5.md +++ b/distributedlog-tutorials/distributedlog-basic/basic-5.md @@ -90,7 +90,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create the stream under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-basic/basic-6.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-basic/basic-6.md b/distributedlog-tutorials/distributedlog-basic/basic-6.md index 3edd596..b06998f 100644 --- a/distributedlog-tutorials/distributedlog-basic/basic-6.md +++ b/distributedlog-tutorials/distributedlog-basic/basic-6.md @@ -88,7 +88,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create the stream under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-messaging/messaging-1.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-messaging/messaging-1.md b/distributedlog-tutorials/distributedlog-messaging/messaging-1.md index d2c2d7f..4f72d25 100644 --- a/distributedlog-tutorials/distributedlog-messaging/messaging-1.md +++ b/distributedlog-tutorials/distributedlog-messaging/messaging-1.md @@ -44,7 +44,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create the stream under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/distributedlog-tutorials/distributedlog-messaging/messaging-2.md ---------------------------------------------------------------------- diff --git a/distributedlog-tutorials/distributedlog-messaging/messaging-2.md b/distributedlog-tutorials/distributedlog-messaging/messaging-2.md index 5622888..7c3e79a 100644 --- a/distributedlog-tutorials/distributedlog-messaging/messaging-2.md +++ b/distributedlog-tutorials/distributedlog-messaging/messaging-2.md @@ -76,7 +76,7 @@ Run the example in the following steps: 2. Start the write proxy, listening on port 8000. ``` // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf ``` 3. Create the stream under the distributedlog uri. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/admin_guide/bookkeeper.rst ---------------------------------------------------------------------- diff --git a/docs/admin_guide/bookkeeper.rst b/docs/admin_guide/bookkeeper.rst index c60fbf1..8155c14 100644 --- a/docs/admin_guide/bookkeeper.rst +++ b/docs/admin_guide/bookkeeper.rst @@ -55,10 +55,10 @@ To build bookkeeper, run: $ cd bookkeeper $ mvn clean package assembly:single -DskipTests -However, since `bookkeeper-server` is one of the dependency of `distributedlog-service`. -You could simply run bookkeeper using same set of scripts provided in `distributedlog-service`. +However, since `bookkeeper-server` is one of the dependency of `distributedlog-proxy-server`. +You could simply run bookkeeper using same set of scripts provided in `distributedlog-proxy-server`. In the following sections, we will describe how to run bookkeeper using the scripts provided in -`distributedlog-service`. +`distributedlog-proxy-server`. Run from distributedlog source ------------------------------ @@ -76,7 +76,7 @@ First of all, build DistributedLog: Configuration +++++++++++++ -The configuration file `bookie.conf` under `distributedlog-service/conf` is a template of production +The configuration file `bookie.conf` under `distributedlog-proxy-server/conf` is a template of production configuration to run a bookie node. Most of the configuration settings are good for production usage. You might need to configure following settings according to your environment and hardware platform. @@ -191,21 +191,21 @@ It is recommended to configure following settings to align with the cpu cores of Run +++ -As `bookkeeper-server` is shipped as part of `distributedlog-service`, you could use the `dlog-daemon.sh` +As `bookkeeper-server` is shipped as part of `distributedlog-proxy-server`, you could use the `dlog-daemon.sh` script to start `bookie` as daemon thread. Start the bookie: .. code-block:: bash - $ ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf /path/to/bookie/conf + $ ./distributedlog-proxy-server/bin/dlog-daemon.sh start bookie --conf /path/to/bookie/conf Stop the bookie: .. code-block:: bash - $ ./distributedlog-service/bin/dlog-daemon.sh stop bookie + $ ./distributedlog-proxy-server/bin/dlog-daemon.sh stop bookie Please check bookkeeper_ website for more details. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/admin_guide/operations.rst ---------------------------------------------------------------------- diff --git a/docs/admin_guide/operations.rst b/docs/admin_guide/operations.rst index e56c596..99826c4 100644 --- a/docs/admin_guide/operations.rst +++ b/docs/admin_guide/operations.rst @@ -158,7 +158,7 @@ A CLI is provided for inspecting DistributedLog streams and metadata. or command is the full name of a class with a defined main() method. Environment variables: - DLOG_LOG_CONF Log4j configuration file (default $HOME/src/distributedlog/distributedlog-service/conf/log4j.properties) + DLOG_LOG_CONF Log4j configuration file (default $HOME/src/distributedlog/distributedlog-proxy-server/conf/log4j.properties) DLOG_EXTRA_OPTS Extra options to be passed to the jvm DLOG_EXTRA_CLASSPATH Add extra paths to the dlog classpath http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/admin_guide/zookeeper.rst ---------------------------------------------------------------------- diff --git a/docs/admin_guide/zookeeper.rst b/docs/admin_guide/zookeeper.rst index 0fee9de..8f5d7bf 100644 --- a/docs/admin_guide/zookeeper.rst +++ b/docs/admin_guide/zookeeper.rst @@ -32,10 +32,10 @@ zookeeper version `3.4.8`. Run ZooKeeper from distributedlog source ---------------------------------------- -Since `zookeeper` is one of the dependency of `distributedlog-service`. You could simply -run `zookeeper` servers using same set of scripts provided in `distributedlog-service`. +Since `zookeeper` is one of the dependency of `distributedlog-proxy-server`. You could simply +run `zookeeper` servers using same set of scripts provided in `distributedlog-proxy-server`. In the following sections, we will describe how to run zookeeper using the scripts provided -in `distributedlog-service`. +in `distributedlog-proxy-server`. Build +++++ @@ -49,7 +49,7 @@ First of all, build DistributedLog: Configuration +++++++++++++ -The configuration file `zookeeper.conf.template` under `distributedlog-service/conf` is a template of +The configuration file `zookeeper.conf.template` under `distributedlog-proxy-server/conf` is a template of production configuration to run a zookeeper node. Most of the configuration settings are good for production usage. You might need to configure following settings according to your environment and hardware platform. @@ -84,20 +84,20 @@ It is recommended to put `dataLogDir` under a separated disk from others for per Run +++ -As `zookeeper` is shipped as part of `distributedlog-service`, you could use the `dlog-daemon.sh` +As `zookeeper` is shipped as part of `distributedlog-proxy-server`, you could use the `dlog-daemon.sh` script to start `zookeeper` as daemon thread. Start the zookeeper: .. code-block:: bash - $ ./distributedlog-service/bin/dlog-daemon.sh start zookeeper /path/to/zookeeper.conf + $ ./distributedlog-proxy-server/bin/dlog-daemon.sh start zookeeper /path/to/zookeeper.conf Stop the zookeeper: .. code-block:: bash - $ ./distributedlog-service/bin/dlog-daemon.sh stop zookeeper + $ ./distributedlog-proxy-server/bin/dlog-daemon.sh stop zookeeper Please check zookeeper_ website for more details. http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/deployment/cluster.rst ---------------------------------------------------------------------- diff --git a/docs/deployment/cluster.rst b/docs/deployment/cluster.rst index b4caf3f..6edff4e 100644 --- a/docs/deployment/cluster.rst +++ b/docs/deployment/cluster.rst @@ -35,7 +35,7 @@ To build DistributedLog, run: Or run `./scripts/snapshot` to build the release packages from current source. The released -packages contain the binaries for running `distributedlog-service`, `distributedlog-benchmark` +packages contain the binaries for running `distributedlog-proxy-server`, `distributedlog-benchmark` and `distributedlog-tutorials`. NOTE: we run the following instructions from distributedlog source code after @@ -54,7 +54,7 @@ Create a `zookeeper.conf` from the `zookeeper.conf.template`. .. code-block:: bash - $ cp distributedlog-service/conf/zookeeper.conf.template distributedlog-service/conf/zookeeper.conf + $ cp distributedlog-proxy-server/conf/zookeeper.conf.template distributedlog-proxy-server/conf/zookeeper.conf Configure the settings in `zookeeper.conf`. By default, it will use `/tmp/data/zookeeper` for storing the zookeeper data. Let's create the data directories for zookeeper. @@ -73,14 +73,14 @@ Start the zookeeper daemon using `dlog-daemon.sh`. .. code-block:: bash - $ ./distributedlog-service/bin/dlog-daemon.sh start zookeeper ${DL_HOME}/distributedlog-service/conf/zookeeper.conf + $ ./distributedlog-proxy-server/bin/dlog-daemon.sh start zookeeper ${DL_HOME}/distributedlog-proxy-server/conf/zookeeper.conf You could verify the zookeeper setup using `zkshell`. .. code-block:: bash - // ./distributedlog-service/bin/dlog zkshell ${zkservers} - $ ./distributedlog-service/bin/dlog zkshell localhost:2181 + // ./distributedlog-proxy-server/bin/dlog zkshell ${zkservers} + $ ./distributedlog-proxy-server/bin/dlog zkshell localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled @@ -109,9 +109,9 @@ bookkeeper cluster locally. Let's make three copies of `bookie.conf.template`. .. code-block:: bash - $ cp distributedlog-service/conf/bookie.conf.template distributedlog-service/conf/bookie-1.conf - $ cp distributedlog-service/conf/bookie.conf.template distributedlog-service/conf/bookie-2.conf - $ cp distributedlog-service/conf/bookie.conf.template distributedlog-service/conf/bookie-3.conf + $ cp distributedlog-proxy-server/conf/bookie.conf.template distributedlog-proxy-server/conf/bookie-1.conf + $ cp distributedlog-proxy-server/conf/bookie.conf.template distributedlog-proxy-server/conf/bookie-2.conf + $ cp distributedlog-proxy-server/conf/bookie.conf.template distributedlog-proxy-server/conf/bookie-3.conf Configure the settings in the bookie configuraiont files. @@ -141,7 +141,7 @@ So using `zkshell` to create the `zkLedgersRootPath`. :: - $ ./distributedlog-service/bin/dlog zkshell localhost:2181 + $ ./distributedlog-proxy-server/bin/dlog zkshell localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled @@ -162,7 +162,7 @@ If the `zkLedgersRootPath`, run `metaformat` to format the bookkeeper metadata. :: - $ BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-1.conf ./distributedlog-service/bin/dlog bkshell metaformat + $ BOOKIE_CONF=${DL_HOME}/distributedlog-proxy-server/conf/bookie-1.conf ./distributedlog-proxy-server/bin/dlog bkshell metaformat Are you sure to format bookkeeper metadata ? (Y or N) Y Add Bookies @@ -259,9 +259,9 @@ Once the disk directories are configured correctly in the configuration file, us :: - BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-1.conf ./distributedlog-service/bin/dlog bkshell bookieformat - BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-2.conf ./distributedlog-service/bin/dlog bkshell bookieformat - BOOKIE_CONF=${DL_HOME}/distributedlog-service/conf/bookie-3.conf ./distributedlog-service/bin/dlog bkshell bookieformat + BOOKIE_CONF=${DL_HOME}/distributedlog-proxy-server/conf/bookie-1.conf ./distributedlog-proxy-server/bin/dlog bkshell bookieformat + BOOKIE_CONF=${DL_HOME}/distributedlog-proxy-server/conf/bookie-2.conf ./distributedlog-proxy-server/bin/dlog bkshell bookieformat + BOOKIE_CONF=${DL_HOME}/distributedlog-proxy-server/conf/bookie-3.conf ./distributedlog-proxy-server/bin/dlog bkshell bookieformat Start bookie @@ -271,16 +271,16 @@ Start the bookie using `dlog-daemon.sh`. :: - SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-1.conf - SERVICE_PORT=3182 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-2.conf - SERVICE_PORT=3183 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-3.conf + SERVICE_PORT=3181 ./distributedlog-proxy-server/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-proxy-server/conf/bookie-1.conf + SERVICE_PORT=3182 ./distributedlog-proxy-server/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-proxy-server/conf/bookie-2.conf + SERVICE_PORT=3183 ./distributedlog-proxy-server/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-proxy-server/conf/bookie-3.conf Verify whether the bookie is setup correctly. You could simply check whether the bookie is showed up in zookeeper `zkLedgersRootPath`/available znode. :: - $ ./distributedlog-service/bin/dlog zkshell localhost:2181 + $ ./distributedlog-proxy-server/bin/dlog zkshell localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled @@ -310,9 +310,9 @@ Stop the bookie using `dlog-daemon.sh`. :: - $ ./distributedlog-service/bin/dlog-daemon.sh stop bookie + $ ./distributedlog-proxy-server/bin/dlog-daemon.sh stop bookie // Example: - $ SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh stop bookie + $ SERVICE_PORT=3181 ./distributedlog-proxy-server/bin/dlog-daemon.sh stop bookie doing stop bookie ... stopping bookie Shutdown is in progress... Please wait... @@ -325,13 +325,13 @@ Start the bookie in `readonly` mode. :: - $ SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-service/conf/bookie-1.conf --readonly + $ SERVICE_PORT=3181 ./distributedlog-proxy-server/bin/dlog-daemon.sh start bookie --conf ${DL_HOME}/distributedlog-proxy-server/conf/bookie-1.conf --readonly Verify if the bookie is running in `readonly` mode. :: - $ ./distributedlog-service/bin/dlog zkshell localhost:2181 + $ ./distributedlog-proxy-server/bin/dlog zkshell localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled @@ -379,7 +379,7 @@ bookkeeper cluster we just created above. :: - $ distributedlog-service/bin/dlog admin bind \\ + $ distributedlog-proxy-server/bin/dlog admin bind \\ -dlzr 127.0.0.1:2181 \\ -dlzw 127.0.0.1:2181 \\ -s 127.0.0.1:2181 \\ @@ -419,16 +419,16 @@ Different from bookkeeper, DistributedLog tries not to configure any environment in configuration files. Any environment related settings are stored and configured via `namespace binding`. The configuration file should contain non-environment related settings. -There is a `write_proxy.conf` template file available under `distributedlog-service` module. +There is a `write_proxy.conf` template file available under `distributedlog-proxy-server` module. Run write proxy +++++++++++++++ -A write proxy could be started using `dlog-daemon.sh` script under `distributedlog-service`. +A write proxy could be started using `dlog-daemon.sh` script under `distributedlog-proxy-server`. :: - WP_SHARD_ID=${WP_SHARD_ID} WP_SERVICE_PORT=${WP_SERVICE_PORT} WP_STATS_PORT=${WP_STATS_PORT} ./distributedlog-service/bin/dlog-daemon.sh start writeproxy + WP_SHARD_ID=${WP_SHARD_ID} WP_SERVICE_PORT=${WP_SERVICE_PORT} WP_STATS_PORT=${WP_STATS_PORT} ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy - `WP_SHARD_ID`: A non-negative integer. You don't need to guarantee uniqueness of shard id, as it is just an indicator to the client for routing the requests. If you are running the `write proxy` using a cluster scheduler @@ -436,7 +436,7 @@ A write proxy could be started using `dlog-daemon.sh` script under `distributedl - `WP_SERVICE_PORT`: The port that write proxy listens on. - `WP_STATS_PORT`: The port that write proxy exposes stats to a http endpoint. -Please check `distributedlog-service/conf/dlogenv.sh` for more environment variables on configuring write proxy. +Please check `distributedlog-proxy-server/conf/dlogenv.sh` for more environment variables on configuring write proxy. - `WP_CONF_FILE`: The path to the write proxy configuration file. - `WP_NAMESPACE`: The distributedlog namespace that the write proxy is serving for. @@ -445,9 +445,9 @@ For example, we start 3 write proxies locally and point to the namespace created :: - $ WP_SHARD_ID=1 WP_SERVICE_PORT=4181 WP_STATS_PORT=20001 ./distributedlog-service/bin/dlog-daemon.sh start writeproxy - $ WP_SHARD_ID=2 WP_SERVICE_PORT=4182 WP_STATS_PORT=20002 ./distributedlog-service/bin/dlog-daemon.sh start writeproxy - $ WP_SHARD_ID=3 WP_SERVICE_PORT=4183 WP_STATS_PORT=20003 ./distributedlog-service/bin/dlog-daemon.sh start writeproxy + $ WP_SHARD_ID=1 WP_SERVICE_PORT=4181 WP_STATS_PORT=20001 ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy + $ WP_SHARD_ID=2 WP_SERVICE_PORT=4182 WP_STATS_PORT=20002 ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy + $ WP_SHARD_ID=3 WP_SERVICE_PORT=4183 WP_STATS_PORT=20003 ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy The write proxy will announce itself to the zookeeper path `.write_proxy` under the dl namespace path. @@ -455,7 +455,7 @@ We could verify that the write proxy is running correctly by checking the zookee :: - $ ./distributedlog-service/bin/dlog zkshell localhost:2181 + $ ./distributedlog-proxy-server/bin/dlog zkshell localhost:2181 Connecting to localhost:2181 Welcome to ZooKeeper! JLine support is enabled @@ -480,7 +480,7 @@ Removing a write proxy is pretty straightforward by just killing the process. :: - WP_SHARD_ID=1 WP_SERVICE_PORT=4181 WP_STATS_PORT=10001 ./distributedlog-service/bin/dlog-daemon.sh stop writeproxy + WP_SHARD_ID=1 WP_SERVICE_PORT=4181 WP_STATS_PORT=10001 ./distributedlog-proxy-server/bin/dlog-daemon.sh stop writeproxy Adding a new write proxy is just adding a new host and starting the write proxy @@ -501,7 +501,7 @@ Create 10 streams. :: - $ ./distributedlog-service/bin/dlog tool create -u distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace -r stream- -e 0-10 + $ ./distributedlog-proxy-server/bin/dlog tool create -u distributedlog://127.0.0.1:2181/messaging/distributedlog/mynamespace -r stream- -e 0-10 You are going to create streams : [stream-0, stream-1, stream-2, stream-3, stream-4, stream-5, stream-6, stream-7, stream-8, stream-9, stream-10] (Y or N) Y http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/start/download.rst ---------------------------------------------------------------------- diff --git a/docs/start/download.rst b/docs/start/download.rst index 308f36a..4b39321 100644 --- a/docs/start/download.rst +++ b/docs/start/download.rst @@ -76,6 +76,41 @@ You can add the following dependencies to your `pom.xml` to include Apache Distr <!-- use core library to access DL storage --> <dependency> + <groupId>org.apache.distributedlog</groupId> + <artifactId>distributedlog-core_2.11</artifactId> + <version>{{ site.DL_VERSION_STABLE }}</version> + </dependency> + <!-- use thin proxy client to access DL via write proxy --> + <dependency> + <groupId>org.apache.distributedlog</groupId> + <artifactId>distributedlog-proxy-client_2.11</artifactId> + <version>{{ site.DL_VERSION_STABLE }}</version> + </dependency> + + +Version: 0.4.0 (the package is repackaged from com.twitter to org.apache.distributedlog) + +.. code-block:: xml + + <!-- use core library to access DL storage --> + <dependency> + <groupId>org.apache.distributedlog</groupId> + <artifactId>distributedlog-core_2.11</artifactId> + <version>{{ site.DL_VERSION_STABLE }}</version> + </dependency> + <!-- use thin proxy client to access DL via write proxy --> + <dependency> + <groupId>org.apache.distributedlog</groupId> + <artifactId>distributedlog-client_2.11</artifactId> + <version>{{ site.DL_VERSION_STABLE }}</version> + </dependency> + +Version: 0.3.x + +.. code-block:: xml + + <!-- use core library to access DL storage --> + <dependency> <groupId>com.twitter</groupId> <artifactId>distributedlog-core_2.11</artifactId> <version>{{ site.DL_VERSION_STABLE }}</version> http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/start/quickstart.rst ---------------------------------------------------------------------- diff --git a/docs/start/quickstart.rst b/docs/start/quickstart.rst index 010d323..91d3ba7 100644 --- a/docs/start/quickstart.rst +++ b/docs/start/quickstart.rst @@ -35,13 +35,13 @@ Step 2: Start ZooKeeper & BookKeeper DistributedLog uses `ZooKeeper` as the metadata store and `BookKeeper` as the log segment store. So you need to first start a zookeeper server and a few bookies if you don't already have one. You can -use the `dlog` script in `distributedlog-service` package to get a standalone bookkeeper sandbox. It +use the `dlog` script in `distributedlog-proxy-server` package to get a standalone bookkeeper sandbox. It starts a zookeeper server and `N` bookies (N is 3 by default). :: // Start the local sandbox instance at port `7000` - > ./distributedlog-service/bin/dlog local 7000 + > ./distributedlog-proxy-server/bin/dlog local 7000 DistributedLog Sandbox is running now. You could access distributedlog://127.0.0.1:7000 @@ -54,7 +54,7 @@ streams. The zkServer for the local sandbox is `127.0.0.1:7000` and the bookkeep :: - > ./distributedlog-service/bin/dlog admin bind -l /ledgers -s 127.0.0.1:7000 -c distributedlog://127.0.0.1:7000/messaging/my_namespace + > ./distributedlog-proxy-server/bin/dlog admin bind -l /ledgers -s 127.0.0.1:7000 -c distributedlog://127.0.0.1:7000/messaging/my_namespace No bookkeeper is bound to distributedlog://127.0.0.1:7000/messaging/my_namespace Created binding on distributedlog://127.0.0.1:7000/messaging/my_namespace. @@ -69,14 +69,14 @@ Let's create 5 log streams, prefixed with `messaging-stream-`. :: - > ./distributedlog-service/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/my_namespace -r messaging-stream- -e 1-5 + > ./distributedlog-proxy-server/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/my_namespace -r messaging-stream- -e 1-5 We can now see the streams if we run the `list` command from the tool. :: - > ./distributedlog-service/bin/dlog tool list -u distributedlog://127.0.0.1:7000/messaging/my_namespace + > ./distributedlog-proxy-server/bin/dlog tool list -u distributedlog://127.0.0.1:7000/messaging/my_namespace Streams under distributedlog://127.0.0.1:7000/messaging/my_namespace : -------------------------------- messaging-stream-1 @@ -94,13 +94,13 @@ Now, lets start a write proxy server that serves writes to distributedlog namesp :: - > ./distributedlog-service/bin/dlog-daemon.sh start writeproxy -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/my_namespace -mx -c `pwd`/distributedlog-service/conf/distributedlog_proxy.conf + > ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/my_namespace -mx -c `pwd`/distributedlog-proxy-server/conf/distributedlog_proxy.conf From 0.3.51-RC1 and onwards, use the below command to start the write proxy :: - > WP_SHARD_ID=1 WP_SERVICE_PORT=8000 WP_STATS_PORT=8001 WP_NAMESPACE='distributedlog://127.0.0.1:7000/messaging/my_namespace' ./distributedlog-service/bin/dlog-daemon.sh start writeproxy + > WP_SHARD_ID=1 WP_SERVICE_PORT=8000 WP_STATS_PORT=8001 WP_NAMESPACE='distributedlog://127.0.0.1:7000/messaging/my_namespace' ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy Step 6: Tail reading records ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/basic-2.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/basic-2.rst b/docs/tutorials/basic-2.rst index ef4c7b6..ae9af4f 100644 --- a/docs/tutorials/basic-2.rst +++ b/docs/tutorials/basic-2.rst @@ -125,7 +125,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create the stream http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/basic-3.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/basic-3.rst b/docs/tutorials/basic-3.rst index 55de7b0..5f0aa08 100644 --- a/docs/tutorials/basic-3.rst +++ b/docs/tutorials/basic-3.rst @@ -172,7 +172,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create multiple streams http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/basic-4.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/basic-4.rst b/docs/tutorials/basic-4.rst index bcbacd1..7bc63b0 100644 --- a/docs/tutorials/basic-4.rst +++ b/docs/tutorials/basic-4.rst @@ -150,7 +150,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create the stream http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/basic-5.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/basic-5.rst b/docs/tutorials/basic-5.rst index 94220e8..dfa6d33 100644 --- a/docs/tutorials/basic-5.rst +++ b/docs/tutorials/basic-5.rst @@ -149,7 +149,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create the stream http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/basic-6.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/basic-6.rst b/docs/tutorials/basic-6.rst index 4993fe4..91185d2 100644 --- a/docs/tutorials/basic-6.rst +++ b/docs/tutorials/basic-6.rst @@ -139,7 +139,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create the stream http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/messaging-1.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/messaging-1.rst b/docs/tutorials/messaging-1.rst index 5536811..7c89916 100644 --- a/docs/tutorials/messaging-1.rst +++ b/docs/tutorials/messaging-1.rst @@ -73,7 +73,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create the stream http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/docs/tutorials/messaging-2.rst ---------------------------------------------------------------------- diff --git a/docs/tutorials/messaging-2.rst b/docs/tutorials/messaging-2.rst index dc1cf5d..36ccdb5 100644 --- a/docs/tutorials/messaging-2.rst +++ b/docs/tutorials/messaging-2.rst @@ -112,7 +112,7 @@ Start the write proxy, listening on port 8000. :: // DistributedLogServerApp -p ${service-port} --shard-id ${shard-id} -sp ${stats-port} -u {distributedlog-uri} -mx -c ${conf-file} - ./distributedlog-service/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-service/conf/distributedlog_proxy.conf + ./distributedlog-proxy-server/bin/dlog org.apache.distributedlog.service.DistributedLogServerApp -p 8000 --shard-id 1 -sp 8001 -u distributedlog://127.0.0.1:7000/messaging/distributedlog -mx -c ${distributedlog-repo}/distributedlog-proxy-server/conf/distributedlog_proxy.conf Create the stream http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/pom.xml ---------------------------------------------------------------------- diff --git a/pom.xml b/pom.xml index dafda4a..a24577c 100644 --- a/pom.xml +++ b/pom.xml @@ -125,6 +125,7 @@ <maven-compiler-plugin.version>3.1</maven-compiler-plugin.version> <maven-jar-plugin.version>2.2</maven-jar-plugin.version> <maven-javadoc-plugin.version>2.8</maven-javadoc-plugin.version> + <maven-shade-plugin.version>2.4.3</maven-shade-plugin.version> <maven-source-plugin.version>2.2.1</maven-source-plugin.version> <maven-surefire-plugin.version>2.19.1</maven-surefire-plugin.version> <scrooge-maven-plugin.version>3.17.0</scrooge-maven-plugin.version> http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/scripts/dev/repackage.sh ---------------------------------------------------------------------- diff --git a/scripts/dev/repackage.sh b/scripts/dev/repackage.sh index cb55399..d2bbb2e 100755 --- a/scripts/dev/repackage.sh +++ b/scripts/dev/repackage.sh @@ -17,7 +17,7 @@ # limitations under the License. # -for module in distributedlog-benchmark distributedlog-client distributedlog-core distributedlog-protocol distributedlog-service distributedlog-tutorials/distributedlog-basic distributedlog-tutorials/distributedlog-kafka distributedlog-tutorials/distributedlog-messaging distributedlog-tutorials/distributedlog-mapreduce; do +for module in distributedlog-benchmark distributedlog-proxy-client distributedlog-core distributedlog-protocol distributedlog-proxy-server distributedlog-tutorials/distributedlog-basic distributedlog-tutorials/distributedlog-kafka distributedlog-tutorials/distributedlog-messaging distributedlog-tutorials/distributedlog-mapreduce; do echo "Repackaging module ${module}" if [[ -d "${module}/src/main/java/com/twitter/distributedlog" ]]; then mkdir -p ${module}/src/main/java/org/apache/distributedlog http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/scripts/integration/smoketest.sh ---------------------------------------------------------------------- diff --git a/scripts/integration/smoketest.sh b/scripts/integration/smoketest.sh index 586f21e..cece4fd 100755 --- a/scripts/integration/smoketest.sh +++ b/scripts/integration/smoketest.sh @@ -41,7 +41,7 @@ SMOKESTREAM_PREFIX="smoketest-stream-" # start the sandbox echo "start sandbox" -nohup ${DLOG_ROOT}/distributedlog-service/bin/dlog local ${ZK_PORT} > ${LOG_DIR}/sandbox.out 2>&1& +nohup ${DLOG_ROOT}/distributedlog-proxy-server/bin/dlog local ${ZK_PORT} > ${LOG_DIR}/sandbox.out 2>&1& if [ $? -ne 0 ]; then echo "Failed to start sandbox" @@ -60,7 +60,7 @@ fi # create streams echo "create streams" -${DLOG_ROOT}/distributedlog-service/bin/dlog tool create -u distributedlog://127.0.0.1:${ZK_PORT}/messaging/${NAMESPACE} -r ${SMOKESTREAM_PREFIX} -e 1-5 -f +${DLOG_ROOT}/distributedlog-proxy-server/bin/dlog tool create -u distributedlog://127.0.0.1:${ZK_PORT}/messaging/${NAMESPACE} -r ${SMOKESTREAM_PREFIX} -e 1-5 -f if [ $? -ne 0 ]; then echo "Failed to create streams prefixed with '${SMOKESTREAM_PREFIX}'." @@ -69,7 +69,7 @@ fi # list streams echo "list streams" -STREAM_OUTPUT=$(${DLOG_ROOT}/distributedlog-service/bin/dlog tool list -u distributedlog://127.0.0.1:${ZK_PORT}/messaging/${NAMESPACE} | grep "${SMOKESTREAM_PREFIX}") +STREAM_OUTPUT=$(${DLOG_ROOT}/distributedlog-proxy-server/bin/dlog tool list -u distributedlog://127.0.0.1:${ZK_PORT}/messaging/${NAMESPACE} | grep "${SMOKESTREAM_PREFIX}") echo "Create streams : ${STREAM_OUTPUT}." @@ -82,7 +82,7 @@ done # start a write proxy echo "start a write proxy" -${DLOG_ROOT}/distributedlog-service/bin/dlog-daemon.sh start writeproxy +${DLOG_ROOT}/distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy # tail the streams echo "tail the streams" @@ -132,7 +132,7 @@ fi # stop the write proxy echo "stop the write proxy" -${DLOG_ROOT}/distributedlog-service/bin/dlog-daemon.sh stop writeproxy +${DLOG_ROOT}/distributedlog-proxy-server/bin/dlog-daemon.sh stop writeproxy # stop the sandbox echo "stop the sandbox" http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/scripts/snapshot ---------------------------------------------------------------------- diff --git a/scripts/snapshot b/scripts/snapshot index 54e88d4..bd61ca3 100755 --- a/scripts/snapshot +++ b/scripts/snapshot @@ -29,18 +29,18 @@ find . -name lib | xargs rm -r rm -rf dist/* # create the layout -for module in distributedlog-service distributedlog-benchmark distributedlog-tutorials/distributedlog-basic distributedlog-tutorials/distributedlog-messaging; do +for module in distributedlog-proxy-server distributedlog-benchmark distributedlog-tutorials/distributedlog-basic distributedlog-tutorials/distributedlog-messaging; do mkdir -p "dist/release/${module}"/{bin,conf,lib} done # build the package mvn clean install -Ptwitter-ostrich-provider -DskipTests -# build the distributedlog-service -cp -rf distributedlog-service/bin/* dist/release/distributedlog-service/bin -cp -rf distributedlog-service/conf/* dist/release/distributedlog-service/conf -cp -rf distributedlog-service/lib/* dist/release/distributedlog-service/lib -cp -rf distributedlog-service/target/distributedlog-service-*.jar dist/release/distributedlog-service/lib +# build the distributedlog-proxy-server +cp -rf distributedlog-proxy-server/bin/* dist/release/distributedlog-proxy-server/bin +cp -rf distributedlog-proxy-server/conf/* dist/release/distributedlog-proxy-server/conf +cp -rf distributedlog-proxy-server/lib/* dist/release/distributedlog-proxy-server/lib +cp -rf distributedlog-proxy-server/target/distributedlog-proxy-server-*.jar dist/release/distributedlog-proxy-server/lib cp -rf vagrant dist/release/ cp -rf scripts dist/release/ @@ -64,7 +64,7 @@ cp -rf distributedlog-tutorials/distributedlog-messaging/target/distributedlog-m # release cd dist/release -zip -r "distributedlog-all-${GITSHA}.zip" distributedlog-service distributedlog-benchmark distributedlog-tutorials > /dev/null -zip -r "distributedlog-service-${GITSHA}.zip" distributedlog-service >/dev/null +zip -r "distributedlog-all-${GITSHA}.zip" distributedlog-proxy-server distributedlog-benchmark distributedlog-tutorials > /dev/null +zip -r "distributedlog-proxy-server-${GITSHA}.zip" distributedlog-proxy-server >/dev/null zip -r "distributedlog-benchmark-${GITSHA}.zip" distributedlog-benchmark >/dev/null zip -r "distributedlog-tutorials-${GITSHA}.zip" distributedlog-tutorials >/dev/null http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/vagrant/bk.sh ---------------------------------------------------------------------- diff --git a/vagrant/bk.sh b/vagrant/bk.sh index 1eb5ab1..671fe09 100755 --- a/vagrant/bk.sh +++ b/vagrant/bk.sh @@ -44,34 +44,34 @@ cd $log_dir #Stop the running bookie echo "Killing bookkeeper server" -./distributedlog-service/bin/dlog-daemon.sh stop bookie +./distributedlog-proxy-server/bin/dlog-daemon.sh stop bookie -cp ./distributedlog-service/conf/bookie.conf.template ./distributedlog-service/conf/bookie-$BROKER_ID.conf +cp ./distributedlog-proxy-server/conf/bookie.conf.template ./distributedlog-proxy-server/conf/bookie-$BROKER_ID.conf sed \ -e 's/zkServers=localhost:2181/'zkServers=$PUBLIC_ZOOKEEPER_ADDRESSES'/' \ - ./distributedlog-service/conf/bookie.conf.template > ./distributedlog-service/conf/bookie-$BROKER_ID.conf + ./distributedlog-proxy-server/conf/bookie.conf.template > ./distributedlog-proxy-server/conf/bookie-$BROKER_ID.conf sleep 5 if [ $DEPLOY_BK == true ]; then if [ $BROKER_ID -eq "1" ]; then - echo "create /messaging" | ./distributedlog-service/bin/dlog zkshell $PUBLIC_ZOOKEEPER_ADDRESSES - echo "create /messaging/bookkeeper" | ./distributedlog-service/bin/dlog zkshell $PUBLIC_ZOOKEEPER_ADDRESSES - echo "create /messaging/bookkeeper/ledgers" | ./distributedlog-service/bin/dlog zkshell $PUBLIC_ZOOKEEPER_ADDRESSES + echo "create /messaging" | ./distributedlog-proxy-server/bin/dlog zkshell $PUBLIC_ZOOKEEPER_ADDRESSES + echo "create /messaging/bookkeeper" | ./distributedlog-proxy-server/bin/dlog zkshell $PUBLIC_ZOOKEEPER_ADDRESSES + echo "create /messaging/bookkeeper/ledgers" | ./distributedlog-proxy-server/bin/dlog zkshell $PUBLIC_ZOOKEEPER_ADDRESSES echo "Metaformatting bookie" - export BOOKIE_CONF=$log_dir/distributedlog-service/conf/bookie-$BROKER_ID.conf - ./distributedlog-service/bin/dlog bkshell metaformat -n -f + export BOOKIE_CONF=$log_dir/distributedlog-proxy-server/conf/bookie-$BROKER_ID.conf + ./distributedlog-proxy-server/bin/dlog bkshell metaformat -n -f fi echo "Configuring bookkeeper" - BOOKIE_CONF=$log_dir/distributedlog-service/conf/bookie-$BROKER_ID.conf ./distributedlog-service/bin/dlog bkshell bookieformat + BOOKIE_CONF=$log_dir/distributedlog-proxy-server/conf/bookie-$BROKER_ID.conf ./distributedlog-proxy-server/bin/dlog bkshell bookieformat echo "Starting server" - SERVICE_PORT=3181 ./distributedlog-service/bin/dlog-daemon.sh start bookie -c $log_dir/distributedlog-service/conf/bookie-$BROKER_ID.conf + SERVICE_PORT=3181 ./distributedlog-proxy-server/bin/dlog-daemon.sh start bookie -c $log_dir/distributedlog-proxy-server/conf/bookie-$BROKER_ID.conf if [ $BROKER_ID -eq "1" ]; then - ./distributedlog-service/bin/dlog admin bind \ + ./distributedlog-proxy-server/bin/dlog admin bind \ -dlzr $PUBLIC_ZOOKEEPER_ADDRESSES \ -dlzw $PUBLIC_ZOOKEEPER_ADDRESSES \ -s $PUBLIC_ZOOKEEPER_ADDRESSES \ @@ -88,5 +88,5 @@ if [ $DEPLOY_WP == true ]; then WP_SHARD_ID=$BROKER_ID \ WP_SERVICE_PORT=4181 \ WP_STATS_PORT=20001 \ - ./distributedlog-service/bin/dlog-daemon.sh start writeproxy + ./distributedlog-proxy-server/bin/dlog-daemon.sh start writeproxy fi http://git-wip-us.apache.org/repos/asf/incubator-distributedlog/blob/0cb775c9/vagrant/zk.sh ---------------------------------------------------------------------- diff --git a/vagrant/zk.sh b/vagrant/zk.sh index c690149..8e110d9 100755 --- a/vagrant/zk.sh +++ b/vagrant/zk.sh @@ -25,23 +25,23 @@ JMX_PORT=$3 log_dir=/opt/distributedlog-trunk/dist/release cd $log_dir -cp $log_dir/distributedlog-service/conf/zookeeper.conf.template $log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties -echo "initLimit=5" >> $log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties -echo "syncLimit=2" >> $log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties -echo "quorumListenOnAllIPs=true" >> $log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties +cp $log_dir/distributedlog-proxy-server/conf/zookeeper.conf.template $log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties +echo "initLimit=5" >> $log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties +echo "syncLimit=2" >> $log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties +echo "quorumListenOnAllIPs=true" >> $log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties sed '/server.1/d' \ - ./distributedlog-service/conf/bookie.conf.template > ./distributedlog-service/conf/zookeeper-$ZKID.conf + ./distributedlog-proxy-server/conf/bookie.conf.template > ./distributedlog-proxy-server/conf/zookeeper-$ZKID.conf for i in $(seq 1 $NUM_ZK); do - echo "server.${i}=zk${i}:2888:3888" >>$log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties + echo "server.${i}=zk${i}:2888:3888" >>$log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties done mkdir -p /tmp/data/zookeeper echo "$ZKID" > /tmp/data/zookeeper/myid echo "Killing ZooKeeper" -./distributedlog-service/bin/dlog-daemon.sh stop zookeeper || true +./distributedlog-proxy-server/bin/dlog-daemon.sh stop zookeeper || true sleep 5 echo "Starting ZooKeeper" -echo "./distributedlog-service/bin/dlog-daemon.sh start zookeeper $log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties" -./distributedlog-service/bin/dlog-daemon.sh start zookeeper $log_dir/distributedlog-service/conf/zookeeper-$ZKID.properties +echo "./distributedlog-proxy-server/bin/dlog-daemon.sh start zookeeper $log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties" +./distributedlog-proxy-server/bin/dlog-daemon.sh start zookeeper $log_dir/distributedlog-proxy-server/conf/zookeeper-$ZKID.properties
