[GitHub] incubator-hawq issue #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev into H...
Github user guofengrichard commented on the issue: https://github.com/apache/incubator-hawq/pull/1073 These commits have been checked into master of 'apache/incubator-hawq'. So close this pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard closed the pull request at: https://github.com/apache/incubator-hawq/pull/1073 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev into H...
Github user guofengrichard commented on the issue: https://github.com/apache/incubator-hawq/pull/1073 Removed unnecessary configure files. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev into H...
Github user guofengrichard commented on the issue: https://github.com/apache/incubator-hawq/pull/1073 Installation of libhdfs3 has been removed for CentOS 6. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq issue #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev into H...
Github user guofengrichard commented on the issue: https://github.com/apache/incubator-hawq/pull/1073 Based on the comments, I have added in license header and also made amendments to Makefile and README. Other improvements will be made in the future. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1073#discussion_r94710415 --- Diff: contrib/hawq-docker/README.md --- @@ -0,0 +1,96 @@ +# hawq-docker + +hawq-docker is based on *wangzw's* repo *hawq-devel-env*. It is the docker images and scripts to help developers of Apache HAWQ to setup building and testing environment with docker. + +Both CentOS 7 and CentOS 6 are supported. +Change variable **OS_VERSION** (:= centos7 OR centos6) in Makefile to switch between CentOS 7 and CentOS 6. + +Take CentOS 7 as an example below. + +# Install docker +* following the instructions to install docker. +https://docs.docker.com/ + +# Setup build and test environment +* clone this repository +``` +git clone https://github.com/guofengrichard/hawq-docker.git . +``` +* Get the docker images +``` + cd hawq-docker + make pull (recommended) +OR + make build --- End diff -- Agree with Paul. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1073#discussion_r94548045 --- Diff: contrib/hawq-docker/README.md --- @@ -0,0 +1,96 @@ +# hawq-docker + +hawq-docker is based on *wangzw's* repo *hawq-devel-env*. It is the docker images and scripts to help developers of Apache HAWQ to setup building and testing environment with docker. + +Both CentOS 7 and CentOS 6 are supported. +Change variable **OS_VERSION** (:= centos7 OR centos6) in Makefile to switch between CentOS 7 and CentOS 6. + +Take CentOS 7 as an example below. + +# Install docker +* following the instructions to install docker. +https://docs.docker.com/ + +# Setup build and test environment +* clone this repository +``` +git clone https://github.com/guofengrichard/hawq-docker.git . --- End diff -- Right. I forgot to change this link. Will correct it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1073#discussion_r94547555
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with
other containers
+LOCAL :=
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all:
+ @echo " Usage:"
+ @echo "To setup a build and test environment: make run"
+ @echo "To start all containers: make start"
+ @echo "To stop all containers:make stop"
+ @echo "To remove hdfs containers: make clean"
+ @echo "To remove all containers: make
distclean"
+ @echo ""
+ @echo "To build images locally: make build"
+ @echo "To pull latest images: make pull"
+
+build:
+ @make -f $(THIS_MAKEFILE_PATH) build-hawq-dev-$(OS_VERSION)
+ @make -f $(THIS_MAKEFILE_PATH) build-hawq-test-$(OS_VERSION)
+ @echo "Done!"
+
+build-hawq-dev-$(OS_VERSION):
$(TOP_DIR)/$(OS_VERSION)-docker/hawq-dev/Dockerfile
+ @echo build hawq-dev:$(OS_VERSION) image
+ docker build -t hawq/hawq-dev:$(OS_VERSION)
$(TOP_DIR)/$(OS_VERSION)-docker/hawq-dev/
+
+build-hawq-test-$(OS_VERSION):
$(TOP_DIR)/$(OS_VERSION)-docker/hawq-test/Dockerfile
+ @echo build hawq-test:$(OS_VERSION) image
+ docker build -t hawq/hawq-test:$(OS_VERSION)
$(TOP_DIR)/$(OS_VERSION)-docker/hawq-test/
+
+create-data-container:
+ @echo create ${CLUSTER_ID}-data container
+ @if [ ! -z "$(LOCAL)" -a ! -d "$(LOCAL)" ]; then \
+ echo "LOCAL must be set to a directory!"; \
+ exit 1; \
+ fi
+ @if [ -z "`docker ps -a --filter="name=${CLUSTER_ID}-data$$" | grep -v
CONTAINER`" ]; then \
+ if [ -z "$(LOCAL)" ]; then \
+ docker create -v /data --name=${CLUSTER_ID}-data
hawq/hawq-dev:$(OS_VERSION) /bin/true; \
+ else \
+ docker create -v $(LOCAL):/data
--name=${CLUSTER_ID}-data hawq/hawq-dev:$(OS_VERSION) /bin/true; \
+ fi \
+ else \
+ echo "${CLUSTER_ID}-data container already exist!"; \
+ fi
+
+run:
+ @if [ -z "`docker network ls 2>/dev/null`" ]; then \
+make -f $(THIS_MAKEFILE_PATH) NETWORK=default
create-data-container && \
+make -f $(THIS_MAKEFILE_PATH) NETWORK=default run-hdfs; \
+ else \
+ if [ -z "`docker network ls 2>/dev/null | grep $(NETWORK)`" ];
then \
+ echo create network $(NETWORK) && \
+ docker network create --driver bridge $(NETWORK); \
+ fi && \
+ make -f $(THIS_MAKEFILE_PATH) create-data-container && \
+make -f $(THIS_MAKEFILE_PATH) run-hdfs; \
--- End diff --
Thanks for reminding.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request: https://github.com/apache/incubator-hawq/pull/1073#discussion_r94525426 --- Diff: contrib/hawq-docker/README.md --- @@ -0,0 +1,96 @@ +# hawq-docker + +hawq-docker is based on *wangzw's* repo *hawq-devel-env*. It is the docker images and scripts to help developers of Apache HAWQ to setup building and testing environment with docker. + +Both CentOS 7 and CentOS 6 are supported. +Change variable **OS_VERSION** (:= centos7 OR centos6) in Makefile to switch between CentOS 7 and CentOS 6. + +Take CentOS 7 as an example below. + +# Install docker +* following the instructions to install docker. +https://docs.docker.com/ + +# Setup build and test environment +* clone this repository +``` +git clone https://github.com/guofengrichard/hawq-docker.git . +``` +* Get the docker images +``` + cd hawq-docker + make pull (recommended) +OR + make build +``` +* setup a 5 nodes virtual cluster for Apache HAWQ build and test. +``` +make run +``` +Now let's have a look about what we creted. +``` +[root@localhost hawq-docker]# docker ps -a +CONTAINER IDIMAGE COMMAND CREATED STATUS PORTS NAMES +382b2b3360d1hawq/hawq-test:centos7 "entrypoint.sh bash" 2 minutes ago Up 2 minutescentos7-datanode3 +86513c331d45hawq/hawq-test:centos7 "entrypoint.sh bash" 2 minutes ago Up 2 minutescentos7-datanode2 +c0ab10e46e4ahawq/hawq-test:centos7 "entrypoint.sh bash" 2 minutes ago Up 2 minutescentos7-datanode1 +e27beea63953hawq/hawq-test:centos7 "entrypoint.sh bash" 2 minutes ago Up 2 minutescentos7-namenode +1f986959bd04hawq/hawq-dev:centos7"/bin/true"2 minutes ago Created centos7-data +``` +**centos7-data** is a data container and mounted to /data directory on all other containers to provide a shared storage for the cluster. + +# Build and Test Apache HAWQ +* attach to namenode +``` +docker exec -it centos7-namenode bash +``` +* check if HDFS working well +``` +sudo -u hdfs hdfs dfsadmin -report +``` +* clone Apache HAWQ code to /data direcotry +``` +git clone https://github.com/apache/incubator-hawq.git /data/hawq +``` +* build Apache HAWQ +``` +cd /data/hawq +./configure --prefix=/data/hawq-dev +make +make install +``` +(When you are using CentOS 6, run command `scl enable devtoolset-2 bash` before +configuring hawq and run command `exit` after installing hawq.) --- End diff -- The 'exit' command is to exit from the bash environment setup by previous 'scl' command. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1073#discussion_r94525211
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with
other containers
+LOCAL :=
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all:
+ @echo " Usage:"
+ @echo "To setup a build and test environment: make run"
+ @echo "To start all containers: make start"
+ @echo "To stop all containers:make stop"
+ @echo "To remove hdfs containers: make clean"
+ @echo "To remove all containers: make
distclean"
+ @echo ""
+ @echo "To build images locally: make build"
+ @echo "To pull latest images: make pull"
+
+build:
+ @make -f $(THIS_MAKEFILE_PATH) build-hawq-dev-$(OS_VERSION)
+ @make -f $(THIS_MAKEFILE_PATH) build-hawq-test-$(OS_VERSION)
+ @echo "Done!"
--- End diff --
That's right.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1073#discussion_r94525000
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with
other containers
+LOCAL :=
+# networks used in docker
+NETWORK := $(CLUSTER_ID)_hawq_network
+
+all:
+ @echo " Usage:"
+ @echo "To setup a build and test environment: make run"
+ @echo "To start all containers: make start"
+ @echo "To stop all containers:make stop"
+ @echo "To remove hdfs containers: make clean"
+ @echo "To remove all containers: make
distclean"
+ @echo ""
+ @echo "To build images locally: make build"
--- End diff --
No cleanup command for 'make build'. Removing docker images is not
encouraged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1073#discussion_r94524665
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
+OS_VERSION := centos7
+# Do not use underscore "_" in CLUSTER_ID
+CLUSTER_ID := $(OS_VERSION)
+# Monut this local directory to /data in data container and share with
other containers
+LOCAL :=
--- End diff --
Yes. '/data' is in containers and will be shared among all the containers.
'LOCAL' is supposed to be a local directory on the host. If it is defined,
the directory will be mounted to '/data' in all containers.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
Github user guofengrichard commented on a diff in the pull request:
https://github.com/apache/incubator-hawq/pull/1073#discussion_r94524175
--- Diff: contrib/hawq-docker/Makefile ---
@@ -0,0 +1,205 @@
+#!/usr/bin/make all
+
+THIS_MAKEFILE_PATH := $(abspath $(lastword $(MAKEFILE_LIST)))
+TOP_DIR := $(abspath $(dir ${THIS_MAKEFILE_PATH}))
+NDATANODES := 3
+CUR_DATANODE := 1
--- End diff --
@stanlyxiang is right about NDATANODES and CUR_DATANODE.
Good suggestion on configuring the number. For now user can define the
value via parameters to 'make', for example
$ make NDATANODES=6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request #1073: HAWQ-1248. Merge Dockerfiles for HAWQ Dev...
GitHub user guofengrichard opened a pull request: https://github.com/apache/incubator-hawq/pull/1073 HAWQ-1248. Merge Dockerfiles for HAWQ Dev into HAWQ code base. This commit is to merge Dockerfiles into HAWQ code base. The Dockerfiles are used to create docker images, which provide an out-of-box way for developers to setup build and test environment for HAWQ. You can merge this pull request into a Git repository by running: $ git pull https://github.com/guofengrichard/incubator-hawq hawq-docker Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-hawq/pull/1073.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1073 commit e33743e12fc20cdd7496fb40ee5a3778786ee78e Author: Richard Guo <guofengli...@gmail.com> Date: 2017-01-03T09:12:59Z HAWQ-1248. Merge Dockerfiles for HAWQ Dev into HAWQ code base. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
