This is an automated email from the ASF dual-hosted git repository.
ming pushed a commit to branch master
in repository
https://gitbox.apache.org/repos/asf/incubator-hugegraph-toolchain.git
The following commit(s) were added to refs/heads/master by this push:
new e8fcdd3c feat: support docker for loader (#530)
e8fcdd3c is described below
commit e8fcdd3c35eeba8d50b51a146c4d955cee7a6b71
Author: Dandelion <[email protected]>
AuthorDate: Mon Nov 13 19:32:58 2023 +0800
feat: support docker for loader (#530)
* feat: support docker for loader
* add empty line at end of file
* supplement license
* supplement readme
* supplement readme
* change dockerfile cmd and readme
* opt doc and docker compose
---
README.md | 8 +-
hugegraph-hubble/README.md | 3 +-
hugegraph-loader/Dockerfile | 51 +++++++++
hugegraph-loader/README.md | 124 ++++++++++++++++++++-
hugegraph-loader/docker/example/docker-compose.yml | 37 ++++++
5 files changed, 216 insertions(+), 7 deletions(-)
diff --git a/README.md b/README.md
index 2651d2f4..71e8f1d8 100644
--- a/README.md
+++ b/README.md
@@ -12,10 +12,16 @@
## Modules
- [hugegraph-loader](./hugegraph-loader): Loading datasets into the HugeGraph
from multiple data sources.
-- [hugegraph-hubble](./hugegraph-hubble): Online HugeGraph management and
analysis dashboard (Include: data loading, schema management, graph traverser
and display). We can use `docker run -itd --name=hubble -p 8088:8088
hugegraph/hubble` to quickly start
[hubble](https://hub.docker.com/r/hugegraph/hubble) or we can follow
[this](hugegraph-hubble/README.md#quick-start) to use docker-compose to start
`hubble` with `server`.
+- [hugegraph-hubble](./hugegraph-hubble): Online HugeGraph management and
analysis dashboard (Include: data loading, schema management, graph traverser
and display).
- [hugegraph-tools](./hugegraph-tools): Command line tool for deploying,
managing and backing-up/restoring graphs from HugeGraph.
- [hugegraph-client](./hugegraph-client): A Java-written client for HugeGraph,
providing `RESTful` APIs for accessing graph
vertex/edge/schema/gremlin/variables and traversals etc.
+## Usage
+
+- [hugegraph-loader](./hugegraph-loader): We can use `docker run -itd --name
loader hugegraph/loader` to quickly start
[loader](https://hub.docker.com/r/hugegraph/loader) or we can follow
[this](./hugegraph-loader/README.md#212-docker-compose) to use docker-compose
to start `loader` with `server`. And we can find more details in the
[doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/).
+- [hugegraph-hubble](./hugegraph-hubble): We can use `docker run -itd
--name=hubble -p 8088:8088 hugegraph/hubble` to quickly start
[hubble](https://hub.docker.com/r/hugegraph/hubble) or we can follow
[this](hugegraph-hubble/README.md#quick-start) to use docker-compose to start
`hubble` with `server`. And we can find more details in the
[doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-hubble/).
+- [hugegraph-client](./hugegraph-client): We can follow the
[doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-client/) to learn
how to quick start with `client`.
+
## Maven Dependencies
You could use import the dependencies in `maven` like this:
diff --git a/hugegraph-hubble/README.md b/hugegraph-hubble/README.md
index e19fd280..6de7f686 100644
--- a/hugegraph-hubble/README.md
+++ b/hugegraph-hubble/README.md
@@ -20,8 +20,7 @@ We can quickly start `hubble` in two ways:
1. We can use `docker run -itd --name=hubble -p 8088:8088 hugegraph/hubble` to
quickly start [hubble](https://hub.docker.com/r/hugegraph/hubble).
2. Or we can use the `docker-compose.yml` to start `hubble` with
`hugegraph-server`. If we set `PRELOAD=true`, we can preload the example graph
when starting `hugegraph-server`:
-
- ```
+ ```yaml
version: '3'
services:
server:
diff --git a/hugegraph-loader/Dockerfile b/hugegraph-loader/Dockerfile
new file mode 100644
index 00000000..06985e70
--- /dev/null
+++ b/hugegraph-loader/Dockerfile
@@ -0,0 +1,51 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM maven:3.9.0-eclipse-temurin-11 AS build
+
+COPY . /pkg
+WORKDIR /pkg
+
+RUN set -x \
+ && mvn install -pl hugegraph-client,hugegraph-loader -am
-Dmaven.javadoc.skip=true -DskipTests -ntp
+
+
+RUN set -x \
+ && cd /pkg/hugegraph-loader/ \
+ && echo "$(ls)" \
+ && mvn clean package -DskipTests
+
+
+FROM openjdk:11-slim
+
+COPY --from=build /pkg/hugegraph-loader/apache-hugegraph-loader-incubating-*/
/loader
+WORKDIR /loader/
+
+RUN set -x \
+ && apt-get -q update \
+ && apt-get -q install -y --no-install-recommends --no-install-suggests \
+ dumb-init \
+ procps \
+ curl \
+ lsof \
+ && apt-get clean \
+ && rm -rf /var/lib/apt/lists/*
+
+VOLUME /loader
+
+ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+CMD ["tail","-f","/dev/null"]
diff --git a/hugegraph-loader/README.md b/hugegraph-loader/README.md
index 036b56bf..e81c2b2f 100644
--- a/hugegraph-loader/README.md
+++ b/hugegraph-loader/README.md
@@ -7,7 +7,7 @@
hugegraph-loader is a customizable command line utility for loading small to
medium size graph datasets into the HugeGraph database from multiple data
sources with various input formats.
-## Features
+## 1. Features
- Multiple data sources, such as local file(path), HDFS file(path), MySQL
- Various input formats, such as json, csv, and text with any delimiters.
@@ -15,7 +15,123 @@ hugegraph-loader is a customizable command line utility for
loading small to med
- Detecting schema from data automatically, reduce the complex work of schema
management.
- Advanced customized operations with groovy script, users can configure how
to construct vertices and edges by themselves.
-## Building
+## 2. Usage for Docker(Recommand)
+
+- Run `loader` with Docker
+ - Docker run
+ - Docker-compose
+- Load data in docker container `loader`
+
+### 2.1 Start with Docker
+
+#### 2.1.1 Docker run
+
+Use the command `docker run -itd --name loader hugegraph/loader` to start
loader.
+
+If you want to load your data, you can mount the data folder like `-v
/path/to/data/file:/loader/file`
+
+
+#### 2.1.2 Docker-compose
+
+The example `docker-compose.yml` is [here](./docker/example/docker-compose.yml)
+
+If you want to load your data, you can mount the data folder like:
+```yaml
+volumes:
+ - /path/to/data/file:/loader/file
+```
+
+Use the command `docker-compose up -d` to deploy `loader` with `server` and
`hubble`.
+
+### 2.2 Load data with docker container
+
+#### 2.2.1 load data with docker
+
+> If the `loader` and `server` is in the same docker network (for example, you
deploy `loader` and `server` with `docker-compose`), we can set `-h
{server_container_name}`. In our example, the container name of `server` is
`graph`
+>
+> If `loader` is deployed alone, the `-h` should be set to the ip of the host
of `server`. Other parameter description is
[here](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/#341-parameter-description)
+
+```bash
+docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f
example/file/struct.json -s example/file/schema.groovy -h graph -p 8080
+```
+
+Then we can see the result.
+
+```bash
+HugeGraphLoader worked in NORMAL MODE
+vertices/edges loaded this time : 8/6
+--------------------------------------------------
+count metrics
+ input read success : 14
+ input read failure : 0
+ vertex parse success : 8
+ vertex parse failure : 0
+ vertex insert success : 8
+ vertex insert failure : 0
+ edge parse success : 6
+ edge parse failure : 0
+ edge insert success : 6
+ edge insert failure : 0
+--------------------------------------------------
+meter metrics
+ total time : 0.199s
+ read time : 0.046s
+ load time : 0.153s
+ vertex load time : 0.077s
+ vertex load rate(vertices/s) : 103
+ edge load time : 0.112s
+ edge load rate(edges/s) : 53
+```
+
+Then you can use `curl` or `hubble` to see the result.
+
+```bash
+> curl "http://localhost:8080/graphs/hugegraph/graph/vertices"; | gunzip
+{"vertices":[{"id":1,"label":"software","type":"vertex","properties":{"name":"lop","lang":"java","price":328.0}},{"id":2,"label":"software","type":"vertex","properties":{"name":"ripple","lang":"java","price":199.0}},{"id":"1:tom","label":"person","type":"vertex","properties":{"name":"tom"}},{"id":"1:josh","label":"person","type":"vertex","properties":{"name":"josh","age":32,"city":"Beijing"}},{"id":"1:marko","label":"person","type":"vertex","properties":{"name":"marko","age":29,"city":"B
[...]
+```
+
+If you want to check the edges, use `curl
"http://localhost:8080/graphs/hugegraph/graph/edges"; | gunzip`
+
+#### 2.2.2 enter the docker container to load data
+
+If you want to do some additional operation in the container, you can enter
the container as follows:
+
+```bash
+docker exec -it loader bash
+```
+
+Then, you can load data as follows:
+
+```bash
+sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s
example/file/schema.groovy -h graph -p 8080
+```
+
+The result is as same as above.
+
+## 3. Use loader directly
+
+> notice: currently, version is `1.0.0`
+
+Download and unzip the compiled archive
+
+```bash
+wget
https://downloads.apache.org/incubator/hugegraph/{version}/apache-hugegraph-toolchain-incubating-{version}.tar.gz
+tar zxf *hugegraph*.tar.gz
+```
+
+Then, load data with example file:
+
+```bash
+cd apache-hugegraph-toolchain-incubating-{version}
+cd apache-hugegraph-loader-incubating-{version}
+sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s
example/file/schema.groovy
+```
+
+More details is in the
[doc](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/)
+
+## 4. Building
+
+You can also build the `loader` by yourself.
Required:
@@ -34,10 +150,10 @@ To build with default tests:
mvn clean install
```
-## Doc
+## 5. Doc
The [loader
homepage](https://hugegraph.apache.org/docs/quickstart/hugegraph-loader/)
contains more information about it.
-## License
+## 6. License
hugegraph-loader is licensed under Apache 2.0 License.
diff --git a/hugegraph-loader/docker/example/docker-compose.yml
b/hugegraph-loader/docker/example/docker-compose.yml
new file mode 100644
index 00000000..7c9a1b90
--- /dev/null
+++ b/hugegraph-loader/docker/example/docker-compose.yml
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+version: '3'
+services:
+ server:
+ image: hugegraph/hugegraph
+ container_name: graph
+ ports:
+ - 8080:8080
+
+ hubble:
+ image: hugegraph/hubble
+ container_name: hubble
+ ports:
+ - 8088:8088
+
+ loader:
+ image: hugegraph/loader
+ container_name: loader
+ # mount your own data here
+ # volumes:
+ # - /path/to/data/file:/loader/file
