xxubai commented on code in PR #3667:
URL: https://github.com/apache/amoro/pull/3667#discussion_r2204809894
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
Review Comment:
It's configured by toolchain.xml, we can remove this.
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
+| Quick Build | `mvn clean package -DskipTests` | Skip
tests for faster build |
+| Skip Frontend Build | `mvn clean package -Pskip-dashboard-build` | Skip Web
Dashboard build |
+| Hadoop 2.x Build | `mvn clean package -DskipTests -Phadoop2` | Use
Hadoop 2.x dependencies (default is 3.x) |
+
+### Advanced Build Options
+
+<details>
+<summary>Click to expand advanced build options</summary>
+
+#### Special Configuration Builds
+
+| Build Scenario | Command
| Description |
+|----------------------|------------------------------------------------------------|---------------------------------------------------|
+| Disable Disk Storage | `mvn clean package -DskipTests
-Pno-extented-disk-storage` | Do not introduce RocksDB to avoid memory overflow
|
+| Enable Aliyun OSS | `mvn clean package -DskipTests -Paliyun-oss-sdk`
| Enable Aliyun OSS SDK support |
+
+#### Engine Version Specific Builds
Review Comment:
This refers to the execution engine versions used by optimizer
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
+| Quick Build | `mvn clean package -DskipTests` | Skip
tests for faster build |
+| Skip Frontend Build | `mvn clean package -Pskip-dashboard-build` | Skip Web
Dashboard build |
+| Hadoop 2.x Build | `mvn clean package -DskipTests -Phadoop2` | Use
Hadoop 2.x dependencies (default is 3.x) |
+
+### Advanced Build Options
+
+<details>
+<summary>Click to expand advanced build options</summary>
+
+#### Special Configuration Builds
+
+| Build Scenario | Command
| Description |
+|----------------------|------------------------------------------------------------|---------------------------------------------------|
+| Disable Disk Storage | `mvn clean package -DskipTests
-Pno-extented-disk-storage` | Do not introduce RocksDB to avoid memory overflow
|
+| Enable Aliyun OSS | `mvn clean package -DskipTests -Paliyun-oss-sdk`
| Enable Aliyun OSS SDK support |
+
+#### Engine Version Specific Builds
+
+| Engine Type | Command Example
| Description
|
+|------------------|---------------------------------------------------------------------------------------------------|--------------------------------------------------------|
+| Flink Optimizer | `mvn clean package -DskipTests
-Dflink-optimizer.flink-version=1.20.0` | Specify
Flink version (default 1.20.0) |
+| Flink Below 1.15 | `mvn clean package -DskipTests -Pflink-optimizer-pre-1.15
-Dflink-optimizer.flink-version=1.14.6` | Flink versions below 1.15 require
additional parameter |
+| Spark Optimizer | `mvn clean package -DskipTests
-Dspark-optimizer.spark-version=3.3.3` | Specify
Spark version (default 3.3.3) |
+
+#### Complete Build Options
Review Comment:
Support building different table formats:
- -Psupport-all-formats
- -Psupport-paimon-format
- -Psupport-hudi-format
support Iceberg by default
##########
README.md:
##########
@@ -80,21 +92,27 @@ For details, please refer to: [Iceberg
Docs](https://iceberg.apache.org/docs/lat
Amoro support multiple processing engines for Mixed format as below:
-| Processing Engine | Version | Batch Read | Batch Write |
Batch Overwrite | Streaming Read | Streaming Write | Create Table | Alter Table
|
-|-------------------|------------------------|-------------|-------------|-----------------|----------------|-----------------|--------------|-------------|
-| Flink | 1.15.x, 1.16.x, 1.17.x | ✔ | ✔ |
✖ | ✔ | ✔ | ✔ | ✖
|
-| Spark | 3.1, 3.2, 3.3 | ✔ | ✔ |
✔ | ✖ | ✖ | ✔ | ✔
|
-| Hive | 2.x, 3.x | ✔ | ✖ |
✔ | ✖ | ✖ | ✖ | ✔ |
-| Trino | 406 | ✔ | ✖ |
✔ | ✖ | ✖ | ✖ | ✔ |
+| Processing Engine | Version | Batch Read | Batch Write |
Batch Overwrite | Streaming Read | Streaming Write | Create Table | Alter Table
|
+|-------------------|------------------------|------------|-------------|-----------------|----------------|-----------------|--------------|-------------|
+| Flink | 1.15.x, 1.16.x, 1.17.x | ✔ | ✔ |
✖ | ✔ | ✔ | ✔ | ✖
|
+| Spark | 3.1, 3.2, 3.3,3.5 | ✔ | ✔ |
✔ | ✖ | ✖ | ✔ | ✔
|
Review Comment:
Currently, mixed format only supports spark 3.3, spark 3.5
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
Review Comment:
only `amoro-mixed-trino` can build with jdk17
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
+| Quick Build | `mvn clean package -DskipTests` | Skip
tests for faster build |
+| Skip Frontend Build | `mvn clean package -Pskip-dashboard-build` | Skip Web
Dashboard build |
+| Hadoop 2.x Build | `mvn clean package -DskipTests -Phadoop2` | Use
Hadoop 2.x dependencies (default is 3.x) |
+
+### Advanced Build Options
+
+<details>
+<summary>Click to expand advanced build options</summary>
+
+#### Special Configuration Builds
+
+| Build Scenario | Command
| Description |
+|----------------------|------------------------------------------------------------|---------------------------------------------------|
+| Disable Disk Storage | `mvn clean package -DskipTests
-Pno-extented-disk-storage` | Do not introduce RocksDB to avoid memory overflow
|
+| Enable Aliyun OSS | `mvn clean package -DskipTests -Paliyun-oss-sdk`
| Enable Aliyun OSS SDK support |
+
+#### Engine Version Specific Builds
+
+| Engine Type | Command Example
| Description
|
+|------------------|---------------------------------------------------------------------------------------------------|--------------------------------------------------------|
+| Flink Optimizer | `mvn clean package -DskipTests
-Dflink-optimizer.flink-version=1.20.0` | Specify
Flink version (default 1.20.0) |
+| Flink Below 1.15 | `mvn clean package -DskipTests -Pflink-optimizer-pre-1.15
-Dflink-optimizer.flink-version=1.14.6` | Flink versions below 1.15 require
additional parameter |
+| Spark Optimizer | `mvn clean package -DskipTests
-Dspark-optimizer.spark-version=3.3.3` | Specify
Spark version (default 3.3.3) |
+
+#### Complete Build Options
+
+| Build Scenario | Command
| Description |
+|-----------------------|-------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------|
+| Build Trino Module | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Requires JDK 17
|
+| Build All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Requires toolchains.xml configuration |
+| Complete Distribution | `mvn clean package -Psupport-all-formats`
| Distribution package with all formats |
+| Paimon Format Package | `mvn clean package -Psupport-paimon-format`
| Include Apache Paimon format |
+| Hudi Format Package | `mvn clean package -Psupport-hudi-format`
| Include Apache Hudi format |
+
+</details>
+
+### JDK 17 Build Configuration
+
+If you need to build all modules (including `amoro-mixed-trino`), create a
`toolchains.xml` file in
+the `${user.home}/.m2/` directory:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<toolchains>
+ <toolchain>
+ <type>jdk</type>
+ <provides>
+ <version>17</version>
+ <vendor>sun</vendor>
+ </provides>
+ <configuration>
+ <jdkHome>${YourJDK17Home}</jdkHome>
+ </configuration>
+ </toolchain>
</toolchains>
-```
Review Comment:
Why removing the code block?
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
Review Comment:
It's enough to just list command with different jdk versions
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
+| Quick Build | `mvn clean package -DskipTests` | Skip
tests for faster build |
+| Skip Frontend Build | `mvn clean package -Pskip-dashboard-build` | Skip Web
Dashboard build |
+| Hadoop 2.x Build | `mvn clean package -DskipTests -Phadoop2` | Use
Hadoop 2.x dependencies (default is 3.x) |
+
+### Advanced Build Options
+
+<details>
+<summary>Click to expand advanced build options</summary>
+
+#### Special Configuration Builds
+
+| Build Scenario | Command
| Description |
+|----------------------|------------------------------------------------------------|---------------------------------------------------|
+| Disable Disk Storage | `mvn clean package -DskipTests
-Pno-extented-disk-storage` | Do not introduce RocksDB to avoid memory overflow
|
Review Comment:
Disk storage does not refer to the storage system. In addition, this
configuration is rarely modified and can be omitted.
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
Review Comment:
Need to explain the default configuration, such as HADOOP version, SPARK,
FLINK version, etc.
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
+| Quick Build | `mvn clean package -DskipTests` | Skip
tests for faster build |
Review Comment:
```suggestion
| Skip Unit Tests | `mvn clean package -DskipTests` | Skip
tests for faster build |
```
##########
README.md:
##########
@@ -109,45 +127,118 @@ Amoro contains modules as below:
- `amoro-format-paimon` contains integration of Apache Paimon format
- `amoro-format-mixed` provides Mixed format implementation
- `amoro-mixed-hive` integrates with Apache Hive and implements Mixed Hive
format
- - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded version)
- - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded version)
+ - `amoro-mixed-flink` provides Flink connectors for Mixed format tables
(use amoro-flink-runtime for a shaded
+ version)
+ - `amoro-mixed-spark` provides Spark connectors for Mixed format tables
(use amoro-spark-runtime for a shaded
+ version)
- `amoro-mixed-trino` provides Trino connectors for Mixed format tables
-
## Building
-Amoro is built using Maven with JDK 8, 11 and 17(required for
`amoro-format-mixed/amoro-mixed-trino` module).
-
-* Build all modules without `amoro-mixed-trino`: `mvn clean package`
-* Build and skip tests: `mvn clean package -DskipTests`
-* Build and skip dashboard: `mvn clean package -Pskip-dashboard-build`
-* Build and disable disk storage, RocksDB will NOT be introduced to avoid
memory overflow: `mvn clean package -DskipTests -Pno-extented-disk-storage`
-* Build and enable aliyun-oss-sdk: `mvn clean package -DskipTests
-Paliyun-oss-sdk`
-* Build with hadoop 2.x(the default is 3.x) dependencies: `mvn clean package
-DskipTests -Phadoop2`
-* Specify Flink version for Flink optimizer(the default is 1.20.0): `mvn clean
package -DskipTests -Dflink-optimizer.flink-version=1.20.0`
- * If the version of Flink is below 1.15.0, you also need to add the
`-Pflink-optimizer-pre-1.15` parameter: `mvn clean package -DskipTests
-Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6`
-* Specify Spark version for Spark optimizer(the default is 3.3.3): `mvn clean
package -DskipTests -Dspark-optimizer.spark-version=3.3.3`
-* Build `amoro-mixed-trino` module under JDK 17: `mvn clean package
-DskipTests -Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am`.
-* Build all modules: `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`, besides you need config `toolchains.xml`
in `${user.home}/.m2/` dir with content below.
-* Build a distribution package with all formats integrated: `mvn clean package
-Psupport-all-formats`
- * Build a distribution package with Apache Paimon format: `mvn clean package
-Psupport-paimon-format`
- * Build a distribution package with Apache Hudi format: `mvn clean package
-Psupport-hudi-format`
-
-```
-<?xml version="1.0" encoding="UTF-8"?>
-<toolchains>
- <toolchain>
- <type>jdk</type>
- <provides>
- <version>17</version>
- <vendor>sun</vendor>
- </provides>
- <configuration>
- <jdkHome>${YourJDK17Home}</jdkHome>
- </configuration>
- </toolchain>
+### JDK Version Requirements
+
+Amoro supports multiple JDK versions for building, with different modules
having different JDK version requirements:
+
+| JDK Version | Supported Modules | Notes
|
+|-------------|----------------------------------------|------------------------------------------|
+| JDK 8 | All modules except `amoro-mixed-trino` | Minimum required
version |
+| JDK 11 | All modules | Default compilation
version, recommended |
+| JDK 17 | All modules | Required for
`amoro-mixed-trino` module |
+
+### How to Build with Different JDK Versions
+
+#### Building with JDK 8
+
+To build with JDK 8, use the `java8` profile:
+
+| Build Scenario | Command |
Description |
+|-----------------------|-------------------------------------------------|-----------------------------------------------------------|
+| Standard JDK 8 Build | `mvn clean package -Pjava8` |
Build all modules (except `amoro-mixed-trino`) with JDK 8 |
+| JDK 8 with Hadoop 2.x | `mvn clean package -DskipTests -Pjava8,hadoop2` |
Build with JDK 8 and Hadoop 2.x dependencies |
+| JDK 8 Quick Build | `mvn clean package -DskipTests -Pjava8` |
Skip tests and build with JDK 8 |
+
+#### Building with JDK 11
+
+JDK 11 is the default compilation version, so no special profile is needed:
+
+| Build Scenario | Command |
Description |
+|------------------------|-------------------------------------------|------------------------------------------------------------|
+| Standard JDK 11 Build | `mvn clean package` | Build
all modules (except `amoro-mixed-trino`) with JDK 11 |
+| JDK 11 with Hadoop 2.x | `mvn clean package -DskipTests -Phadoop2` | Build
with JDK 11 and Hadoop 2.x dependencies |
+| JDK 11 Quick Build | `mvn clean package -DskipTests` | Skip
tests and build with JDK 11 |
+
+#### Building with JDK 17
+
+For JDK 17, you can build all modules including `amoro-mixed-trino`:
+
+| Build Scenario | Command
| Description |
+|------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|
+| JDK 17 All Modules | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino`
| Build all modules including Trino with JDK 17 |
+| JDK 17 Trino Only | `mvn clean package -DskipTests
-Pformat-mixed-format-trino,build-mixed-format-trino -pl
'amoro-format-mixed/amoro-mixed-trino' -am` | Build only Trino module with JDK
17 |
+| JDK 17 with Hadoop 2.x | `mvn clean package -DskipTests
-Ptoolchain,build-mixed-format-trino,hadoop2`
| Build all modules with JDK 17 and Hadoop 2.x |
+
+### Basic Build Options
+
+The most commonly used build commands:
+
+| Build Scenario | Command |
Description |
+|---------------------|--------------------------------------------|------------------------------------------------|
+| Standard Build | `mvn clean package` | Build all
modules (except `amoro-mixed-trino`) |
+| Quick Build | `mvn clean package -DskipTests` | Skip
tests for faster build |
+| Skip Frontend Build | `mvn clean package -Pskip-dashboard-build` | Skip Web
Dashboard build |
+| Hadoop 2.x Build | `mvn clean package -DskipTests -Phadoop2` | Use
Hadoop 2.x dependencies (default is 3.x) |
+
+### Advanced Build Options
Review Comment:
Here we suggest showing the impact of different Maven profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
