abstractdog commented on code in PR #477:
URL: https://github.com/apache/tez/pull/477#discussion_r3072749066
##########
README.md:
##########
@@ -15,18 +15,227 @@
Apache Tez
==========
-Apache Tez is a generic data-processing pipeline engine envisioned as a
low-level engine for higher abstractions
-such as Apache Hadoop Map-Reduce, Apache Pig, Apache Hive etc.
+Apache Tez is a generic data-processing pipeline engine envisioned as a
+low-level engine for higher abstractions such as Apache Hadoop Map-Reduce,
+Apache Pig, Apache Hive etc.
At its heart, tez is very simple and has just two components:
-* The data-processing pipeline engine where-in one can plug-in input,
processing and output implementations to
- perform arbitrary data-processing. Every 'task' in tez has the following:
- - Input to consume key/value pairs from.
- - Processor to process them.
- - Output to collect the processed key/value pairs.
+* The data-processing pipeline engine where-in one can plug-in input,
+ processing and output implementations to perform arbitrary data-processing.
+ Every 'task' in tez has the following:
+* Input to consume key/value pairs from.
+* Processor to process them.
+* Output to collect the processed key/value pairs.
-* A master for the data-processing application, where-by one can put together
arbitrary data-processing 'tasks'
- described above into a task-DAG to process data as desired.
+* A master for the data-processing application, where-by one can put together
+ arbitrary data-processing 'tasks' described above into a task-DAG to process
+ data as desired.
The generic master is implemented as a Apache Hadoop YARN ApplicationMaster.
+
+Building Tez
+------------
+
+For instructions on how to contribute to Tez, refer to:
+[Tez Wiki - How to Contribute](https://cwiki.apache.org/confluence/display/TEZ)
+
+Requirements
+------------
+
+* JDK 21+
+* Maven 3.6.3 or later
+* spotbugs 4.9.3 or later (if running spotbugs)
+* ProtocolBuffer 3.25.5
+* Internet connection for first build (to fetch all dependencies)
+* Hadoop 3.x
+
+Maven Modules
+-------------
+
+* **tez** (Main Tez project)
+ * **tez-api**: Tez API
+ * **tez-common**: Tez common
+ * **tez-runtime-internals**: Tez runtime internals
+ * **tez-runtime-library**: Tez runtime library
+ * **tez-mapreduce**: Tez mapreduce
+ * **tez-dag**: Tez dag
+ * **tez-examples**: Tez examples
+ * **tez-plugins**: Tez plugins
+ * **tez-tests**: Tez tests and additional test examples
+ * **tez-dist**: Tez dist
+ * **tez-ui**: Tez web user interface
+
+Maven Build Goals
+-----------------
+
+* Clean: `mvn clean`
+* Compile: `mvn compile`
+* Run tests: `mvn test`
+* Create JAR: `mvn package`
+* Run spotbugs: `mvn compile spotbugs:spotbugs`
+* Run checkstyle: `mvn compile checkstyle:checkstyle`
+* Install JAR in M2 cache: `mvn install`
+* Deploy JAR to Maven repo: `mvn deploy`
+* Run jacoco: `mvn test -Pjacoco`
+* Run Rat: `mvn apache-rat:check`
+* Build javadocs: `mvn javadoc:javadoc`
+* Build distribution: `mvn package -Dhadoop.version=3.4.2`
+* Visualize state machines: `mvn compile -Pvisualize -DskipTests=true`
+
+Build Options
+-------------
+
+* Use `-Dpackage.format` to create distributions with a format other than
+ .tar.gz (mvn-assembly-plugin formats).
+* Use `-Dhadoop.version` to specify the version of Hadoop to build Tez against.
+* Use `-Dprotoc.path` to specify the path to `protoc`.
+* Use `-Dallow.root.build` to root build `tez-ui` components.
+
+Test Options
+------------
+
+* Use `-DskipTests` to skip tests when running Maven goals like `package`,
+ `install`, `deploy`, or `verify`.
+* Specific tests: `-Dtest=<TESTCLASSNAME>,<TESTCLASSNAME#METHODNAME>,....`
+* Exclude tests: `-Dtest.exclude=<TESTCLASSNAME>`
+* Exclude pattern:
+ `-Dtest.exclude.pattern=**/<TESTCLASSNAME1>.java,**/<TESTCLASSNAME2>.java`
+
+Building against a Specific Version of Hadoop
+----------------------------------------------
+
+Tez runs on top of Apache Hadoop YARN and requires Hadoop 3.x.
+
+By default, it can be compiled against other compatible Hadoop versions by
+specifying `hadoop.version`:
+
+```bash
+mvn package -Dhadoop.version=3.4.2
+```
+
+To skip tests and Javadocs:
+
+```bash
+mvn package -Dhadoop.version=3.4.2 -DskipTests -Dmaven.javadoc.skip=true
+```
+
+For recent versions of Hadoop (which do not bundle AWS and Azure by default),
+you can bundle AWS-S3 or Azure support:
+
+```bash
+mvn package -Dhadoop.version=3.4.2 -Paws -Pazure
+```
+
+Tez also has shims to provide version-specific implementations for various
APIs.
+For more details, refer to
+[Hadoop Shims](https://cwiki.apache.org/confluence/display/TEZ/HadoopShims).
+
+UI Build Issues
Review Comment:
does it make sense to group the UI-related stuff together?
- Tez UI
- UI Build Issues
- Skip UI Build
- Issue with PhantomJS on building in PowerPC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
