Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/22084#discussion_r209481255
--- Diff: dev/make-distribution.sh ---
@@ -188,6 +190,23 @@ if [ -f
"$SPARK_HOME"/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar
cp
"$SPARK_HOME"/common/network-yarn/target/scala*/spark-*-yarn-shuffle.jar
"$DISTDIR/yarn"
fi
+# Only copy external jars if built
+if [ -f
"$SPARK_HOME"/external/avro/target/spark-avro_${SCALA_VERSION}-${VERSION}.jar
]; then
+ cp
"$SPARK_HOME"/external/avro/target/spark-avro_${SCALA_VERSION}-${VERSION}.jar
"$DISTDIR/external/jars/"
+fi
+if [ -f
"$SPARK_HOME"/external/kafka-0-10/target/spark-streaming-kafka-0-10_${SCALA_VERSION}-${VERSION}.jar
]; then
+ cp
"$SPARK_HOME"/external/kafka-0-10/target/spark-streaming-kafka-0-10_${SCALA_VERSION}-${VERSION}.jar
"$DISTDIR/external/jars/"
--- End diff --
When building such external jar, assembly jar will also be built
accordingly. And the assembly jar can be used directly.
Jars provided here still not so useful because it lacks third-party
dependencies like Kafka, so I'm not sure if it is more convenient compared to
pull from maven repo directly.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
