[2 BUG REPORT] failed to run make-distribution.sh when a older version maven installed in system and run VersionsSuite test hang
BUG 1:
I have installed maven 3.0.2 in system, When I using make-distribution.sh
, it seem not use maven 3.2.2 but use /usr/local/bin/mvn to build spark. So
I add --force option in make-distribution.sh like this：
line 130:
VERSION=$("$MVN" *--force* help:evaluate -Dexpression=project.version $@
2>/dev/null | grep -v "INFO" | tail -n 1)
SCALA_VERSION=$("$MVN"* --force* help:evaluate
-Dexpression=scala.binary.version $@ 2>/dev/null\
| grep -v "INFO"\
| tail -n 1)
SPARK_HADOOP_VERSION=$("$MVN" *--force* help:evaluate
-Dexpression=hadoop.version $@ 2>/dev/null\
| grep -v "INFO"\
| tail -n 1)
SPARK_HIVE=$("$MVN"* --force* help:evaluate
-Dexpression=project.activeProfiles -pl sql/hive $@ 2>/dev/null\
| grep -v "INFO"\
| fgrep --count "hive";\
# Reset exit status to 0, otherwise the script stops here if the last
grep finds nothing\
# because we use "set -o pipefail"
echo -n)
line 170:
BUILD_COMMAND=("$MVN" *--force* clean package -DskipTests $@)
that will force spark to use build/mvn and solve this problem.
BUG 2:
When I run running unit test VersionsSuite, it will hang for one night or
more. I use jstack and lsof and find it try to send a http request. That
seems not be a good item when runing test in terrible network.
I use jstack and finally find out reason:
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
- locked <0x0007440224d8> (a java.io.BufferedInputStream)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1323)
- locked <0x000744022530> (a
sun.net.www.protocol.http.HttpURLConnection)
at
java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)
...
and I use lsof:
java32082 user 247u IPv4 527001934 TCP 8.8.8.8:33233 (LISTEN)
java32082 user 267u IPv4 527001979 TCP 8.8.8.8:52301 (LISTEN)
java32082 user 316u IPv4 527001999 TCP *:51993 (LISTEN)
java32082 user 521u IPv4 527111590 TCP 8.8.8.8:53286
->butan141.server4you.de:http (ESTABLISHED)
This test suite try to connect butan141.server4you.de, The process will
hang when network is terrible .
what should I do when spark ut hang?
Hi, all: I compile spark-1.6.1 in redhat 5.7(I have installed spark-1.6.0-cdh5.7.0 hive-1.1.0+cdh5.7.0 and hadoop-2.6.0+cdh5.7.0 in this machine). my compile cmd is: build/mvn --force -Psparkr -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.7.0 -Phive -Phive-thriftserver when I use this cmd to run unit test. test hanged for a whole night(nearly 10 hour): build/mvn --force -Psparkr -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.7.0 -Phive -Phive-thriftserver test That is the last log: Discovery completed in 25 seconds, 803 milliseconds. Run starting. Expected test count is: 1716 StatisticsSuite: - parse analyze commands Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. log4j:WARN No appenders could be found for logger (hive.log). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. Warning: fs.defaultFS is not set when running "chgrp" command. Warning: fs.defaultFS is not set when running "chmod" command. - analyze MetastoreRelations - estimates the size of a test MetastoreRelation - auto converts to broadcast hash join, by size estimate of a relation - auto converts to broadcast left semi join, by size estimate of a relation VersionsSuite: I modify ./sql/hive/target/scala-2.10/test-classes/org/apache/spark/sql/hive/client/VersionsSuite.class and add print statement in each beginning of function, but no message print. Now what should I do to solve this problem? Thank you!
