[
https://issues.apache.org/jira/browse/SPARK-7819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14572270#comment-14572270
]
Fi commented on SPARK-7819:
---------------------------
@Nathan: No, haven't tried spark-submit. That's for Scala/Java based jobs,
right? I am running pyspark jobs, so no. However, while poking around in how
this system works, I believe most of the bits that makes spark-submit work is
done in the python case as well (i.e. load up spark-env.sh and read in
spark-defaults.conf, as well as setup various classpaths).
BTW, the stacktrace is attached to this ticket as invalidClassException.log
@Yin: The driver and workers (running as a mesos framework), were all running
the exact same spark tarball, deployed to SPARK_HOME=/opt/spark/current on the
driver and each of the slaves.
I built from the most recent revision in
https://github.com/apache/spark/commits/branch-1.4
(84da653192a2d9edb82d0dbe50f577c4dc6a0c78)
which already contains the SPARK-8020 fix.
Anyway, looking at that ticket, I don't recall seeing error messages in the log
matching any of the noted stacktraces.
> Isolated Hive Client Loader appears to cause Native Library
> libMapRClient.4.0.2-mapr.so already loaded in another classloader error
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-7819
> URL: https://issues.apache.org/jira/browse/SPARK-7819
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.4.0
> Reporter: Fi
> Priority: Critical
> Attachments: invalidClassException.log, stacktrace.txt, test.py
>
>
> In reference to the pull request: https://github.com/apache/spark/pull/5876
> I have been running the Spark 1.3 branch for some time with no major hiccups,
> and recently switched to the Spark 1.4 branch.
> I build my spark distribution with the following build command:
> {noformat}
> make-distribution.sh --tgz --skip-java-test --with-tachyon -Phive
> -Phive-0.13.1 -Pmapr4 -Pspark-ganglia-lgpl -Pkinesis-asl -Phive-thriftserver
> {noformat}
> When running a python script containing a series of smoke tests I use to
> validate the build, I encountered an error under the following conditions:
> * start a spark context
> * start a hive context
> * run any hive query
> * stop the spark context
> * start a second spark context
> * run any hive query
> ** ERROR
> From what I can tell, the Isolated Class Loader is hitting a MapR class that
> is loading its native library (presumedly as part of a static initializer).
> Unfortunately, the JVM prohibits this the second time around.
> I would think that shutting down the SparkContext would clear out any
> vestigials of the JVM, so I'm surprised that this would even be a problem.
> Note: all other smoke tests we are running passes fine.
> I will attach the stacktrace and a python script reproducing the issue (at
> least for my environment and build).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
