Latest Advancement: I found the cause of NoClassDef exception: I wasn't using spark-submit, instead I tried to run the spark application directly with SparkConf set in the code. (this is handy in local debugging). However the old problem remains: Even my maven-shade plugin doesn't give any warning of duplicate, it still gives me the same error:
14/06/21 16:43:59 ERROR ExecutorUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-2,5,main] java.lang.NoSuchFieldError: INSTANCE at org.apache.http.entity.ContentType.parse(ContentType.java:229) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-throws-NoSuchFieldError-when-testing-on-cluster-mode-tp8064p8078.html Sent from the Apache Spark User List mailing list archive at Nabble.com.