hi,XueFu,thanks a lot for your help,now i will provide more detail to
reproduce this ssue:
1),i checkout a spark branch from hive
github(https://github.com/apache/hive/tree/spark on Nov 29,becasue of
for version now it will give something wrong about:Caused by:
java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ),
and built command:mvn clean package -DskipTests -Phadoop-2 -Pdist
after built i get package from
:/home/ym/hive-on-spark/hive1129/hive/packaging/target(apache-hive-0.15.0-SNAPSHOT-bin.tar.gz)
2)i checkout spark from
https://github.com/apache/spark/tree/v1.2.0-snapshot0,becasue of spark
branch-1.2 is with spark parent(1.2.1-SNAPSHOT),so i chose
v1.2.0-snapshot0 and i compare this spark's pom.xml with
spark-parent-1.2.0-SNAPSHOT.pom(get from
http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and
there is only difference is spark-parent name,and built command is :
|mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean
package|
3)comand i execute in hive-shell:
./hive --auxpath
/opt/hispark/spark/assembly/target/scala-2.10/spark-assembly-1.2.0-hadoop2.4.0.jar(copy
this jar to hive dir lib already)
create table student(sno int,sname string,sage int,ssex string) row
format delimited FIELDS TERMINATED BY ',';
create table score(sno int,cno int,sage int) row format delimited FIELDS
TERMINATED BY ',';
load data local inpath
'/home/hive-on-spark/temp/spark-1.2.0/examples/src/main/resources/student.txt'
into table student;
load data local inpath
'/home/hive-on-spark/temp/spark-1.2.0/examples/src/main/resources/score.txt'
into table score;
set hive.execution.engine=spark;
set spark.master=spark://10.175.xxx.xxx:7077;
set spark.eventLog.enabled=true;
set spark.executor.memory=9086m;
set spark.serializer=org.apache.spark.serializer.KryoSerializer;
select distinct st.sno,sname from student st join score sc
on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28;(work in mr)
4)
studdent.txt file
1,rsh,27,female
2,kupo,28,male
3,astin,29,female
4,beike,30,male
5,aili,31,famle
score.txt file
1,10,80
2,11,85
3,12,90
4,13,95
5,14,100
On 2014/12/2 23:28, Xuefu Zhang wrote:
Could you provide details on how to reproduce the issue? such as the
exact spark branch, the command to build Spark, how you build Hive,
and what queries/commands you run.
We are running Hive on Spark all the time. Our pre-commit test runs
without any issue.
Thanks,
Xuefu
On Tue, Dec 2, 2014 at 4:13 AM, yuemeng1 <yueme...@huawei.com
<mailto:yueme...@huawei.com>> wrote:
hi,XueFu
i checkout a spark branch from
sparkgithub(tags:v1.2.0-snapshot0)and i compare this spark's
pom.xml with spark-parent-1.2.0-SNAPSHOT.pom(get from
http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and
there is only difference is follow:
in spark-parent-1.2.0-SNAPSHOT.pom
<artifactId>spark-parent</artifactId>
<version>1.2.0-SNAPSHOT</version>
and in v1.2.0-snapshot0
<artifactId>spark-parent</artifactId>
<version>1.2.0</version>
i think there is no essence diff,and i built v1.2.0-snapshot0 and
deploy it as my spark clusters
when i run query about join two table ,it still give some error
what i show u earlier
Job aborted due to stage failure: Task 0 in stage 1.0 failed 4
times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7,
datasight18): java.lang.NullPointerException+details
Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most
recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18):
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Driver stacktrace:
i think my spark clusters did't had any problem,but why always
give me such error
On 2014/12/2 13:39, Xuefu Zhang wrote:
You need to build your spark assembly from spark 1.2 branch. this
should give your both a spark build as well as spark-assembly
jar, which you need to copy to Hive lib directory. Snapshot is
fine, and spark 1.2 hasn't been released yet.
--Xuefu
On Mon, Dec 1, 2014 at 7:41 PM, yuemeng1 <yueme...@huawei.com
<mailto:yueme...@huawei.com>> wrote:
hi.XueFu,
thanks a lot for your inforamtion,but as far as i know ,the
latest spark version on github is spark-snapshot-1.3,but
there is no spark-1.2,only have a branch-1.2 with
spark-snapshot-1.2,can u tell me which spark version i should
built,and for now,that's
spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar produce error
like that
On 2014/12/2 11:03, Xuefu Zhang wrote:
It seems that wrong class, HiveInputFormat, is loaded. The
stacktrace is way off the current Hive code. You need to
build Spark 1.2 and copy spark-assembly jar to Hive's lib
directory and that it.
--Xuefu
On Mon, Dec 1, 2014 at 6:22 PM, yuemeng1
<yueme...@huawei.com <mailto:yueme...@huawei.com>> wrote:
hi,i built a hive on spark package and my spark assembly
jar is
spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar,when i run
a query in hive shell,before execute this query,
i set all the require which hive need with spark.and i
execute a join query :
select distinct st.sno,sname from student st join score
sc on(st.sno=sc.sno) where sc.cno IN(11,12,13) and
st.sage > 28;
but it failed,
get follow error in spark webUI:
Job aborted due to stage failure: Task 0 in stage 1.0
failed 4 times, most recent failure: Lost task 0.3 in
stage 1.0 (TID 7, datasight18):
java.lang.NullPointerException+details
Job aborted due to stage failure: Task 0 in stage 1.0 failed 4
times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18):
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430)
at
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587)
at
org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210)
at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
at
org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:230)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
Driver stacktrace:
can u give me a help to deal this probelm,and i think my
built was succussed!
