Hi Yuemeng, I'm glad that Hive on Spark finally works for you. As you know, this project is still in development and yet to be released. Thus, please forgive about the lack of proper documentation. We have a "Get Started" page that's linked in HIVE-7292. If you can improve the document there, it would be very helpful for other Hive users.
Thanks, Xuefu On Wed, Dec 3, 2014 at 5:42 PM, yuemeng1 <yueme...@huawei.com> wrote: > hi,thanks a lot for your help,with your help ,my hive-on-spark can work > well now > it take me long time to install and deploy.here are some advice,i think > we need to improve the installation documentation, allowing users to use > the least amount of time to compile and install > 1)add which spark version we should pick from spark github if we select > built spark instead of download a spark pre-built,tell them the right built > commad!(not include Pyarn ,Phive) > 2)if they get some error during built ,such as [ERRO > /hive/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobStatus.java: > [22,24] cannot find symbol > [ERROR] symbol: class JobExecutionStatus,tell them what they can do? > for our users,first to use it ,then feel good or bad? > and if u need,i can add something to start document > > > thanks > yuemeng > > > > > > > On 2014/12/3 11:03, Xuefu Zhang wrote: > > When you build Spark, remove -Phive as well as -Pyarn. When you run hive > queries, you may need to run "set spark.home=/path/to/spark/dir"; > > Thanks, > Xuefu > > On Tue, Dec 2, 2014 at 6:29 PM, yuemeng1 <yueme...@huawei.com> wrote: > >> hi,XueFu,thanks a lot for your help,now i will provide more detail to >> reproduce this ssue: >> 1),i checkout a spark branch from hive github( >> https://github.com/apache/hive/tree/spark on Nov 29,becasue of for >> version now it will give something wrong about:Caused by: >> java.lang.RuntimeException: Unable to instantiate >> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ), >> and built command:mvn clean package -DskipTests -Phadoop-2 -Pdist >> after built i get package from >> :/home/ym/hive-on-spark/hive1129/hive/packaging/target(apache-hive-0.15.0-SNAPSHOT-bin.tar.gz) >> 2)i checkout spark from >> https://github.com/apache/spark/tree/v1.2.0-snapshot0,becasue of spark >> branch-1.2 is with spark parent(1.2.1-SNAPSHOT),so i chose >> v1.2.0-snapshot0 and i compare this spark's pom.xml with >> spark-parent-1.2.0-SNAPSHOT.pom(get from >> http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and >> there is only difference is spark-parent name,and built command is : >> >> mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean >> package >> >> 3)comand i execute in hive-shell: >> ./hive --auxpath >> /opt/hispark/spark/assembly/target/scala-2.10/spark-assembly-1.2.0-hadoop2.4.0.jar(copy >> this jar to hive dir lib already) >> create table student(sno int,sname string,sage int,ssex string) row >> format delimited FIELDS TERMINATED BY ','; >> create table score(sno int,cno int,sage int) row format delimited FIELDS >> TERMINATED BY ','; >> load data local inpath >> '/home/hive-on-spark/temp/spark-1.2.0/examples/src/main/resources/student.txt' >> into table student; >> load data local inpath >> '/home/hive-on-spark/temp/spark-1.2.0/examples/src/main/resources/score.txt' >> into table score; >> set hive.execution.engine=spark; >> set spark.master=spark://10.175.xxx.xxx:7077; >> set spark.eventLog.enabled=true; >> set spark.executor.memory=9086m; >> set spark.serializer=org.apache.spark.serializer.KryoSerializer; >> select distinct st.sno,sname from student st join score sc >> on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28;(work in mr) >> 4) >> studdent.txt file >> 1,rsh,27,female >> 2,kupo,28,male >> 3,astin,29,female >> 4,beike,30,male >> 5,aili,31,famle >> >> score.txt file >> 1,10,80 >> 2,11,85 >> 3,12,90 >> 4,13,95 >> 5,14,100 >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> On 2014/12/2 23:28, Xuefu Zhang wrote: >> >> Could you provide details on how to reproduce the issue? such as the >> exact spark branch, the command to build Spark, how you build Hive, and >> what queries/commands you run. >> >> We are running Hive on Spark all the time. Our pre-commit test runs >> without any issue. >> >> Thanks, >> Xuefu >> >> On Tue, Dec 2, 2014 at 4:13 AM, yuemeng1 <yueme...@huawei.com> wrote: >> >>> hi,XueFu >>> i checkout a spark branch from sparkgithub(tags:v1.2.0-snapshot0)and i >>> compare this spark's pom.xml with spark-parent-1.2.0-SNAPSHOT.pom(get from >>> http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark_2.10-1.2-SNAPSHOT/org/apache/spark/spark-parent/1.2.0-SNAPSHOT/),and >>> there is only difference is follow: >>> in spark-parent-1.2.0-SNAPSHOT.pom >>> <artifactId>spark-parent</artifactId> >>> <version>1.2.0-SNAPSHOT</version> >>> and in v1.2.0-snapshot0 >>> <artifactId>spark-parent</artifactId> >>> <version>1.2.0</version> >>> i think there is no essence diff,and i built v1.2.0-snapshot0 and deploy >>> it as my spark clusters >>> when i run query about join two table ,it still give some error what i >>> show u earlier >>> >>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, >>> most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): >>> java.lang.NullPointerException+details >>> >>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most >>> recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): >>> java.lang.NullPointerException >>> at >>> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) >>> at >>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437) >>> at >>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430) >>> at >>> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) >>> at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233) >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210) >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99) >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) >>> at >>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) >>> at >>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>> at org.apache.spark.scheduler.Task.run(Task.scala:56) >>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>> at java.lang.Thread.run(Thread.java:722) >>> >>> Driver stacktrace: >>> >>> >>> >>> i think my spark clusters did't had any problem,but why always give me >>> such error >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On 2014/12/2 13:39, Xuefu Zhang wrote: >>> >>> You need to build your spark assembly from spark 1.2 branch. this >>> should give your both a spark build as well as spark-assembly jar, which >>> you need to copy to Hive lib directory. Snapshot is fine, and spark 1.2 >>> hasn't been released yet. >>> >>> --Xuefu >>> >>> On Mon, Dec 1, 2014 at 7:41 PM, yuemeng1 <yueme...@huawei.com> wrote: >>> >>>> >>>> >>>> hi.XueFu, >>>> thanks a lot for your inforamtion,but as far as i know ,the latest >>>> spark version on github is spark-snapshot-1.3,but there is no >>>> spark-1.2,only have a branch-1.2 with spark-snapshot-1.2,can u tell me >>>> which spark version i should built,and for now,that's >>>> spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar produce error like that >>>> >>>> >>>> On 2014/12/2 11:03, Xuefu Zhang wrote: >>>> >>>> It seems that wrong class, HiveInputFormat, is loaded. The stacktrace >>>> is way off the current Hive code. You need to build Spark 1.2 and copy >>>> spark-assembly jar to Hive's lib directory and that it. >>>> >>>> --Xuefu >>>> >>>> On Mon, Dec 1, 2014 at 6:22 PM, yuemeng1 <yueme...@huawei.com> wrote: >>>> >>>>> hi,i built a hive on spark package and my spark assembly jar is >>>>> spark-assembly-1.2.0-SNAPSHOT-hadoop2.4.0.jar,when i run a query in hive >>>>> shell,before execute this query, >>>>> i set all the require which hive need with spark.and i execute a >>>>> join query : >>>>> select distinct st.sno,sname from student st join score sc >>>>> on(st.sno=sc.sno) where sc.cno IN(11,12,13) and st.sage > 28; >>>>> but it failed, >>>>> get follow error in spark webUI: >>>>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, >>>>> most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): >>>>> java.lang.NullPointerException+details >>>>> >>>>> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, >>>>> most recent failure: Lost task 0.3 in stage 1.0 (TID 7, datasight18): >>>>> java.lang.NullPointerException >>>>> at >>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:255) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:437) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:430) >>>>> at >>>>> org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:587) >>>>> at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:233) >>>>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:210) >>>>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:99) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) >>>>> at >>>>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) >>>>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:263) >>>>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:230) >>>>> at >>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) >>>>> at >>>>> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) >>>>> at org.apache.spark.scheduler.Task.run(Task.scala:56) >>>>> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:196) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) >>>>> at java.lang.Thread.run(Thread.java:722) >>>>> >>>>> Driver stacktrace: >>>>> >>>>> >>>>> can u give me a help to deal this probelm,and i think my built was >>>>> succussed! >>>>> >>>> >>>> >>>> >>> >>> >> >> > >
