thanks I think the problem is that the TEZ user group is exceptionally quiet. Just sent an email to Hive user group to see anyone has managed to built a vendor independent version.
Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 29 May 2016 at 21:23, Jörn Franke <jornfra...@gmail.com> wrote: > Well I think it is different from MR. It has some optimizations which you > do not find in MR. Especially the LLAP option in Hive2 makes it > interesting. > > I think hive 1.2 works with 0.7 and 2.0 with 0.8 . At least for 1.2 it is > integrated in the Hortonworks distribution. > > > On 29 May 2016, at 21:43, Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: > > Hi Jorn, > > I started building apache-tez-0.8.2 but got few errors. Couple of guys > from TEZ user group kindly gave a hand but I could not go very far (or may > be I did not make enough efforts) making it work. > > That TEZ user group is very quiet as well. > > My understanding is TEZ is MR with DAG but of course Spark has both plus > in-memory capability. > > It would be interesting to see what version of TEZ works as execution > engine with Hive. > > Vendors are divided on this (use Hive with TEZ) or use Impala instead of > Hive etc as I am sure you already know. > > Cheers, > > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 29 May 2016 at 20:19, Jörn Franke <jornfra...@gmail.com> wrote: > >> Very interesting do you plan also a test with TEZ? >> >> On 29 May 2016, at 13:40, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >> Hi, >> >> I did another study of Hive using Spark engine compared to Hive with MR. >> >> Basically took the original table imported using Sqoop and created and >> populated a new ORC table partitioned by year and month into 48 partitions >> as follows: >> >> <sales_partition.PNG> >> >> Connections use JDBC via beeline. Now for each partition using MR it >> takes an average of 17 minutes as seen below for each PARTITION.. Now that >> is just an individual partition and there are 48 partitions. >> >> In contrast doing the same operation with Spark engine took 10 minutes >> all inclusive. I just gave up on MR. You can see the StartTime and >> FinishTime from below >> >> <image.png> >> >> This is by no means indicate that Spark is much better than MR but shows >> that some very good results can ve achieved using Spark engine. >> >> >> Dr Mich Talebzadeh >> >> >> >> LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >> >> >> >> http://talebzadehmich.wordpress.com >> >> >> >> On 24 May 2016 at 08:03, Mich Talebzadeh <mich.talebza...@gmail.com> >> wrote: >> >>> Hi, >>> >>> We use Hive as the database and use Spark as an all purpose query tool. >>> >>> Whether Hive is the write database for purpose or one is better off with >>> something like Phoenix on Hbase, well the answer is it depends and your >>> mileage varies. >>> >>> So fit for purpose. >>> >>> Ideally what wants is to use the fastest method to get the results. How >>> fast we confine it to our SLA agreements in production and that helps us >>> from unnecessary further work as we technologists like to play around. >>> >>> So in short, we use Spark most of the time and use Hive as the backend >>> engine for data storage, mainly ORC tables. >>> >>> We use Hive on Spark and with Hive 2 on Spark 1.3.1 for now we have a >>> combination that works. Granted it helps to use Hive 2 on Spark 1.6.1 but >>> at the moment it is one of my projects. >>> >>> We do not use any vendor's products as it enables us to move away from >>> being tied down after years of SAP, Oracle and MS dependency to yet another >>> vendor. Besides there is some politics going on with one promoting Tez and >>> another Spark as a backend. That is fine but obviously we prefer an >>> independent assessment ourselves. >>> >>> My gut feeling is that one needs to look at the use case. Recently we >>> had to import a very large table from Oracle to Hive and decided to use >>> Spark 1.6.1 with Hive 2 on Spark 1.3.1 and that worked fine. We just used >>> JDBC connection with temp table and it was good. We could have used sqoop >>> but decided to settle for Spark so it all depends on use case. >>> >>> HTH >>> >>> >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> >>> On 24 May 2016 at 03:11, ayan guha <guha.a...@gmail.com> wrote: >>> >>>> Hi >>>> >>>> Thanks for very useful stats. >>>> >>>> Did you have any benchmark for using Spark as backend engine for Hive >>>> vs using Spark thrift server (and run spark code for hive queries)? We are >>>> using later but it will be very useful to remove thriftserver, if we can. >>>> >>>> On Tue, May 24, 2016 at 9:51 AM, Jörn Franke <jornfra...@gmail.com> >>>> wrote: >>>> >>>>> >>>>> Hi Mich, >>>>> >>>>> I think these comparisons are useful. One interesting aspect could be >>>>> hardware scalability in this context. Additionally different type of >>>>> computations. Furthermore, one could compare Spark and Tez+llap as >>>>> execution engines. I have the gut feeling that each one can be justified >>>>> by different use cases. >>>>> Nevertheless, there should be always a disclaimer for such >>>>> comparisons, because Spark and Hive are not good for a lot of concurrent >>>>> lookups of single rows. They are not good for frequently write small >>>>> amounts of data (eg sensor data). Here hbase could be more interesting. >>>>> Other use cases can justify graph databases, such as Titan, or text >>>>> analytics/ data matching using Solr on Hadoop. >>>>> Finally, even if you have a lot of data you need to think if you >>>>> always have to process everything. For instance, I have found valid use >>>>> cases in practice where we decided to evaluate 10 machine learning models >>>>> in parallel on only a sample of data and only evaluate the "winning" model >>>>> of the total of data. >>>>> >>>>> As always it depends :) >>>>> >>>>> Best regards >>>>> >>>>> P.s.: at least Hortonworks has in their distribution spark 1.5 with >>>>> hive 1.2 and spark 1.6 with hive 1.2. Maybe they have somewhere described >>>>> how to manage bringing both together. You may check also Apache Bigtop >>>>> (vendor neutral distribution) on how they managed to bring both together. >>>>> >>>>> On 23 May 2016, at 01:42, Mich Talebzadeh <mich.talebza...@gmail.com> >>>>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> I have done a number of extensive tests using Spark-shell with Hive DB >>>>> and ORC tables. >>>>> >>>>> >>>>> >>>>> Now one issue that we typically face is and I quote: >>>>> >>>>> >>>>> >>>>> Spark is fast as it uses Memory and DAG. Great but when we save data >>>>> it is not fast enough >>>>> >>>>> OK but there is a solution now. If you use Spark with Hive and you are >>>>> on a descent version of Hive >= 0.14, then you can also deploy Spark as >>>>> execution engine for Hive. That will make your application run pretty fast >>>>> as you no longer rely on the old Map-Reduce for Hive engine. In a nutshell >>>>> what you are gaining speed in both querying and storage. >>>>> >>>>> >>>>> >>>>> I have made some comparisons on this set-up and I am sure some of you >>>>> will find it useful. >>>>> >>>>> >>>>> >>>>> The version of Spark I use for Spark queries (Spark as query tool) is >>>>> 1.6. >>>>> >>>>> The version of Hive I use in Hive 2 >>>>> >>>>> The version of Spark I use as Hive execution engine is 1.3.1 It works >>>>> and frankly Spark 1.3.1 as an execution engine is adequate (until we sort >>>>> out the Hadoop libraries mismatch). >>>>> >>>>> >>>>> >>>>> An example I am using Hive on Spark engine to find the min and max of >>>>> IDs for a table with 1 billion rows: >>>>> >>>>> >>>>> >>>>> 0: jdbc:hive2://rhes564:10010/default> select min(id), >>>>> max(id),avg(id), stddev(id) from oraclehadoop.dummy; >>>>> >>>>> Query ID = hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Starting Spark Job = 5e092ef9-d798-4952-b156-74df49da9151 >>>>> >>>>> >>>>> >>>>> INFO : Completed compiling >>>>> command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006); >>>>> Time taken: 1.911 seconds >>>>> >>>>> INFO : Executing >>>>> command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006): >>>>> select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy >>>>> >>>>> INFO : Query ID = >>>>> hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006 >>>>> >>>>> INFO : Total jobs = 1 >>>>> >>>>> INFO : Launching Job 1 out of 1 >>>>> >>>>> INFO : Starting task [Stage-1:MAPRED] in serial mode >>>>> >>>>> >>>>> >>>>> Query Hive on Spark job[0] stages: >>>>> >>>>> 0 >>>>> >>>>> 1 >>>>> >>>>> Status: Running (Hive on Spark job[0]) >>>>> >>>>> Job Progress Format >>>>> >>>>> CurrentTime StageId_StageAttemptId: >>>>> SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount >>>>> [StageCost] >>>>> >>>>> 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1 >>>>> >>>>> 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1 >>>>> >>>>> 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1 >>>>> >>>>> 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22 Stage-1_0: 0/1 >>>>> >>>>> INFO : >>>>> >>>>> Query Hive on Spark job[0] stages: >>>>> >>>>> INFO : 0 >>>>> >>>>> INFO : 1 >>>>> >>>>> INFO : >>>>> >>>>> Status: Running (Hive on Spark job[0]) >>>>> >>>>> INFO : Job Progress Format >>>>> >>>>> CurrentTime StageId_StageAttemptId: >>>>> SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount >>>>> [StageCost] >>>>> >>>>> INFO : 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1 >>>>> >>>>> INFO : 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1 >>>>> >>>>> INFO : 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1 >>>>> >>>>> INFO : 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22 Stage-1_0: 0/1 >>>>> >>>>> 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished Stage-1_0: >>>>> 0(+1)/1 >>>>> >>>>> 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished Stage-1_0: 1/1 >>>>> Finished >>>>> >>>>> Status: Finished successfully in 53.25 seconds >>>>> >>>>> OK >>>>> >>>>> INFO : 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished >>>>> Stage-1_0: 0(+1)/1 >>>>> >>>>> INFO : 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished >>>>> Stage-1_0: 1/1 Finished >>>>> >>>>> INFO : Status: Finished successfully in 53.25 seconds >>>>> >>>>> INFO : Completed executing >>>>> command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006); >>>>> Time taken: 56.337 seconds >>>>> >>>>> INFO : OK >>>>> >>>>> +-----+------------+---------------+-----------------------+--+ >>>>> >>>>> | c0 | c1 | c2 | c3 | >>>>> >>>>> +-----+------------+---------------+-----------------------+--+ >>>>> >>>>> | 1 | 100000000 | 5.00000005E7 | 2.8867513459481288E7 | >>>>> >>>>> +-----+------------+---------------+-----------------------+--+ >>>>> >>>>> 1 row selected (58.529 seconds) >>>>> >>>>> >>>>> >>>>> 58 seconds first run with cold cache is pretty good >>>>> >>>>> >>>>> >>>>> And let us compare it with running the same query on map-reduce engine >>>>> >>>>> >>>>> >>>>> : jdbc:hive2://rhes564:10010/default> set hive.execution.engine=mr; >>>>> >>>>> Hive-on-MR is deprecated in Hive 2 and may not be available in the >>>>> future versions. Consider using a different execution engine (i.e. spark, >>>>> tez) or using Hive 1.X releases. >>>>> >>>>> No rows affected (0.007 seconds) >>>>> >>>>> 0: jdbc:hive2://rhes564:10010/default> select min(id), >>>>> max(id),avg(id), stddev(id) from oraclehadoop.dummy; >>>>> >>>>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available >>>>> in the future versions. Consider using a different execution engine (i.e. >>>>> spark, tez) or using Hive 1.X releases. >>>>> >>>>> Query ID = hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc >>>>> >>>>> Total jobs = 1 >>>>> >>>>> Launching Job 1 out of 1 >>>>> >>>>> Number of reduce tasks determined at compile time: 1 >>>>> >>>>> In order to change the average load for a reducer (in bytes): >>>>> >>>>> set hive.exec.reducers.bytes.per.reducer=<number> >>>>> >>>>> In order to limit the maximum number of reducers: >>>>> >>>>> set hive.exec.reducers.max=<number> >>>>> >>>>> In order to set a constant number of reducers: >>>>> >>>>> set mapreduce.job.reduces=<number> >>>>> >>>>> Starting Job = job_1463956731753_0005, Tracking URL = >>>>> http://localhost.localdomain:8088/proxy/application_1463956731753_0005/ >>>>> >>>>> Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job -kill >>>>> job_1463956731753_0005 >>>>> >>>>> Hadoop job information for Stage-1: number of mappers: 22; number of >>>>> reducers: 1 >>>>> >>>>> 2016-05-23 00:26:38,127 Stage-1 map = 0%, reduce = 0% >>>>> >>>>> INFO : Compiling >>>>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc): >>>>> select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy >>>>> >>>>> INFO : Semantic Analysis Completed >>>>> >>>>> INFO : Returning Hive schema: >>>>> Schema(fieldSchemas:[FieldSchema(name:c0, type:int, comment:null), >>>>> FieldSchema(name:c1, type:int, comment:null), FieldSchema(name:c2, >>>>> type:double, comment:null), FieldSchema(name:c3, type:double, >>>>> comment:null)], properties:null) >>>>> >>>>> INFO : Completed compiling >>>>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc); >>>>> Time taken: 0.144 seconds >>>>> >>>>> INFO : Executing >>>>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc): >>>>> select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy >>>>> >>>>> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in >>>>> the future versions. Consider using a different execution engine (i.e. >>>>> spark, tez) or using Hive 1.X releases. >>>>> >>>>> INFO : WARNING: Hive-on-MR is deprecated in Hive 2 and may not be >>>>> available in the future versions. Consider using a different execution >>>>> engine (i.e. spark, tez) or using Hive 1.X releases. >>>>> >>>>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available >>>>> in the future versions. Consider using a different execution engine (i.e. >>>>> spark, tez) or using Hive 1.X releases. >>>>> >>>>> INFO : Query ID = >>>>> hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc >>>>> >>>>> INFO : Total jobs = 1 >>>>> >>>>> INFO : Launching Job 1 out of 1 >>>>> >>>>> INFO : Starting task [Stage-1:MAPRED] in serial mode >>>>> >>>>> INFO : Number of reduce tasks determined at compile time: 1 >>>>> >>>>> INFO : In order to change the average load for a reducer (in bytes): >>>>> >>>>> INFO : set hive.exec.reducers.bytes.per.reducer=<number> >>>>> >>>>> INFO : In order to limit the maximum number of reducers: >>>>> >>>>> INFO : set hive.exec.reducers.max=<number> >>>>> >>>>> INFO : In order to set a constant number of reducers: >>>>> >>>>> INFO : set mapreduce.job.reduces=<number> >>>>> >>>>> WARN : Hadoop command-line option parsing not performed. Implement >>>>> the Tool interface and execute your application with ToolRunner to remedy >>>>> this. >>>>> >>>>> INFO : number of splits:22 >>>>> >>>>> INFO : Submitting tokens for job: job_1463956731753_0005 >>>>> >>>>> INFO : The url to track the job: >>>>> http://localhost.localdomain:8088/proxy/application_1463956731753_0005/ >>>>> >>>>> INFO : Starting Job = job_1463956731753_0005, Tracking URL = >>>>> http://localhost.localdomain:8088/proxy/application_1463956731753_0005/ >>>>> >>>>> INFO : Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job -kill >>>>> job_1463956731753_0005 >>>>> >>>>> INFO : Hadoop job information for Stage-1: number of mappers: 22; >>>>> number of reducers: 1 >>>>> >>>>> INFO : 2016-05-23 00:26:38,127 Stage-1 map = 0%, reduce = 0% >>>>> >>>>> 2016-05-23 00:26:44,367 Stage-1 map = 5%, reduce = 0%, Cumulative CPU >>>>> 4.56 sec >>>>> >>>>> INFO : 2016-05-23 00:26:44,367 Stage-1 map = 5%, reduce = 0%, >>>>> Cumulative CPU 4.56 sec >>>>> >>>>> 2016-05-23 00:26:50,558 Stage-1 map = 9%, reduce = 0%, Cumulative CPU >>>>> 9.17 sec >>>>> >>>>> INFO : 2016-05-23 00:26:50,558 Stage-1 map = 9%, reduce = 0%, >>>>> Cumulative CPU 9.17 sec >>>>> >>>>> 2016-05-23 00:26:56,747 Stage-1 map = 14%, reduce = 0%, Cumulative >>>>> CPU 14.04 sec >>>>> >>>>> INFO : 2016-05-23 00:26:56,747 Stage-1 map = 14%, reduce = 0%, >>>>> Cumulative CPU 14.04 sec >>>>> >>>>> 2016-05-23 00:27:02,944 Stage-1 map = 18%, reduce = 0%, Cumulative >>>>> CPU 18.64 sec >>>>> >>>>> INFO : 2016-05-23 00:27:02,944 Stage-1 map = 18%, reduce = 0%, >>>>> Cumulative CPU 18.64 sec >>>>> >>>>> 2016-05-23 00:27:08,105 Stage-1 map = 23%, reduce = 0%, Cumulative >>>>> CPU 23.25 sec >>>>> >>>>> INFO : 2016-05-23 00:27:08,105 Stage-1 map = 23%, reduce = 0%, >>>>> Cumulative CPU 23.25 sec >>>>> >>>>> 2016-05-23 00:27:14,298 Stage-1 map = 27%, reduce = 0%, Cumulative >>>>> CPU 27.84 sec >>>>> >>>>> INFO : 2016-05-23 00:27:14,298 Stage-1 map = 27%, reduce = 0%, >>>>> Cumulative CPU 27.84 sec >>>>> >>>>> 2016-05-23 00:27:20,484 Stage-1 map = 32%, reduce = 0%, Cumulative >>>>> CPU 32.56 sec >>>>> >>>>> INFO : 2016-05-23 00:27:20,484 Stage-1 map = 32%, reduce = 0%, >>>>> Cumulative CPU 32.56 sec >>>>> >>>>> 2016-05-23 00:27:26,659 Stage-1 map = 36%, reduce = 0%, Cumulative >>>>> CPU 37.1 sec >>>>> >>>>> INFO : 2016-05-23 00:27:26,659 Stage-1 map = 36%, reduce = 0%, >>>>> Cumulative CPU 37.1 sec >>>>> >>>>> 2016-05-23 00:27:32,839 Stage-1 map = 41%, reduce = 0%, Cumulative >>>>> CPU 41.74 sec >>>>> >>>>> INFO : 2016-05-23 00:27:32,839 Stage-1 map = 41%, reduce = 0%, >>>>> Cumulative CPU 41.74 sec >>>>> >>>>> 2016-05-23 00:27:39,003 Stage-1 map = 45%, reduce = 0%, Cumulative >>>>> CPU 46.32 sec >>>>> >>>>> INFO : 2016-05-23 00:27:39,003 Stage-1 map = 45%, reduce = 0%, >>>>> Cumulative CPU 46.32 sec >>>>> >>>>> 2016-05-23 00:27:45,173 Stage-1 map = 50%, reduce = 0%, Cumulative >>>>> CPU 50.93 sec >>>>> >>>>> 2016-05-23 00:27:50,316 Stage-1 map = 55%, reduce = 0%, Cumulative >>>>> CPU 55.55 sec >>>>> >>>>> INFO : 2016-05-23 00:27:45,173 Stage-1 map = 50%, reduce = 0%, >>>>> Cumulative CPU 50.93 sec >>>>> >>>>> INFO : 2016-05-23 00:27:50,316 Stage-1 map = 55%, reduce = 0%, >>>>> Cumulative CPU 55.55 sec >>>>> >>>>> 2016-05-23 00:27:56,482 Stage-1 map = 59%, reduce = 0%, Cumulative >>>>> CPU 60.25 sec >>>>> >>>>> INFO : 2016-05-23 00:27:56,482 Stage-1 map = 59%, reduce = 0%, >>>>> Cumulative CPU 60.25 sec >>>>> >>>>> 2016-05-23 00:28:02,642 Stage-1 map = 64%, reduce = 0%, Cumulative >>>>> CPU 64.86 sec >>>>> >>>>> INFO : 2016-05-23 00:28:02,642 Stage-1 map = 64%, reduce = 0%, >>>>> Cumulative CPU 64.86 sec >>>>> >>>>> 2016-05-23 00:28:08,814 Stage-1 map = 68%, reduce = 0%, Cumulative >>>>> CPU 69.41 sec >>>>> >>>>> INFO : 2016-05-23 00:28:08,814 Stage-1 map = 68%, reduce = 0%, >>>>> Cumulative CPU 69.41 sec >>>>> >>>>> 2016-05-23 00:28:14,977 Stage-1 map = 73%, reduce = 0%, Cumulative >>>>> CPU 74.06 sec >>>>> >>>>> INFO : 2016-05-23 00:28:14,977 Stage-1 map = 73%, reduce = 0%, >>>>> Cumulative CPU 74.06 sec >>>>> >>>>> 2016-05-23 00:28:21,134 Stage-1 map = 77%, reduce = 0%, Cumulative >>>>> CPU 78.72 sec >>>>> >>>>> INFO : 2016-05-23 00:28:21,134 Stage-1 map = 77%, reduce = 0%, >>>>> Cumulative CPU 78.72 sec >>>>> >>>>> 2016-05-23 00:28:27,282 Stage-1 map = 82%, reduce = 0%, Cumulative >>>>> CPU 83.32 sec >>>>> >>>>> INFO : 2016-05-23 00:28:27,282 Stage-1 map = 82%, reduce = 0%, >>>>> Cumulative CPU 83.32 sec >>>>> >>>>> 2016-05-23 00:28:33,437 Stage-1 map = 86%, reduce = 0%, Cumulative >>>>> CPU 87.9 sec >>>>> >>>>> INFO : 2016-05-23 00:28:33,437 Stage-1 map = 86%, reduce = 0%, >>>>> Cumulative CPU 87.9 sec >>>>> >>>>> 2016-05-23 00:28:38,579 Stage-1 map = 91%, reduce = 0%, Cumulative >>>>> CPU 92.52 sec >>>>> >>>>> INFO : 2016-05-23 00:28:38,579 Stage-1 map = 91%, reduce = 0%, >>>>> Cumulative CPU 92.52 sec >>>>> >>>>> 2016-05-23 00:28:44,759 Stage-1 map = 95%, reduce = 0%, Cumulative >>>>> CPU 97.35 sec >>>>> >>>>> INFO : 2016-05-23 00:28:44,759 Stage-1 map = 95%, reduce = 0%, >>>>> Cumulative CPU 97.35 sec >>>>> >>>>> 2016-05-23 00:28:49,915 Stage-1 map = 100%, reduce = 0%, Cumulative >>>>> CPU 99.6 sec >>>>> >>>>> INFO : 2016-05-23 00:28:49,915 Stage-1 map = 100%, reduce = 0%, >>>>> Cumulative CPU 99.6 sec >>>>> >>>>> 2016-05-23 00:28:54,043 Stage-1 map = 100%, reduce = 100%, Cumulative >>>>> CPU 101.4 sec >>>>> >>>>> MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 msec >>>>> >>>>> Ended Job = job_1463956731753_0005 >>>>> >>>>> MapReduce Jobs Launched: >>>>> >>>>> Stage-Stage-1: Map: 22 Reduce: 1 Cumulative CPU: 101.4 sec HDFS >>>>> Read: 5318569 HDFS Write: 46 SUCCESS >>>>> >>>>> Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec >>>>> >>>>> OK >>>>> >>>>> INFO : 2016-05-23 00:28:54,043 Stage-1 map = 100%, reduce = 100%, >>>>> Cumulative CPU 101.4 sec >>>>> >>>>> INFO : MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 >>>>> msec >>>>> >>>>> INFO : Ended Job = job_1463956731753_0005 >>>>> >>>>> INFO : MapReduce Jobs Launched: >>>>> >>>>> INFO : Stage-Stage-1: Map: 22 Reduce: 1 Cumulative CPU: 101.4 >>>>> sec HDFS Read: 5318569 HDFS Write: 46 SUCCESS >>>>> >>>>> INFO : Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec >>>>> >>>>> INFO : Completed executing >>>>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc); >>>>> Time taken: 142.525 seconds >>>>> >>>>> INFO : OK >>>>> >>>>> +-----+------------+---------------+-----------------------+--+ >>>>> >>>>> | c0 | c1 | c2 | c3 | >>>>> >>>>> +-----+------------+---------------+-----------------------+--+ >>>>> >>>>> | 1 | 100000000 | 5.00000005E7 | 2.8867513459481288E7 | >>>>> >>>>> +-----+------------+---------------+-----------------------+--+ >>>>> >>>>> 1 row selected (142.744 seconds) >>>>> >>>>> >>>>> >>>>> OK Hive on map-reduce engine took 142 seconds compared to 58 seconds >>>>> with Hive on Spark. So you can obviously gain pretty well by using Hive on >>>>> Spark. >>>>> >>>>> >>>>> >>>>> Please also note that I did not use any vendor's build for this >>>>> purpose. I compiled Spark 1.3.1 myself. >>>>> >>>>> >>>>> >>>>> HTH >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Dr Mich Talebzadeh >>>>> >>>>> >>>>> >>>>> LinkedIn >>>>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>>>> >>>>> >>>>> >>>>> http://talebzadehmich.wordpress.com/ >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Best Regards, >>>> Ayan Guha >>>> >>> >>> >> >
