Hi Timur and everyone.
I will answer your first question as it is very relevant
1) How to make 2 versions of Spark live together on the same cluster
(libraries clash, paths, etc.) ?
Most of the Spark users perform ETL, ML operations on Spark as well. So, we
may have 3 Spark installations simultaneously
There are two distinct points here.
Using Spark as a query engine. That is BAU and most forum members use it
everyday. You run Spark with either Standalone, Yarn or Mesos as Cluster
managers. You start master that does the management of resources and you
start slaves to create workers.
You deploy Spark either by Spark-shell, Spark-sql or submit jobs through
spark-submit etc. You may or may not use Hive as your database. You may use
Hbase via Phoenix etc
If you choose to use Hive as your database, on every host of cluster
including your master host, you ensure that Hive APIs are installed
(meaning Hive installed). In $SPARK_HOME/conf, you create a soft link to
cd $SPARK_HOME/conf
hduser@rhes564: /usr/lib/spark-1.6.1-bin-hadoop2.6/conf> ltr hive-site.xml
lrwxrwxrwx 1 hduser hadoop 32 May 3 17:48 *hive-site.xml ->
/usr/lib/hive/conf/hive-site.xml*
Now in hive-site.xml you can define all the parameters needed for Spark
connectivity. Remember we are making Hive use spark1.3.1 engine. WE ARE
NOT RUNNING SPARK 1.3.1 AS A QUERY TOOL. We do not need to start master or
workers for Spark 1.3.1! It is just an execution engine like mr etc.
Let us look at how we do that in hive-site,xml. Noting the settings for
hive.execution.engine=spark and spark.home=/usr/lib/spark-1.3.1-bin-hadoop2
below. That tells Hive to use spark 1.3.1 as the execution engine. You just
install spark 1.3.1 on the host just the binary download it is
/usr/lib/spark-1.3.1-bin-hadoop2.6
In hive-site.xml, you set the properties.
<property>
<name>hive.execution.engine</name>
<value>spark</value>
<description>
Expects one of [mr, tez, spark].
Chooses execution engine. Options are: mr (Map reduce, default), tez,
spark. While MR
remains the default engine for historical reasons, it is itself a
historical engine
and is deprecated in Hive 2 line. It may be removed without further
warning.
</description>
</property>
<property>
<name>spark.home</name>
<value>/usr/lib/spark-1.3.1-bin-hadoop2</value>
<description>something</description>
</property>
<property>
<name>hive.merge.sparkfiles</name>
<value>false</value>
<description>Merge small files at the end of a Spark DAG
Transformation</description>
</property>
<property>
<name>hive.spark.client.future.timeout</name>
<value>60s</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec,
us/usec, ns/nsec), which is sec if not specified.
Timeout for requests from Hive client to remote Spark driver.
</description>
</property>
<property>
<name>hive.spark.job.monitor.timeout</name>
<value>60s</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec,
us/usec, ns/nsec), which is sec if not specified.
Timeout for job monitor to get Spark job state.
</description>
</property>
<property>
<name>hive.spark.client.connect.timeout</name>
<value>1000ms</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec,
us/usec, ns/nsec), which is msec if not specified.
Timeout for remote Spark driver in connecting back to Hive client.
</description>
</property>
<property>
<name>hive.spark.client.server.connect.timeout</name>
<value>90000ms</value>
<description>
Expects a time value with unit (d/day, h/hour, m/min, s/sec, ms/msec,
us/usec, ns/nsec), which is msec if not specified.
Timeout for handshake between Hive client and remote Spark driver.
Checked by both processes.
</description>
</property>
<property>
<name>hive.spark.client.secret.bits</name>
<value>256</value>
<description>Number of bits of randomness in the generated secret for
communication between Hive client and remote Spark driver. Rounded down to
the nearest multiple of 8.</description>
</property>
<property>
<name>hive.spark.client.rpc.threads</name>
<value>8</value>
<description>Maximum number of threads for remote Spark driver's RPC
event loop.</description>
</property>
And other settings as well
That was the Hive stuff for your Spark BAU. So there are two distinct
things. Now going to Hive itself, you will need to add the correct assembly
jar file for Hadoop. These are called
spark-assembly-x.y.z-hadoop2.4.0.jar
Where x.y.z in this case is 1.3.1
The assembly file is
spark-assembly-1.3.1-hadoop2.4.0.jar
So you add that spark-assembly-1.3.1-hadoop2.4.0.jar to $HIVE_HOME/libs
ls $HIVE_HOME/lib/spark-assembly-1.3.1-hadoop2.4.0.jar
/usr/lib/hive/lib/spark-assembly-1.3.1-hadoop2.4.0.jar
And you need to compile spark from source excluding Hadoop dependencies
./make-distribution.sh --name "hadoop2-without-hive" --tgz
"-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"
So Hive uses spark engine by default
If you want to use mr in hive you just do
0: jdbc:hive2://rhes564:10010/default>
*set hive.execution.engine=mr;*Hive-on-MR is deprecated in Hive 2 and may
not be available in the future versions. Consider using a different
execution engine (i.e. spark, tez) or using Hive 1.X releases.
No rows affected (0.007 seconds)
With regard to the second question
*2) How stable such construction is on INSERT / UPDATE / CTAS operations?
Any problems with writing into specific tables / directories, ORC / Parquet
peculiarities, memory / timeout parameters tuning ?*
With this set up that is Hive using spark as execution engine, my tests
look OK. Basically I can do whatever I do with Hive using map-reduceengine.
The caveat as usual is the amount of memory used by Spark for in-memory
work. I am afraid that resource constraint will be there no matter how you
want to deploy Spark
*3)) How stable such construction is in multi-user / multi-tenant
production environment when several people make different queries
simultaneously?*
This is subjective how you are going to deploy it and how scalable it is.
Your mileage varies and you really need to test it for yourself to find
out.
Also worth noting that with Spark app using Hive ORC tables you may have
issues with ORC tables defined ass transactional. You do not have that
issue with Hive on Spark engine. There are certainly limitations with
HiveSql construct. For example some properties are not implemented. Case in
point with Spark-sql
spark-sql> CREATE TEMPORARY TABLE tmp as select * from oraclehadoop.sales
limit 10;
Error in query: Unhandled clauses: TEMPORARY 1, 2,2, 7
.
You are likely trying to use an unsupported Hive feature.";
However, no issue with hive on spark engine
set hive.execution.engine=spark;
0: jdbc:hive2://rhes564:10010/default> CREATE TEMPORARY TABLE tmp as select
* from oraclehadoop.sales limit 10;
Starting Spark Job = d87e6c68-03f1-4c37-a9d4-f77e117039a4
Query Hive on Spark job[0] stages:
INFO : Completed executing
command(queryId=hduser_20160523090757_a474efb8-cea8-473e-8899-60bc7934a887);
Time taken: 43.894 seconds
INFO : OK
HTH
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 23 May 2016 at 05:57, Mohanraj Ragupathiraj <mohanaug...@gmail.com>
wrote:
> Great Comparison !! thanks
>
> On Mon, May 23, 2016 at 7:42 AM, Mich Talebzadeh <
> mich.talebza...@gmail.com> wrote:
>
>> Hi,
>>
>>
>>
>> I have done a number of extensive tests using Spark-shell with Hive DB
>> and ORC tables.
>>
>>
>>
>> Now one issue that we typically face is and I quote:
>>
>>
>>
>> Spark is fast as it uses Memory and DAG. Great but when we save data it
>> is not fast enough
>>
>> OK but there is a solution now. If you use Spark with Hive and you are on
>> a descent version of Hive >= 0.14, then you can also deploy Spark as
>> execution engine for Hive. That will make your application run pretty fast
>> as you no longer rely on the old Map-Reduce for Hive engine. In a nutshell
>> what you are gaining speed in both querying and storage.
>>
>>
>>
>> I have made some comparisons on this set-up and I am sure some of you
>> will find it useful.
>>
>>
>>
>> The version of Spark I use for Spark queries (Spark as query tool) is 1.6.
>>
>> The version of Hive I use in Hive 2
>>
>> The version of Spark I use as Hive execution engine is 1.3.1 It works and
>> frankly Spark 1.3.1 as an execution engine is adequate (until we sort out
>> the Hadoop libraries mismatch).
>>
>>
>>
>> An example I am using Hive on Spark engine to find the min and max of IDs
>> for a table with 1 billion rows:
>>
>>
>>
>> 0: jdbc:hive2://rhes564:10010/default> select min(id), max(id),avg(id),
>> stddev(id) from oraclehadoop.dummy;
>>
>> Query ID = hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006
>>
>>
>>
>>
>>
>> Starting Spark Job = 5e092ef9-d798-4952-b156-74df49da9151
>>
>>
>>
>> INFO : Completed compiling
>> command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006);
>> Time taken: 1.911 seconds
>>
>> INFO : Executing
>> command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006):
>> select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy
>>
>> INFO : Query ID =
>> hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006
>>
>> INFO : Total jobs = 1
>>
>> INFO : Launching Job 1 out of 1
>>
>> INFO : Starting task [Stage-1:MAPRED] in serial mode
>>
>>
>>
>> Query Hive on Spark job[0] stages:
>>
>> 0
>>
>> 1
>>
>> Status: Running (Hive on Spark job[0])
>>
>> Job Progress Format
>>
>> CurrentTime StageId_StageAttemptId:
>> SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
>> [StageCost]
>>
>> 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1
>>
>> 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1
>>
>> 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1
>>
>> 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22 Stage-1_0: 0/1
>>
>> INFO :
>>
>> Query Hive on Spark job[0] stages:
>>
>> INFO : 0
>>
>> INFO : 1
>>
>> INFO :
>>
>> Status: Running (Hive on Spark job[0])
>>
>> INFO : Job Progress Format
>>
>> CurrentTime StageId_StageAttemptId:
>> SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
>> [StageCost]
>>
>> INFO : 2016-05-23 00:21:19,062 Stage-0_0: 0/22 Stage-1_0: 0/1
>>
>> INFO : 2016-05-23 00:21:20,070 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1
>>
>> INFO : 2016-05-23 00:21:23,119 Stage-0_0: 0(+12)/22 Stage-1_0: 0/1
>>
>> INFO : 2016-05-23 00:21:26,156 Stage-0_0: 13(+9)/22 Stage-1_0: 0/1
>>
>> 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished Stage-1_0: 0(+1)/1
>>
>> 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished Stage-1_0: 1/1
>> Finished
>>
>> Status: Finished successfully in 53.25 seconds
>>
>> OK
>>
>> INFO : 2016-05-23 00:21:29,181 Stage-0_0: 22/22 Finished
>> Stage-1_0: 0(+1)/1
>>
>> INFO : 2016-05-23 00:21:30,189 Stage-0_0: 22/22 Finished
>> Stage-1_0: 1/1 Finished
>>
>> INFO : Status: Finished successfully in 53.25 seconds
>>
>> INFO : Completed executing
>> command(queryId=hduser_20160523002031_3e22e26e-4293-4e90-ae8b-72fe9683c006);
>> Time taken: 56.337 seconds
>>
>> INFO : OK
>>
>> +-----+------------+---------------+-----------------------+--+
>>
>> | c0 | c1 | c2 | c3 |
>>
>> +-----+------------+---------------+-----------------------+--+
>>
>> | 1 | 100000000 | 5.00000005E7 | 2.8867513459481288E7 |
>>
>> +-----+------------+---------------+-----------------------+--+
>>
>> 1 row selected (58.529 seconds)
>>
>>
>>
>> 58 seconds first run with cold cache is pretty good
>>
>>
>>
>> And let us compare it with running the same query on map-reduce engine
>>
>>
>>
>> : jdbc:hive2://rhes564:10010/default> set hive.execution.engine=mr;
>>
>> Hive-on-MR is deprecated in Hive 2 and may not be available in the future
>> versions. Consider using a different execution engine (i.e. spark, tez) or
>> using Hive 1.X releases.
>>
>> No rows affected (0.007 seconds)
>>
>> 0: jdbc:hive2://rhes564:10010/default> select min(id), max(id),avg(id),
>> stddev(id) from oraclehadoop.dummy;
>>
>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in
>> the future versions. Consider using a different execution engine (i.e.
>> spark, tez) or using Hive 1.X releases.
>>
>> Query ID = hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc
>>
>> Total jobs = 1
>>
>> Launching Job 1 out of 1
>>
>> Number of reduce tasks determined at compile time: 1
>>
>> In order to change the average load for a reducer (in bytes):
>>
>> set hive.exec.reducers.bytes.per.reducer=<number>
>>
>> In order to limit the maximum number of reducers:
>>
>> set hive.exec.reducers.max=<number>
>>
>> In order to set a constant number of reducers:
>>
>> set mapreduce.job.reduces=<number>
>>
>> Starting Job = job_1463956731753_0005, Tracking URL =
>> http://localhost.localdomain:8088/proxy/application_1463956731753_0005/
>>
>> Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job -kill
>> job_1463956731753_0005
>>
>> Hadoop job information for Stage-1: number of mappers: 22; number of
>> reducers: 1
>>
>> 2016-05-23 00:26:38,127 Stage-1 map = 0%, reduce = 0%
>>
>> INFO : Compiling
>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc):
>> select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy
>>
>> INFO : Semantic Analysis Completed
>>
>> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:c0,
>> type:int, comment:null), FieldSchema(name:c1, type:int, comment:null),
>> FieldSchema(name:c2, type:double, comment:null), FieldSchema(name:c3,
>> type:double, comment:null)], properties:null)
>>
>> INFO : Completed compiling
>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc);
>> Time taken: 0.144 seconds
>>
>> INFO : Executing
>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc):
>> select min(id), max(id),avg(id), stddev(id) from oraclehadoop.dummy
>>
>> WARN : Hive-on-MR is deprecated in Hive 2 and may not be available in
>> the future versions. Consider using a different execution engine (i.e.
>> spark, tez) or using Hive 1.X releases.
>>
>> INFO : WARNING: Hive-on-MR is deprecated in Hive 2 and may not be
>> available in the future versions. Consider using a different execution
>> engine (i.e. spark, tez) or using Hive 1.X releases.
>>
>> WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in
>> the future versions. Consider using a different execution engine (i.e.
>> spark, tez) or using Hive 1.X releases.
>>
>> INFO : Query ID =
>> hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc
>>
>> INFO : Total jobs = 1
>>
>> INFO : Launching Job 1 out of 1
>>
>> INFO : Starting task [Stage-1:MAPRED] in serial mode
>>
>> INFO : Number of reduce tasks determined at compile time: 1
>>
>> INFO : In order to change the average load for a reducer (in bytes):
>>
>> INFO : set hive.exec.reducers.bytes.per.reducer=<number>
>>
>> INFO : In order to limit the maximum number of reducers:
>>
>> INFO : set hive.exec.reducers.max=<number>
>>
>> INFO : In order to set a constant number of reducers:
>>
>> INFO : set mapreduce.job.reduces=<number>
>>
>> WARN : Hadoop command-line option parsing not performed. Implement the
>> Tool interface and execute your application with ToolRunner to remedy this.
>>
>> INFO : number of splits:22
>>
>> INFO : Submitting tokens for job: job_1463956731753_0005
>>
>> INFO : The url to track the job:
>> http://localhost.localdomain:8088/proxy/application_1463956731753_0005/
>>
>> INFO : Starting Job = job_1463956731753_0005, Tracking URL =
>> http://localhost.localdomain:8088/proxy/application_1463956731753_0005/
>>
>> INFO : Kill Command = /home/hduser/hadoop-2.6.0/bin/hadoop job -kill
>> job_1463956731753_0005
>>
>> INFO : Hadoop job information for Stage-1: number of mappers: 22; number
>> of reducers: 1
>>
>> INFO : 2016-05-23 00:26:38,127 Stage-1 map = 0%, reduce = 0%
>>
>> 2016-05-23 00:26:44,367 Stage-1 map = 5%, reduce = 0%, Cumulative CPU
>> 4.56 sec
>>
>> INFO : 2016-05-23 00:26:44,367 Stage-1 map = 5%, reduce = 0%,
>> Cumulative CPU 4.56 sec
>>
>> 2016-05-23 00:26:50,558 Stage-1 map = 9%, reduce = 0%, Cumulative CPU
>> 9.17 sec
>>
>> INFO : 2016-05-23 00:26:50,558 Stage-1 map = 9%, reduce = 0%,
>> Cumulative CPU 9.17 sec
>>
>> 2016-05-23 00:26:56,747 Stage-1 map = 14%, reduce = 0%, Cumulative CPU
>> 14.04 sec
>>
>> INFO : 2016-05-23 00:26:56,747 Stage-1 map = 14%, reduce = 0%,
>> Cumulative CPU 14.04 sec
>>
>> 2016-05-23 00:27:02,944 Stage-1 map = 18%, reduce = 0%, Cumulative CPU
>> 18.64 sec
>>
>> INFO : 2016-05-23 00:27:02,944 Stage-1 map = 18%, reduce = 0%,
>> Cumulative CPU 18.64 sec
>>
>> 2016-05-23 00:27:08,105 Stage-1 map = 23%, reduce = 0%, Cumulative CPU
>> 23.25 sec
>>
>> INFO : 2016-05-23 00:27:08,105 Stage-1 map = 23%, reduce = 0%,
>> Cumulative CPU 23.25 sec
>>
>> 2016-05-23 00:27:14,298 Stage-1 map = 27%, reduce = 0%, Cumulative CPU
>> 27.84 sec
>>
>> INFO : 2016-05-23 00:27:14,298 Stage-1 map = 27%, reduce = 0%,
>> Cumulative CPU 27.84 sec
>>
>> 2016-05-23 00:27:20,484 Stage-1 map = 32%, reduce = 0%, Cumulative CPU
>> 32.56 sec
>>
>> INFO : 2016-05-23 00:27:20,484 Stage-1 map = 32%, reduce = 0%,
>> Cumulative CPU 32.56 sec
>>
>> 2016-05-23 00:27:26,659 Stage-1 map = 36%, reduce = 0%, Cumulative CPU
>> 37.1 sec
>>
>> INFO : 2016-05-23 00:27:26,659 Stage-1 map = 36%, reduce = 0%,
>> Cumulative CPU 37.1 sec
>>
>> 2016-05-23 00:27:32,839 Stage-1 map = 41%, reduce = 0%, Cumulative CPU
>> 41.74 sec
>>
>> INFO : 2016-05-23 00:27:32,839 Stage-1 map = 41%, reduce = 0%,
>> Cumulative CPU 41.74 sec
>>
>> 2016-05-23 00:27:39,003 Stage-1 map = 45%, reduce = 0%, Cumulative CPU
>> 46.32 sec
>>
>> INFO : 2016-05-23 00:27:39,003 Stage-1 map = 45%, reduce = 0%,
>> Cumulative CPU 46.32 sec
>>
>> 2016-05-23 00:27:45,173 Stage-1 map = 50%, reduce = 0%, Cumulative CPU
>> 50.93 sec
>>
>> 2016-05-23 00:27:50,316 Stage-1 map = 55%, reduce = 0%, Cumulative CPU
>> 55.55 sec
>>
>> INFO : 2016-05-23 00:27:45,173 Stage-1 map = 50%, reduce = 0%,
>> Cumulative CPU 50.93 sec
>>
>> INFO : 2016-05-23 00:27:50,316 Stage-1 map = 55%, reduce = 0%,
>> Cumulative CPU 55.55 sec
>>
>> 2016-05-23 00:27:56,482 Stage-1 map = 59%, reduce = 0%, Cumulative CPU
>> 60.25 sec
>>
>> INFO : 2016-05-23 00:27:56,482 Stage-1 map = 59%, reduce = 0%,
>> Cumulative CPU 60.25 sec
>>
>> 2016-05-23 00:28:02,642 Stage-1 map = 64%, reduce = 0%, Cumulative CPU
>> 64.86 sec
>>
>> INFO : 2016-05-23 00:28:02,642 Stage-1 map = 64%, reduce = 0%,
>> Cumulative CPU 64.86 sec
>>
>> 2016-05-23 00:28:08,814 Stage-1 map = 68%, reduce = 0%, Cumulative CPU
>> 69.41 sec
>>
>> INFO : 2016-05-23 00:28:08,814 Stage-1 map = 68%, reduce = 0%,
>> Cumulative CPU 69.41 sec
>>
>> 2016-05-23 00:28:14,977 Stage-1 map = 73%, reduce = 0%, Cumulative CPU
>> 74.06 sec
>>
>> INFO : 2016-05-23 00:28:14,977 Stage-1 map = 73%, reduce = 0%,
>> Cumulative CPU 74.06 sec
>>
>> 2016-05-23 00:28:21,134 Stage-1 map = 77%, reduce = 0%, Cumulative CPU
>> 78.72 sec
>>
>> INFO : 2016-05-23 00:28:21,134 Stage-1 map = 77%, reduce = 0%,
>> Cumulative CPU 78.72 sec
>>
>> 2016-05-23 00:28:27,282 Stage-1 map = 82%, reduce = 0%, Cumulative CPU
>> 83.32 sec
>>
>> INFO : 2016-05-23 00:28:27,282 Stage-1 map = 82%, reduce = 0%,
>> Cumulative CPU 83.32 sec
>>
>> 2016-05-23 00:28:33,437 Stage-1 map = 86%, reduce = 0%, Cumulative CPU
>> 87.9 sec
>>
>> INFO : 2016-05-23 00:28:33,437 Stage-1 map = 86%, reduce = 0%,
>> Cumulative CPU 87.9 sec
>>
>> 2016-05-23 00:28:38,579 Stage-1 map = 91%, reduce = 0%, Cumulative CPU
>> 92.52 sec
>>
>> INFO : 2016-05-23 00:28:38,579 Stage-1 map = 91%, reduce = 0%,
>> Cumulative CPU 92.52 sec
>>
>> 2016-05-23 00:28:44,759 Stage-1 map = 95%, reduce = 0%, Cumulative CPU
>> 97.35 sec
>>
>> INFO : 2016-05-23 00:28:44,759 Stage-1 map = 95%, reduce = 0%,
>> Cumulative CPU 97.35 sec
>>
>> 2016-05-23 00:28:49,915 Stage-1 map = 100%, reduce = 0%, Cumulative CPU
>> 99.6 sec
>>
>> INFO : 2016-05-23 00:28:49,915 Stage-1 map = 100%, reduce = 0%,
>> Cumulative CPU 99.6 sec
>>
>> 2016-05-23 00:28:54,043 Stage-1 map = 100%, reduce = 100%, Cumulative
>> CPU 101.4 sec
>>
>> MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 msec
>>
>> Ended Job = job_1463956731753_0005
>>
>> MapReduce Jobs Launched:
>>
>> Stage-Stage-1: Map: 22 Reduce: 1 Cumulative CPU: 101.4 sec HDFS
>> Read: 5318569 HDFS Write: 46 SUCCESS
>>
>> Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec
>>
>> OK
>>
>> INFO : 2016-05-23 00:28:54,043 Stage-1 map = 100%, reduce = 100%,
>> Cumulative CPU 101.4 sec
>>
>> INFO : MapReduce Total cumulative CPU time: 1 minutes 41 seconds 400 msec
>>
>> INFO : Ended Job = job_1463956731753_0005
>>
>> INFO : MapReduce Jobs Launched:
>>
>> INFO : Stage-Stage-1: Map: 22 Reduce: 1 Cumulative CPU: 101.4 sec
>> HDFS Read: 5318569 HDFS Write: 46 SUCCESS
>>
>> INFO : Total MapReduce CPU Time Spent: 1 minutes 41 seconds 400 msec
>>
>> INFO : Completed executing
>> command(queryId=hduser_20160523002632_9f91d42a-ea46-4a66-a589-7d39c23b41dc);
>> Time taken: 142.525 seconds
>>
>> INFO : OK
>>
>> +-----+------------+---------------+-----------------------+--+
>>
>> | c0 | c1 | c2 | c3 |
>>
>> +-----+------------+---------------+-----------------------+--+
>>
>> | 1 | 100000000 | 5.00000005E7 | 2.8867513459481288E7 |
>>
>> +-----+------------+---------------+-----------------------+--+
>>
>> 1 row selected (142.744 seconds)
>>
>>
>>
>> OK Hive on map-reduce engine took 142 seconds compared to 58 seconds with
>> Hive on Spark. So you can obviously gain pretty well by using Hive on Spark.
>>
>>
>>
>> Please also note that I did not use any vendor's build for this purpose.
>> I compiled Spark 1.3.1 myself.
>>
>>
>>
>> HTH
>>
>>
>>
>>
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>>
>>
>>
>> http://talebzadehmich.wordpress.com/
>>
>>
>>
>
>
>
> --
> Thanks and Regards
> Mohan
> VISA Pte Limited, Singapore.
>
