Hi Marcin,
It is the speed really. The speed in which data is digested into Hive.
Sqoop is two stage as I understand.
1. Take the data out of RDMSD via JADB and put in on an external HDFS
file
2. Read that file and insert into a Hive table
The issue is the second part. In general I use Hive 2 with Spark 1.3.1
engine to put data into Hive table. I wondered if there was such a
parameter in Sqoop to use Spark engine.
Well I gather this is easier said that done. I am importing 1 billion rows
table from Oracle
sqoop import --connect "jdbc:oracle:thin:@rhes564:1521:mydb12" --username
scratchpad -P \
--query "select * from scratchpad.dummy where \
\$CONDITIONS" \
--split-by ID \
--hive-import --hive-table "oraclehadoop.dummy" --target-dir
"dummy"
Now the fact that in hive-site.xml I have set hive.execution.engine=spark
does not matter. Sqoop seems to internally set hive.execution.engine=mr
anyway.
May be there should be an option --hive-execution-engine='mr/tez/spak' etc
in above command?
Cheers,
Mich
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 30 April 2016 at 14:51, Marcin Tustin <[email protected]> wrote:
> They're not simply interchangeable. sqoop is written to use mapreduce.
>
> I actually implemented my own replacement for sqoop-export in spark, which
> was extremely simple. It wasn't any faster, because the bottleneck was the
> receiving database.
>
> Is your motivation here speed? Or correctness?
>
> On Sat, Apr 30, 2016 at 8:45 AM, Mich Talebzadeh <
> [email protected]> wrote:
>
>> Hi,
>>
>> What is the simplest way of making sqoop import use spark engine as
>> opposed to the default mapreduce when putting data into hive table. I did
>> not see any parameter for this in sqoop command line doc.
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>
> Want to work at Handy? Check out our culture deck and open roles
> <http://www.handy.com/careers>
> Latest news <http://www.handy.com/press> at Handy
> Handy just raised $50m
> <http://venturebeat.com/2015/11/02/on-demand-home-service-handy-raises-50m-in-round-led-by-fidelity/>
> led
> by Fidelity
>
>
