I tried all the approaches.
1.Partitioned by year,month,day on hive table with parquet format when
table is created in impala.
2. Dataset from hive is not partitioned. used insert overwrite
hivePartitonedTable partition(year,month,day) select * from
tempViewOFDataset . Also tried
Dataset.write.mode(overwrite).insertInto(hivePartitonedTable )
3. Tried approach of repartitioning dataset before inserting into hive
table as below.
unionedDS.repartition(unionedDS("year"),unionedDS("month"),unionedDS("day"))
None of the approaches helped me with performance.
On Sun, Aug 20, 2017 at 1:35 PM, ayan guha <[email protected]> wrote:
> Just curious - is your dataset partitioned on your partition columns?
>
> On Mon, 21 Aug 2017 at 3:54 am, KhajaAsmath Mohammed <
> [email protected]> wrote:
>
>> We are in cloudera CDH5.10 and we are using spark 2 that comes with
>> cloudera.
>>
>> Coming to second solution, creating a temporary view on dataframe but it
>> didnt improve my performance too.
>>
>> I do remember performance was very fast when doing whole overwrite table
>> without partitons but the problem started after using partitions.
>>
>> On Sun, Aug 20, 2017 at 12:46 PM, Jörn Franke <[email protected]>
>> wrote:
>>
>>> Ah i see then I would check also directly in Hive if you have issues to
>>> insert data in the Hive table. Alternatively you can try to register
>>> the df as temptable and do a insert into the Hive table from the temptable
>>> using Spark sql ("insert into table hivetable select * from temptable")
>>>
>>>
>>> You seem to use Cloudera so you probably have a very outdated Hive
>>> version. So you could switch to a distribution having a recent version of
>>> Hive 2 with Tez+llap - these are much more performant with much more
>>> features.
>>>
>>> Alternatively you can try to register the df as temptable and do a
>>> insert into the Hive table from the temptable using Spark sql ("insert into
>>> table hivetable select * from temptable")
>>>
>>> On 20. Aug 2017, at 18:47, KhajaAsmath Mohammed <[email protected]>
>>> wrote:
>>>
>>> Hi,
>>>
>>> I have created hive table in impala first with storage format as
>>> parquet. With dataframe from spark I am tryinig to insert into the same
>>> table with below syntax.
>>>
>>> Table is partitoned by year,month,day
>>> ds.write.mode(SaveMode.Overwrite).insertInto("db.parqut_table")
>>>
>>> https://issues.apache.org/jira/browse/SPARK-20049
>>>
>>> I saw something in the above link not sure if that is same thing in my
>>> case.
>>>
>>> Thanks,
>>> Asmath
>>>
>>> On Sun, Aug 20, 2017 at 11:42 AM, Jörn Franke <[email protected]>
>>> wrote:
>>>
>>>> Have you made sure that the saveastable stores them as parquet?
>>>>
>>>> On 20. Aug 2017, at 18:07, KhajaAsmath Mohammed <
>>>> [email protected]> wrote:
>>>>
>>>> we are using parquet tables, is it causing any performance issue?
>>>>
>>>> On Sun, Aug 20, 2017 at 9:09 AM, Jörn Franke <[email protected]>
>>>> wrote:
>>>>
>>>>> Improving the performance of Hive can be also done by switching to
>>>>> Tez+llap as an engine.
>>>>> Aside from this : you need to check what is the default format that it
>>>>> writes to Hive. One issue for the slow storing into a hive table could be
>>>>> that it writes by default to csv/gzip or csv/bzip2
>>>>>
>>>>> > On 20. Aug 2017, at 15:52, KhajaAsmath Mohammed <
>>>>> [email protected]> wrote:
>>>>> >
>>>>> > Yes we tried hive and want to migrate to spark for better
>>>>> performance. I am using paraquet tables . Still no better performance
>>>>> while
>>>>> loading.
>>>>> >
>>>>> > Sent from my iPhone
>>>>> >
>>>>> >> On Aug 20, 2017, at 2:24 AM, Jörn Franke <[email protected]>
>>>>> wrote:
>>>>> >>
>>>>> >> Have you tried directly in Hive how the performance is?
>>>>> >>
>>>>> >> In which Format do you expect Hive to write? Have you made sure it
>>>>> is in this format? It could be that you use an inefficient format (e.g.
>>>>> CSV
>>>>> + bzip2).
>>>>> >>
>>>>> >>> On 20. Aug 2017, at 03:18, KhajaAsmath Mohammed <
>>>>> [email protected]> wrote:
>>>>> >>>
>>>>> >>> Hi,
>>>>> >>>
>>>>> >>> I have written spark sql job on spark2.0 by using scala . It is
>>>>> just pulling the data from hive table and add extra columns , remove
>>>>> duplicates and then write it back to hive again.
>>>>> >>>
>>>>> >>> In spark ui, it is taking almost 40 minutes to write 400 go of
>>>>> data. Is there anything that I need to improve performance .
>>>>> >>>
>>>>> >>> Spark.sql.partitions is 2000 in my case with executor memory of
>>>>> 16gb and dynamic allocation enabled.
>>>>> >>>
>>>>> >>> I am doing insert overwrite on partition by
>>>>> >>> Da.write.mode(overwrite).insertinto(table)
>>>>> >>>
>>>>> >>> Any suggestions please ??
>>>>> >>>
>>>>> >>> Sent from my iPhone
>>>>> >>> ------------------------------------------------------------
>>>>> ---------
>>>>> >>> To unsubscribe e-mail: [email protected]
>>>>> >>>
>>>>>
>>>>
>>>>
>>>
>> --
> Best Regards,
> Ayan Guha
>
