The query looks like this:
INSERT overwrite TABLE our_schema.our_table
SELECT wh.*
FROM our_schema.our_table wh
LEFT JOIN our_schema.our_table p
ON p.product_id = wh.product_id
WHERE p.is_deleted = FALSE OR p.is_deleted IS NULL
it is a non transactional table. could it be the reason a combination between
DbTransactionManager
and the fact that the query performs an overwrite of the table with itself ?
We found similar issues: https://issues.apache.org/jira/browse/HIVE-18922
or
issues.apache.org/jira/browse/HIVE-13850<https://issues.apache.org/jira/browse/HIVE-13850>
Thank you for any hints on how to address this issue.
Julien
Julien Tane
Big Data Engineer
[Tel.] +49 721 98993-393
[Fax] +49 721 98993-66
[E-Mail] [email protected]<mailto:[email protected]>
solute GmbH
Zeppelinstraße 15
76185 Karlsruhe
Germany
[Logo Solute]
Marken der solute GmbH | brands of solute GmbH
[Marken]
Geschäftsführer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten
Webseite | www.solute.de <http://www.solute.de/>
Sitz | Registered Office: Karlsruhe
Registergericht | Register Court: Amtsgericht Mannheim
Registernummer | Register No.: HRB 110579
USt-ID | VAT ID: DE234663798
Informationen zum Datenschutz | Information about privacy policy
https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php
________________________________
Von: Julien Tane <[email protected]>
Gesendet: Dienstag, 30. Juni 2020 17:39:45
An: [email protected]
Betreff: Error with hive-staging.staging
Hello,
we are having trouble with a query which exits infrequently:
org.apache.hadoop.hive.ql.exec.tez.MergeFileRecordProcessor.close(MergeFileRecordProcessor.java:180)\n\t...
17 more\nCaused by: java.io.IOException:
hdfs://ourserver:8020/apps/hive/warehouse/our_schema.db/our_table/.hive-staging_hive_2020-06-30_14-58-28_289_2174922658316991535-9493/_task_tmp.-ext-10000/_tmp.000002_0
to
hdfs://ourserver:8020/apps/hive/warehouse/our_schema.db/our_table/.hive-staging_hive_2020-06-30_14-58-28_289_2174922658316991535-9493/_tmp.-ext-10000/000002_0
We are not completely sure how to solve this issue... or even why it happens.
Julien Tane
Big Data Engineer
[Tel.] +49 721 98993-393
[Fax] +49 721 98993-66
[E-Mail] [email protected]<mailto:[email protected]>
solute GmbH
Zeppelinstraße 15
76185 Karlsruhe
Germany
[Logo Solute]
Marken der solute GmbH | brands of solute GmbH
[Marken]
Geschäftsführer | Managing Director: Dr. Thilo Gans, Bernd Vermaaten
Webseite | www.solute.de <http://www.solute.de/>
Sitz | Registered Office: Karlsruhe
Registergericht | Register Court: Amtsgericht Mannheim
Registernummer | Register No.: HRB 110579
USt-ID | VAT ID: DE234663798
Informationen zum Datenschutz | Information about privacy policy
https://www.solute.de/ger/datenschutz/grundsaetze-der-datenverarbeitung.php
