[jira] [Updated] (HIVE-28087) Hive Iceberg: Insert into partitioned table fails if the data is not clustered
[
https://issues.apache.org/jira/browse/HIVE-28087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simhadri Govindappa updated HIVE-28087:
---
Description:
Insert into partitioned table fails with the following error if the data is not
clustered.
*Using cluster by clause it succeeds :*
{noformat}
0: jdbc:hive2://localhost:10001/> insert into table partition_transform_4
select t, ts from t1 cluster by ts;
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. container SUCCEEDED 1 100
0 0
Reducer 2 .. container SUCCEEDED 1 100
0 0
--
VERTICES: 02/02 [==>>] 100% ELAPSED TIME: 9.47 s
--
INFO : Starting task [Stage-2:DEPENDENCY_COLLECTION] in serial mode
INFO : Starting task [Stage-0:MOVE] in serial mode
INFO : Completed executing
command(queryId=root_20240222123244_0c448b32-4fd9-420d-be31-e39e2972af82);
Time taken: 10.534 seconds
100 rows affected (10.696 seconds){noformat}
*Without cluster By it fails:*
{noformat}
0: jdbc:hive2://localhost:10001/> insert into table partition_transform_4
select t, ts from t1;
--
VERTICES MODESTATUS TOTAL COMPLETED RUNNING PENDING
FAILED KILLED
--
Map 1 .. container SUCCEEDED 1 100
0 0
Reducer 2container RUNNING 1 010
2 0
--
VERTICES: 01/02 [=>>-] 50% ELAPSED TIME: 9.53 s
--
Caused by: java.lang.IllegalStateException: Incoming records violate the writer
assumption that records are clustered by spec and by partition within each
spec. Either cluster the incoming records or switch to fanout writers.
Encountered records that belong to already closed files:
partition 'ts_month=2027-03' in spec [
1000: ts_month: month(2)
]
at org.apache.iceberg.io.ClusteredWriter.write(ClusteredWriter.java:96)
at
org.apache.iceberg.io.ClusteredDataWriter.write(ClusteredDataWriter.java:31)
at
org.apache.iceberg.mr.hive.writer.HiveIcebergRecordWriter.write(HiveIcebergRecordWriter.java:53)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1181)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
at
org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:919)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:158)
at
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.processVectorGroup(ReduceRecordSource.java:502)
... 20 more{noformat}
A simple repro, using the attached csv file:
[^query-hive-377.csv]
{noformat}
create database t3;
use t3;
create table vector1k(
t int,
si int,
i int,
b bigint,
f float,
d double,
dc decimal(38,18),
bo boolean,
s string,
s2 string,
ts timestamp,
ts2 timestamp,
dt date)
row format delimited fields terminated by ',';
load data local inpath "/query-hive-377.csv" OVERWRITE into table vector1k;
select * from vector1k; create table vectortab10k(
t int,
si int,
i int,
b bigint,
f float,
d double,
dc decimal(38,18),
bo boolean,
s string,
s2 string,
ts timestamp,
ts2 timestamp,
dt date)
stored by iceberg
stored as orc;
insert into vectortab10k select * from vector1k;
select count(*) from vectortab10k ;
create table partition_transform_4(t int, ts timestamp) partitioned by
spec(month(ts)) stored by iceberg;
insert into table partition_transform_4 select t, ts from vectortab10k ;
{noformat}
was:
Insert into partitioned table fails with the following error if the data is not
clustered.
*Using cluster by clause it succeeds :*
{noformat}
0: jdbc:hive2://localhost:10001/> insert into table partition_transform_4
select t, ts from t1 cluster by ts;
[jira] [Comment Edited] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826830#comment-17826830
]
Peter Vary edited comment on HIVE-26882 at 3/13/24 6:52 PM:
[~lirui]: You could try this:
{code}
query.executeUpdate()
{code}
https://github.com/apache/hive/blob/4b01a607091581ac9bdb372f8b47c1efca4d4bb4/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java#L587
was (Author: pvary):
[~lirui]: You could try this:
```
query.executeUpdate()
```
https://github.com/apache/hive/blob/4b01a607091581ac9bdb372f8b47c1efca4d4bb4/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java#L587
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.10, 4.0.0-beta-1
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826830#comment-17826830 ] Peter Vary commented on HIVE-26882: --- [~lirui]: You could try this: ``` query.executeUpdate() ``` https://github.com/apache/hive/blob/4b01a607091581ac9bdb372f8b47c1efca4d4bb4/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java#L587 > Allow transactional check of Table parameter before altering the Table > -- > > Key: HIVE-26882 > URL: https://issues.apache.org/jira/browse/HIVE-26882 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 2.3.10, 4.0.0-beta-1 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > We should add the possibility to transactionally check if a Table parameter > is changed before altering the table in the HMS. > This would provide an alternative, less error-prone and faster way to commit > an Iceberg table, as the Iceberg table currently needs to: > - Create an exclusive lock > - Get the table metadata to check if the current snapshot is not changed > - Update the table metadata > - Release the lock > After the change these 4 HMS calls could be substituted with a single alter > table call. > Also we could avoid cases where the locks are left hanging by failed processes -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work started] (HIVE-28118) Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
[
https://issues.apache.org/jira/browse/HIVE-28118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-28118 started by Palakur Eshwitha Sai.
---
> Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
> -
>
> Key: HIVE-28118
> URL: https://issues.apache.org/jira/browse/HIVE-28118
> Project: Hive
> Issue Type: Bug
>Reporter: Palakur Eshwitha Sai
>Assignee: Palakur Eshwitha Sai
>Priority: Major
>
> With Viewfs overload scheme enabled in the cluster and mount point configured
> to S3, Hive insert into operation fails with MoveTask error.
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
> to destination
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> (state=08S01,code=1)
> Caused by: java.io.FileNotFoundException: No such file or directory:
> s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
> ~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28118) Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
[
https://issues.apache.org/jira/browse/HIVE-28118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Palakur Eshwitha Sai updated HIVE-28118:
Description:
With Viewfs overload scheme enabled in the cluster and mount point configured
to S3, Hive insert into operation fails with MoveTask error.
{code:java}
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
to destination
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
(state=08S01,code=1)
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]
at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]{code}
was:
With Viewfs overload scheme enabled in the cluster and mount point configured
to S3, Hive insert into operation fails with MoveTask error.
{code:java}
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
to destination
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
(state=08S01,code=1)
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]{code}
> Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
> -
>
> Key: HIVE-28118
> URL: https://issues.apache.org/jira/browse/HIVE-28118
> Project: Hive
> Issue Type: Bug
>Reporter: Palakur Eshwitha Sai
>Assignee: Palakur Eshwitha Sai
>Priority: Major
>
> With Viewfs overload scheme enabled in the cluster and mount point configured
> to S3, Hive insert into operation fails with MoveTask error.
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
> to destination
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> (state=08S01,code=1)
> Caused by: java.io.FileNotFoundException: No such file or directory:
> s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
>
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]
> at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
> ~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28118) Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
[
https://issues.apache.org/jira/browse/HIVE-28118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Palakur Eshwitha Sai updated HIVE-28118:
Description:
With Viewfs overload scheme enabled in the cluster and mount point configured
to S3, Hive insert into operation fails with MoveTask error.
{code:java}
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
to destination
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
(state=08S01,code=1)
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]{code}
was:
With Viewfs overload scheme enabled in the cluster and mount point configured
to S3, Hive insert into operation fails with MoveTask error.
{code:java}
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
to destination
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
(state=08S01,code=1)
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]
{code}
> Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
> -
>
> Key: HIVE-28118
> URL: https://issues.apache.org/jira/browse/HIVE-28118
> Project: Hive
> Issue Type: Bug
>Reporter: Palakur Eshwitha Sai
>Assignee: Palakur Eshwitha Sai
>Priority: Major
>
> With Viewfs overload scheme enabled in the cluster and mount point configured
> to S3, Hive insert into operation fails with MoveTask error.
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
> to destination
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> (state=08S01,code=1)
> Caused by: java.io.FileNotFoundException: No such file or directory:
> s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
> ~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]{code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28118) Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
[
https://issues.apache.org/jira/browse/HIVE-28118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Palakur Eshwitha Sai updated HIVE-28118:
Description:
With Viewfs overload scheme enabled in the cluster and mount point configured
to S3, Hive insert into operation fails with MoveTask error.
{code:java}
Error: Error while processing statement: FAILED: Execution Error, return code 1
from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
to destination
hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
(state=08S01,code=1)
Caused by: java.io.FileNotFoundException: No such file or directory:
s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
at
org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]
{code}
was:With Viewfs overload scheme enabled in the cluster and mount point
configured to S3, Hive insert into operation fails with MoveTask error.
> Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
> -
>
> Key: HIVE-28118
> URL: https://issues.apache.org/jira/browse/HIVE-28118
> Project: Hive
> Issue Type: Bug
>Reporter: Palakur Eshwitha Sai
>Assignee: Palakur Eshwitha Sai
>Priority: Major
>
> With Viewfs overload scheme enabled in the cluster and mount point configured
> to S3, Hive insert into operation fails with MoveTask error.
> {code:java}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask. Unable to move source
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-10002
> to destination
> hdfs://s3cluster/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> (state=08S01,code=1)
> Caused by: java.io.FileNotFoundException: No such file or directory:
> s3a://testhadoop/warehouse/tablespace/external/hive/sales_by_state/state=CA/.hive-staging_hive_2024-01-05_05-52-09_785_7084847051463431810-8/-ext-1
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:2344)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:2226)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2160)
> ~[hadoop-aws-3.2.2.3.2.2.4-8.jar:?]at
> org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:888)
> ~[hadoop-common-3.2.2.3.2.2.4-6.jar:?]
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28118) Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
[ https://issues.apache.org/jira/browse/HIVE-28118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Palakur Eshwitha Sai updated HIVE-28118: Description: With Viewfs overload scheme enabled in the cluster and mount point configured to S3, Hive insert into operation fails with MoveTask error. > Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error > - > > Key: HIVE-28118 > URL: https://issues.apache.org/jira/browse/HIVE-28118 > Project: Hive > Issue Type: Bug >Reporter: Palakur Eshwitha Sai >Assignee: Palakur Eshwitha Sai >Priority: Major > > With Viewfs overload scheme enabled in the cluster and mount point configured > to S3, Hive insert into operation fails with MoveTask error. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28118) Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error
Palakur Eshwitha Sai created HIVE-28118: --- Summary: Hive Insert Into S3 with Viewfs overload scheme fails with MoveTask Error Key: HIVE-28118 URL: https://issues.apache.org/jira/browse/HIVE-28118 Project: Hive Issue Type: Bug Reporter: Palakur Eshwitha Sai Assignee: Palakur Eshwitha Sai -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table
[
https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826001#comment-17826001
]
Rui Li commented on HIVE-26882:
---
Hi [~pvary], I tried writing direct SQL with JDO:
{code:Java}
String dml = "update ...";
openTransaction();
query = pm.newQuery("javax.jdo.query.SQL", dml);
long numUpdated = (long) query.execute();
...
commitTransaction();
{code}
But I got an error:
{noformat}
javax.jdo.JDOUserException: JDOQL Single-String query should always start with
SELECT
{noformat}
So it seems JDO only allows direct SELECT statements? I also tried appending a
SELECT before the UPDATE, but then I got another error indicating it doesn't
support multiple statements in the query string. Please let me know if I'm not
using the correct APIs.
> Allow transactional check of Table parameter before altering the Table
> --
>
> Key: HIVE-26882
> URL: https://issues.apache.org/jira/browse/HIVE-26882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.10, 4.0.0-beta-1
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> We should add the possibility to transactionally check if a Table parameter
> is changed before altering the table in the HMS.
> This would provide an alternative, less error-prone and faster way to commit
> an Iceberg table, as the Iceberg table currently needs to:
> - Create an exclusive lock
> - Get the table metadata to check if the current snapshot is not changed
> - Update the table metadata
> - Release the lock
> After the change these 4 HMS calls could be substituted with a single alter
> table call.
> Also we could avoid cases where the locks are left hanging by failed processes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-28116) I'm curious about the "hive.metastore.disallow.incompatible.col.type.changes" settings.
[ https://issues.apache.org/jira/browse/HIVE-28116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-28116. Fix Version/s: Not Applicable Resolution: Invalid > I'm curious about the "hive.metastore.disallow.incompatible.col.type.changes" > settings. > --- > > Key: HIVE-28116 > URL: https://issues.apache.org/jira/browse/HIVE-28116 > Project: Hive > Issue Type: Task >Reporter: Juhyun Seo >Priority: Major > Fix For: Not Applicable > > > Hello. > I'm leaving an inquiry because I want to understand the > "hive.metastore.disallow.incompatible.col.type.changes" setting in detail. > When I read the > [document|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.disallow.incompatible.col.type.changes] > it was found to be related to the possibility of converting the data type of > the column and compatibility. However, if you use `ALTER TABLE > iceberg.iceberg_db.iceberg_table DROP COLUMN 'name'`, the same error occurs. > If you delete a column here, the data related to that column will be removed > from the table, which I think is separate from the compatibility issue of > data type, and I wonder what it has to do with this setting. > can I check the code related to this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28116) I'm curious about the "hive.metastore.disallow.incompatible.col.type.changes" settings.
[ https://issues.apache.org/jira/browse/HIVE-28116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825967#comment-17825967 ] Stamatis Zampetakis commented on HIVE-28116: Hey, [~wngus606] for questions you should use the appropriate mailing lists: [https://hive.apache.org/community/mailinglists/] > I'm curious about the "hive.metastore.disallow.incompatible.col.type.changes" > settings. > --- > > Key: HIVE-28116 > URL: https://issues.apache.org/jira/browse/HIVE-28116 > Project: Hive > Issue Type: Task >Reporter: Juhyun Seo >Priority: Major > > Hello. > I'm leaving an inquiry because I want to understand the > "hive.metastore.disallow.incompatible.col.type.changes" setting in detail. > When I read the > [document|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.disallow.incompatible.col.type.changes] > it was found to be related to the possibility of converting the data type of > the column and compatibility. However, if you use `ALTER TABLE > iceberg.iceberg_db.iceberg_table DROP COLUMN 'name'`, the same error occurs. > If you delete a column here, the data related to that column will be removed > from the table, which I think is separate from the compatibility issue of > data type, and I wonder what it has to do with this setting. > can I check the code related to this? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HIVE-27897) Backport of HIVE-22373, HIVE-25553, HIVE-23561, HIVE-24321, HIVE-22856, HIVE-22973, HIVE-21729, HIVE-23022, HIVE-23034, HIVE-22221, HIVE-23230 to branch-3
[ https://issues.apache.org/jira/browse/HIVE-27897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-27897. - Fix Version/s: 3.2.0 Resolution: Fixed > Backport of HIVE-22373, HIVE-25553, HIVE-23561, HIVE-24321, HIVE-22856, > HIVE-22973, HIVE-21729, HIVE-23022, HIVE-23034, HIVE-1, HIVE-23230 to > branch-3 > -- > > Key: HIVE-27897 > URL: https://issues.apache.org/jira/browse/HIVE-27897 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.2.0 >Reporter: Aman Raj >Assignee: Aman Raj >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
