[
https://issues.apache.org/jira/browse/IMPALA-13244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17867242#comment-17867242
]
Gabor Kaszab commented on IMPALA-13244:
---------------------------------------
Ok, so I guess the issue is with the order of the partition cols. So you
defined the table cols this order: 'xxx', 'code', 'updatetime'. While the
partition cols are defined in the opposite order. So when you insert with
providing also the table structure then there the columns aren't in the order
how the actual order of the cols in the table are (in the order of how you
defined the partition cols).
I'm not sure this is a real issue here.
> Timestamp partition error in catalogd when insert data into iceberg table
> --------------------------------------------------------------------------
>
> Key: IMPALA-13244
> URL: https://issues.apache.org/jira/browse/IMPALA-13244
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.4.0
> Environment: centos7.9
> Reporter: Pain Sun
> Priority: Major
>
> create table sql like this:
>
> CREATE TABLE test111.table1 (
> xxx STRING,
> code STRING,
> updatetime TIMESTAMP
> ) PARTITIONED BY spec(
> month(updatetime),
> bucket(10, code),
> bucket(10, xxx)
> ) STORED AS ICEBERG TBLPROPERTIES(
> 'iceberg.catalog' = 'hadoop.catalog',
> 'iceberg.catalog_location' = '/impalatable',
> 'iceberg.table_identifier' = 'middle.table1',
> 'write.metadata.previous-versions-max' = '3',
> 'write.metadata.delete-after-commit.enabled' = 'true',
> 'commit.manifest.min-count-to-merge' = '3',
> 'commit.manifest-merge.enabled' = 'true',
> 'format-version' = '1'
> );
>
>
>
> then insert data into this table like this:
> insert into
> test111.table1 (
> xxx,
> code,
> updatetime
> )
> select
> 'm1' as xxx,
> 'c1' as code,
> '2024-07-17 13:44:01' as updatetime;
> Catalogd error like this :
> E0719 09:50:57.458815 126128 JniUtil.java:183]
> 964d388b63170b6b:7c6e06c200000000] Error in Update catalog for
> test111.table1. Time spent: 6ms
> I0719 09:50:57.459015 126128 jni-util.cc:302]
> 964d388b63170b6b:7c6e06c200000000] java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at
> org.apache.impala.util.IcebergUtil.parseMonthToTransformMonth(IcebergUtil.java:882)
> at
> org.apache.impala.util.IcebergUtil.getPartitionValue(IcebergUtil.java:826)
> at
> org.apache.impala.util.IcebergUtil.partitionDataFromDataFile(IcebergUtil.java:800)
> at
> org.apache.impala.service.IcebergCatalogOpExecutor.createDataFile(IcebergCatalogOpExecutor.java:445)
> at
> org.apache.impala.service.IcebergCatalogOpExecutor.appendFiles(IcebergCatalogOpExecutor.java:487)
> at
> org.apache.impala.service.IcebergCatalogOpExecutor.execute(IcebergCatalogOpExecutor.java:366)
> at
> org.apache.impala.service.CatalogOpExecutor.updateCatalogImpl(CatalogOpExecutor.java:7443)
> at
> org.apache.impala.service.CatalogOpExecutor.updateCatalog(CatalogOpExecutor.java:7180)
> at
> org.apache.impala.service.JniCatalog.lambda$updateCatalog$15(JniCatalog.java:504)
> at
> org.apache.impala.service.JniCatalogOp.lambda$execAndSerialize$1(JniCatalogOp.java:90)
> at org.apache.impala.service.JniCatalogOp.execOp(JniCatalogOp.java:58)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:89)
> at
> org.apache.impala.service.JniCatalogOp.execAndSerialize(JniCatalogOp.java:100)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:245)
> at
> org.apache.impala.service.JniCatalog.execAndSerialize(JniCatalog.java:259)
> at
> org.apache.impala.service.JniCatalog.updateCatalog(JniCatalog.java:503)
> I0719 09:50:57.459033 126128 status.cc:129]
> 964d388b63170b6b:7c6e06c200000000] IllegalStateException: null
> @ 0x10546b4
> @ 0x1b94d34
> @ 0x10040ab
> @ 0xfa1c27
> @ 0xf61f84
> @ 0xf4acc3
> @ 0xf5278b
> @ 0x14486aa
> @ 0x143b0fa
> @ 0x1c78d39
> @ 0x1c79fd1
> @ 0x256da47
> @ 0x7fabd2eb8ea5
> @ 0x7fabcfe939fd
> E0719 09:50:57.459059 126128 catalog-server.cc:324]
> 964d388b63170b6b:7c6e06c200000000] IllegalStateException: null
>
> but spark insert success.
>
> versions :
> impala: 4.4.0
> jar in impala: iceberg-api-1.3.1.7.2.18.0-369.jar
> spark: 3.3.4
> iceberg: apache 1.3.1
> iceberg-spark jar: iceberg-spark-runtime-3.3_2.12-1.3.1.jar
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
