soumilshah1995 opened a new issue, #362:
URL: https://github.com/apache/incubator-xtable/issues/362
Hello im trying to translate Hudi metadata into ICEBERG
I was able to do Hudi to delta
```
sourceFormat: HUDI
targetFormats:
- DELTA
datasets:
-
tableBasePath: s3://soumil-dev-bucket-1995/silver/table_name=orders/
tableName: orders
```
Following above works
```
sourceFormat: HUDI
targetFormats:
- ICEBERG
datasets:
-
tableBasePath: s3://soumil-dev-bucket-1995/silver/table_name=orders/
tableName: orders
```
Hudi version : 0.12
Spark Version : 3.3.0-amzn-1
Java Version
sh-4.2$ java -version
openjdk version "1.8.0_392"
OpenJDK Runtime Environment Corretto-8.392.08.1 (build 1.8.0_392-b08)
OpenJDK 64-Bit Server VM Corretto-8.392.08.1 (build 25.392-b08, mixed mode)
sh-4.2$
I see following error
```
sh-4.2$ java -jar ./utilities-0.1.0-beta1-bundled.jar --dataset
./my_config.yaml
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
SLF4J: Class path contains SLF4J bindings targeting slf4j-api versions 1.7.x
or earlier.
SLF4J: Ignoring binding found at
[jar:file:/home/glue_user/workspace/jupyter_workspace/utilities-0.1.0-beta1-bundled.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See https://www.slf4j.org/codes.html#ignoredBindings for an
explanation.
2024-03-02 14:06:30 INFO io.onetable.utilities.RunSync:141 - Running sync
for basePath s3://soumil-dev-bucket-1995/silver/table_name=orders/ for
following table formats [ICEBERG]
2024-03-02 14:06:32 INFO io.onetable.client.OneTableClient:264 - No
previous OneTable sync for target. Falling back to snapshot sync.
# WARNING: Unable to get Instrumentation. Dynamic Attach failed. You may add
this JAR as -javaagent manually, or supply -Djdk.attach.allowAttachSelf
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with
module exceptions: [org.openjdk.jol.vm.sa.SASupportException: Sense failed.,
org.openjdk.jol.vm.sa.SASupportException: Sense failed.,
org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
2024-03-02 14:06:35 ERROR io.onetable.spi.sync.TableFormatSync:61 - Failed
to sync snapshot
java.lang.IllegalArgumentException: Cannot add field order_id as an
identifier field: not a required field
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkArgument(Preconditions.java:220)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
org.apache.iceberg.Schema.validateIdentifierField(Schema.java:126)
~[utilities-0.1.0-beta1-bundled.jar:?]
at org.apache.iceberg.Schema.lambda$new$0(Schema.java:106)
~[utilities-0.1.0-beta1-bundled.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_392]
at org.apache.iceberg.Schema.<init>(Schema.java:106)
~[utilities-0.1.0-beta1-bundled.jar:?]
at org.apache.iceberg.Schema.<init>(Schema.java:91)
~[utilities-0.1.0-beta1-bundled.jar:?]
at org.apache.iceberg.Schema.<init>(Schema.java:83)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
io.onetable.iceberg.IcebergSchemaExtractor.toIceberg(IcebergSchemaExtractor.java:90)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
io.onetable.iceberg.IcebergClient.initializeTableIfRequired(IcebergClient.java:125)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
io.onetable.iceberg.IcebergClient.beginSync(IcebergClient.java:113)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
io.onetable.spi.sync.TableFormatSync.getSyncResult(TableFormatSync.java:107)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
io.onetable.spi.sync.TableFormatSync.syncSnapshot(TableFormatSync.java:54)
~[utilities-0.1.0-beta1-bundled.jar:?]
at
io.onetable.client.OneTableClient.lambda$syncSnapshot$4(OneTableClient.java:167)
~[utilities-0.1.0-beta1-bundled.jar:?]
at java.util.HashMap.forEach(HashMap.java:1290) ~[?:1.8.0_392]
at
io.onetable.client.OneTableClient.syncSnapshot(OneTableClient.java:165)
~[utilities-0.1.0-beta1-bundled.jar:?]
at io.onetable.client.OneTableClient.sync(OneTableClient.java:122)
~[utilities-0.1.0-beta1-bundled.jar:?]
at io.onetable.utilities.RunSync.main(RunSync.java:162)
~[utilities-0.1.0-beta1-bundled.jar:?]
2024-03-02 14:06:35 INFO io.onetable.client.OneTableClient:127 - OneTable
Sync is successful for the following formats [ICEBERG]
sh-4.2$
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
