abmo-x opened a new issue, #9997:
URL: https://github.com/apache/iceberg/issues/9997
### Apache Iceberg version
1.3.1
### Query engine
Flink
### Please describe the bug 🐞
Iceberg table points to a metadata.json that failed to upload to s3. Due to
this the table was not loadable as latest metadata json was not found
`The main method caused an error: Failed to open input stream for file:
s3a://bucket/prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json`
### Expected:
when s3 upload to metadata json failed, iceberg's hive commit should have
failed as write metadata failed.
### Actual:
Iceberg logs show that commit was successful
`2024-03-06T23:06:36.160+00:00 INFO
org.apache.iceberg.hive.HiveTableOperations
Committed to table iceberg.default.some_table with the new metadata
location
s3a://bucket/prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json
`
#### However the upload to s3 failed!
`
06/Mar/2024:23:06:01 +0000 REST.PUT.OBJECT
prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json
"PUT
/bucket/prefix/default.db/some_table/metadata/52949-1e28478a-bf5e-4ec0-8d2c-......metadata.json
HTTP/1.1" **400 IncompleteBody**`
#### Flink pipeline fails around same time as upload
`2024-03-06T23:06:36.110+00:00 WARN
org.apache.flink.runtime.taskmanager.Task
message: Source: table[11] -> PythonCalc[12] -> (Calc[13] ->
IcebergStreamWriter, Calc[15] -> IcebergStreamWriter) (2/4)#0 switched from
INITIALIZING to FAILED with failure cause:`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
