Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859711825 I'm here to submit a PR for a fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859711384 I have identified the problem, during the execution of deleteRemovedMetadataFiles(base, metadata), if there is some unexpected problem such as OOM, it will cause the datafile to be c

[I] Hive: partitioning is not working [iceberg]

2023-12-17 Thread via GitHub
bluzy opened a new issue, #9329: URL: https://github.com/apache/iceberg/issues/9329 ### Apache Iceberg version 1.3.1 ### Query engine Hive ### Please describe the bug 🐞 I have a question when querying a partitioned table in Hive. I have hourly partiti

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
Fokko commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429583185 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859649309 @RussellSpitzer I have slightly modified HadoopTableOperations.commit(). What do you think about this modification? ``` @Override public void commit(TableMetadata bas

Re: [I] Is iceberg writing metadata atomic [iceberg]

2023-12-17 Thread via GitHub
pvary commented on issue #9328: URL: https://github.com/apache/iceberg/issues/9328#issuecomment-1859610834 I do not really understand your question, but the Flink commit logic is described here: https://github.com/apache/iceberg/issues/9089#issuecomment-1843199768 This might help you

Re: [PR] Spark: Add tests for select using tag and branch identifier [iceberg]

2023-12-17 Thread via GitHub
wypoon commented on PR #9286: URL: https://github.com/apache/iceberg/pull/9286#issuecomment-1859545554 @nastra would be great if you reviewed this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859523079 @RussellSpitzer --User has mutliple tables homed in the same directory, Remove orphan files for one deletes files for the other. This is generally unrecoverable and you'll no

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
RussellSpitzer commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859515258 When I see this sort of thing it's usually one of two issues 1. The user has accidentally run some command which deletes files without talking to iceberg, the snapshot r

[I] Is iceberg writing metadata atomic [iceberg]

2023-12-17 Thread via GitHub
lpy148145 opened a new issue, #9328: URL: https://github.com/apache/iceberg/issues/9328 ### Query engine Flink1.16 ### Question Is iceberg writing metadata atomic,Why do I write with flink, and there is a snapshot record in the Metadata file, but no Snapshot file exists

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859497363 Slack link: https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1700291064787019 -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859496024 @chennurchaitanya -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859495578 I also found another situation where I have a table that has never had a CALL command executed on it. All I did was to run a MERGE INTO once a day, but after the OOM, the table did h

Re: [I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird commented on issue #9327: URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859492897 @RussellSpitzer @nastra can you help me? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

[I] FileNotFoundException It can occur in some scenarios. (data file) [iceberg]

2023-12-17 Thread via GitHub
BsoBird opened a new issue, #9327: URL: https://github.com/apache/iceberg/issues/9327 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 SPARK 3.4.1. ``` Caused by: java.io.FileNotFoundException: File

[I] Spark by default loading/considering spark_catalog catalog instead of given catalog. [iceberg]

2023-12-17 Thread via GitHub
vimalKeshu opened a new issue, #9326: URL: https://github.com/apache/iceberg/issues/9326 ### Apache Iceberg version 1.4.2 (latest release) ### Query engine Spark ### Please describe the bug 🐞 I am trying spark iceberg with rest catalog in local mode. Belo

Re: [PR] Core: Add param to limit manifest parallel reader queue size [iceberg]

2023-12-17 Thread via GitHub
Heltman commented on code in PR #7844: URL: https://github.com/apache/iceberg/pull/7844#discussion_r1429368849 ## core/src/main/java/org/apache/iceberg/util/ParallelIterable.java: ## @@ -67,10 +70,17 @@ private ParallelIterator( try (Closeable ignore

Re: [PR] Core: Fixed certain operations failing to add new data files during retries [iceberg]

2023-12-17 Thread via GitHub
rdblue commented on PR #9230: URL: https://github.com/apache/iceberg/pull/9230#issuecomment-1859376491 @jasonf20, sorry for being unclear in my reply earlier, but I don't think that the tests in this PR reproduce the error. The tests here reproduce similar errors, but do it by manually call

Re: [I] Support write to identifier branch when WAP is enabled [iceberg]

2023-12-17 Thread via GitHub
github-actions[bot] commented on issue #7103: URL: https://github.com/apache/iceberg/issues/7103#issuecomment-1859338991 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache Gi

Re: [I] Support to read iceberg table's metadata column with Flink [iceberg]

2023-12-17 Thread via GitHub
github-actions[bot] commented on issue #7110: URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1859338977 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [I] Support write to identifier branch when WAP is enabled [iceberg]

2023-12-17 Thread via GitHub
github-actions[bot] closed issue #7103: Support write to identifier branch when WAP is enabled URL: https://github.com/apache/iceberg/issues/7103 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Flink will throw MetaException when hive table property value is larger than 4000 [iceberg]

2023-12-17 Thread via GitHub
github-actions[bot] commented on issue #7113: URL: https://github.com/apache/iceberg/issues/7113#issuecomment-1859338957 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Parquet: Add system config for unsafe Parquet ID fallback. [iceberg]

2023-12-17 Thread via GitHub
rdblue commented on PR #9324: URL: https://github.com/apache/iceberg/pull/9324#issuecomment-1859284853 @jackye1995, this is the fix I mentioned for Parquet data files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use th

[PR] Parquet: Add system config for unsafe Parquet ID fallback. [iceberg]

2023-12-17 Thread via GitHub
rdblue opened a new pull request, #9324: URL: https://github.com/apache/iceberg/pull/9324 This adds a `SystemConfig` setting to enable or disable unsafe Parquet ID fallback. Unsafe Parquet ID fallback (`ParquetSchemaUtil.pruneColumnsFallback`) is Netflix-specific behavior that assign

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429261008 ## tests/table/test_name_mapping.py: ## @@ -0,0 +1,291 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements.

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429260719 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429260415 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429260289 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429259170 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429255597 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429253148 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429252665 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429252046 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429249534 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429247634 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. S

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429246371 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429246160 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429245920 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429245410 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429245053 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429244908 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429244651 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241718 ## pyiceberg/table/__init__.py: ## @@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int: snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -

Re: [PR] Add name-mapping [iceberg-python]

2023-12-17 Thread via GitHub
Fokko commented on code in PR #212: URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429241706 ## pyiceberg/table/name_mapping.py: ## @@ -0,0 +1,203 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. Se

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241217 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241217 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429240582 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241025 ## pyiceberg/table/__init__.py: ## @@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction: """ return self._append_updates(Set

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429240582 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429240582 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429238390 ## pyiceberg/table/__init__.py: ## @@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]: def update_schema(self, allow_incompatible_changes: bool = F

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429236348 ## pyiceberg/table/__init__.py: ## @@ -583,7 +637,7 @@ class AssertRefSnapshotId(TableRequirement): """ type: Literal["assert-ref-snapshot-id"] = Field(d

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429233871 ## pyiceberg/table/__init__.py: ## @@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction: """ return self._append_updates(Set

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429233139 ## pyiceberg/table/__init__.py: ## @@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction: """ return self._append_updates(Set

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429230197 ## pyiceberg/table/__init__.py: ## @@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction: """ return self._append_updates(Set

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429229851 ## pyiceberg/table/__init__.py: ## @@ -110,6 +120,8 @@ ALWAYS_TRUE = AlwaysTrue() TABLE_ROOT_ID = -1 +_JAVA_LONG_MAX = 9223372036854775807 Review Comment: Loo

Re: [PR] Flink: Fix TestIcebergSourceWithWatermarkExtractor flakiness [iceberg]

2023-12-17 Thread via GitHub
stevenzwu commented on code in PR #9309: URL: https://github.com/apache/iceberg/pull/9309#discussion_r1429229476 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -95,8 +97,7 @@ public class TestIcebergSourceWi

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429229166 ## pyiceberg/manifest.py: ## @@ -920,9 +927,9 @@ def prepare_manifest(self, manifest_file: ManifestFile) -> ManifestFile: # To validate this, check that

Re: [PR] Write support [iceberg-python]

2023-12-17 Thread via GitHub
rdblue commented on code in PR #41: URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429228120 ## pyiceberg/manifest.py: ## @@ -897,9 +902,11 @@ def prepare_manifest(self, manifest_file: ManifestFile) -> ManifestFile: class ManifestListWriterV2(ManifestList

Re: [PR] Flink: Fix TestIcebergSourceWithWatermarkExtractor flakiness [iceberg]

2023-12-17 Thread via GitHub
pvary commented on code in PR #9309: URL: https://github.com/apache/iceberg/pull/9309#discussion_r142906 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -304,6 +322,11 @@ public void testThrottling() thro

Re: [PR] Flink: Fix TestIcebergSourceWithWatermarkExtractor flakiness [iceberg]

2023-12-17 Thread via GitHub
pvary commented on code in PR #9309: URL: https://github.com/apache/iceberg/pull/9309#discussion_r1429221701 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -95,8 +97,7 @@ public class TestIcebergSourceWithWa

Re: [PR] Flink: Fix TestIcebergSourceWithWatermarkExtractor flakiness [iceberg]

2023-12-17 Thread via GitHub
stevenzwu commented on code in PR #9309: URL: https://github.com/apache/iceberg/pull/9309#discussion_r1429218088 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -95,8 +97,7 @@ public class TestIcebergSourceWi

Re: [PR] API: New API For sequential / streaming updates [iceberg]

2023-12-17 Thread via GitHub
jasonf20 commented on PR #9323: URL: https://github.com/apache/iceberg/pull/9323#issuecomment-1859197491 **Benchmark** The following test was run locally just to demonstrate that the difference in IO performance is very significant. While the transaction approach IO grows linearly with t

[PR] API: New API For sequential / streaming updates [iceberg]

2023-12-17 Thread via GitHub
jasonf20 opened a new pull request, #9323: URL: https://github.com/apache/iceberg/pull/9323 **Explanation** Certain data production patterns can result in a bunch of micro-batch updates that need to be applied to the table sequentially. If these batches include updates they need to be c

Re: [PR] Flink: Support watermark alignment of source splits [iceberg]

2023-12-17 Thread via GitHub
pvary commented on PR #9308: URL: https://github.com/apache/iceberg/pull/9308#issuecomment-1859147694 @stevenzwu: Here is what I have found: `wakeUp` is called whenever there is a new message concerning the `SplitReader`: - new split added/removed - split paused/resumed These

Re: [PR] Flink: Support watermark alignment of source splits [iceberg]

2023-12-17 Thread via GitHub
pvary commented on code in PR #9308: URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429133489 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceSplitReader.java: ## @@ -80,6 +87,25 @@ public RecordsWithSplitIds> fetch() throws

Re: [PR] Flink: Support watermark alignment of source splits [iceberg]

2023-12-17 Thread via GitHub
pvary commented on code in PR #9308: URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429133074 ## flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java: ## @@ -273,23 +271,40 @@ public void apply( public voi

Re: [PR] Flink: Support watermark alignment of source splits [iceberg]

2023-12-17 Thread via GitHub
pvary commented on code in PR #9308: URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429132495 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceSplitReader.java: ## @@ -113,7 +139,13 @@ public void handleSplitsChanges(SplitsCh

Re: [PR] Flink: Support watermark alignment of source splits [iceberg]

2023-12-17 Thread via GitHub
pvary commented on code in PR #9308: URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429132559 ## flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceSplitReader.java: ## @@ -113,7 +139,13 @@ public void handleSplitsChanges(SplitsCh

Re: [PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.4.0 [iceberg-go]

2023-12-17 Thread via GitHub
dependabot[bot] closed pull request #29: build(deps): bump github.com/google/uuid from 1.3.1 to 1.4.0 URL: https://github.com/apache/iceberg-go/pull/29 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t

[PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.5.0 [iceberg-go]

2023-12-17 Thread via GitHub
dependabot[bot] opened a new pull request, #46: URL: https://github.com/apache/iceberg-go/pull/46 Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.3.1 to 1.5.0. Release notes Sourced from https://github.com/google/uuid/releases";>github.com/google/uuid's relea

Re: [PR] build(deps): bump github.com/google/uuid from 1.3.1 to 1.4.0 [iceberg-go]

2023-12-17 Thread via GitHub
dependabot[bot] commented on PR #29: URL: https://github.com/apache/iceberg-go/pull/29#issuecomment-1859071850 Superseded by #46. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comm