BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859711825
I'm here to submit a PR for a fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859711384
I have identified the problem, during the execution of
deleteRemovedMetadataFiles(base, metadata), if there is some unexpected problem
such as OOM, it will cause the datafile to be c
bluzy opened a new issue, #9329:
URL: https://github.com/apache/iceberg/issues/9329
### Apache Iceberg version
1.3.1
### Query engine
Hive
### Please describe the bug 🐞
I have a question when querying a partitioned table in Hive.
I have hourly partiti
Fokko commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429583185
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. Se
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859649309
@RussellSpitzer
I have slightly modified HadoopTableOperations.commit(). What do you think
about this modification?
```
@Override
public void commit(TableMetadata bas
pvary commented on issue #9328:
URL: https://github.com/apache/iceberg/issues/9328#issuecomment-1859610834
I do not really understand your question, but the Flink commit logic is
described here:
https://github.com/apache/iceberg/issues/9089#issuecomment-1843199768
This might help you
wypoon commented on PR #9286:
URL: https://github.com/apache/iceberg/pull/9286#issuecomment-1859545554
@nastra would be great if you reviewed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859523079
@RussellSpitzer
--User has mutliple tables homed in the same directory, Remove orphan files
for one deletes files for the other. This is generally unrecoverable and you'll
no
RussellSpitzer commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859515258
When I see this sort of thing it's usually one of two issues
1. The user has accidentally run some command which deletes files without
talking to iceberg, the snapshot r
lpy148145 opened a new issue, #9328:
URL: https://github.com/apache/iceberg/issues/9328
### Query engine
Flink1.16
### Question
Is iceberg writing metadata atomic,Why do I write with flink, and there is a
snapshot record in the Metadata file, but no Snapshot file exists
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859497363
Slack link:
https://apache-iceberg.slack.com/archives/C025PH0G1D4/p1700291064787019
--
This is an automated message from the Apache Git Service.
To respond to the message, please
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859496024
@chennurchaitanya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859495578
I also found another situation where I have a table that has never had a
CALL command executed on it. All I did was to run a MERGE INTO once a day, but
after the OOM, the table did h
BsoBird commented on issue #9327:
URL: https://github.com/apache/iceberg/issues/9327#issuecomment-1859492897
@RussellSpitzer @nastra
can you help me?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above t
BsoBird opened a new issue, #9327:
URL: https://github.com/apache/iceberg/issues/9327
### Apache Iceberg version
1.4.2 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
SPARK 3.4.1.
```
Caused by: java.io.FileNotFoundException: File
vimalKeshu opened a new issue, #9326:
URL: https://github.com/apache/iceberg/issues/9326
### Apache Iceberg version
1.4.2 (latest release)
### Query engine
Spark
### Please describe the bug 🐞
I am trying spark iceberg with rest catalog in local mode.
Belo
Heltman commented on code in PR #7844:
URL: https://github.com/apache/iceberg/pull/7844#discussion_r1429368849
##
core/src/main/java/org/apache/iceberg/util/ParallelIterable.java:
##
@@ -67,10 +70,17 @@ private ParallelIterator(
try (Closeable ignore
rdblue commented on PR #9230:
URL: https://github.com/apache/iceberg/pull/9230#issuecomment-1859376491
@jasonf20, sorry for being unclear in my reply earlier, but I don't think
that the tests in this PR reproduce the error. The tests here reproduce similar
errors, but do it by manually call
github-actions[bot] commented on issue #7103:
URL: https://github.com/apache/iceberg/issues/7103#issuecomment-1859338991
This issue has been closed because it has not received any activity in the
last 14 days since being marked as 'stale'
--
This is an automated message from the Apache Gi
github-actions[bot] commented on issue #7110:
URL: https://github.com/apache/iceberg/issues/7110#issuecomment-1859338977
This issue has been automatically marked as stale because it has been open
for 180 days with no activity. It will be closed in next 14 days if no further
activity occurs.
github-actions[bot] closed issue #7103: Support write to identifier branch when
WAP is enabled
URL: https://github.com/apache/iceberg/issues/7103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
github-actions[bot] commented on issue #7113:
URL: https://github.com/apache/iceberg/issues/7113#issuecomment-1859338957
This issue has been automatically marked as stale because it has been open
for 180 days with no activity. It will be closed in next 14 days if no further
activity occurs.
rdblue commented on PR #9324:
URL: https://github.com/apache/iceberg/pull/9324#issuecomment-1859284853
@jackye1995, this is the fix I mentioned for Parquet data files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
rdblue opened a new pull request, #9324:
URL: https://github.com/apache/iceberg/pull/9324
This adds a `SystemConfig` setting to enable or disable unsafe Parquet ID
fallback.
Unsafe Parquet ID fallback (`ParquetSchemaUtil.pruneColumnsFallback`) is
Netflix-specific behavior that assign
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429261008
##
tests/table/test_name_mapping.py:
##
@@ -0,0 +1,291 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429260719
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429260415
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429260289
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429259170
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429255597
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429253148
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429252665
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429252046
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429249534
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429247634
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. S
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429246371
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429246160
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429245920
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429245410
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429245053
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429244908
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429244651
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241718
##
pyiceberg/table/__init__.py:
##
@@ -1904,3 +2001,144 @@ def _generate_snapshot_id() -> int:
snapshot_id = snapshot_id if snapshot_id >= 0 else snapshot_id * -
Fokko commented on code in PR #212:
URL: https://github.com/apache/iceberg-python/pull/212#discussion_r1429241706
##
pyiceberg/table/name_mapping.py:
##
@@ -0,0 +1,203 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. Se
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241217
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241217
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429240582
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429241025
##
pyiceberg/table/__init__.py:
##
@@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction:
"""
return self._append_updates(Set
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429240582
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429240582
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429238390
##
pyiceberg/table/__init__.py:
##
@@ -830,6 +884,49 @@ def history(self) -> List[SnapshotLogEntry]:
def update_schema(self, allow_incompatible_changes: bool = F
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429236348
##
pyiceberg/table/__init__.py:
##
@@ -583,7 +637,7 @@ class AssertRefSnapshotId(TableRequirement):
"""
type: Literal["assert-ref-snapshot-id"] =
Field(d
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429233871
##
pyiceberg/table/__init__.py:
##
@@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction:
"""
return self._append_updates(Set
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429233139
##
pyiceberg/table/__init__.py:
##
@@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction:
"""
return self._append_updates(Set
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429230197
##
pyiceberg/table/__init__.py:
##
@@ -209,6 +221,48 @@ def set_properties(self, **updates: str) -> Transaction:
"""
return self._append_updates(Set
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429229851
##
pyiceberg/table/__init__.py:
##
@@ -110,6 +120,8 @@
ALWAYS_TRUE = AlwaysTrue()
TABLE_ROOT_ID = -1
+_JAVA_LONG_MAX = 9223372036854775807
Review Comment:
Loo
stevenzwu commented on code in PR #9309:
URL: https://github.com/apache/iceberg/pull/9309#discussion_r1429229476
##
flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java:
##
@@ -95,8 +97,7 @@ public class TestIcebergSourceWi
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429229166
##
pyiceberg/manifest.py:
##
@@ -920,9 +927,9 @@ def prepare_manifest(self, manifest_file: ManifestFile) ->
ManifestFile:
# To validate this, check that
rdblue commented on code in PR #41:
URL: https://github.com/apache/iceberg-python/pull/41#discussion_r1429228120
##
pyiceberg/manifest.py:
##
@@ -897,9 +902,11 @@ def prepare_manifest(self, manifest_file: ManifestFile) ->
ManifestFile:
class ManifestListWriterV2(ManifestList
pvary commented on code in PR #9309:
URL: https://github.com/apache/iceberg/pull/9309#discussion_r142906
##
flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java:
##
@@ -304,6 +322,11 @@ public void testThrottling() thro
pvary commented on code in PR #9309:
URL: https://github.com/apache/iceberg/pull/9309#discussion_r1429221701
##
flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java:
##
@@ -95,8 +97,7 @@ public class TestIcebergSourceWithWa
stevenzwu commented on code in PR #9309:
URL: https://github.com/apache/iceberg/pull/9309#discussion_r1429218088
##
flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java:
##
@@ -95,8 +97,7 @@ public class TestIcebergSourceWi
jasonf20 commented on PR #9323:
URL: https://github.com/apache/iceberg/pull/9323#issuecomment-1859197491
**Benchmark**
The following test was run locally just to demonstrate that the difference
in IO performance is very significant. While the transaction approach IO grows
linearly with t
jasonf20 opened a new pull request, #9323:
URL: https://github.com/apache/iceberg/pull/9323
**Explanation**
Certain data production patterns can result in a bunch of micro-batch
updates that need to be applied to the table sequentially. If these batches
include updates they need to be c
pvary commented on PR #9308:
URL: https://github.com/apache/iceberg/pull/9308#issuecomment-1859147694
@stevenzwu: Here is what I have found:
`wakeUp` is called whenever there is a new message concerning the
`SplitReader`:
- new split added/removed
- split paused/resumed
These
pvary commented on code in PR #9308:
URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429133489
##
flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceSplitReader.java:
##
@@ -80,6 +87,25 @@ public RecordsWithSplitIds> fetch()
throws
pvary commented on code in PR #9308:
URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429133074
##
flink/v1.18/flink/src/test/java/org/apache/iceberg/flink/source/TestIcebergSourceWithWatermarkExtractor.java:
##
@@ -273,23 +271,40 @@ public void apply(
public voi
pvary commented on code in PR #9308:
URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429132495
##
flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceSplitReader.java:
##
@@ -113,7 +139,13 @@ public void
handleSplitsChanges(SplitsCh
pvary commented on code in PR #9308:
URL: https://github.com/apache/iceberg/pull/9308#discussion_r1429132559
##
flink/v1.18/flink/src/main/java/org/apache/iceberg/flink/source/reader/IcebergSourceSplitReader.java:
##
@@ -113,7 +139,13 @@ public void
handleSplitsChanges(SplitsCh
dependabot[bot] closed pull request #29: build(deps): bump
github.com/google/uuid from 1.3.1 to 1.4.0
URL: https://github.com/apache/iceberg-go/pull/29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go t
dependabot[bot] opened a new pull request, #46:
URL: https://github.com/apache/iceberg-go/pull/46
Bumps [github.com/google/uuid](https://github.com/google/uuid) from 1.3.1 to
1.5.0.
Release notes
Sourced from https://github.com/google/uuid/releases";>github.com/google/uuid's
relea
dependabot[bot] commented on PR #29:
URL: https://github.com/apache/iceberg-go/pull/29#issuecomment-1859071850
Superseded by #46.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comm
72 matches
Mail list logo