Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-03-28 Thread via GitHub
harishch1998 commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1544095514 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -78,6 +80,8 @@ public class HTTPClient implements RESTClient { private static final int

Re: [PR] Introduce two properties for reading the connection timeout and socke… [iceberg]

2024-03-28 Thread via GitHub
harishch1998 commented on code in PR #10053: URL: https://github.com/apache/iceberg/pull/10053#discussion_r1544095303 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -448,6 +443,43 @@ static HttpRequestInterceptor loadInterceptorDynamically( return

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-28 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1544090262 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +368,53 @@ public void renameTable(TableIdentifier from, TableIdentifier

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-28 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1544090262 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +368,53 @@ public void renameTable(TableIdentifier from, TableIdentifier

Re: [PR] Hive: Add View support for HIVE catalog [iceberg]

2024-03-28 Thread via GitHub
szehon-ho commented on code in PR #9852: URL: https://github.com/apache/iceberg/pull/9852#discussion_r1544090262 ## hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java: ## @@ -250,29 +368,53 @@ public void renameTable(TableIdentifier from, TableIdentifier

Re: [PR] Flink: backport PR #9321 for range partitioner on map statistics [iceberg]

2024-03-28 Thread via GitHub
pvary commented on PR #10061: URL: https://github.com/apache/iceberg/pull/10061#issuecomment-2026633035 @stevenzwu : is this clean? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
syun64 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1544039629 ## pyiceberg/table/__init__.py: ## @@ -3108,3 +3138,127 @@ def snapshots(self) -> "pa.Table": snapshots, schema=snapshots_schema,

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
syun64 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1544015221 ## pyiceberg/table/__init__.py: ## @@ -2526,25 +2537,44 @@ def _dataframe_to_data_files( """ from pyiceberg.io.pyarrow import bin_pack_arrow_table,

Re: [PR] Add Pagination To List Apis [iceberg]

2024-03-28 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1544047140 ## core/src/test/java/org/apache/iceberg/rest/responses/TestListTablesResponse.java: ## @@ -105,6 +105,14 @@ public void testBuilderDoesNotCreateInvalidObjects() {

Re: [PR] Add Pagination To List Apis [iceberg]

2024-03-28 Thread via GitHub
rahil-c commented on code in PR #9782: URL: https://github.com/apache/iceberg/pull/9782#discussion_r1520289302 ## core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java: ## @@ -224,6 +229,12 @@ public void initialize(String name, Map unresolved) {

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1543796831 ## tests/integration/test_partitioned_write.py: ## @@ -0,0 +1,533 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] [0.6.x] Backport PR #523 to cast data to iceberg table's pyarrow schema [iceberg-python]

2024-03-28 Thread via GitHub
HonahX commented on code in PR #559: URL: https://github.com/apache/iceberg-python/pull/559#discussion_r1544024793 ## pyiceberg/table/__init__.py: ## @@ -1010,6 +1053,13 @@ def append(self, df: pa.Table) -> None: if len(self.spec().fields) > 0: raise

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
liurenjie1024 commented on PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#issuecomment-2026487818 Hi, @marvinlanhenke Thanks for your contribution, the overall design looks good to me! Please take your time and enjoy time with family -- This is an automated message from

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1543956147 ## crates/iceberg/src/transform/mod.rs: ## @@ -37,6 +36,15 @@ pub trait TransformFunction: Send { fn transform(, input: ArrayRef) -> Result; ///

Re: [I] Non-nullable columns marked as nullable during table creation [iceberg]

2024-03-28 Thread via GitHub
zhongyujiang commented on issue #7771: URL: https://github.com/apache/iceberg/issues/7771#issuecomment-2026461665 > Do you know if this feature will added anytime soon? @va-bo-101 I'm not sure, I think I can investigate more on this later and file a PR to support it, so the

Re: [I] UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1

2024-03-28 Thread via GitHub
Zelin1203 commented on issue #10056: URL: https://github.com/apache/iceberg/issues/10056#issuecomment-2026450467 > You are using MySQL according to `Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server

Re: [I] UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1

2024-03-28 Thread via GitHub
Zelin1203 commented on issue #10056: URL: https://github.com/apache/iceberg/issues/10056#issuecomment-2026449330 > @Zelin1203 your stack trace indicates that you're using MySql, can you confirm that and also please share your full catalog configuration? /cc @jbonofre Yes, i am using

Re: [PR] [Bug fix] update name mapping in Transaction.update_schema [iceberg-python]

2024-03-28 Thread via GitHub
HonahX commented on PR #508: URL: https://github.com/apache/iceberg-python/pull/508#issuecomment-2026418611 Remove this from 0.6.1 because this depends on https://github.com/apache/iceberg-python/pull/441, which is a new feature. -- This is an automated message from the Apache Git

Re: [I] Lock remains in HMS if HiveTableOperations gets killed (direct process shutdown - no signals) after lock is acquired [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2301: URL: https://github.com/apache/iceberg/issues/2301#issuecomment-2026356909 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Build a utility to infer partitions at a given path [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2300: URL: https://github.com/apache/iceberg/issues/2300#issuecomment-2026356894 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Catalog Migration transaction [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2288: URL: https://github.com/apache/iceberg/issues/2288#issuecomment-2026356862 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Consider both delete file size and data file size when planing tasks [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2298: URL: https://github.com/apache/iceberg/issues/2298#issuecomment-2026356876 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Hive: Support identifiers with catalog [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2274: URL: https://github.com/apache/iceberg/issues/2274#issuecomment-2026356831 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] hive create external table for iceberg have error. [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2277: URL: https://github.com/apache/iceberg/issues/2277#issuecomment-2026356842 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity

Re: [I] Can not write data to hive catalog iceberg table, why ? [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2076: URL: https://github.com/apache/iceberg/issues/2076#issuecomment-2026356717 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Can not write data to hive catalog iceberg table, why ? [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] closed issue #2076: Can not write data to hive catalog iceberg table, why ? URL: https://github.com/apache/iceberg/issues/2076 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Integrity of Iceberg tables [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2073: URL: https://github.com/apache/iceberg/issues/2073#issuecomment-2026356706 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Could Hive alter iceberg table? [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2063: URL: https://github.com/apache/iceberg/issues/2063#issuecomment-2026356697 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Integrity of Iceberg tables [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] closed issue #2073: Integrity of Iceberg tables URL: https://github.com/apache/iceberg/issues/2073 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Could Hive alter iceberg table? [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] closed issue #2063: Could Hive alter iceberg table? URL: https://github.com/apache/iceberg/issues/2063 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [I] Authenticated encryption stream [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2060: URL: https://github.com/apache/iceberg/issues/2060#issuecomment-2026356683 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] String representation of startsWith often has too many quotes around the string value [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] closed issue #2059: String representation of startsWith often has too many quotes around the string value URL: https://github.com/apache/iceberg/issues/2059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [I] String representation of startsWith often has too many quotes around the string value [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] commented on issue #2059: URL: https://github.com/apache/iceberg/issues/2059#issuecomment-2026356664 This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale' -- This is an automated message from the Apache

Re: [I] Authenticated encryption stream [iceberg]

2024-03-28 Thread via GitHub
github-actions[bot] closed issue #2060: Authenticated encryption stream URL: https://github.com/apache/iceberg/issues/2060 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] [0.6.x] Backport PR #324 and #493 for fixing dead links in docs [iceberg-python]

2024-03-28 Thread via GitHub
HonahX opened a new pull request, #556: URL: https://github.com/apache/iceberg-python/pull/556 This PR backports #324 and #493 to 0.6.x branch, resolving merge conflicts caused by cherry-picking -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [PR] Core: add a new task-type field to task JSON serialization. add data task JSON serialization imp. [iceberg]

2024-03-28 Thread via GitHub
stevenzwu commented on PR #9728: URL: https://github.com/apache/iceberg/pull/9728#issuecomment-2026251811 @nastra can you help take another look? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1543797231 ## tests/integration/test_partitioned_write.py: ## @@ -0,0 +1,533 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1543796831 ## tests/integration/test_partitioned_write.py: ## @@ -0,0 +1,533 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1543796652 ## tests/integration/test_partitioned_write.py: ## @@ -0,0 +1,533 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license

Re: [PR] Partitioned Append on Identity Transform [iceberg-python]

2024-03-28 Thread via GitHub
jqin61 commented on code in PR #555: URL: https://github.com/apache/iceberg-python/pull/555#discussion_r1543793736 ## pyiceberg/table/__init__.py: ## @@ -2526,25 +2537,44 @@ def _dataframe_to_data_files( """ from pyiceberg.io.pyarrow import bin_pack_arrow_table,

Re: [PR] Spark: Clarify schema behavior when working with branches [iceberg]

2024-03-28 Thread via GitHub
rdblue commented on code in PR #10055: URL: https://github.com/apache/iceberg/pull/10055#discussion_r1543785185 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/sql/TestSelect.java: ## @@ -348,6 +349,64 @@ public void testBranchReference() {

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543752014 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -93,6 +95,16 @@ static void toJson(Snapshot snapshot, JsonGenerator generator) throws

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1543730206 ## crates/iceberg/src/spec/transform.rs: ## @@ -398,6 +679,80 @@ mod tests { } } +#[test] +fn test_none_projection() -> Result<()> {

Re: [I] branch schema affected by main table schema [iceberg]

2024-03-28 Thread via GitHub
namrathamyske commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2026055049 Looks like we are disabling the workaround from https://github.com/apache/iceberg/pull/10059. More future fix is where we track `schemaId` from snapshot in `SnapshotRef`

Re: [I] branch schema affected by main table schema [iceberg]

2024-03-28 Thread via GitHub
danielcweeks commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2026009473 @namrathamyske it was pointed out to me that workaround may not be working correctly for branches, which is something we might need to address. -- This is an automated

Re: [I] branch schema affected by main table schema [iceberg]

2024-03-28 Thread via GitHub
namrathamyske commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2026003001 Thanks @danielcweeks @nastra ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-28 Thread via GitHub
Fokko merged PR #444: URL: https://github.com/apache/iceberg-python/pull/444 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-28 Thread via GitHub
Fokko commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-2025890254 @kevinjqliu Thanks for adding the examples. I think in general we want to have slightly bigger files. A simple heuristic I can think of is that we put an upper bound on the

Re: [PR] Bin-pack Writes Operation into multiple parquet files, and parallelize writing `WriteTask`s [iceberg-python]

2024-03-28 Thread via GitHub
kevinjqliu commented on PR #444: URL: https://github.com/apache/iceberg-python/pull/444#issuecomment-2025883841 @Fokko resolved merge conflict. Please take another look. I've also updated the description to show the results of parallelized writes -- This is an automated message from the

Re: [I] Using time-travel query with timestamp against branch should return the branch's schema [iceberg]

2024-03-28 Thread via GitHub
nastra commented on issue #10058: URL: https://github.com/apache/iceberg/issues/10058#issuecomment-2025711190 Looking back at https://github.com/apache/iceberg/pull/9219, we should probably not allow `branch_` usage with `TIMESTAMP AS OF` -- This is an automated message from the Apache

Re: [I] branch schema affected by main table schema [iceberg]

2024-03-28 Thread via GitHub
nastra commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2025665097 > @namrathamyske you can force reading with the snapshot id on a branch by using the time travel statement. > > ``` > select * from default.t.branch_b1 for timestamp as of

[I] Using time-travel query with timestamp against branch should return the branch's schema [iceberg]

2024-03-28 Thread via GitHub
nastra opened a new issue, #10058: URL: https://github.com/apache/iceberg/issues/10058 ### Apache Iceberg version 1.5.0 (latest release) ### Query engine None ### Please describe the bug  Running `select * from tbl.branch_b1 for timestamp as of now()`

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543278782 ## core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543273755 ## core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543275450 ## core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java: ## @@ -0,0 +1,127 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543271894 ## core/src/test/java/org/apache/iceberg/TestManifestEncryption.java: ## @@ -85,7 +85,7 @@ public class TestManifestEncryption { private static final

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543265194 ## core/src/main/java/org/apache/iceberg/SnapshotProducer.java: ## @@ -237,10 +243,36 @@ public Snapshot apply() { OutputFile manifestList =

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-03-28 Thread via GitHub
flyrain commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2025617889 Thanks @laithalzyoud for taking lead for the copy table action. Agreed with @amogh-jahagirdar, can we separated the PR to interface only PR, and implementation PRs. That way, we can

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543263029 ## core/src/main/java/org/apache/iceberg/SnapshotParser.java: ## @@ -93,6 +95,16 @@ static void toJson(Snapshot snapshot, JsonGenerator generator) throws

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543238877 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -143,7 +192,38 @@ private void cacheManifests(FileIO fileIO) { if (allManifests ==

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543238877 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -143,7 +192,38 @@ private void cacheManifests(FileIO fileIO) { if (allManifests ==

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543237158 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -143,7 +192,38 @@ private void cacheManifests(FileIO fileIO) { if (allManifests ==

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543229877 ## api/src/main/java/org/apache/iceberg/Snapshot.java: ## @@ -162,6 +162,25 @@ default Iterable removedDeleteFiles(FileIO io) { */ String

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543221066 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -62,15 +73,41 @@ class BaseSnapshot implements Snapshot { Map summary, Integer

Re: [PR] Manifest list encryption [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #7770: URL: https://github.com/apache/iceberg/pull/7770#discussion_r1543221066 ## core/src/main/java/org/apache/iceberg/BaseSnapshot.java: ## @@ -62,15 +73,41 @@ class BaseSnapshot implements Snapshot { Map summary, Integer

[PR] feat: support uri redirect in rest client [iceberg-rust]

2024-03-28 Thread via GitHub
TennyZhuang opened a new pull request, #310: URL: https://github.com/apache/iceberg-rust/pull/310 See the test code for details -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] branch schema affected by main table schema [iceberg]

2024-03-28 Thread via GitHub
danielcweeks commented on issue #9737: URL: https://github.com/apache/iceberg/issues/9737#issuecomment-2025526690 @namrathamyske you can force reading with the snapshot id on a branch by using the time travel statement. ``` select * from default.t.branch_b1 for timestamp as of

Re: [PR] Extend HTTPClient Builder to allow setting a proxy server [iceberg]

2024-03-28 Thread via GitHub
amogh-jahagirdar commented on code in PR #10052: URL: https://github.com/apache/iceberg/pull/10052#discussion_r1543187257 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -493,7 +525,17 @@ public HTTPClient build() { interceptor =

Re: [PR] Extend HTTPClient Builder to allow setting a proxy server [iceberg]

2024-03-28 Thread via GitHub
amogh-jahagirdar commented on code in PR #10052: URL: https://github.com/apache/iceberg/pull/10052#discussion_r1543178811 ## core/src/main/java/org/apache/iceberg/rest/HTTPClient.java: ## @@ -120,6 +128,13 @@ private HTTPClient( int maxRetries =

Re: [PR] Build: disable link-check for all medium blog posts [iceberg]

2024-03-28 Thread via GitHub
nastra merged PR #10057: URL: https://github.com/apache/iceberg/pull/10057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-03-28 Thread via GitHub
amogh-jahagirdar commented on code in PR #10024: URL: https://github.com/apache/iceberg/pull/10024#discussion_r1543096333 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/BaseCopyTableSparkAction.java: ## @@ -0,0 +1,871 @@ +/* + * Licensed to the Apache

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-28 Thread via GitHub
manuzhang commented on PR #10042: URL: https://github.com/apache/iceberg/pull/10042#issuecomment-2025455007 @nastra just created #10057 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [I] Flaky spark-3.3-scala-2.13-java-8 tests due to `No space left on device` [iceberg]

2024-03-28 Thread via GitHub
jbonofre commented on issue #10040: URL: https://github.com/apache/iceberg/issues/10040#issuecomment-2025432289 The `No space left on device` looks weird to me. Maybe a large temp file is created ? -- This is an automated message from the Apache Git Service. To respond to the message,

Re: [I] [JDBC Catalog] Table commit fails if iceberg_type field is NULL [iceberg]

2024-03-28 Thread via GitHub
jbonofre commented on issue #10046: URL: https://github.com/apache/iceberg/issues/10046#issuecomment-2025430576 I think I might have found the problem, I'm doing the test to reproduce. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to

Re: [I] UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1

2024-03-28 Thread via GitHub
jbonofre commented on issue #10056: URL: https://github.com/apache/iceberg/issues/10056#issuecomment-2025428144 You are using MySQL according to ` Caused by: java.sql.SQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server

Re: [I] UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1

2024-03-28 Thread via GitHub
nastra commented on issue #10056: URL: https://github.com/apache/iceberg/issues/10056#issuecomment-2025420554 @Zelin1203 your stack trace indicates that you're using MySql, can you confirm that and also please share your full catalog configuration? /cc @jbonofre -- This is an automated

[I] UncheckedSQLException: Failed to execute exists query: SELECT table_namespace FROM iceberg_tables WHERE catalog_name = ? AND (table_namespace = ? OR table_namespace LIKE ? ESCAPE '\') LIMIT 1 [ic

2024-03-28 Thread via GitHub
Zelin1203 opened a new issue, #10056: URL: https://github.com/apache/iceberg/issues/10056 ### Apache Iceberg version 1.5.0 (latest release) ### Query engine Flink ### Please describe the bug  I use jdbc catalog. When I update flink from 1.17.1 to 1.18.1

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543108275 ## spark/v3.4/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1429,40 +1429,28 @@ public void showViews() throws

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543102376 ## core/src/test/java/org/apache/iceberg/jdbc/TestJdbcUtil.java: ## @@ -45,4 +47,10 @@ public void testFilterAndRemovePrefix() {

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543101785 ## spark/v3.5/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestViews.java: ## @@ -1428,40 +1428,28 @@ public void showViews() throws

Re: [PR] Build: disable link-check for existing medium blog posts [iceberg]

2024-03-28 Thread via GitHub
nastra commented on PR #10042: URL: https://github.com/apache/iceberg/pull/10042#issuecomment-2025364451 @manuzhang can you open up a PR to fix the remaining links please? Also IMO we should be running this for PRs and on main -- This is an automated message from the Apache Git Service.

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
marvinlanhenke commented on PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#issuecomment-2025357488 @liurenjie1024 I think I covered most of your suggestions. PTAL if the overall design and implementation in general is fine? ### Unresolved Issues: - [ ] How to

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
nastra commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543085683 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -288,6 +288,7 @@ public List listNamespaces(Namespace namespace) throws NoSuchNamespac

Re: [I] iceberg-core: Switch tests to JUnit5 + AssertJ-style assertions [iceberg]

2024-03-28 Thread via GitHub
tomtongue commented on issue #9085: URL: https://github.com/apache/iceberg/issues/9085#issuecomment-2025354178 Yes, the subclasses of `TableTestBase` are related to several packages like Flink, Spark etc. I will create PRs. -- This is an automated message from the Apache Git Service. To

Re: [PR] partitioned write support [iceberg-python]

2024-03-28 Thread via GitHub
jqin61 commented on PR #353: URL: https://github.com/apache/iceberg-python/pull/353#issuecomment-2025344888 As discussed in the monthly meeting, this will be broken down into 4 prs of: 1. Partitioned append with identity transform 2. Dynamic overwrite using delete + append, 2 snapshots

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-03-28 Thread via GitHub
nastra commented on PR #9793: URL: https://github.com/apache/iceberg/pull/9793#issuecomment-2025343375 @igoradulian are you planning on updating this PR? There are a bunch of import changes that would need to be reverted. Also make sure to run `./gradlew spotlessApply` locally. -- This

Re: [PR] #9073 Junit 4 tests switched to JUnit 5 [iceberg]

2024-03-28 Thread via GitHub
nastra commented on code in PR #9793: URL: https://github.com/apache/iceberg/pull/9793#discussion_r1543077889 ## data/src/test/java/org/apache/iceberg/TestGenericAppenderFactory.java: ## @@ -18,13 +18,13 @@ */ package org.apache.iceberg; +import

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543073405 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -288,6 +288,7 @@ public List listNamespaces(Namespace namespace) throws

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543070548 ## core/src/main/java/org/apache/iceberg/jdbc/JdbcUtil.java: ## @@ -468,6 +468,10 @@ private JdbcUtil() {} static Namespace stringToNamespace(String

Re: [PR] Core: Add tests for catalogs supporting empty namespaces [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on code in PR #9890: URL: https://github.com/apache/iceberg/pull/9890#discussion_r1543069444 ## core/src/main/java/org/apache/iceberg/inmemory/InMemoryCatalog.java: ## @@ -288,6 +288,7 @@ public List listNamespaces(Namespace namespace) throws

Re: [PR] Migrate WAP, Metrics and FilterFiles in Core to JUnit5 [iceberg]

2024-03-28 Thread via GitHub
nastra merged PR #10039: URL: https://github.com/apache/iceberg/pull/10039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail:

Re: [PR] Spark: Clarify schema behavior when working with branches [iceberg]

2024-03-28 Thread via GitHub
nastra commented on code in PR #10055: URL: https://github.com/apache/iceberg/pull/10055#discussion_r1543062498 ## docs/docs/spark-queries.md: ## @@ -124,6 +124,27 @@ SELECT * FROM prod.db.table.`tag_historical-snapshot`; Note that the identifier with branch or tag may not

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-03-28 Thread via GitHub
RussellSpitzer commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2025287831 @flyrain You should take a look at this as well -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Hive: Use EnvironmentContext instead of Hive Locks to provide transactional commits after HIVE-26882 [iceberg]

2024-03-28 Thread via GitHub
pvary commented on PR #6570: URL: https://github.com/apache/iceberg/pull/6570#issuecomment-2025231799 @chenwyi2: Maybe we should wait a bit and see if we have a fix and document that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Implement `project` for `Transform`. [iceberg-rust]

2024-03-28 Thread via GitHub
ZENOTME commented on issue #264: URL: https://github.com/apache/iceberg-rust/issues/264#issuecomment-2025194854 > @liurenjie1024 @ZENOTME ...the `fn project(...)` is getting kinda extensive (lengthy). I think it'd be appropriate to introduce some helper functions here instead of writing a

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
liurenjie1024 commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1542937628 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +270,174 @@ impl Transform { _ => self == other, } } + +/// Projects a

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1542923641 ## crates/iceberg/src/spec/values.rs: ## @@ -683,6 +684,47 @@ impl Datum { pub fn data_type() -> { #type } + +/// Create a new

Re: [I] Iceberg Materialized Views [iceberg]

2024-03-28 Thread via GitHub
wmoustafa commented on issue #10043: URL: https://github.com/apache/iceberg/issues/10043#issuecomment-2025131612 Implementation PR for Option 1 in the proposal doc: https://github.com/apache/iceberg/pull/9830. -- This is an automated message from the Apache Git Service. To respond to the

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1542923641 ## crates/iceberg/src/spec/values.rs: ## @@ -683,6 +684,47 @@ impl Datum { pub fn data_type() -> { #type } + +/// Create a new

Re: [PR] feat: Project transform [iceberg-rust]

2024-03-28 Thread via GitHub
marvinlanhenke commented on code in PR #309: URL: https://github.com/apache/iceberg-rust/pull/309#discussion_r1542916299 ## crates/iceberg/src/spec/transform.rs: ## @@ -261,6 +270,174 @@ impl Transform { _ => self == other, } } + +/// Projects a

  1   2   >