Re: [PR] Core: Make namespace separator configurable [iceberg]

2024-08-12 Thread via GitHub
nastra commented on PR #10877: URL: https://github.com/apache/iceberg/pull/10877#issuecomment-2285442828 @jackye1995 could you take a look at this PR please? It would be great to get this in, so that we can can continue with https://github.com/apache/iceberg/pull/10904 / https://github.com

Re: [I] Check if dependencies in libs.versions.toml are the latest supported ones for JDK11 [iceberg]

2024-08-12 Thread via GitHub
nastra commented on issue #10852: URL: https://github.com/apache/iceberg/issues/10852#issuecomment-2285406734 @imneerajsharma thanks for looking into this. While 1 and 2 sound good, I don't think we can actually enforce 3 as I don't see any option to tell dependabot to only look for depende

Re: [PR] Support convert orc timestamptz [iceberg]

2024-08-12 Thread via GitHub
ming95 commented on PR #9905: URL: https://github.com/apache/iceberg/pull/9905#issuecomment-2285382336 > > > @ming95: What happens if someone writes to this ORC table some new rows after migration? > > > @pvary > > > > > > Sorry I've had a recent job change, so I don't

[I] Clean up TODOs in the codebase [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu opened a new issue, #1052: URL: https://github.com/apache/iceberg-python/issues/1052 ### Feature Request / Improvement Opening this to track a few todos I've seen around the codebase. For example, https://github.com/search?q=repo%3Aapache%2Ficeberg-python+github.com

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-12 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1714692537 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,213 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-12 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1714691005 ## data/src/main/java/org/apache/iceberg/data/PartitionStatsWriterUtil.java: ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under o

Re: [PR] Support convert orc timestamptz [iceberg]

2024-08-12 Thread via GitHub
pvary commented on PR #9905: URL: https://github.com/apache/iceberg/pull/9905#issuecomment-2285375168 > > @ming95: What happens if someone writes to this ORC table some new rows after migration? > > @pvary > > Sorry I've had a recent job change, so I don't have an environment t

Re: [PR] Support convert orc timestamptz [iceberg]

2024-08-12 Thread via GitHub
pvary commented on code in PR #9905: URL: https://github.com/apache/iceberg/pull/9905#discussion_r1714689354 ## orc/src/test/java/org/apache/iceberg/orc/TestExpressionToSearchArgument.java: ## @@ -396,6 +402,7 @@ public void testModifiedSimpleSchemaNameMapping() { @Test

Re: [PR] Support convert orc timestamptz [iceberg]

2024-08-12 Thread via GitHub
pvary commented on code in PR #9905: URL: https://github.com/apache/iceberg/pull/9905#discussion_r1714688639 ## orc/src/test/java/org/apache/iceberg/orc/TestExpressionToSearchArgument.java: ## @@ -332,6 +334,7 @@ public void testEvolvedSchema() { @Test public void testOr

Re: [PR] Support convert orc timestamptz [iceberg]

2024-08-12 Thread via GitHub
pvary commented on code in PR #9905: URL: https://github.com/apache/iceberg/pull/9905#discussion_r1714689121 ## orc/src/test/java/org/apache/iceberg/orc/TestExpressionToSearchArgument.java: ## @@ -358,6 +363,7 @@ public void testOriginalSchemaNameMapping() { @Test public

Re: [PR] Data: Add a util to read write partition stats [iceberg]

2024-08-12 Thread via GitHub
ajantha-bhat commented on code in PR #10176: URL: https://github.com/apache/iceberg/pull/10176#discussion_r1714682727 ## core/src/main/java/org/apache/iceberg/PartitionStatsUtil.java: ## @@ -0,0 +1,240 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or

Re: [PR] Simplify PrimitiveLiteral [iceberg-rust]

2024-08-12 Thread via GitHub
liurenjie1024 merged PR #502: URL: https://github.com/apache/iceberg-rust/pull/502 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Core: create an empty Hadoop config if not provided in constructor [iceberg]

2024-08-12 Thread via GitHub
stevenzwu commented on code in PR #10926: URL: https://github.com/apache/iceberg/pull/10926#discussion_r1714643866 ## core/src/main/java/org/apache/iceberg/hadoop/HadoopFileIO.java: ## @@ -63,7 +63,11 @@ public class HadoopFileIO implements HadoopConfigurable, DelegateFileIO {

[PR] Core: create an empty Hadoop config if not provided in constructor [iceberg]

2024-08-12 Thread via GitHub
stevenzwu opened a new pull request, #10926: URL: https://github.com/apache/iceberg/pull/10926 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

[PR] Core: (unit test) Set partition to the right PartitionKey [iceberg]

2024-08-12 Thread via GitHub
hsiang-c opened a new pull request, #10925: URL: https://github.com/apache/iceberg/pull/10925 - Found a few minor typos where the right partition key is not set. Coincidentally, they don't break unit tests so it is a bit tricky to detect. - Fix comments on partition values -- This is

Re: [PR] Spark: Add CopyTable action interface [iceberg]

2024-08-12 Thread via GitHub
manuzhang commented on PR #10920: URL: https://github.com/apache/iceberg/pull/10920#issuecomment-2285284628 I'm a bit lost in the context. If I run this action without copying the table (e.g. rewrite `s3` to `gcs`), isn't it in a corrupt status? -- This is an automated message from the Ap

Re: [PR] Support convert orc timestamptz [iceberg]

2024-08-12 Thread via GitHub
ming95 commented on PR #9905: URL: https://github.com/apache/iceberg/pull/9905#issuecomment-2285146229 > @ming95: What happens if someone writes to this ORC table some new rows after migration? @pvary Sorry I've had a recent job change, so I don't have an environment to te

Re: [PR] core: support support move a column with same name after rename column [iceberg]

2024-08-12 Thread via GitHub
FANNG1 commented on PR #10862: URL: https://github.com/apache/iceberg/pull/10862#issuecomment-2285136976 @RussellSpitzer @Fokko do you have time to reivew this PR? thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and us

Re: [I] Getting Original Schema of a DataFile in a FileScanTask? [iceberg-python]

2024-08-12 Thread via GitHub
github-actions[bot] commented on issue #401: URL: https://github.com/apache/iceberg-python/issues/401#issuecomment-2285113501 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity oc

Re: [I] whether flink.actions.RewriteDataFilesAction does not implement option [iceberg]

2024-08-12 Thread via GitHub
github-actions[bot] commented on issue #4970: URL: https://github.com/apache/iceberg/issues/4970#issuecomment-2285111603 This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs.

Re: [PR] Add watermark in flink table [iceberg]

2024-08-12 Thread via GitHub
github-actions[bot] commented on PR #3681: URL: https://github.com/apache/iceberg/pull/3681#issuecomment-2285110699 This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If y

Re: [PR] Add watermark in flink table [iceberg]

2024-08-12 Thread via GitHub
github-actions[bot] closed pull request #3681: Add watermark in flink table URL: https://github.com/apache/iceberg/pull/3681 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
danielcweeks commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714476750 ## open-api/src/test/java/org/apache/iceberg/rest/RESTCompatibilityKitCatalogTests.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
danielcweeks commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714478569 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RCKUtils.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [I] Support for timestamp downcasting when loading data to iceberg tables [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu commented on issue #1045: URL: https://github.com/apache/iceberg-python/issues/1045#issuecomment-2285080564 Here's the setup I currently use * Git clone `apache/iceberg-python` repo (this repo) * Fork the repo (`kevinjqliu/iceberg-python`) * Add the forked repo as a remot

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
danielcweeks commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714476750 ## open-api/src/test/java/org/apache/iceberg/rest/RESTCompatibilityKitCatalogTests.java: ## @@ -0,0 +1,77 @@ +/* + * Licensed to the Apache Software Foundation (

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
danielcweeks commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714476101 ## build.gradle: ## @@ -952,6 +952,36 @@ project(':iceberg-snowflake') { } project(':iceberg-open-api') { + apply plugin: 'java-test-fixtures' Review Comme

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
danielcweeks commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714474602 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RCKUtils.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
danielcweeks commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714474262 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RCKUtils.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2285052713 > you're benchmarking the fsspec FileIO path in pyiceberg, which if I understand correctly is using fsspec s3fs directly with a lot of defaults. Probably it keeps the defa

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714462484 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1452 @@ +/* + * Licensed to the Apache Software

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2285046586 > I have one more question regarding the read_parquet from awswrangler. Do you know why it's faster than the other methods? Is there any optimization on their end or som

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714454918 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1452 @@ +/* + * Licensed to the Apache Software

Re: [I] SqlCatalog and other catalog works different on load_namespace_properties [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu commented on issue #769: URL: https://github.com/apache/iceberg-python/issues/769#issuecomment-2285038209 > In the documentation of the abstract method list_namespace_properties there is a reference to an exception being thrown if the namespace does not exist, but the current imp

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714447553 ## tests/integration/test_add_files.py: ## @@ -732,3 +732,76 @@ def test_add_files_subset_of_schema(spark: SparkSession, session_catalog: Catalo for col

[PR] Test: Add rowDelete test in TestChangeLogReader [iceberg]

2024-08-12 Thread via GitHub
dramaticlly opened a new pull request, #10924: URL: https://github.com/apache/iceberg/pull/10924 Refactor and improve test for TestChangeLogReader - added test case for use CDC for reading MOR table - simplified row sorting with Comparator - extracted common logic into rowFromScan

[PR] Bump pyspark from 3.5.1 to 3.5.2 [iceberg-python]

2024-08-12 Thread via GitHub
dependabot[bot] opened a new pull request, #1048: URL: https://github.com/apache/iceberg-python/pull/1048 Bumps [pyspark](https://github.com/apache/spark) from 3.5.1 to 3.5.2. Commits https://github.com/apache/spark/commit/bb7846dd487f259994fdc69e18e03382e3f64f42";>bb7846d Prep

[PR] Bump deptry from 0.19.0 to 0.19.1 [iceberg-python]

2024-08-12 Thread via GitHub
dependabot[bot] opened a new pull request, #1047: URL: https://github.com/apache/iceberg-python/pull/1047 Bumps [deptry](https://github.com/fpgmaas/deptry) from 0.19.0 to 0.19.1. Release notes Sourced from https://github.com/fpgmaas/deptry/releases";>deptry's releases. 0.19.

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
amitgilad3 commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714418419 ## pyiceberg/table/__init__.py: ## @@ -630,7 +630,20 @@ def add_files(self, file_paths: List[str], snapshot_properties: Dict[str, str] = Raises:

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714417771 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1452 @@ +/* + * Licensed to the Apache Software

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714415965 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1452 @@ +/* + * Licensed to the Apache Software

Re: [PR] Build: Bump software.amazon.awssdk:bom from 2.26.29 to 2.27.2 [iceberg]

2024-08-12 Thread via GitHub
Fokko merged PR #10913: URL: https://github.com/apache/iceberg/pull/10913 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump com.google.cloud:libraries-bom from 26.43.0 to 26.44.0 [iceberg]

2024-08-12 Thread via GitHub
Fokko merged PR #10916: URL: https://github.com/apache/iceberg/pull/10916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump org.apache.commons:commons-compress from 1.26.2 to 1.27.0 [iceberg]

2024-08-12 Thread via GitHub
Fokko merged PR #10914: URL: https://github.com/apache/iceberg/pull/10914 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [I] Create AWS Glue table from table JSON [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu commented on issue #1025: URL: https://github.com/apache/iceberg-python/issues/1025#issuecomment-2284966898 > I think it would be handy if we could put table schemas json into config file, and simply load them if needed. What would that look like? I'm not too familiar with

Re: [PR] [bug] fix reading with `to_arrow_batch_reader` and `limit` [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu merged PR #1042: URL: https://github.com/apache/iceberg-python/pull/1042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@i

Re: [I] [bug] `to_arrow_batch_reader` does not respect the given limit, returning more records than specified [iceberg-python]

2024-08-12 Thread via GitHub
kevinjqliu closed issue #1040: [bug] `to_arrow_batch_reader` does not respect the given limit, returning more records than specified URL: https://github.com/apache/iceberg-python/issues/1040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2024-08-12 Thread via GitHub
SandeepSinghGahir commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-2284961922 As a workaround, I'm now saving dataframe into an iceberg table and not seeing this issue. -- This is an automated message from the Apache Git Service. To respond to the

Re: [I] javax.net.ssl.SSLException: Connection reset on S3 w/ S3FileIO and Apache HTTP client [iceberg]

2024-08-12 Thread via GitHub
SandeepSinghGahir commented on issue #10340: URL: https://github.com/apache/iceberg/issues/10340#issuecomment-2284958450 Hi, This issue/bug has been open for a while now. Do we know when can we expect a fix? Or is there any workaround? Background: I'm joining multiple iceberg tabl

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2024-08-12 Thread via GitHub
SandeepSinghGahir commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-2284956384 Please find the email below. On Monday, August 12, 2024 at 02:42:06 PM PDT, §æ § ***@***.***> wrote: Hi, This issue/bug has been open for a while now -

Re: [I] Unable to merge CDC data into snapshot data. java.lang.ClassCastException: org.apache.spark.unsafe.types.UTF8String cannot be cast to java.lang.Long [iceberg]

2024-08-12 Thread via GitHub
SandeepSinghGahir commented on issue #8333: URL: https://github.com/apache/iceberg/issues/8333#issuecomment-2284955150 Hi, This issue/bug has been open for a while now ->  https://github.com/apache/iceberg/issues/10340 Do we know when can we expect a fix? Or is there any workaround?

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
sungwy commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714392026 ## pyiceberg/table/__init__.py: ## @@ -630,7 +630,20 @@ def add_files(self, file_paths: List[str], snapshot_properties: Dict[str, str] = Raises:

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
sungwy commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714391548 ## pyiceberg/table/__init__.py: ## @@ -630,7 +630,20 @@ def add_files(self, file_paths: List[str], snapshot_properties: Dict[str, str] = Raises:

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
amitgilad3 commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714382905 ## pyiceberg/table/__init__.py: ## @@ -621,6 +621,13 @@ def delete(self, delete_filter: Union[str, BooleanExpression], snapshot_properti if not dele

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
amitgilad3 commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714382905 ## pyiceberg/table/__init__.py: ## @@ -621,6 +621,13 @@ def delete(self, delete_filter: Union[str, BooleanExpression], snapshot_properti if not dele

Re: [PR] prevent adding duplicate files [iceberg-python]

2024-08-12 Thread via GitHub
sungwy commented on code in PR #1036: URL: https://github.com/apache/iceberg-python/pull/1036#discussion_r1714372318 ## pyiceberg/table/__init__.py: ## @@ -621,6 +621,13 @@ def delete(self, delete_filter: Union[str, BooleanExpression], snapshot_properti if not delete_s

Re: [I] Check if dependencies in libs.versions.toml are the latest supported ones for JDK11 [iceberg]

2024-08-12 Thread via GitHub
imneerajsharma commented on issue #10852: URL: https://github.com/apache/iceberg/issues/10852#issuecomment-2284906414 Hi team, I’m working on the issue where Dependabot suggests updates without considering the JDK version used in our project. Since Iceberg is now running on JDK 11+,

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714330295 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1452 @@ +/* + * Licensed to the Apache Software

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714320355 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/sink/committer/TestIcebergCommitter.java: ## @@ -0,0 +1,1452 @@ +/* + * Licensed to the Apache Software

Re: [I] SqlCatalog and other catalog works different on load_namespace_properties [iceberg-python]

2024-08-12 Thread via GitHub
soumya-ghosh commented on issue #769: URL: https://github.com/apache/iceberg-python/issues/769#issuecomment-2284821254 @kevinjqliu Seems like behavior of SqlCatalog for load_namespace_properties is rectified in current version. I got it fixed in PR - https://github.com/apache/iceberg-pyth

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714305902 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,761 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714293215 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/ManifestOutputFileFactory.java: ## @@ -41,6 +42,19 @@ class ManifestOutputFileFactory { private

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
sdd commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1714292148 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,344 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Execute st

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
sdd commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1714279852 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,344 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Execute st

Re: [PR] Add REST Compatibility Kit [iceberg]

2024-08-12 Thread via GitHub
dimas-b commented on code in PR #10908: URL: https://github.com/apache/iceberg/pull/10908#discussion_r1714266569 ## open-api/src/testFixtures/java/org/apache/iceberg/rest/RCKUtils.java: ## @@ -0,0 +1,116 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * o

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
sdd commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1714277890 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,344 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Execute st

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
sdd commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1714265481 ## crates/catalog/sql/src/catalog.rs: ## @@ -167,43 +177,344 @@ impl SqlCatalog { .await .map_err(from_sqlx_error) } + +/// Execute st

Re: [PR] feat(storage): support aws session token [iceberg-rust]

2024-08-12 Thread via GitHub
Xuanwo merged PR #530: URL: https://github.com/apache/iceberg-rust/pull/530 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.a

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
sdd commented on PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#issuecomment-2284722772 I'm also ok with this being SQL rather than Diesel. Diesel feels like a large dependency to bring in for a small number of quite straightforward queries. -- This is an automated message

Re: [I] FileIO S3: accept session_token [iceberg-rust]

2024-08-12 Thread via GitHub
Xuanwo closed issue #528: FileIO S3: accept session_token URL: https://github.com/apache/iceberg-rust/issues/528 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe,

Re: [PR] Docs: Add Trademark symbol where appropriate [iceberg]

2024-08-12 Thread via GitHub
RussellSpitzer commented on PR #10921: URL: https://github.com/apache/iceberg/pull/10921#issuecomment-2284697719 Thanks @Fokko and @singhpk234 for Review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [PR] Docs: Add Trademark symbol where appropriate [iceberg]

2024-08-12 Thread via GitHub
RussellSpitzer merged PR #10921: URL: https://github.com/apache/iceberg/pull/10921 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@ic

Re: [PR] Access delegation [iceberg-python]

2024-08-12 Thread via GitHub
guitcastro commented on PR #1033: URL: https://github.com/apache/iceberg-python/pull/1033#issuecomment-2284664084 @Fokko done :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comme

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714189646 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,811 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Introduces the new IcebergSink based on the new V2 Flink Sink Abstraction [iceberg]

2024-08-12 Thread via GitHub
rodmeneses commented on code in PR #10179: URL: https://github.com/apache/iceberg/pull/10179#discussion_r1714184380 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/sink/IcebergSink.java: ## @@ -0,0 +1,811 @@ +/* + * Licensed to the Apache Software Foundation (ASF) un

Re: [PR] Spark: Add CopyTable action interface [iceberg]

2024-08-12 Thread via GitHub
flyrain commented on PR #10920: URL: https://github.com/apache/iceberg/pull/10920#issuecomment-2284587459 > Isn't an implementation of this interface actually copying the table? NO. I think it makes sense to go with a name like `RewriteFileLocation`, or `RewriteFilePaths` -- This is an

Re: [PR] Spark: Add CopyTable action interface [iceberg]

2024-08-12 Thread via GitHub
flyrain commented on PR #10920: URL: https://github.com/apache/iceberg/pull/10920#issuecomment-2284579356 cc @szehon-ho @huaxingao -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific co

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
callum-ryan commented on PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#issuecomment-2284572219 > Hi, I need more time to review the SQL individually. Here are some additional comments that I believe should take priority. > > By the way, have you considered using `diese

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1714153827 ## crates/catalog/sql/src/catalog.rs: ## @@ -141,21 +142,24 @@ impl SqlCatalog { } /// SQLX Any does not implement PostgresSQL bindings, so we have t

Re: [PR] feat: SQL Catalog - namespaces [iceberg-rust]

2024-08-12 Thread via GitHub
callum-ryan commented on code in PR #534: URL: https://github.com/apache/iceberg-rust/pull/534#discussion_r1714153256 ## crates/catalog/sql/src/catalog.rs: ## @@ -141,21 +142,24 @@ impl SqlCatalog { } /// SQLX Any does not implement PostgresSQL bindings, so we have t

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-12 Thread via GitHub
c-thiel commented on PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#issuecomment-2284445481 @liurenjie1024 ready for another round! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

Re: [I] Peformance question for to_arrow, to_pandas, to_duckdb [iceberg-python]

2024-08-12 Thread via GitHub
jkleinkauff commented on issue #1032: URL: https://github.com/apache/iceberg-python/issues/1032#issuecomment-2284442737 @kevinjqliu that's awesome! Thank you so much ! I have one more question regarding the **read_parquet** from awswrangler. Do you know why it's faster than the other

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-12 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1714069854 ## crates/iceberg/src/spec/partition.rs: ## @@ -136,6 +145,427 @@ impl UnboundPartitionSpec { } } +impl From for UnboundPartitionField { +fn from(field:

Re: [PR] Build: Bump nessie from 0.94.4 to 0.95.0 [iceberg]

2024-08-12 Thread via GitHub
Fokko merged PR #10910: URL: https://github.com/apache/iceberg/pull/10910 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Build: Bump datamodel-code-generator from 0.25.8 to 0.25.9 [iceberg]

2024-08-12 Thread via GitHub
Fokko merged PR #10917: URL: https://github.com/apache/iceberg/pull/10917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apa

Re: [PR] Spark: Add CopyTable action interface [iceberg]

2024-08-12 Thread via GitHub
laithalzyoud commented on PR #10920: URL: https://github.com/apache/iceberg/pull/10920#issuecomment-2284319501 > > Since it's just rewriting metadata, manifest and position delete files with the new location prefix and not actually copying the table > > Isn't an implementation of this

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-12 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1713998496 ## crates/iceberg/src/spec/partition.rs: ## @@ -136,6 +145,427 @@ impl UnboundPartitionSpec { } } +impl From for UnboundPartitionField { +fn from(field:

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-12 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1713993768 ## crates/iceberg/src/spec/partition.rs: ## @@ -136,6 +145,427 @@ impl UnboundPartitionSpec { } } +impl From for UnboundPartitionField { +fn from(field:

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-12 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1713990348 ## crates/iceberg/src/spec/partition.rs: ## @@ -136,6 +145,427 @@ impl UnboundPartitionSpec { } } +impl From for UnboundPartitionField { +fn from(field:

Re: [PR] feat: Partition Binding and safe PartitionSpecBuilder [iceberg-rust]

2024-08-12 Thread via GitHub
c-thiel commented on code in PR #491: URL: https://github.com/apache/iceberg-rust/pull/491#discussion_r1713982541 ## crates/iceberg/src/spec/partition.rs: ## @@ -44,22 +44,27 @@ pub struct PartitionField { pub transform: Transform, } +impl PartitionField { +/// To un

Re: [PR] Fix tracing existing entries when there are deletes [iceberg-python]

2024-08-12 Thread via GitHub
Fokko commented on code in PR #1046: URL: https://github.com/apache/iceberg-python/pull/1046#discussion_r1713980714 ## pyiceberg/table/__init__.py: ## @@ -3379,13 +3379,14 @@ def _copy_with_new_status(entry: ManifestEntry, status: ManifestEntryStatus) ->

Re: [PR] Fix tracing existing entries when there are deletes [iceberg-python]

2024-08-12 Thread via GitHub
Fokko commented on PR #1046: URL: https://github.com/apache/iceberg-python/pull/1046#issuecomment-2284270583 @sungwy Thanks for the quick follow-up, appreciate it 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

Re: [PR] Access delegation [iceberg-python]

2024-08-12 Thread via GitHub
Fokko commented on PR #1033: URL: https://github.com/apache/iceberg-python/pull/1033#issuecomment-2284256995 @guitcastro Can you rebase this one, this looks good to me 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-12 Thread via GitHub
Fokko commented on PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029#issuecomment-2284241240 Thanks @guitcastro for picking this up, and thanks @ndrluis and @kevinjqliu for the review 🙌 -- This is an automated message from the Apache Git Service. To respond to the message

Re: [PR] Support s3.signer.endpoint for nessie [iceberg-python]

2024-08-12 Thread via GitHub
Fokko merged PR #1029: URL: https://github.com/apache/iceberg-python/pull/1029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceber

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-08-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1713959444 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TagBasedLockFactory.java: ## @@ -0,0 +1,186 @@ +/* + * Licensed to the Apache Software

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-08-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1713958997 ## flink/v1.19/flink/src/test/java/org/apache/iceberg/flink/maintenance/operator/TestTagBasedLockFactory.java: ## @@ -0,0 +1,165 @@ +/* + * Licensed to the Apache Softw

Re: [PR] Flink: Maintenance - TriggerManager [iceberg]

2024-08-12 Thread via GitHub
pvary commented on code in PR #10484: URL: https://github.com/apache/iceberg/pull/10484#discussion_r1713948194 ## flink/v1.19/flink/src/main/java/org/apache/iceberg/flink/maintenance/operator/TriggerManager.java: ## @@ -0,0 +1,308 @@ +/* + * Licensed to the Apache Software Found

Re: [PR] Spark: Add CopyTable action interface [iceberg]

2024-08-12 Thread via GitHub
manuzhang commented on PR #10920: URL: https://github.com/apache/iceberg/pull/10920#issuecomment-2284213638 > Since it's just rewriting metadata, manifest and position delete files with the new location prefix and not actually copying the table Isn't an implementation of this interfa

Re: [PR] Spark: Add CopyTable spark action [iceberg]

2024-08-12 Thread via GitHub
laithalzyoud commented on PR #10024: URL: https://github.com/apache/iceberg/pull/10024#issuecomment-2284175571 I created the [PR](https://github.com/apache/iceberg/pull/10920) to just add the interface, please feel to review it and provide feedback! -- This is an automated message from th

  1   2   >