Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1822113350 Thanks for your patience, @przemekd! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi merged PR #9029: URL: https://github.com/apache/iceberg/pull/9029 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

przemekd commented on code in PR #9029: URL: https://github.com/apache/iceberg/pull/9029#discussion_r1401297380 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -237,7 +236,14 @@ public EncryptionManager encryption() { @Override public LocationProvi

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on code in PR #9029: URL: https://github.com/apache/iceberg/pull/9029#discussion_r1401282710 ## core/src/main/java/org/apache/iceberg/SerializableTable.java: ## @@ -237,7 +236,14 @@ public EncryptionManager encryption() { @Override public LocationPr

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1821791818 You are right, the data analyst doesn't have to create new locations in that case. I am convinced now. Let me review the change itself. -- This is an automated message from the A

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

przemekd commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1820779641 @aokolnychyi We assume the data analyst uses a completely separate cluster, JAR is not populated on executors and not supplied with the job. With the fix provided she still can do her jo

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1820198393 Does the data analyst connect to a remote already running cluster? Do we assume the jar is populated on executors but not supplied by the analyst in the local shell? -- This is an

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

przemekd commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1814274741 @aokolnychyi Sure. So, the problem is shortly described in the linked issue: https://github.com/apache/iceberg/issues/9025 I am not directly creating `SerializableTable` instance in m

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1813526621 I am not sure I entirely understood how `SerializableTable` instances are created in your case. Is it created from the primary table before sending the table info to remote nodes?

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

przemekd commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1807918280 @aokolnychyi Sure. This one is not about serialization. The idea behind this fix is to allow to query (read) tables in Spark (and potentially other query engines using `SerializableTable

Re: [PR] Core: lazily create locationProvider in SerializableTable [iceberg]

aokolnychyi commented on PR #9029: URL: https://github.com/apache/iceberg/pull/9029#issuecomment-1806884352 @przemekd, could you elaborate a bit on the custom `LocationProvider`? All implementations must be `Serializable`. Are you seeing issues with Java or Kryo serialization? -- This is