Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-16 Thread via GitHub
wmoustafa commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1454299740 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -69,62 +99,108 @@ public void setRowPositionSupplier(Supplier posSupplier) { @Overri

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on PR #9366: URL: https://github.com/apache/iceberg/pull/9366#issuecomment-1873441376 Thanks for reviewing, @Fokko! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific c

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue merged PR #9366: URL: https://github.com/apache/iceberg/pull/9366 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apac

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
Fokko commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439066996 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -381,6 +550,45 @@ public BigDecimal read(Decoder decoder, Object ignored) throws IOException {

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r143909 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -381,6 +550,45 @@ public BigDecimal read(Decoder decoder, Object ignored) throws IOException {

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439066126 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -381,6 +550,45 @@ public BigDecimal read(Decoder decoder, Object ignored) throws IOException {

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439064941 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -78,7 +79,8 @@ public CloseableIterator iterator() { if (start != null) { if (reader i

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439065448 ## core/src/main/java/org/apache/iceberg/avro/GenericAvroReader.java: ## @@ -69,62 +99,107 @@ public void setRowPositionSupplier(Supplier posSupplier) { @Override

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439064941 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -78,7 +79,8 @@ public CloseableIterator iterator() { if (start != null) { if (reader i

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2024-01-01 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1439064577 ## core/src/main/java/org/apache/iceberg/avro/Avro.java: ## @@ -683,23 +698,34 @@ public ReadBuilder classLoader(ClassLoader classLoader) { return this; }

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-27 Thread via GitHub
Fokko commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1436375930 ## core/src/main/java/org/apache/iceberg/avro/AvroIterable.java: ## @@ -78,7 +79,8 @@ public CloseableIterator iterator() { if (start != null) { if (reader in

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-22 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435370739 ## core/src/test/java/org/apache/iceberg/avro/TestAvroNameMapping.java: ## @@ -80,10 +80,7 @@ public void testMapProjections() throws IOException { Record project

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-22 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435370465 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -739,17 +956,18 @@ protected void set(R struct, int pos, Object value) { } } - static c

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-22 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435370012 ## core/src/main/java/org/apache/iceberg/avro/ValueReaders.java: ## @@ -564,6 +678,115 @@ public Map read(Decoder decoder, Object reuse) throws IOException { }

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-22 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435369249 ## core/src/main/java/org/apache/iceberg/avro/ValueReader.java: ## @@ -23,4 +23,8 @@ public interface ValueReader { T read(Decoder decoder, Object reuse) throws IO

Re: [PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-22 Thread via GitHub
rdblue commented on code in PR #9366: URL: https://github.com/apache/iceberg/pull/9366#discussion_r1435368996 ## core/src/main/java/org/apache/iceberg/avro/SupportsCustomRecords.java: ## @@ -0,0 +1,33 @@ +/* + * + * * Licensed to the Apache Software Foundation (ASF) under one +

[PR] Core: Refactor internal Avro reader to resolve schemas directly [iceberg]

2023-12-22 Thread via GitHub
rdblue opened a new pull request, #9366: URL: https://github.com/apache/iceberg/pull/9366 This refactors the Avro generic reader so that it resolves schemas directly (like PyIceberg) rather than creating an Avro schema to trick Avro's `ResolvingDecoder` into projecting columns correctly.