[GitHub] [incubator-iceberg] chenjunjiedada closed pull request #786: replace SparkDataFile with DataFile

chenjunjiedada closed pull request #786: replace SparkDataFile with DataFile URL: https://github.com/apache/incubator-iceberg/pull/786 This is an automated message from the Apache Git Service. To respond to the message, pleas

[GitHub] [incubator-iceberg] chenjunjiedada opened a new pull request #786: replace SparkDataFile with DataFile

chenjunjiedada opened a new pull request #786: replace SparkDataFile with DataFile URL: https://github.com/apache/incubator-iceberg/pull/786 This fixes https://github.com/apache/incubator-iceberg/issues/763 This is an automat

[GitHub] [incubator-iceberg] shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues

shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#discussion_r378502028 ## File path: data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java

[GitHub] [incubator-iceberg] shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues

shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#discussion_r378501690 ## File path: data/src/main/java/org/apache/iceberg/data/orc/GenericOrcReader.java

[GitHub] [incubator-iceberg] shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues

shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#discussion_r378501824 ## File path: data/src/main/java/org/apache/iceberg/data/orc/GenericOrcWriter.java

[GitHub] [incubator-iceberg] shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues

shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#discussion_r378501653 ## File path: data/src/main/java/org/apache/iceberg/data/orc/GenericOrcReader.java

[GitHub] [incubator-iceberg] shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues

shardulm94 commented on a change in pull request #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#discussion_r378475886 ## File path: data/src/main/java/org/apache/iceberg/data/orc/GenericOrcReader.java

[GitHub] [incubator-iceberg] vrozov commented on a change in pull request #784: Allow caller to construct HadoopInputFile and HadoopOutputFile using an existing instance of FileSystem object.

vrozov commented on a change in pull request #784: Allow caller to construct HadoopInputFile and HadoopOutputFile using an existing instance of FileSystem object. URL: https://github.com/apache/incubator-iceberg/pull/784#discussion_r378465876 ## File path: core/src/main/java/org/ap

[GitHub] [incubator-iceberg] rdblue commented on issue #751: Add an option to decide whether to delete data files in Catalog.dropTable()

rdblue commented on issue #751: Add an option to decide whether to delete data files in Catalog.dropTable() URL: https://github.com/apache/incubator-iceberg/issues/751#issuecomment-585359978 I think that keeping both data and metadata is the correct behavior. That way, drop removes a refe

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #784: Allow caller to construct HadoopInputFile and HadoopOutputFile using an existing instance of FileSystem object.

rdblue commented on a change in pull request #784: Allow caller to construct HadoopInputFile and HadoopOutputFile using an existing instance of FileSystem object. URL: https://github.com/apache/incubator-iceberg/pull/784#discussion_r378425260 ## File path: core/src/main/java/org/ap

[GitHub] [incubator-iceberg] sudssf commented on issue #792: AvroSchemaUtil.toIceberg does not process logicalType during schema conversion

sudssf commented on issue #792: AvroSchemaUtil.toIceberg does not process logicalType during schema conversion URL: https://github.com/apache/incubator-iceberg/issues/792#issuecomment-585338034 Thank you @rdsr changes you suggested are correct way to write schema. I will close this issue

[GitHub] [incubator-iceberg] sudssf closed issue #792: AvroSchemaUtil.toIceberg does not process logicalType during schema conversion

sudssf closed issue #792: AvroSchemaUtil.toIceberg does not process logicalType during schema conversion URL: https://github.com/apache/incubator-iceberg/issues/792 This is an automated message from the Apache Git Service. T

[GitHub] [incubator-iceberg] rdsr edited a comment on issue #778: ORC: Implement TestGenericData and fix reader and writer issues

rdsr edited a comment on issue #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#issuecomment-585294292 >However, another approach can be to just store the timestamp forced to the UTC timezone as a "Timestamp Ins

[GitHub] [incubator-iceberg] Fokko commented on issue #795: Support from Python 3.6+

Fokko commented on issue #795: Support from Python 3.6+ URL: https://github.com/apache/incubator-iceberg/pull/795#issuecomment-585322587 Merged master into the branch, it should be green now 🤞 This is an automated message fro

[GitHub] [incubator-iceberg] rdblue commented on issue #795: Support from Python 3.6+

rdblue commented on issue #795: Support from Python 3.6+ URL: https://github.com/apache/incubator-iceberg/pull/795#issuecomment-585321700 Reopening to trigger Travis. I tried restarting the python test, but it didn't pick up the fix from #794. --

[GitHub] [incubator-iceberg] Fokko opened a new pull request #795: Support from Python 3.6+

Fokko opened a new pull request #795: Support from Python 3.6+ URL: https://github.com/apache/incubator-iceberg/pull/795 We test against Python 3.6. So it makes sense to support from 3.6 and onwards. This is an automated mess

[GitHub] [incubator-iceberg] rdblue closed pull request #795: Support from Python 3.6+

rdblue closed pull request #795: Support from Python 3.6+ URL: https://github.com/apache/incubator-iceberg/pull/795 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [incubator-iceberg] rdblue commented on issue #793: Updating travis to use bionic

rdblue commented on issue #793: Updating travis to use bionic URL: https://github.com/apache/incubator-iceberg/pull/793#issuecomment-585321090 Looks like the Java test failure is caused by a new Java default version that isn't Java 8. Can we pin to Java 8 in the matrix? ---

[GitHub] [incubator-iceberg] rdblue commented on issue #794: Fix the CI

rdblue commented on issue #794: Fix the CI URL: https://github.com/apache/incubator-iceberg/pull/794#issuecomment-585319948 Huge thanks, @Fokko! This is an automated message from the Apache Git Service. To respond to the messa

[GitHub] [incubator-iceberg] rdblue merged pull request #794: Fix the CI

rdblue merged pull request #794: Fix the CI URL: https://github.com/apache/incubator-iceberg/pull/794 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the UR

[GitHub] [incubator-iceberg] rdblue commented on issue #789: Fix race condition in SnapshotProducer

rdblue commented on issue #789: Fix race condition in SnapshotProducer URL: https://github.com/apache/incubator-iceberg/pull/789#issuecomment-585319156 I think the current solution is a reasonable one. If we want to take the time to get rid of the synchronized block, then initializing this

[GitHub] [incubator-iceberg] vrozov commented on issue #789: Fix race condition in SnapshotProducer

vrozov commented on issue #789: Fix race condition in SnapshotProducer URL: https://github.com/apache/incubator-iceberg/pull/789#issuecomment-585307748 @jun-he I don't think that using `Suppliers.memoize()` avoids double locking, it makes it less explicit, moving double checked locking from

[GitHub] [incubator-iceberg] rdsr edited a comment on issue #778: ORC: Implement TestGenericData and fix reader and writer issues

rdsr edited a comment on issue #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#issuecomment-585294292 >However, another approach can be to just store the timestamp forced to the UTC timezone as a "Timestamp Ins

[GitHub] [incubator-iceberg] rdsr commented on issue #778: ORC: Implement TestGenericData and fix reader and writer issues

rdsr commented on issue #778: ORC: Implement TestGenericData and fix reader and writer issues URL: https://github.com/apache/incubator-iceberg/pull/778#issuecomment-585294292 >However, another approach can be to just store the timestamp forced to the UTC timezone as a "Timestamp Instant" s

[GitHub] [incubator-iceberg] waterlx commented on issue #751: Add an option to decide whether to delete data files in Catalog.dropTable()

waterlx commented on issue #751: Add an option to decide whether to delete data files in Catalog.dropTable() URL: https://github.com/apache/incubator-iceberg/issues/751#issuecomment-585286984 @rdblue got your idea. Thanks! I might need to check with you about the expected behavior of dr

[GitHub] [incubator-iceberg] chenjunjiedada commented on issue #588: [WIP] Add sequence number for supporting row level delete

chenjunjiedada commented on issue #588: [WIP] Add sequence number for supporting row level delete URL: https://github.com/apache/incubator-iceberg/pull/588#issuecomment-585252478 @rdblue @aokolnychyi, Base on the comments, I assume the next steps are - bump the table format version

[GitHub] [incubator-iceberg] Fokko commented on issue #794: Fix the CI

Fokko commented on issue #794: Fix the CI URL: https://github.com/apache/incubator-iceberg/pull/794#issuecomment-585233554 My pleasure @TGooch44 . All green again 👍 This is an automated message from the Apache Git Service. To

[GitHub] [incubator-iceberg] TGooch44 commented on issue #794: Fix the CI

TGooch44 commented on issue #794: Fix the CI URL: https://github.com/apache/incubator-iceberg/pull/794#issuecomment-585211402 @Fokko thanks for taking a look and submitting the PR! This is an automated message from the Apache

[GitHub] [incubator-iceberg] Fokko commented on issue #795: Support from Python 3.6+

Fokko commented on issue #795: Support from Python 3.6+ URL: https://github.com/apache/incubator-iceberg/pull/795#issuecomment-585194414 CI fails because of https://github.com/apache/incubator-iceberg/pull/794 This is an autom

[GitHub] [incubator-iceberg] Fokko commented on issue #793: Updating travis to use bionic

Fokko commented on issue #793: Updating travis to use bionic URL: https://github.com/apache/incubator-iceberg/pull/793#issuecomment-585193694 Related: https://github.com/apache/incubator-iceberg/pull/795 This is an automated m

[GitHub] [incubator-iceberg] Fokko opened a new pull request #795: Support from Python 3.6+

Fokko opened a new pull request #795: Support from Python 3.6+ URL: https://github.com/apache/incubator-iceberg/pull/795 This is an automated message from the Apache Git Service. To respond to the message, please log on to Gi

[GitHub] [incubator-iceberg] Fokko opened a new pull request #794: Fix the CI

Fokko opened a new pull request #794: Fix the CI URL: https://github.com/apache/incubator-iceberg/pull/794 There is a new release of virtualenv that broke the support with Tox: https://pypi.org/project/virtualenv/#history Using the previous version is the best solution for now.

[GitHub] [incubator-iceberg] Fokko commented on issue #793: Updating travis to use bionic

Fokko commented on issue #793: Updating travis to use bionic URL: https://github.com/apache/incubator-iceberg/pull/793#issuecomment-585184764 I would support from 3.6 and higher. At this moment, it is only from 3.4 and onwards: https://github.com/apache/incubator-iceberg/blob/master/python

[GitHub] [incubator-iceberg] openinx commented on issue #788: Integrate the Apache Flink into Apache Iceberg

openinx commented on issue #788: Integrate the Apache Flink into Apache Iceberg URL: https://github.com/apache/incubator-iceberg/issues/788#issuecomment-585167992 @rdblue Thanks for your roadmap confirmation, so the next thing to do would be how will we design and implement it. Indeed, I