[GitHub] [incubator-iceberg] jun-he commented on issue #243: Fix mkdocs and add missing footnotes

2019-07-03 Thread GitBox
jun-he commented on issue #243: Fix mkdocs and add missing footnotes URL: https://github.com/apache/incubator-iceberg/pull/243#issuecomment-508357997 @rdblue thanks for the comment! Yeah, this is a good idea and I have updated the PR accordingly. ---

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #250: Update Spark runtime Jar contents

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #250: Update Spark runtime Jar contents URL: https://github.com/apache/incubator-iceberg/pull/250#discussion_r300161023 ## File path: build.gradle ## @@ -218,8 +218,11 @@ project(':iceberg-core') { dependencies { compile

[GitHub] [incubator-iceberg] rdblue commented on issue #103: Add Table.metadataAsTable

2019-07-03 Thread GitBox
rdblue commented on issue #103: Add Table.metadataAsTable URL: https://github.com/apache/incubator-iceberg/pull/103#issuecomment-508261531 I'm closing this. See #252. This is an automated message from the Apache Git Service. T

[GitHub] [incubator-iceberg] rdblue closed pull request #103: Add Table.metadataAsTable

2019-07-03 Thread GitBox
rdblue closed pull request #103: Add Table.metadataAsTable URL: https://github.com/apache/incubator-iceberg/pull/103 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [incubator-iceberg] aokolnychyi commented on a change in pull request #239: Update iceberg-spark to use HiveTables

2019-07-03 Thread GitBox
aokolnychyi commented on a change in pull request #239: Update iceberg-spark to use HiveTables URL: https://github.com/apache/incubator-iceberg/pull/239#discussion_r300154842 ## File path: hive/src/main/java/org/apache/iceberg/hive/HiveCatalogs.java ## @@ -0,0 +1,40 @@ +/*

[GitHub] [incubator-iceberg] rdblue opened a new pull request #252: Add entries metadata table

2019-07-03 Thread GitBox
rdblue opened a new pull request #252: Add entries metadata table URL: https://github.com/apache/incubator-iceberg/pull/252 This adds a metadata table that uses another table's manifests as data files. This is an easy way to view Iceberg metadata from a query engine. This also adds a

[GitHub] [incubator-iceberg] aokolnychyi opened a new issue #251: Register existing tables in Iceberg HiveCatalog

2019-07-03 Thread GitBox
aokolnychyi opened a new issue #251: Register existing tables in Iceberg HiveCatalog URL: https://github.com/apache/incubator-iceberg/issues/251 Right now, we can keep metadata and data files while dropping tables from Iceberg `HiveCatalog`. However, there is no way to register those table

[GitHub] [incubator-iceberg] rdblue commented on issue #240: Add HiveCatalog implementation

2019-07-03 Thread GitBox
rdblue commented on issue #240: Add HiveCatalog implementation URL: https://github.com/apache/incubator-iceberg/pull/240#issuecomment-508237625 I agree with the idea to add a UUID to table metadata. I've been meaning to do that. Are you suggesting that we purge data but not metadata?

[GitHub] [incubator-iceberg] mccheah commented on issue #194: Add basic arrow schema conversion

2019-07-03 Thread GitBox
mccheah commented on issue #194: Add basic arrow schema conversion URL: https://github.com/apache/incubator-iceberg/pull/194#issuecomment-508237685 Ping on this - are we continuing to make progress here? This is an automated m

[GitHub] [incubator-iceberg] mccheah commented on issue #241: Use gradle-consistent-versions to manage dependencies

2019-07-03 Thread GitBox
mccheah commented on issue #241: Use gradle-consistent-versions to manage dependencies URL: https://github.com/apache/incubator-iceberg/pull/241#issuecomment-508203294 Addressed that. This is an automated message from the Apa

[GitHub] [incubator-iceberg] rdsr commented on a change in pull request #250: Update Spark runtime Jar contents

2019-07-03 Thread GitBox
rdsr commented on a change in pull request #250: Update Spark runtime Jar contents URL: https://github.com/apache/incubator-iceberg/pull/250#discussion_r300094252 ## File path: build.gradle ## @@ -218,8 +218,11 @@ project(':iceberg-core') { dependencies { compile p

[GitHub] [incubator-iceberg] goldentriangle commented on issue #179: Use Iceberg tables as sources for Spark Structured Streaming

2019-07-03 Thread GitBox
goldentriangle commented on issue #179: Use Iceberg tables as sources for Spark Structured Streaming URL: https://github.com/apache/incubator-iceberg/issues/179#issuecomment-508202418 @prodeezy is there an ETA for this feature to be completed? -

[GitHub] [incubator-iceberg] puneetzaroo commented on a change in pull request #248: Add support for returning hive partitions that match a filter express…

2019-07-03 Thread GitBox
puneetzaroo commented on a change in pull request #248: Add support for returning hive partitions that match a filter express… URL: https://github.com/apache/incubator-iceberg/pull/248#discussion_r300093566 ## File path: hive/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.j

[GitHub] [incubator-iceberg] puneetzaroo commented on a change in pull request #248: Add support for returning hive partitions that match a filter express…

2019-07-03 Thread GitBox
puneetzaroo commented on a change in pull request #248: Add support for returning hive partitions that match a filter express… URL: https://github.com/apache/incubator-iceberg/pull/248#discussion_r300093566 ## File path: hive/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.j

[GitHub] [incubator-iceberg] rdsr commented on issue #236: iceberg: don't bundle hdfs with iceberg runtime

2019-07-03 Thread GitBox
rdsr commented on issue #236: iceberg: don't bundle hdfs with iceberg runtime URL: https://github.com/apache/incubator-iceberg/pull/236#issuecomment-508201373 Sounds good to me. Lets review #250 and after #250 is committed, we can do the remaining work here if any.

[GitHub] [incubator-iceberg] aokolnychyi commented on issue #249: Convert from Spark table to Iceberg table

2019-07-03 Thread GitBox
aokolnychyi commented on issue #249: Convert from Spark table to Iceberg table URL: https://github.com/apache/incubator-iceberg/issues/249#issuecomment-508187961 [Here](https://lists.apache.org/thread.html/0f55e50fe48c15a55b31277b48e70500c642ea50cf8d6a3f12dfafe4@%3Cdev.iceberg.apache.org%3

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #246: Support dynamic partition overwrite

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #246: Support dynamic partition overwrite URL: https://github.com/apache/incubator-iceberg/pull/246#discussion_r300075364 ## File path: spark/src/test/java/org/apache/iceberg/spark/source/TestParquetWrite.java ## @@ -114,4 +114

[GitHub] [incubator-iceberg] aokolnychyi commented on a change in pull request #250: Update Spark runtime Jar contents

2019-07-03 Thread GitBox
aokolnychyi commented on a change in pull request #250: Update Spark runtime Jar contents URL: https://github.com/apache/incubator-iceberg/pull/250#discussion_r300072552 ## File path: build.gradle ## @@ -218,8 +218,11 @@ project(':iceberg-core') { dependencies { co

[GitHub] [incubator-iceberg] aokolnychyi commented on a change in pull request #246: Support dynamic partition overwrite

2019-07-03 Thread GitBox
aokolnychyi commented on a change in pull request #246: Support dynamic partition overwrite URL: https://github.com/apache/incubator-iceberg/pull/246#discussion_r300069138 ## File path: spark/src/test/java/org/apache/iceberg/spark/source/TestParquetWrite.java ## @@ -114,4

[GitHub] [incubator-iceberg] rdblue commented on issue #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on issue #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#issuecomment-508179219 Thanks for the updates, @edgarRd! I think this is getting close to merging. This is an automat

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300068351 ## File path: orc/src/main/java/org/apache/iceberg/orc/OrcFileAppender.java ## @@ -95,24 +97,16 @@ public Metrics metrics()

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300067717 ## File path: orc/src/main/java/org/apache/iceberg/orc/OrcIterable.java ## @@ -59,10 +60,22 @@ @SuppressWarnings("uncheck

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300067457 ## File path: orc/src/main/java/org/apache/iceberg/orc/TypeConversion.java ## @@ -159,21 +163,45 @@ Type convertOrcToType(Ty

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300062920 ## File path: site/docs/spec.md ## @@ -473,7 +473,7 @@ Lists must use the [3-level representation](https://github.com/apach

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300061844 ## File path: orc/src/main/java/org/apache/iceberg/orc/TypeConversion.java ## @@ -159,21 +163,45 @@ Type convertOrcToType(Ty

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300060980 ## File path: orc/src/main/java/org/apache/iceberg/orc/TypeConversion.java ## @@ -122,11 +124,13 @@ static TypeDescription t

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300060696 ## File path: orc/src/main/java/org/apache/iceberg/orc/TypeConversion.java ## @@ -35,19 +41,21 @@ * @return the ORC sche

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300060612 ## File path: orc/src/main/java/org/apache/iceberg/orc/TypeConversion.java ## @@ -35,19 +41,21 @@ * @return the ORC sche

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300058882 ## File path: orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300058882 ## File path: orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #227: ORC column map fix

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #227: ORC column map fix URL: https://github.com/apache/incubator-iceberg/pull/227#discussion_r300056977 ## File path: orc/src/main/java/org/apache/iceberg/orc/ORCSchemaUtil.java ## @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache

[GitHub] [incubator-iceberg] rdblue commented on issue #241: Use gradle-consistent-versions to manage dependencies

2019-07-03 Thread GitBox
rdblue commented on issue #241: Use gradle-consistent-versions to manage dependencies URL: https://github.com/apache/incubator-iceberg/pull/241#issuecomment-508168221 @mccheah, @CRogers, this looks good to me other than the mockito problem that @rdsr pointed out. Can you address that comme

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #241: Use gradle-consistent-versions to manage dependencies

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #241: Use gradle-consistent-versions to manage dependencies URL: https://github.com/apache/incubator-iceberg/pull/241#discussion_r300054806 ## File path: build.gradle ## @@ -253,24 +240,31 @@ project(':iceberg-hive') { depend

[GitHub] [incubator-iceberg] rdblue commented on issue #247: Fixes `withPartitionPath` API for transform functions that consider `toHumanString` path representations incompatible as type and value wit

2019-07-03 Thread GitBox
rdblue commented on issue #247: Fixes `withPartitionPath` API for transform functions that consider `toHumanString` path representations incompatible as type and value with the partition value preserved in the spec URL: https://github.com/apache/incubator-iceberg/pull/247#issuecomment-508165479

[GitHub] [incubator-iceberg] rdblue commented on issue #236: iceberg: don't bundle hdfs with iceberg runtime

2019-07-03 Thread GitBox
rdblue commented on issue #236: iceberg: don't bundle hdfs with iceberg runtime URL: https://github.com/apache/incubator-iceberg/pull/236#issuecomment-508164720 @SurenNihalani and @rdsr, as part of adding user docs and testing with Spark 2.4, I updated the contents of the Spark runtime Jar

[GitHub] [incubator-iceberg] rdblue opened a new pull request #250: Update Spark runtime Jar contents

2019-07-03 Thread GitBox
rdblue opened a new pull request #250: Update Spark runtime Jar contents URL: https://github.com/apache/incubator-iceberg/pull/250 This removes the shadow configuration and minimizes the shaded dependencies for Spark using exclusions. ---

[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #248: Add support for returning hive partitions that match a filter express…

2019-07-03 Thread GitBox
rdblue commented on a change in pull request #248: Add support for returning hive partitions that match a filter express… URL: https://github.com/apache/incubator-iceberg/pull/248#discussion_r300047098 ## File path: hive/src/test/java/org/apache/iceberg/hive/TestHiveMetastore.java

[GitHub] [incubator-iceberg] chenjunjiedada opened a new issue #249: Convert from Spark table to Iceberg table

2019-07-03 Thread GitBox
chenjunjiedada opened a new issue #249: Convert from Spark table to Iceberg table URL: https://github.com/apache/incubator-iceberg/issues/249 Currently, Iceberg-spark doesn't have API to convert from a Spark table to iceberg table. User may want to migrate Spark or Hive tables to iceberg t

[GitHub] [incubator-iceberg] puneetzaroo opened a new pull request #248: Add support for returning hive partitions that match a filter express…

2019-07-03 Thread GitBox
puneetzaroo opened a new pull request #248: Add support for returning hive partitions that match a filter express… URL: https://github.com/apache/incubator-iceberg/pull/248 …ion. Also added some unit tests for the partition scanning code.

[GitHub] [incubator-iceberg] fbocse opened a new pull request #247: Fixes `withPartitionPath` API for transform functions that consider `toHumanString` path representations incompatible as type and va

2019-07-03 Thread GitBox
fbocse opened a new pull request #247: Fixes `withPartitionPath` API for transform functions that consider `toHumanString` path representations incompatible as type and value with the partition value preserved in the spec URL: https://github.com/apache/incubator-iceberg/pull/247