Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient

2021-01-08 Thread OpenInx
OK, there's a try-with-resource to close the TableLoader in FlinkInputFormat [1]. so we don't have to do the extra try-with-resource in PR 2051 ( I will close that). Under my host, I did not reproduce your connection leak issues when running TestFlinkInputFormatReaderDeletes. Did you have an

Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient

2021-01-08 Thread Steven Wu
I use the try-with-resource pattern in the FLIP-27 dev branch. I saw this problem in Flink tests with the master branch too (although less likely). With the FLIP-27 dev branch and an additional DeleteReadTests, it almost happened 100%. Also, the Spark module (in the master branch) also has this fl

Re: test flakiness with SocketException of broken pipe in HiveMetaStoreClient

2021-01-08 Thread Ryan Blue
It could be that there are two separate flaky test issues with not releasing connections in Flink and Spark. I don't think that the HiveCatalog code has been changed much recently, which would point toward problems elsewhere. I think one good reason to use HiveCatalog is to catch problems like the

Re: About proposal for including Partition stats for Iceberg tables

2021-01-08 Thread Ryan Blue
Hi Piyush, You might want to consider having a separate partition stats file for each partition spec. That way, each stats file contains just one partition struct type and you can keep the struct unmodified. There is a way to convert a partition struct to a string (PartitionSpec.partitionToPath) b