[ https://issues.apache.org/jira/browse/HIVE-5454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-5454: ------------------------------ Attachment: D13317.3.patch QwertyManiac updated the revision "HIVE-5454 [jira] HCatalog runs a partition listing with an empty filter". Fixed 4 reported checkstyle violations. Lint reported fine from arc, so didn't notice these earlier. Reviewers: JIRA REVISION DETAIL https://reviews.facebook.net/D13317 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D13317?vs=41043&id=41049#toc AFFECTED FILES hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/transfer/impl/HCatInputFormatReader.java hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/HCatMapReduceTest.java hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java hcatalog/src/docs/src/documentation/content/xdocs/inputoutput.xml hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hcatalog/utils/HBaseReadWrite.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/GroupByAge.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadJson.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadRC.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadText.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/ReadWrite.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/SimpleRead.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreComplex.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreDemo.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/StoreNumbers.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/SumNumbers.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/TypeDataCheck.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteJson.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteRC.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteText.java hcatalog/src/test/e2e/hcatalog/udfs/java/org/apache/hive/hcatalog/utils/WriteTextPartitioned.java hcatalog/storage-handlers/hbase/src/test/org/apache/hive/hcatalog/hbase/TestHBaseInputFormat.java To: JIRA, QwertyManiac > HCatalog runs a partition listing with an empty filter > ------------------------------------------------------ > > Key: HIVE-5454 > URL: https://issues.apache.org/jira/browse/HIVE-5454 > Project: Hive > Issue Type: Bug > Components: HCatalog > Affects Versions: 0.12.0 > Reporter: Harsh J > Attachments: D13317.1.patch, D13317.2.patch, D13317.3.patch > > > This is a HCATALOG-527 caused regression, wherein the HCatLoader's way of > calling HCatInputFormat causes it to do 2x partition lookups - once without > the filter, and then again with the filter. > For tables with large number partitions (100000, say), the non-filter lookup > proves fatal both to the client ("Read timed out" errors from > ThriftMetaStoreClient cause the server doesn't respond) and to the server > (too much data loaded into the cache, OOME, or slowdown). > The fix would be to use a single call that also passes a partition filter > information, as was in the case of HCatalog 0.4 sources before HCATALOG-527. > (HCatalog-release-wise, this affects all 0.5.x users) -- This message was sent by Atlassian JIRA (v6.1#6144)