----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69410/ -----------------------------------------------------------
Review request for hive, Nandor Kollar and Peter Vary. Bugs: HIVE-20330 https://issues.apache.org/jira/browse/HIVE-20330 Repository: hive-git Description ------- The change in this patch is that we're not just serializing and putting one InputJobInfo into JobConf, but rather always append to a list (or create it on the first occurrence) of InputJobInfo instances in it. This ensures that if multiple tables serve as inputs in a job, Pig can retrieve information for each of the tables, not just the last one added. I've also discovered a bug in InputJobInfo.writeObject() where the ObjectOutputStream was closed by mistake after writing partition information in a compressed manner. Closing the compressed writer inevitably closed the OOS on the context and prevented any other objects to be written into OOS - I had to fix that because it prevented serializing InputJobInfo instances inside a list. Diffs ----- hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatUtil.java 8e72a1275a5cdcc2d778080fff6bb82198395f5f hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java 195eaa367933990e3ef0ef879f34049c65822aee hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatBaseInputFormat.java 8d7a8f9df9412105ec7d77fad9af0d7dd18f4323 hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/HCatInputFormat.java ad6f3eb9f93338023863c6239d6af0449b20ff9c hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InitializeInput.java 364382d9ccf6eb9fc29689b0eb5f973f422051b4 hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/InputJobInfo.java ac1dd54be821d32aa008d41514df05a41f16223c hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java 91aa4fa2693e0b0bd65c1667210af340619f552d hcatalog/hcatalog-pig-adapter/src/main/java/org/apache/hive/hcatalog/pig/HCatLoader.java c3bde2d2a3cbd09fb0b1ed758bf4f2b1041a23cb hcatalog/hcatalog-pig-adapter/src/test/java/org/apache/hive/hcatalog/pig/AbstractHCatLoaderTest.java 58981f88ef6abfbf7a4b7ffc3116c53d47e86fde Diff: https://reviews.apache.org/r/69410/diff/1/ Testing ------- Added (true) unit tests to verify my method of adding/retrieving InputJobInfo instances to/from config instances. Added (integration-like) unit tests to mock Pig calling HCatLoader for multiple input tables, and checking the reported input sizes. Thanks, Adam Szita