----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/18459/#review35533 -----------------------------------------------------------
Updated patch with Gunther feedback. trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java <https://reviews.apache.org/r/18459/#comment66124> Didn't update template file, since as Lefty pointed out, soon it won't be required anymore. Updating test hive-site may result in failures in existing test cases which are written for jdbc stats collection (like hashing key etc). Each of those tests need to be examined, then updated etc and given the rate at which patch queue is moving, that will delay this patch endlessly. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java <https://reviews.apache.org/r/18459/#comment66125> Good point. Instead of task attempt id, I have switched to using partition # in filenames. Since, partition # is tied to task # and is independent of attempt #, its guaranteed to give same filename for a task across different attempts. Further, I have changed create() with overwrite option, so once we get same filename we overwrite it again if task is running in its second attempt. See, changes in FSStatsPublisher. trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java <https://reviews.apache.org/r/18459/#comment66126> Updated logging at higher level at various places. - Ashutosh Chauhan On Feb. 26, 2014, 4:37 p.m., Ashutosh Chauhan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/18459/ > ----------------------------------------------------------- > > (Updated Feb. 26, 2014, 4:37 p.m.) > > > Review request for hive and Navis Ryu. > > > Bugs: HIVE-6500 > https://issues.apache.org/jira/browse/HIVE-6500 > > > Repository: hive > > > Description > ------- > > FS based stats collection. > > > Diffs > ----- > > trunk/common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java > 1572127 > trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1572127 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java > 1572127 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 1572127 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java > 1572127 > trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1572127 > trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java > 1572127 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregator.java > 1572127 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsAggregatorTez.java > 1572127 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/CounterStatsPublisher.java > 1572127 > > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsCollectionTaskIndependent.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsFactory.java 1572127 > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java > PRE-CREATION > trunk/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsPublisher.java > PRE-CREATION > trunk/ql/src/test/queries/clientpositive/statsfs.q PRE-CREATION > trunk/ql/src/test/results/clientpositive/statsfs.q.out PRE-CREATION > > Diff: https://reviews.apache.org/r/18459/diff/ > > > Testing > ------- > > Added new tests. > > > Thanks, > > Ashutosh Chauhan > >
