> On Sept. 25, 2015, 6:55 p.m., John Pullokkaran wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java,
> > line 316
> > <https://reviews.apache.org/r/38768/diff/1/?file=1084857#file1084857line316>
> >
> > Also why is this only at the else condition?
> > Are we assuming that if the requested cols & returned cols size matches
> > then order is preserved.
> >
> > This seems rather shaky assumptions.
> > Its better to fix stats API to guarantee the order.
No, I did not assume that. The reasons I put the logic in this else conditiion
are because:
1. only when the requested cols & returned cols size matches, the stats is
mapped to an index and put into the hiveColStatsMap, see following code in
updateColStats (line 363 -367)
==
if (hiveColStats != null && hiveColStats.size() ==
nonPartColNamesThatRqrStats.size()) {
for (int i = 0; i < hiveColStats.size(); i++) {
// the columns in
nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats/hiveColStats
// are in same order
hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i),
hiveColStats.get(i));
}
}
==
2. For partitioned table, the hiveColStats is already built in the order of
nonPartColNamesThatRqrStats. There is not necessary to reorder it.
==
for (String c : nonPartColNamesThatRqrStats) {
ColStatistics cs = stats.getColumnStatisticsFromColName(c);
if (cs != null) {
hiveColStats.add(cs);
} else {
colNamesFailedStats.add(c);
}
}
==
3. HMS stats API might not be a proper place to enforce the order as I
explained in last comment.
- Chaoyu
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38768/#review100655
-----------------------------------------------------------
On Sept. 25, 2015, 6:37 p.m., Chaoyu Tang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38768/
> -----------------------------------------------------------
>
> (Updated Sept. 25, 2015, 6:37 p.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-11964
> https://issues.apache.org/jira/browse/HIVE-11964
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The stats in hiveColStats should be ordered in that of the columns in
> nonPartColIndxsThatRqrStats/nonPartColNamesThatRqrStats before being put in
> the hiveColStatsMap
> ===
> for (int i = 0; i < hiveColStats.size(); i++) {
> hiveColStatsMap.put(nonPartColIndxsThatRqrStats.get(i),
> hiveColStats.get(i));
> }
>
> Since hiveColStats = StatsUtils.getTableColumnStats(hiveTblMetadata,
> hiveNonPartitionCols, nonPartColNamesThatRqrStats); might not return the
> stats in same order as those in nonPartColNamesThatRqrStats.
>
>
> Diffs
> -----
>
>
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/RelOptHiveTable.java
> 6c0bd25
>
> Diff: https://reviews.apache.org/r/38768/diff/
>
>
> Testing
> -------
>
> Precommit tests
>
>
> Thanks,
>
> Chaoyu Tang
>
>
