kasakrisz commented on code in PR #6413:
URL: https://github.com/apache/hive/pull/6413#discussion_r3474202732
##########
ql/src/java/org/apache/hadoop/hive/ql/session/LineageState.java:
##########
@@ -90,6 +91,19 @@ public synchronized void updateDirToOpMap(Path newPath, Path
oldPath) {
}
}
+ /**
+ * Set the lineage information for the associated directory.
+ *
+ * @param dir The directory containing the query results.
+ * @param dc The associated data container.
+ * @param table table metadata used to resolve written columns for lineage
mapping.
+ */
+ public synchronized void setLineage(Path dir, DataContainer dc, Table table)
{
+ List<FieldSchema> cols = table.hasNonNativePartitionSupport() ?
+ table.getAllCols() : table.getCols();
Review Comment:
What is blocking collecting the lineage information of all columns in case
of native tables?
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java:
##########
@@ -274,50 +275,75 @@ private void replaceSelectOperatorProcess(SelectOperator
operator, Operator<? ex
columnExprMap.put(internalName, exprNodeDesc);
signature.add(selRSSig.get(selRSIdx));
}
- // if there is any partition column (in static partition or dynamic
- // partition or mixed case)
- int dynamicPartBegin = -1;
+ int dynPartsCount = 0;
+ if (partSpec != null) {
+ for (Map.Entry<String, String> entry : partSpec.entrySet()) {
+ if (entry.getValue() == null) {
+ dynPartsCount++;
+ }
+ }
+ }
+ // True when the input row already carries the static partition columns as
real columns (e.g. Iceberg),
+ // i.e. it has more columns than just the data columns plus the dynamic
partition columns.
+ boolean inputRRHasStaticParts = (this.columns.size() + dynPartsCount <
columns.size());
+ // Column layout produced for the stats-gathering select:
+ // <-- non-partition cols -->|<-- static partition cols -->|<-- dynamic
partition cols -->
+ // For non-native tables (e.g. Iceberg) partition columns are ordinary
interleaved columns instead.
+ boolean dynamicPartSeen = false;
+ // Count of native static partition columns already emitted as constants.
They are absent from the input
+ // row, so each one shifts the input position of the columns that follow
it.
+ int staticPartShift = 0;
for (int i = 0; i < partitionColumns.size(); i++) {
- ExprNodeDesc exprNodeDesc;
- TypeInfo srcType;
String partColName = partitionColumns.get(i).getName();
+ int tableIndex = tbl.getColumnIndexByName(partColName);
+ boolean isStaticPartition = partSpec != null &&
partSpec.containsKey(partColName)
+ && partSpec.get(partColName) != null;
Review Comment:
Isn't `partSpec.containsKey(partColName)` redundant here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
