[
https://issues.apache.org/jira/browse/IMPALA-15241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-15241:
-------------------------------------
Description:
See discussion in https://gerrit.cloudera.org/#/c/24572/5//COMMIT_MSG@9 for
motivation.
Currently INPUT__FILE__NAME is used in two ways:
1. common: anti join key in IcebergDeleteJoinNode
2. rare: any other use case, e.g. return in select list
INPUT__FILE__NAME is a StringValue where the actual string is deduplicated, but
may get duplicated when going through exchanges and some other operators.
IMPALA-15171 improves this by nulling the string if it is no longer used after
the delete node, but I consider this a bit hacky.
A longer term solution could be to have a global id->path map and store only an
int32 in tuples and use it to look up the name if it is actually needed. A
similar global map already exists in for directed distribution mode of delete
nodes.
The same lookup mechanism could be used for other things too, for example
storing partition columns.
was:
See discussion in https://gerrit.cloudera.org/#/c/24572/5//COMMIT_MSG@9 for
motivation.
Currently INPUT__FILE__NAME is used in two ways:
1. common: anti join key in IcebergDeleteJoinNode
2. rare: any other use case, e.g. return in select list
INPUT__FILE__NAME is a StringValue where the actual string is deduplicated, but
may get duplicated when going through exchanges and some other operators.
IMPALA-15171
improves this by nulling the string if it is no longer used after the delete
node, but I consider this a bit hacky.
A longer term solution could be to have a global id->path map and store only an
int32 in tuples and use it to look up the name if it is actually needed. A
similar global map already exists in for directed distribution mode of delete
nodes.
The same lookup mechanism could be used for other things too, for example
storing partition columns.
> Use file IDs instead of paths to get INPUT__FILE__NAME
> ------------------------------------------------------
>
> Key: IMPALA-15241
> URL: https://issues.apache.org/jira/browse/IMPALA-15241
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: impala-iceberg, performance
>
> See discussion in https://gerrit.cloudera.org/#/c/24572/5//COMMIT_MSG@9 for
> motivation.
> Currently INPUT__FILE__NAME is used in two ways:
> 1. common: anti join key in IcebergDeleteJoinNode
> 2. rare: any other use case, e.g. return in select list
> INPUT__FILE__NAME is a StringValue where the actual string is deduplicated,
> but may get duplicated when going through exchanges and some other operators.
> IMPALA-15171 improves this by nulling the string if it is no longer used
> after the delete node, but I consider this a bit hacky.
> A longer term solution could be to have a global id->path map and store only
> an int32 in tuples and use it to look up the name if it is actually needed. A
> similar global map already exists in for directed distribution mode of delete
> nodes.
> The same lookup mechanism could be used for other things too, for example
> storing partition columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
