[
https://issues.apache.org/jira/browse/IMPALA-12805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated IMPALA-12805:
------------------------------------
Description:
HIVE-27114 adds a new
property,"hive.metastore.partitions.parameters.exclude.pattern", which defaults
to "impala_intermediate_stats_chunk%". This default excludes the Impala
incremental stats from the results returned from the HMS client. I think this
was done to help Hive performance. It means however that Impala cannot
retrieve its own partition stats, which will negatively impact the performance
of many workloads.
One fix for this was IMPALA-12743 which sets a non-empty non-default value for
the new Hive property in the development environment’s hive-site.xml file. In
some customer situations this would require a configuration change that cannot
be deployed simultaneously with the possible HMS upgrade which introduces the
regression.
Fix this by setting a non-empty non-default value for the new Hive property in
the configurations in the MetaStoreClientPool. This avoids the regression. The
natural thing would be to set an empty string as the value, but this is not
enough to override the default. Instead we set the string “do_not_match” which
should never be used for as the name of any partition stats.
was:
HIVE-27114 adds a new
property,"hive.metastore.partitions.parameters.exclude.pattern", which defaults
to "impala_intermediate_stats_chunk%". This default excludes the Impala
incremental stats from the results returned from the HMS client. I think this
was done to help Hive performance. It means however that Impala cannot
retrieve its own partition stats, which will negatively impact the performance
of many workloads.
One fix for this was IMPALA-12743 which sets a non-empty non-default value for
the new Hive property in the development environment’s hive-site.xml file. In
some customer situations this would require a configuration change that cannot
be deployed simultaneously with the possible HMS upgrade which introduces the
regression.
This change sets a non-empty non-default value for the new Hive property in the
configurations in the MetaStoreClientPool. This avoids the regression. The
natural thing would be to set an empty string as the value, but this is not
enough to override the default. Instead we set the string “do_not_match” which
should never be used for as the name of any partition stats.
> Avoid problems of HIVE-27114 without configuration changes
> ----------------------------------------------------------
>
> Key: IMPALA-12805
> URL: https://issues.apache.org/jira/browse/IMPALA-12805
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.4.0
> Reporter: Andrew Sherman
> Assignee: Andrew Sherman
> Priority: Critical
>
> HIVE-27114 adds a new
> property,"hive.metastore.partitions.parameters.exclude.pattern", which
> defaults to "impala_intermediate_stats_chunk%". This default excludes the
> Impala incremental stats from the results returned from the HMS client. I
> think this was done to help Hive performance. It means however that Impala
> cannot retrieve its own partition stats, which will negatively impact the
> performance of many workloads.
> One fix for this was IMPALA-12743 which sets a non-empty non-default value
> for the new Hive property in the development environment’s hive-site.xml
> file. In some customer situations this would require a configuration change
> that cannot be deployed simultaneously with the possible HMS upgrade which
> introduces the regression.
> Fix this by setting a non-empty non-default value for the new Hive property
> in the configurations in the MetaStoreClientPool. This avoids the regression.
> The natural thing would be to set an empty string as the value, but this is
> not enough to override the default. Instead we set the string “do_not_match”
> which should never be used for as the name of any partition stats.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
