[jira] [Resolved] (IMPALA-12462) Do not update unchanged partitions in COMPUTE STATS
[ https://issues.apache.org/jira/browse/IMPALA-12462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Csaba Ringhofer resolved IMPALA-12462. -- Fix Version/s: Impala 4.4.0 Resolution: Done > Do not update unchanged partitions in COMPUTE STATS > > > Key: IMPALA-12462 > URL: https://issues.apache.org/jira/browse/IMPALA-12462 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Csaba Ringhofer >Assignee: Csaba Ringhofer >Priority: Major > Fix For: Impala 4.4.0 > > > Since IMPALA-2201 all partitions are updated in HMS, even if there were no > changes in stats. This was needed duo to an HMS bug but is no longer > necessary. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8675) Remove db/table count metrics from impalad in LocalCatalog mode
[ https://issues.apache.org/jira/browse/IMPALA-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769263#comment-17769263 ] Michael Smith commented on IMPALA-8675: --- RC1 is ready. I guess I could prepare an RC2. > Remove db/table count metrics from impalad in LocalCatalog mode > --- > > Key: IMPALA-8675 > URL: https://issues.apache.org/jira/browse/IMPALA-8675 > Project: IMPALA > Issue Type: Improvement > Components: Catalog >Reporter: Todd Lipcon >Assignee: Quanlong Huang >Priority: Minor > Labels: catalog-v2 > Fix For: Impala 4.4.0 > > > In LocalCatalog there is no need for every coordinator to have the full list > of tables of every database. But, getCatalogMetrics ends up iterating over > every DB and fetching these lists in order to provide a count. The count > isn't particularly relevant -- if someone wants to keep track of the size of > their catalog they are better off looking at that metric from catalogd. We > should remove these catalog metrics. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12434) Isolate shell/pkg_resources.py to keep it off of the PYTHONPATH
[
https://issues.apache.org/jira/browse/IMPALA-12434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-12434:
---
Fix Version/s: Impala 4.4.0
(was: Impala 4.3.0)
> Isolate shell/pkg_resources.py to keep it off of the PYTHONPATH
> ---
>
> Key: IMPALA-12434
> URL: https://issues.apache.org/jira/browse/IMPALA-12434
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
>Affects Versions: Impala 4.3.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Major
> Fix For: Impala 4.4.0
>
>
> In some internal builds on Redhat 8, the pip install of impala-shell into the
> Python 3 virtualenv can fail with this error:
> {noformat}
> Exception:
> Traceback (most recent call last):
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/basecommand.py",
> line 215, in main
> status = self.run(options, args)
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/commands/install.py",
> line 357, in run
> wb.build(autobuilding=True)
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/wheel.py",
> line 753, in build
> self.requirement_set.prepare_files(self.finder)
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py",
> line 381, in prepare_files
> ignore_dependencies=self.ignore_dependencies))
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py",
> line 637, in _prepare_file
> abstract_dist.prep_for_dist()
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_set.py",
> line 130, in prep_for_dist
> self.req_to_install.run_egg_info()
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_install.py",
> line 417, in run_egg_info
> self.setup_py, self.link,
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/pip/req/req_install.py",
> line 387, in setup_py
> import setuptools # noqa
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/setuptools/__init__.py",
> line 12, in
> import setuptools.version
> File
> "impala/shell/build/py3_venv/lib64/python3.6/site-packages/setuptools/version.py",
> line 1, in
> import pkg_resources
> File "impala/shell/pkg_resources.py", line 50
> def _bypass_ensure_directory(name, mode=0777):
> ^
> SyntaxError: invalid token
> make[3]: *** [shell/CMakeFiles/shell_python3_install.dir/build.make:70:
> shell/CMakeFiles/shell_python3_install] Error 2
> {noformat}
> What seems to happen is shell/pkg_resources.py is on the PYTHONPATH and our
> pkg_resources.py doesn't work with Python 3. If I move pkg_resources.py to a
> different directory (like a legacy subdirectory) or if I add a
> WORKING_DIRECTORY for the CMake command, it works fine. In general, keeping
> pkg_resources.py away from our PYTHONPATH (which includes $IMPALA_HOME/shell)
> seems like a good idea.
> This problem doesn't reproduce on the regular upstream Impala build or on my
> local Ubuntu 20 workstation build, but it reliably reproduces on an internal
> build.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12390) Enable performance related clang-tidy checks
[
https://issues.apache.org/jira/browse/IMPALA-12390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith updated IMPALA-12390:
---
Fix Version/s: Impala 4.4.0
> Enable performance related clang-tidy checks
>
>
> Key: IMPALA-12390
> URL: https://issues.apache.org/jira/browse/IMPALA-12390
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Affects Versions: Impala 4.3.0
>Reporter: Joe McDonnell
>Assignee: Joe McDonnell
>Priority: Major
> Fix For: Impala 4.4.0
>
>
> clang-tidy has several performance-related checks that seem like they would
> be useful to enforce. Here are some examples:
> {noformat}
> /home/joemcdonnell/upstream/Impala/be/src/runtime/types.h:313:25: warning:
> loop variable is copied but only used as const reference; consider making it
> a const reference [performance-for-range-copy]
> for (ColumnType child_type : col_type.children) {
> ~~ ^
> const &
> /home/joemcdonnell/upstream/Impala/be/src/catalog/catalog-util.cc:168:34:
> warning: 'find' called with a string literal consisting of a single
> character; consider using the more effective overload accepting a character
> [performance-faster-string-find]
> int pos = object_name.find(".");
> ^~~~
> '.'
> /home/joemcdonnell/upstream/Impala/be/src/util/decimal-util.h:55:53: warning:
> the parameter 'b' is copied for each invocation but only used as a const
> reference; consider making it a const reference
> [performance-unnecessary-value-param]
> static int256_t SafeMultiply(int256_t a, int256_t b, bool may_overflow) {
> ^
> const &
> /home/joemcdonnell/upstream/Impala/be/src/codegen/llvm-codegen.cc:847:5:
> warning: 'push_back' is called inside a loop; consider pre-allocating the
> vector capacity before the loop [performance-inefficient-vector-operation]
> arguments.push_back(args_[i].type);
> ^{noformat}
> In all, they seem to flag things that developers wouldn't ordinarily notice,
> and it doesn't seem to have too many false positives. We should look into
> enabling these.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12318) Use spnego dedicated keytab
[ https://issues.apache.org/jira/browse/IMPALA-12318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Smith resolved IMPALA-12318. Fix Version/s: Impala 4.4.0 Resolution: Fixed > Use spnego dedicated keytab > --- > > Key: IMPALA-12318 > URL: https://issues.apache.org/jira/browse/IMPALA-12318 > Project: IMPALA > Issue Type: Improvement > Components: Security >Reporter: halim kim >Assignee: halim kim >Priority: Minor > Fix For: Impala 4.4.0 > > > Kerberos is one of the authentication methods that impala provides. > Kerberized impala uses its keytab that has impala principal for > authentication. > kerberos authentication can be applied by setting '--principal' and > '--keytab_file' flags. > Further more, It is possible to kerberize impala web console by having > --webserver_require_spnego as true. > The problem is impala uses just one keytab file. Therefore, a keytab must > have both impala and HTTP spnego principal If you want to kerberize web > console too. > > As far as i know, Other service like hadoop, hive and etc provides a option > to use http spnego dedicated keytab file and there are cases that using > seperate http spnego keytab and service keytab. So providing a way to use > another keytab file for http spnego will make users handle kerberos keytab > file more easily. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12449) SSE2NEON is slow compared to native NEON code
[ https://issues.apache.org/jira/browse/IMPALA-12449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Smith updated IMPALA-12449: --- Fix Version/s: Impala 4.4.0 > SSE2NEON is slow compared to native NEON code > - > > Key: IMPALA-12449 > URL: https://issues.apache.org/jira/browse/IMPALA-12449 > Project: IMPALA > Issue Type: Bug >Affects Versions: Impala 4.3.0 >Reporter: Sebastian Pop >Assignee: Sebastian Pop >Priority: Critical > Fix For: Impala 4.4.0 > > > The port to arm64 uses https://github.com/DLTcollab/sse2neon an automatic > tool to convert x86_64 intrinsics to arm64 NEON. > As some of the x86_64 vector instructions do not have a one-to-one > translation to NEON, the sse2neon translation tool falls back to a sequential > emulation of the missing instructions. > To avoid those slow paths, we need to adapt to NEON instructions all the > code-paths with x86_64 intrinsics. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12436) Support containerized toolchain builds on arm
[ https://issues.apache.org/jira/browse/IMPALA-12436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Smith resolved IMPALA-12436. Resolution: Fixed > Support containerized toolchain builds on arm > - > > Key: IMPALA-12436 > URL: https://issues.apache.org/jira/browse/IMPALA-12436 > Project: IMPALA > Issue Type: Task >Reporter: Michael Smith >Assignee: Michael Smith >Priority: Major > Fix For: Not Applicable > > > native-toolchain supports running multiple builds simultaneously in Docker > containers. Those containers are produced from > https://github.com/cloudera/native-toolchain/tree/master/docker, which need > to be updated to support ARM images. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12467) Semantic Search In Impala
Sreenath created IMPALA-12467: - Summary: Semantic Search In Impala Key: IMPALA-12467 URL: https://issues.apache.org/jira/browse/IMPALA-12467 Project: IMPALA Issue Type: Wish Reporter: Sreenath Semantic search is a way for computers to understand the meaning behind words and phrases when you're searching for something. Instead of just looking for exact matches of keywords, it tries to figure out what you're really asking and provides results that are more relevant and meaningful to your question. It's like having a search engine that can understand what you mean, not just what you say, making it easier to find the information you're looking for. This ticket is a wish to have Semantic search in ImpalaSemantic Search In Hive. On the implementation side, semantic search uses an embedding model and any of the similarity distance functions. My proposal is to implement functions for on-the-fly calculation of similarity distance between two values. Once we have them we could easily do semantic search as part of a where clause. * Eg (using a cosine similarity function): “WHERE cos_dist(region, 'europe') > 0.9“. And it could return records with regions like Scandinavia, Nordic, Baltic etc… * We could have functions thats accept values as text or as vector embeddings. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12467) Semantic Search In Impala
[ https://issues.apache.org/jira/browse/IMPALA-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sreenath updated IMPALA-12467: -- Description: Semantic search is a way for computers to understand the meaning behind words and phrases when you're searching for something. Instead of just looking for exact matches of keywords, it tries to figure out what you're really asking and provides results that are more relevant and meaningful to your question. It's like having a search engine that can understand what you mean, not just what you say, making it easier to find the information you're looking for. This ticket is a wish to have semantic search in Impala. On the implementation side, semantic search uses an embedding model and any of the similarity distance functions. My proposal is to implement functions for on-the-fly calculation of similarity distance between two values. Once we have them we could easily do semantic search as part of a where clause. * Eg (using a cosine similarity function): “WHERE cos_dist(region, 'europe') > 0.9“. And it could return records with regions like Scandinavia, Nordic, Baltic etc… * We could have functions thats accept values as text or as vector embeddings. was: Semantic search is a way for computers to understand the meaning behind words and phrases when you're searching for something. Instead of just looking for exact matches of keywords, it tries to figure out what you're really asking and provides results that are more relevant and meaningful to your question. It's like having a search engine that can understand what you mean, not just what you say, making it easier to find the information you're looking for. This ticket is a wish to have Semantic search in ImpalaSemantic Search In Hive. On the implementation side, semantic search uses an embedding model and any of the similarity distance functions. My proposal is to implement functions for on-the-fly calculation of similarity distance between two values. Once we have them we could easily do semantic search as part of a where clause. * Eg (using a cosine similarity function): “WHERE cos_dist(region, 'europe') > 0.9“. And it could return records with regions like Scandinavia, Nordic, Baltic etc… * We could have functions thats accept values as text or as vector embeddings. > Semantic Search In Impala > - > > Key: IMPALA-12467 > URL: https://issues.apache.org/jira/browse/IMPALA-12467 > Project: IMPALA > Issue Type: Wish >Reporter: Sreenath >Priority: Major > > Semantic search is a way for computers to understand the meaning behind words > and phrases when you're searching for something. Instead of just looking for > exact matches of keywords, it tries to figure out what you're really asking > and provides results that are more relevant and meaningful to your question. > It's like having a search engine that can understand what you mean, not just > what you say, making it easier to find the information you're looking for. > This ticket is a wish to have semantic search in Impala. > On the implementation side, semantic search uses an embedding model and any > of the similarity distance functions. > My proposal is to implement functions for on-the-fly calculation of > similarity distance between two values. Once we have them we could easily do > semantic search as part of a where clause. > * Eg (using a cosine similarity function): “WHERE cos_dist(region, 'europe') > > 0.9“. And it could return records with regions like Scandinavia, Nordic, > Baltic etc… > * We could have functions thats accept values as text or as vector > embeddings. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12468) Add the ability to update EventProcessorStatus
Maxwell Guo created IMPALA-12468: Summary: Add the ability to update EventProcessorStatus Key: IMPALA-12468 URL: https://issues.apache.org/jira/browse/IMPALA-12468 Project: IMPALA Issue Type: Improvement Components: be, Catalog, fe Reporter: Maxwell Guo Assignee: Maxwell Guo Once the impala and hive's status is missmatched , and the EventProcessorStatus become NEED_INVALIDATE， we usually use invalidate metadata to reset the catalog instance. And then impala will update the status to ACTIVE . But if impala contains many tables , the cost of invalidate is a bit high for a global invalidate. So we may invalidate metadata for tables one by one for these incremental changed table. For example , we have 1000,000,000,000 tables but only some of the table event process occurs CatalogException and MetastoreNotificationNeedsInvalidateException was thrown. I think there is no need to invalidate all table caches. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12468) Add the ability to update EventProcessorStatus
[ https://issues.apache.org/jira/browse/IMPALA-12468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxwell Guo updated IMPALA-12468: - Description: Once the impala and hive's status is missmatched , and the EventProcessorStatus become NEED_INVALIDATE， we usually use invalidate metadata to reset the catalog instance. And then impala will update the status to ACTIVE . But if impala contains many tables , the cost of invalidate is a bit high for a global invalidate. So we may invalidate metadata for tables one by one for these incremental changed table. For example , we have 1000,000,000,000 tables but only some of the table event process occurs CatalogException and MetastoreNotificationNeedsInvalidateException was thrown. I think there is no need to invalidate all table caches in order to reset the catalog instance see [here |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2088]. MetaStoresProcessor 's async update process will not update the currentStatus when the status is not ACTIVE, see [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L876] So what about add a new SQL grammar : RESET STATUS status ? was: Once the impala and hive's status is missmatched , and the EventProcessorStatus become NEED_INVALIDATE， we usually use invalidate metadata to reset the catalog instance. And then impala will update the status to ACTIVE . But if impala contains many tables , the cost of invalidate is a bit high for a global invalidate. So we may invalidate metadata for tables one by one for these incremental changed table. For example , we have 1000,000,000,000 tables but only some of the table event process occurs CatalogException and MetastoreNotificationNeedsInvalidateException was thrown. I think there is no need to invalidate all table caches. > Add the ability to update EventProcessorStatus > -- > > Key: IMPALA-12468 > URL: https://issues.apache.org/jira/browse/IMPALA-12468 > Project: IMPALA > Issue Type: Improvement > Components: be, Catalog, fe >Reporter: Maxwell Guo >Assignee: Maxwell Guo >Priority: Minor > > Once the impala and hive's status is missmatched , and the > EventProcessorStatus become NEED_INVALIDATE， we usually use invalidate > metadata to reset the catalog instance. And then impala will update the > status to ACTIVE . > But if impala contains many tables , the cost of invalidate is a bit high for > a global invalidate. So we may invalidate metadata for tables one by one for > these incremental changed table. For example , we have 1000,000,000,000 > tables but only some of the table event process occurs CatalogException and > MetastoreNotificationNeedsInvalidateException was thrown. I think there is no > need to invalidate all table caches in order to reset the catalog instance > see [here > |https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L2088]. > > MetaStoresProcessor 's async update process will not update the currentStatus > when the status is not ACTIVE, see > [here|https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java#L876] > So what about add a new SQL grammar : RESET STATUS status ? -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12469) Allow short statements on a single line in clang-format
Peter Rozsa created IMPALA-12469:
Summary: Allow short statements on a single line in clang-format
Key: IMPALA-12469
URL: https://issues.apache.org/jira/browse/IMPALA-12469
Project: IMPALA
Issue Type: Improvement
Components: Infrastructure
Affects Versions: Impala 4.2.0
Reporter: Peter Rozsa
Assignee: Peter Rozsa
Fix For: Impala 4.3.0
Impala's Java codebase uses a lot of oneliner if statements like:
{code:java}
if (ifExists_) { sb.append("IF EXISTS "); } {code}
The .clang-format file does not reflect this, the following rules are missing:
{code:java}
AllowShortIfStatementsOnASingleLine: true
AllowShortBlocksOnASingleLine: true {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12469) Allow short if statements on a single line in clang-format
[
https://issues.apache.org/jira/browse/IMPALA-12469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Rozsa updated IMPALA-12469:
-
Summary: Allow short if statements on a single line in clang-format (was:
Allow short statements on a single line in clang-format)
> Allow short if statements on a single line in clang-format
> --
>
> Key: IMPALA-12469
> URL: https://issues.apache.org/jira/browse/IMPALA-12469
> Project: IMPALA
> Issue Type: Improvement
> Components: Infrastructure
>Affects Versions: Impala 4.2.0
>Reporter: Peter Rozsa
>Assignee: Peter Rozsa
>Priority: Trivial
> Fix For: Impala 4.3.0
>
>
> Impala's Java codebase uses a lot of oneliner if statements like:
> {code:java}
> if (ifExists_) { sb.append("IF EXISTS "); } {code}
> The .clang-format file does not reflect this, the following rules are missing:
> {code:java}
> AllowShortIfStatementsOnASingleLine: true
> AllowShortBlocksOnASingleLine: true {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
