[jira] [Commented] (IMPALA-8786) BufferedPlanRootSink should directly write to a QueryResultSet if one is available
[
https://issues.apache.org/jira/browse/IMPALA-8786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891517#comment-16891517
]
Tim Armstrong commented on IMPALA-8786:
---
Maybe the problem has changed enough that it's a little different, but I
remember last time I thought about it that I had some concerns about handling
the batch boundaries, since the batches coming from query execution and the
batches requested by the client won't generally line up to each other.
I think maybe if we're only doing this as a short-circuit optimisation you
don't run into major problems, since you can just append as many rows will fit
in the current batch, and append the rest to the buffer.
Anyway, it would be nice to do an experiment to see how large the benefit of a
fast path was - it's possible that this part of query execution is fast enough
anyway compared to the network and clients.
> BufferedPlanRootSink should directly write to a QueryResultSet if one is
> available
> --
>
> Key: IMPALA-8786
> URL: https://issues.apache.org/jira/browse/IMPALA-8786
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{BufferedPlanRootSink}} uses a {{RowBatchQueue}} to buffer {{RowBatch}}-es
> and then the consumer thread reads them and writes them to a given
> {{QueryResultSet}}. Implementations of {{RowBatchQueue}} might end up copying
> the buffered {{RowBatch}}-es (e.g. if the queue is backed by a
> {{BufferedTupleStream}}). An optimization would be for the producer thread to
> directly write to the consumer {{QueryResultSet}}. This optimization would
> only be triggered if (1) the queue is empty, and (2) the consumer thread has
> a {{QueryResultSet}} available for writing.
> This "fast path" is useful in a few different scenarios:
> * If the consumer is faster than at reading rows than the producer is at
> sending them; in this case, the overhead of buffering rows in a
> {{RowBatchQueue}} can be completely avoided
> * For queries that return under 1024 its likely that the consumer will
> produce a {{QueryResultSet}} before the first {{RowBatch}} is returned
> (except perhaps for very trivial queries)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7290) Move impala-shell to use HS2
[ https://issues.apache.org/jira/browse/IMPALA-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891511#comment-16891511 ] Tim Armstrong commented on IMPALA-7290: --- [~arodoni_cloudera] the user-facing differences were kept as minimal as possible. The display rounding for floating point numbers may be slightly different, e.g. 8.071 vs 8.072 - the underlying value is the same but they're displayed differently. It does fix a user-facing bug with handling of tab characters: https://issues.apache.org/jira/browse/IMPALA-1840 One minor difference in behaviour that could have consequences on some deployments, is that impala-shell is now connecting to the HS2 endpoint that's shared with more services, so you could exhaust fe_service_threads sooner. Not sure if that's an important enough issue to document. One caveat that's worth documenting is that you can't connect to the HS2 interface of older versions of Impala, because the impala-shell HS2 support depends on some extensions that were only added to this version. > Move impala-shell to use HS2 > > > Key: IMPALA-7290 > URL: https://issues.apache.org/jira/browse/IMPALA-7290 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: maintainability, shell > Fix For: Impala 3.3.0 > > > Most clients have moved to the HS2 interface. impala-shell is one of the > laggards. We should switch impala-shell to use the newer and more standard > interface. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8615) Impala Doc: Doc admission control support for scalable config params
[ https://issues.apache.org/jira/browse/IMPALA-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8615: Description: https://gerrit.cloudera.org/#/c/13906/ > Impala Doc: Doc admission control support for scalable config params > > > Key: IMPALA-8615 > URL: https://issues.apache.org/jira/browse/IMPALA-8615 > Project: IMPALA > Issue Type: Sub-task >Reporter: Bikramjeet Vig >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13906/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8615) Impala Doc: Doc admission control support for scalable config params
[ https://issues.apache.org/jira/browse/IMPALA-8615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8615 started by Alex Rodoni. --- > Impala Doc: Doc admission control support for scalable config params > > > Key: IMPALA-8615 > URL: https://issues.apache.org/jira/browse/IMPALA-8615 > Project: IMPALA > Issue Type: Sub-task >Reporter: Bikramjeet Vig >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-7290) Move impala-shell to use HS2
[ https://issues.apache.org/jira/browse/IMPALA-7290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891499#comment-16891499 ] Alex Rodoni commented on IMPALA-7290: - [~tarmstrong] Besides, the --protocol flag, what difference would users see b/w beeswax and hs2? > Move impala-shell to use HS2 > > > Key: IMPALA-7290 > URL: https://issues.apache.org/jira/browse/IMPALA-7290 > Project: IMPALA > Issue Type: Improvement > Components: Clients >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Labels: maintainability, shell > Fix For: Impala 3.3.0 > > > Most clients have moved to the HS2 interface. impala-shell is one of the > laggards. We should switch impala-shell to use the newer and more standard > interface. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8696) Cmake dependencies for container build do not work correctly
[ https://issues.apache.org/jira/browse/IMPALA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8696. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Cmake dependencies for container build do not work correctly > > > Key: IMPALA-8696 > URL: https://issues.apache.org/jira/browse/IMPALA-8696 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 3.3.0 > > > I've noticed in many cases that containers are not reliably rebuilt when > inputs change, e.g. if the binaries are rebuilt. Investigate and fix this, > it's a major development headache. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8622) Generate list of docker images generated by the build
[
https://issues.apache.org/jira/browse/IMPALA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-8622.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Generate list of docker images generated by the build
> -
>
> Key: IMPALA-8622
> URL: https://issues.apache.org/jira/browse/IMPALA-8622
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: docker
> Fix For: Impala 3.3.0
>
>
> Some consumers of the Impala docker images will want to publish them to a
> repo. Currently they would have to hardcode a list of the docker images
> produced by the Impala build to publish them, or do something brittle like
> "docker image ls" to deduce what images were built.
> It would be useful if the build produced a list of the images in some
> consumable format that would be updated (either manually or automatically)
> when the set of images changes. If it needs to be manually updated, we should
> have some kind of check to verify that it isn't stale.
> One option is to produce JSON, something like:
> {noformat}
> {
> “docker_images”: {
>“impala_base”: “$docker_registry/impala_base”,
>” impalad_coord_exec”: “$docker_registry/impalad_coord_exec”,
>
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8696) Cmake dependencies for container build do not work correctly
[ https://issues.apache.org/jira/browse/IMPALA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Armstrong resolved IMPALA-8696. --- Resolution: Fixed Fix Version/s: Impala 3.3.0 > Cmake dependencies for container build do not work correctly > > > Key: IMPALA-8696 > URL: https://issues.apache.org/jira/browse/IMPALA-8696 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > Fix For: Impala 3.3.0 > > > I've noticed in many cases that containers are not reliably rebuilt when > inputs change, e.g. if the binaries are rebuilt. Investigate and fix this, > it's a major development headache. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (IMPALA-8622) Generate list of docker images generated by the build
[
https://issues.apache.org/jira/browse/IMPALA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong resolved IMPALA-8622.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Generate list of docker images generated by the build
> -
>
> Key: IMPALA-8622
> URL: https://issues.apache.org/jira/browse/IMPALA-8622
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: docker
> Fix For: Impala 3.3.0
>
>
> Some consumers of the Impala docker images will want to publish them to a
> repo. Currently they would have to hardcode a list of the docker images
> produced by the Impala build to publish them, or do something brittle like
> "docker image ls" to deduce what images were built.
> It would be useful if the build produced a list of the images in some
> consumable format that would be updated (either manually or automatically)
> when the set of images changes. If it needs to be manually updated, we should
> have some kind of check to verify that it isn't stale.
> One option is to produce JSON, something like:
> {noformat}
> {
> “docker_images”: {
>“impala_base”: “$docker_registry/impala_base”,
>” impalad_coord_exec”: “$docker_registry/impalad_coord_exec”,
>
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Work started] (IMPALA-8785) DEBUG build images should be tagged differently from release build
[ https://issues.apache.org/jira/browse/IMPALA-8785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8785 started by Tim Armstrong. - > DEBUG build images should be tagged differently from release build > -- > > Key: IMPALA-8785 > URL: https://issues.apache.org/jira/browse/IMPALA-8785 > Project: IMPALA > Issue Type: Improvement > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Critical > Labels: docker > > Currently the container build gives debug and release images the same tag > names. This is fine for development, since you'll generally be working on one > build at a time, but is confusing if you need to integrate this into a more > general build process, since debug and release images are really separate > artifacts. > One example I ran into is a case when I ran ./buildall.sh ... > -release_and_debug, then wanted to build docker images for the release > artifacts. This is not currently possible in a sane way. > I think we should tag images with a -debug suffix if they were generated from > debug artifacts. start-impala-cluster would need to be smart about picking > the image to use. We may want to have different build targets to build the > different flavours of image, too. > Note that there are many flavours of debug images, e.g. ASAN, but I think > simply separating release and debug would avoid a lot of confusion. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8434) Alter Db leads to functions missing unless run "refresh functions"
[
https://issues.apache.org/jira/browse/IMPALA-8434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891493#comment-16891493
]
Quanlong Huang commented on IMPALA-8434:
[~Xiaomeng Zhang], I think your solution make sense but I can't reproduce the
hanging of "invalidate metadata" at first. Finally, I figured out that if you
remove the codes for removing all versions of the tables and functions, the
"invalidate metadata" command do hang. So I guest your changes may be something
like this:
{code:java}
diff --git a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
index 13cb620..9ddb888 100644
--- a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
@@ -19,6 +19,7 @@ package org.apache.impala.catalog;
import java.nio.ByteBuffer;
import java.util.ArrayDeque;
+import java.util.List;
import java.util.Set;
import java.util.concurrent.atomic.AtomicLong;
import java.util.concurrent.atomic.AtomicReference;
@@ -390,11 +391,16 @@ public class ImpaladCatalog extends Catalog implements
FeCatalog {
newDb.setCatalogVersion(catalogVersion);
addDb(newDb);
if (existingDb != null) {
+for (Table tbl : existingDb.getTables()) {
+ newDb.addTable(tbl);
+}
+for (List functionList :
existingDb.getAllFunctions().values()) {
+ for (Function func : functionList) {
+newDb.addFunction(func);
+ }
+}
CatalogObjectVersionSet.INSTANCE.updateVersions(
existingDb.getCatalogVersion(), catalogVersion);
-CatalogObjectVersionSet.INSTANCE.removeAll(existingDb.getTables());
-CatalogObjectVersionSet.INSTANCE.removeAll(
-existingDb.getFunctions(null, new PatternMatcher()));
} else {
CatalogObjectVersionSet.INSTANCE.addVersion(catalogVersion);
}
{code}
Actually, we shouldn't remove the codes calling
"CatalogObjectVersionSet.INSTANCE.removeAll". Because "newDb.addTable" and
"newDb.addFunction" will also add versions into the CatalogObjectVersionSet
which is a *multi-set*. Removing those three lines causes the old table
versions being double counted. So the minVersion_ of CatalogObjectVersionSet
won't be updated since their counters can't decrease to 0.
For "invalidate metadata", in the coordinator side, database updates are
processed prior to the table/function updates:
[https://github.com/apache/impala/blob/eb97c746d2309fcf78ff3b50751cd5e27101539a/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L124-L129].
So even if we retain the old tables/functions of a db, they'll be updated
later in processing the table/function updates of this db. So I think it's no
problems for "invalidate metadata".
Just uploaded a patch for this: [https://gerrit.cloudera.org/c/13904/]
> Alter Db leads to functions missing unless run "refresh functions"
> ---
>
> Key: IMPALA-8434
> URL: https://issues.apache.org/jira/browse/IMPALA-8434
> Project: IMPALA
> Issue Type: Bug
>Reporter: Xiaomeng Zhang
>Assignee: Xiaomeng Zhang
>Priority: Critical
>
> I was testing on master branch. In a database with java and native functions.
> Run queries below, all functions are missing after alter db until run
> "refresh functions" in db.
> {code:java}
> [localhost:21000] xm> show functions;
> Query: show functions
> +-++-+---+
> | return type | signature | binary type | is persistent |
> +-++-+---+
> | STRING | add10impala(STRING) | JAVA | true |
> | STRING | add10udf(STRING) | JAVA | true |
> | INT | add2(INT, INT) | NATIVE | true |
> | INT | add2xm(INT, INT) | NATIVE | true |
> | INT | addtwomaster(INT, INT) | NATIVE | true |
> +-++-+---+
> Fetched 5 row(s) in 0.01s
> [localhost:21000] xm> alter database xm set owner user impala218;
> Query: alter database xm set owner user impala218
> +---+
> | summary |
> +---+
> | Updated database. |
> +---+
> Fetched 1 row(s) in 0.59s
> [localhost:21000] xm> show functions;
> Query: show functions
> Fetched 0 row(s) in 0.01s
> [localhost:21000] xm> refresh functions xm;
> Query: refresh functions xm
> Query submitted at: 2019-04-18 14:19:00 (Coordinator:
> http://xiaomeng-OptiPlex-9020:25000)
> Query progress can be monitored at:
> http://xiaomeng-OptiPlex-9020:25000/query_plan?query_id=fa40cdffde223550:df2a6cc0
> Fetched 0 row(s) in 0.08s
> [localhost:21000] xm> show functions;
> Query: show functions
>
[jira] [Commented] (IMPALA-8622) Generate list of docker images generated by the build
[
https://issues.apache.org/jira/browse/IMPALA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891480#comment-16891480
]
ASF subversion and git services commented on IMPALA-8622:
-
Commit f689daef7f140fb2e227ee96218f31076b5ba257 in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=f689dae ]
IMPALA-8622,IMPALA-8696: fix docker dependencies, add image list
Adds a plain-text space-separated image list in
docker/docker-images.txt. This is generated based on the images built by
CMake, so is kept in sync with images added to or removed from the
CMake file.
Duplicated logic per image is removed - instead there is a helper
function that is called for each daemon image to be built.
Rips out the timestamp mechanism that was intended to avoid unnecessary
container rebuilds, but has turned out to be brittle. Instead the
containers are rebuilt each time the rule is invoked.
This moves some subdirectories so that the image tag matches the
subdirectory, to simplify the build scripts.
Change-Id: I4d8e215e9b07c6491faa4751969a30f0ed373fe3
Reviewed-on: http://gerrit.cloudera.org:8080/13899
Tested-by: Impala Public Jenkins
Reviewed-by: Lars Volker
> Generate list of docker images generated by the build
> -
>
> Key: IMPALA-8622
> URL: https://issues.apache.org/jira/browse/IMPALA-8622
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: docker
>
> Some consumers of the Impala docker images will want to publish them to a
> repo. Currently they would have to hardcode a list of the docker images
> produced by the Impala build to publish them, or do something brittle like
> "docker image ls" to deduce what images were built.
> It would be useful if the build produced a list of the images in some
> consumable format that would be updated (either manually or automatically)
> when the set of images changes. If it needs to be manually updated, we should
> have some kind of check to verify that it isn't stale.
> One option is to produce JSON, something like:
> {noformat}
> {
> “docker_images”: {
>“impala_base”: “$docker_registry/impala_base”,
>” impalad_coord_exec”: “$docker_registry/impalad_coord_exec”,
>
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8696) Cmake dependencies for container build do not work correctly
[ https://issues.apache.org/jira/browse/IMPALA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891481#comment-16891481 ] ASF subversion and git services commented on IMPALA-8696: - Commit f689daef7f140fb2e227ee96218f31076b5ba257 in impala's branch refs/heads/master from Tim Armstrong [ https://gitbox.apache.org/repos/asf?p=impala.git;h=f689dae ] IMPALA-8622,IMPALA-8696: fix docker dependencies, add image list Adds a plain-text space-separated image list in docker/docker-images.txt. This is generated based on the images built by CMake, so is kept in sync with images added to or removed from the CMake file. Duplicated logic per image is removed - instead there is a helper function that is called for each daemon image to be built. Rips out the timestamp mechanism that was intended to avoid unnecessary container rebuilds, but has turned out to be brittle. Instead the containers are rebuilt each time the rule is invoked. This moves some subdirectories so that the image tag matches the subdirectory, to simplify the build scripts. Change-Id: I4d8e215e9b07c6491faa4751969a30f0ed373fe3 Reviewed-on: http://gerrit.cloudera.org:8080/13899 Tested-by: Impala Public Jenkins Reviewed-by: Lars Volker > Cmake dependencies for container build do not work correctly > > > Key: IMPALA-8696 > URL: https://issues.apache.org/jira/browse/IMPALA-8696 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > I've noticed in many cases that containers are not reliably rebuilt when > inputs change, e.g. if the binaries are rebuilt. Investigate and fix this, > it's a major development headache. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8616) Document --disconnected_session_timeout
[ https://issues.apache.org/jira/browse/IMPALA-8616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8616 started by Alex Rodoni. --- > Document --disconnected_session_timeout > --- > > Key: IMPALA-8616 > URL: https://issues.apache.org/jira/browse/IMPALA-8616 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Thomas Tauber-Marshall >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > IMPALA-1653 added a new startup flag, --disconnected_session_timeout, which > we should document. > We might also want to document the new session behavior, that hs2 sessions > aren't automatically closed when the connection is closed, though I don't see > the old behavior documented anywhere so it may be sufficient just to explain > the flag. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8616) Document --disconnected_session_timeout
[ https://issues.apache.org/jira/browse/IMPALA-8616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891470#comment-16891470 ] Alex Rodoni commented on IMPALA-8616: - https://gerrit.cloudera.org/#/c/13903/ > Document --disconnected_session_timeout > --- > > Key: IMPALA-8616 > URL: https://issues.apache.org/jira/browse/IMPALA-8616 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Thomas Tauber-Marshall >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > IMPALA-1653 added a new startup flag, --disconnected_session_timeout, which > we should document. > We might also want to document the new session behavior, that hs2 sessions > aren't automatically closed when the connection is closed, though I don't see > the old behavior documented anywhere so it may be sufficient just to explain > the flag. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8746) Document the new query option DEFAULT_HINTS_INSERT_STATEMENT
[ https://issues.apache.org/jira/browse/IMPALA-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891449#comment-16891449 ] ASF subversion and git services commented on IMPALA-8746: - Commit 55e52287e7d8fb9f3a22a70c26627eabdcaa48d0 in impala's branch refs/heads/master from Alex Rodoni [ https://gitbox.apache.org/repos/asf?p=impala.git;h=55e5228 ] IMPALA-8746: [DOCS] Document the DEFAULT_HINTS_INSERT_STATEMENT query option Change-Id: Ia376721f46eb507901f9f64b5c3341dc0f36475b Reviewed-on: http://gerrit.cloudera.org:8080/13885 Tested-by: Impala Public Jenkins Reviewed-by: Bharath Vissapragada > Document the new query option DEFAULT_HINTS_INSERT_STATEMENT > > > Key: IMPALA-8746 > URL: https://issues.apache.org/jira/browse/IMPALA-8746 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Abhishek Rawat >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > Related to IMPALA-8673. > https://gerrit.cloudera.org/#/c/13885/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8765) Document JSON Udfs
[ https://issues.apache.org/jira/browse/IMPALA-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891451#comment-16891451 ] ASF subversion and git services commented on IMPALA-8765: - Commit 1f4e89e66ed45cb28dfa2c98023464621b3ee7ab in impala's branch refs/heads/master from Alex Rodoni [ https://gitbox.apache.org/repos/asf?p=impala.git;h=1f4e89e ] IMPALA-8765: [DOCS] Document the GET_JSON_OBJECT function Change-Id: I7135528e84f685bfe1c32d81f4cedb6afc133e04 Reviewed-on: http://gerrit.cloudera.org:8080/13886 Tested-by: Impala Public Jenkins Reviewed-by: Quanlong Huang > Document JSON Udfs > -- > > Key: IMPALA-8765 > URL: https://issues.apache.org/jira/browse/IMPALA-8765 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > It looks like we missed documenting the new builtin from the parent JIRA > https://gerrit.cloudera.org/#/c/13886/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8701) Document idle session management after IMPALA-7802
[
https://issues.apache.org/jira/browse/IMPALA-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891450#comment-16891450
]
ASF subversion and git services commented on IMPALA-8701:
-
Commit 1c9af2cb526b038c080f1b09acc403e8d7378364 in impala's branch
refs/heads/master from Alex Rodoni
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=1c9af2c ]
IMPALA-8701: [DOCS] Document --idle_client_poll_time_s flag
Change-Id: I32ace786904f564b9c5fa3ed594e2b679b76d5c6
Reviewed-on: http://gerrit.cloudera.org:8080/13896
Tested-by: Impala Public Jenkins
Reviewed-by: Michael Ho
> Document idle session management after IMPALA-7802
> --
>
> Key: IMPALA-8701
> URL: https://issues.apache.org/jira/browse/IMPALA-8701
> Project: IMPALA
> Issue Type: Task
> Components: Docs
>Affects Versions: Impala 3.3.0
>Reporter: Michael Ho
>Assignee: Alex Rodoni
>Priority: Major
> Labels: future_release_doc, in_33
> Fix For: Impala 3.3.0
>
>
> After IMPALA-7802 is fixed, the network connection of an idle session will be
> closed {{\-\-idle_client_poll_time_s}} seconds latest after the session has
> been declared idle. This is a change of behavior from previous versions in
> which the client connection will hang around until the user explicitly closes
> the session and disconnects.
> By default, {{\-\-idle_client_poll_time_s}} is set to 30 seconds and the
> previous behavior can be restored by setting to 0. In addition, the session
> will only be closed if idle session timeout has been configured to be greater
> than 0 for the session either via the startup flag {{--idle_session_timeout}}
> or the query option {{IDLE_SESSION_TIMEOUT}}. If idle session timeout is not
> configured, a session cannot become idle by definition and therefore its
> connection cannot be closed until the client explicitly closes it.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8765) Document JSON Udfs
[ https://issues.apache.org/jira/browse/IMPALA-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-8765. --- > Document JSON Udfs > -- > > Key: IMPALA-8765 > URL: https://issues.apache.org/jira/browse/IMPALA-8765 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > It looks like we missed documenting the new builtin from the parent JIRA > https://gerrit.cloudera.org/#/c/13886/ -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (IMPALA-8765) Document JSON Udfs
[ https://issues.apache.org/jira/browse/IMPALA-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-8765. --- > Document JSON Udfs > -- > > Key: IMPALA-8765 > URL: https://issues.apache.org/jira/browse/IMPALA-8765 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > It looks like we missed documenting the new builtin from the parent JIRA > https://gerrit.cloudera.org/#/c/13886/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8765) Document JSON Udfs
[ https://issues.apache.org/jira/browse/IMPALA-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni resolved IMPALA-8765. - Resolution: Fixed Fix Version/s: Impala 3.3.0 > Document JSON Udfs > -- > > Key: IMPALA-8765 > URL: https://issues.apache.org/jira/browse/IMPALA-8765 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > It looks like we missed documenting the new builtin from the parent JIRA > https://gerrit.cloudera.org/#/c/13886/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8765) Document JSON Udfs
[ https://issues.apache.org/jira/browse/IMPALA-8765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni resolved IMPALA-8765. - Resolution: Fixed Fix Version/s: Impala 3.3.0 > Document JSON Udfs > -- > > Key: IMPALA-8765 > URL: https://issues.apache.org/jira/browse/IMPALA-8765 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Tim Armstrong >Assignee: Alex Rodoni >Priority: Major > Labels: docs, future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > It looks like we missed documenting the new builtin from the parent JIRA > https://gerrit.cloudera.org/#/c/13886/ -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IMPALA-8787) Add compressed formats test for LZO
Ethan created IMPALA-8787: - Summary: Add compressed formats test for LZO Key: IMPALA-8787 URL: https://issues.apache.org/jira/browse/IMPALA-8787 Project: IMPALA Issue Type: Test Components: Backend Reporter: Ethan Currently, tests/query_test/test_compressed_formats.py, verifies that gzip, snappy, bzip, and deflate compressed RC, sequence, and text files are supported in Impala. There needs to be testing added for reading text files compressed using LZO. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8787) Add compressed formats test for LZO
Ethan created IMPALA-8787: - Summary: Add compressed formats test for LZO Key: IMPALA-8787 URL: https://issues.apache.org/jira/browse/IMPALA-8787 Project: IMPALA Issue Type: Test Components: Backend Reporter: Ethan Currently, tests/query_test/test_compressed_formats.py, verifies that gzip, snappy, bzip, and deflate compressed RC, sequence, and text files are supported in Impala. There needs to be testing added for reading text files compressed using LZO. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (IMPALA-7991) Impala 3.3 Doc: Doc Parquet page index
[ https://issues.apache.org/jira/browse/IMPALA-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-7991 started by Alex Rodoni. --- > Impala 3.3 Doc: Doc Parquet page index > -- > > Key: IMPALA-7991 > URL: https://issues.apache.org/jira/browse/IMPALA-7991 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-7991) Impala 3.3 Doc: Doc Parquet page index
[ https://issues.apache.org/jira/browse/IMPALA-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-7991: Description: https://gerrit.cloudera.org/#/c/13900/ > Impala 3.3 Doc: Doc Parquet page index > -- > > Key: IMPALA-7991 > URL: https://issues.apache.org/jira/browse/IMPALA-7991 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13900/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8470) Impala Doc: Use a more human-readable flag to switch to a different authorization provider
[ https://issues.apache.org/jira/browse/IMPALA-8470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni resolved IMPALA-8470. - Resolution: Fixed Fix Version/s: Impala 3.3.0 An example of the flag added to impala_authorization.xml. > Impala Doc: Use a more human-readable flag to switch to a different > authorization provider > -- > > Key: IMPALA-8470 > URL: https://issues.apache.org/jira/browse/IMPALA-8470 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (IMPALA-8470) Impala Doc: Use a more human-readable flag to switch to a different authorization provider
[ https://issues.apache.org/jira/browse/IMPALA-8470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni resolved IMPALA-8470. - Resolution: Fixed Fix Version/s: Impala 3.3.0 An example of the flag added to impala_authorization.xml. > Impala Doc: Use a more human-readable flag to switch to a different > authorization provider > -- > > Key: IMPALA-8470 > URL: https://issues.apache.org/jira/browse/IMPALA-8470 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8470) Impala Doc: Use a more human-readable flag to switch to a different authorization provider
[ https://issues.apache.org/jira/browse/IMPALA-8470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-8470. --- > Impala Doc: Use a more human-readable flag to switch to a different > authorization provider > -- > > Key: IMPALA-8470 > URL: https://issues.apache.org/jira/browse/IMPALA-8470 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Closed] (IMPALA-8470) Impala Doc: Use a more human-readable flag to switch to a different authorization provider
[ https://issues.apache.org/jira/browse/IMPALA-8470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-8470. --- > Impala Doc: Use a more human-readable flag to switch to a different > authorization provider > -- > > Key: IMPALA-8470 > URL: https://issues.apache.org/jira/browse/IMPALA-8470 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8701) Document idle session management after IMPALA-7802
[
https://issues.apache.org/jira/browse/IMPALA-8701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Rodoni closed IMPALA-8701.
---
Resolution: Fixed
Fix Version/s: Impala 3.3.0
> Document idle session management after IMPALA-7802
> --
>
> Key: IMPALA-8701
> URL: https://issues.apache.org/jira/browse/IMPALA-8701
> Project: IMPALA
> Issue Type: Task
> Components: Docs
>Affects Versions: Impala 3.3.0
>Reporter: Michael Ho
>Assignee: Alex Rodoni
>Priority: Major
> Labels: future_release_doc, in_33
> Fix For: Impala 3.3.0
>
>
> After IMPALA-7802 is fixed, the network connection of an idle session will be
> closed {{\-\-idle_client_poll_time_s}} seconds latest after the session has
> been declared idle. This is a change of behavior from previous versions in
> which the client connection will hang around until the user explicitly closes
> the session and disconnects.
> By default, {{\-\-idle_client_poll_time_s}} is set to 30 seconds and the
> previous behavior can be restored by setting to 0. In addition, the session
> will only be closed if idle session timeout has been configured to be greater
> than 0 for the session either via the startup flag {{--idle_session_timeout}}
> or the query option {{IDLE_SESSION_TIMEOUT}}. If idle session timeout is not
> configured, a session cannot become idle by definition and therefore its
> connection cannot be closed until the client explicitly closes it.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-5278) Add Kerberos HTTP SPNEGO authentication support to Impala web consoles
[ https://issues.apache.org/jira/browse/IMPALA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891367#comment-16891367 ] Chris A commented on IMPALA-5278: - Thank you! > Add Kerberos HTTP SPNEGO authentication support to Impala web consoles > -- > > Key: IMPALA-5278 > URL: https://issues.apache.org/jira/browse/IMPALA-5278 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Clients >Reporter: Chris A >Priority: Major > Labels: kerberos, security, spnego, web-console, web-ui > Fix For: Impala 3.3.0 > > > https://issues.apache.org/jira/browse/HADOOP-7119 and > https://issues.apache.org/jira/browse/HBASE-5291 show SPNEGO being added to > web consoles and web UIs to provide security through Kerberos (as a single > sign-on solution) > According to current documentation > (https://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_kerberos.html), > Kerberos HTTP SPNEGO is not supported in Impala's Web UIs. Is there a way to > implement this in the Impala WebUIs? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8575) Impala Doc: New query options for Parquet writing
[ https://issues.apache.org/jira/browse/IMPALA-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8575 started by Alex Rodoni. --- > Impala Doc: New query options for Parquet writing > - > > Key: IMPALA-8575 > URL: https://issues.apache.org/jira/browse/IMPALA-8575 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8575) Impala Doc: New query options for Parquet writing
[ https://issues.apache.org/jira/browse/IMPALA-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni updated IMPALA-8575: Description: https://gerrit.cloudera.org/#/c/13900/ > Impala Doc: New query options for Parquet writing > - > > Key: IMPALA-8575 > URL: https://issues.apache.org/jira/browse/IMPALA-8575 > Project: IMPALA > Issue Type: Sub-task > Components: Docs >Reporter: Alex Rodoni >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > > https://gerrit.cloudera.org/#/c/13900/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8786) BufferedPlanRootSink should directly write to a QueryResultSet if one is available
Sahil Takiar created IMPALA-8786:
Summary: BufferedPlanRootSink should directly write to a
QueryResultSet if one is available
Key: IMPALA-8786
URL: https://issues.apache.org/jira/browse/IMPALA-8786
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
{{BufferedPlanRootSink}} uses a {{RowBatchQueue}} to buffer {{RowBatch}}-es and
then the consumer thread reads them and writes them to a given
{{QueryResultSet}}. Implementations of {{RowBatchQueue}} might end up copying
the buffered {{RowBatch}}-es (e.g. if the queue is backed by a
{{BufferedTupleStream}}). An optimization would be for the producer thread to
directly write to the consumer {{QueryResultSet}}. This optimization would only
be triggered if (1) the queue is empty, and (2) the consumer thread has a
{{QueryResultSet}} available for writing.
This "fast path" is useful in a few different scenarios:
* If the consumer is faster than at reading rows than the producer is at
sending them; in this case, the overhead of buffering rows in a
{{RowBatchQueue}} can be completely avoided
* For queries that return under 1024 its likely that the consumer will produce
a {{QueryResultSet}} before the first {{RowBatch}} is returned (except perhaps
for very trivial queries)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8786) BufferedPlanRootSink should directly write to a QueryResultSet if one is available
Sahil Takiar created IMPALA-8786:
Summary: BufferedPlanRootSink should directly write to a
QueryResultSet if one is available
Key: IMPALA-8786
URL: https://issues.apache.org/jira/browse/IMPALA-8786
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
{{BufferedPlanRootSink}} uses a {{RowBatchQueue}} to buffer {{RowBatch}}-es and
then the consumer thread reads them and writes them to a given
{{QueryResultSet}}. Implementations of {{RowBatchQueue}} might end up copying
the buffered {{RowBatch}}-es (e.g. if the queue is backed by a
{{BufferedTupleStream}}). An optimization would be for the producer thread to
directly write to the consumer {{QueryResultSet}}. This optimization would only
be triggered if (1) the queue is empty, and (2) the consumer thread has a
{{QueryResultSet}} available for writing.
This "fast path" is useful in a few different scenarios:
* If the consumer is faster than at reading rows than the producer is at
sending them; in this case, the overhead of buffering rows in a
{{RowBatchQueue}} can be completely avoided
* For queries that return under 1024 its likely that the consumer will produce
a {{QueryResultSet}} before the first {{RowBatch}} is returned (except perhaps
for very trivial queries)
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IMPALA-8785) DEBUG build images should be tagged differently from release build
Tim Armstrong created IMPALA-8785: - Summary: DEBUG build images should be tagged differently from release build Key: IMPALA-8785 URL: https://issues.apache.org/jira/browse/IMPALA-8785 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Tim Armstrong Assignee: Tim Armstrong Currently the container build gives debug and release images the same tag names. This is fine for development, since you'll generally be working on one build at a time, but is confusing if you need to integrate this into a more general build process, since debug and release images are really separate artifacts. One example I ran into is a case when I ran ./buildall.sh ... -release_and_debug, then wanted to build docker images for the release artifacts. This is not currently possible in a sane way. I think we should tag images with a -debug suffix if they were generated from debug artifacts. start-impala-cluster would need to be smart about picking the image to use. We may want to have different build targets to build the different flavours of image, too. Note that there are many flavours of debug images, e.g. ASAN, but I think simply separating release and debug would avoid a lot of confusion. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IMPALA-8785) DEBUG build images should be tagged differently from release build
Tim Armstrong created IMPALA-8785: - Summary: DEBUG build images should be tagged differently from release build Key: IMPALA-8785 URL: https://issues.apache.org/jira/browse/IMPALA-8785 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 3.3.0 Reporter: Tim Armstrong Assignee: Tim Armstrong Currently the container build gives debug and release images the same tag names. This is fine for development, since you'll generally be working on one build at a time, but is confusing if you need to integrate this into a more general build process, since debug and release images are really separate artifacts. One example I ran into is a case when I ran ./buildall.sh ... -release_and_debug, then wanted to build docker images for the release artifacts. This is not currently possible in a sane way. I think we should tag images with a -debug suffix if they were generated from debug artifacts. start-impala-cluster would need to be smart about picking the image to use. We may want to have different build targets to build the different flavours of image, too. Note that there are many flavours of debug images, e.g. ASAN, but I think simply separating release and debug would avoid a lot of confusion. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8784) Implement a RowBatchQueue backed by a BufferedTupleStream
Sahil Takiar created IMPALA-8784:
Summary: Implement a RowBatchQueue backed by a BufferedTupleStream
Key: IMPALA-8784
URL: https://issues.apache.org/jira/browse/IMPALA-8784
Project: IMPALA
Issue Type: Sub-task
Components: Backend
Reporter: Sahil Takiar
Assignee: Sahil Takiar
The {{BufferedPlanRootSink}} should use a {{RowBatchQueue}} backed by a
{{BufferedTupleStream}}. This requires the following changes:
* Creating a new {{SpillableRowBatchQueue}} that implements {{RowBatchQueue}}
and internally uses a {{BufferedTupleStream}}
* Changing the implementation of {{RowBatchQueue}} used by
{{BufferedPlanRootSink}} to {{SpillableRowBatchQueue}}
* Update {{PlanRootSink.java}} so that it sets a {{ResourceProfile}} that
should be used by the {{BufferedPlanRootSink}}
* Update {{DataSinks.thrift}} so that it passes {{ResourceProfile}}-s from the
fe/ to the be/
* {{BufferedPlanRootSink}} should Initialize and close a
{{ReservationManager}} to be used by the {{BufferedTupleStream}}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8781) Add additional tests in test_result_spooling.py and validate cancellation logic
[
https://issues.apache.org/jira/browse/IMPALA-8781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated IMPALA-8781:
-
Description: {{test_result_spooling.py}} currently runs a few basic tests
with result spooling enabled. We should add some more to cover all necessary
edge cases (ensure all Impala types are returned correctly, UDFs are evaluated
correctly, etc.) and add tests to validate the cancellation logic in
{{PlanRootSink}}. (was: {{test_result_spooling.py}} currently runs a few basic
tests with result spooling enabled. We should add some more and validate the
cancellation logic in {{PlanRootSink}}.)
> Add additional tests in test_result_spooling.py and validate cancellation
> logic
> ---
>
> Key: IMPALA-8781
> URL: https://issues.apache.org/jira/browse/IMPALA-8781
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{test_result_spooling.py}} currently runs a few basic tests with result
> spooling enabled. We should add some more to cover all necessary edge cases
> (ensure all Impala types are returned correctly, UDFs are evaluated
> correctly, etc.) and add tests to validate the cancellation logic in
> {{PlanRootSink}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-8780) Implementation of BufferedPlanRootSink where FlushFinal blocks until all rows are fetched
[
https://issues.apache.org/jira/browse/IMPALA-8780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated IMPALA-8780:
-
Description:
Implement {{BufferedPlanRootSink}} so that {{FlushFinal}} blocks until all rows
are fetched. The implementation should use the {{RowBatchQueue}} introduced by
IMPALA-8779. By blocking in {{FlushFinal}} all non-coordinator fragments will
be closed if all results fit in the {{RowBatchQueue}}.
{{BufferedPlanRootSink::Send}} should enqueue each given {{RowBatch}} onto the
queue and then return. If the queue is full, it should block until there is
more space left in the queue. {{BufferedPlanRootSink::GetNext}} reads from the
queue and then fills in the given {{QueryResultSet}} by using the {{DataSink}}
{{ScalarExprEvaluator}}-s. Since the producer thread can call
{{BufferedPlanRootSink::Close}} while the consumer is calling
{{BufferedPlanRootSink::GetNext}} the two methods need to be synchronized so
that the {{DataSink}} {{MemTracker}}-s are not closed while {{GetNext}} is
running.
The implementation of {{BufferedPlanRootSink}} should remain the same
regardless of whether a {{std::queue}} backed {{RowBatchQueue}} or a
{{BufferedTupleStream}} backed {{RowBatchQueue}} is used.
{{BufferedPlanRootSink}} and {{BlockingPlanRootSink}} are similar in the sense
that {{BlockingPlanRootSink}} buffers one {{RowBatch}}, so for queries that
return under 1024 rows, all non-coordinator fragments are closed immediately as
well. The advantage of {{BufferedPlanRootSink}} is that allows buffering for 1+
{{RowBatch}}-es.
was:Implement {{BufferedPlanRootSink}} so that {{FlushFinal}} blocks until
all rows are fetched. The implementation should use the {{RowBatchQueue}}
introduced by IMPALA-8779. By blocking in {{FlushFinal}} all non-coordinator
fragments will be closed if all results fit in the {{RowBatchQueue}}.
> Implementation of BufferedPlanRootSink where FlushFinal blocks until all rows
> are fetched
> -
>
> Key: IMPALA-8780
> URL: https://issues.apache.org/jira/browse/IMPALA-8780
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> Implement {{BufferedPlanRootSink}} so that {{FlushFinal}} blocks until all
> rows are fetched. The implementation should use the {{RowBatchQueue}}
> introduced by IMPALA-8779. By blocking in {{FlushFinal}} all non-coordinator
> fragments will be closed if all results fit in the {{RowBatchQueue}}.
> {{BufferedPlanRootSink::Send}} should enqueue each given {{RowBatch}} onto
> the queue and then return. If the queue is full, it should block until there
> is more space left in the queue. {{BufferedPlanRootSink::GetNext}} reads from
> the queue and then fills in the given {{QueryResultSet}} by using the
> {{DataSink}} {{ScalarExprEvaluator}}-s. Since the producer thread can call
> {{BufferedPlanRootSink::Close}} while the consumer is calling
> {{BufferedPlanRootSink::GetNext}} the two methods need to be synchronized so
> that the {{DataSink}} {{MemTracker}}-s are not closed while {{GetNext}} is
> running.
> The implementation of {{BufferedPlanRootSink}} should remain the same
> regardless of whether a {{std::queue}} backed {{RowBatchQueue}} or a
> {{BufferedTupleStream}} backed {{RowBatchQueue}} is used.
> {{BufferedPlanRootSink}} and {{BlockingPlanRootSink}} are similar in the
> sense that {{BlockingPlanRootSink}} buffers one {{RowBatch}}, so for queries
> that return under 1024 rows, all non-coordinator fragments are closed
> immediately as well. The advantage of {{BufferedPlanRootSink}} is that allows
> buffering for 1+ {{RowBatch}}-es.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8746) Document the new query option DEFAULT_HINTS_INSERT_STATEMENT
[ https://issues.apache.org/jira/browse/IMPALA-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-8746. --- > Document the new query option DEFAULT_HINTS_INSERT_STATEMENT > > > Key: IMPALA-8746 > URL: https://issues.apache.org/jira/browse/IMPALA-8746 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Abhishek Rawat >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > Related to IMPALA-8673. > https://gerrit.cloudera.org/#/c/13885/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-8746) Document the new query option DEFAULT_HINTS_INSERT_STATEMENT
[ https://issues.apache.org/jira/browse/IMPALA-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni resolved IMPALA-8746. - Resolution: Fixed Fix Version/s: Impala 3.3.0 > Document the new query option DEFAULT_HINTS_INSERT_STATEMENT > > > Key: IMPALA-8746 > URL: https://issues.apache.org/jira/browse/IMPALA-8746 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Abhishek Rawat >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > Related to IMPALA-8673. > https://gerrit.cloudera.org/#/c/13885/ -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (IMPALA-8746) Document the new query option DEFAULT_HINTS_INSERT_STATEMENT
[ https://issues.apache.org/jira/browse/IMPALA-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni resolved IMPALA-8746. - Resolution: Fixed Fix Version/s: Impala 3.3.0 > Document the new query option DEFAULT_HINTS_INSERT_STATEMENT > > > Key: IMPALA-8746 > URL: https://issues.apache.org/jira/browse/IMPALA-8746 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Abhishek Rawat >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > Related to IMPALA-8673. > https://gerrit.cloudera.org/#/c/13885/ -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-8746) Document the new query option DEFAULT_HINTS_INSERT_STATEMENT
[ https://issues.apache.org/jira/browse/IMPALA-8746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Rodoni closed IMPALA-8746. --- > Document the new query option DEFAULT_HINTS_INSERT_STATEMENT > > > Key: IMPALA-8746 > URL: https://issues.apache.org/jira/browse/IMPALA-8746 > Project: IMPALA > Issue Type: Task > Components: Docs >Reporter: Abhishek Rawat >Assignee: Alex Rodoni >Priority: Major > Labels: future_release_doc, in_33 > Fix For: Impala 3.3.0 > > > Related to IMPALA-8673. > https://gerrit.cloudera.org/#/c/13885/ -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IMPALA-8783) Add Kerberos SPNEGO support to the http hs2 server
Thomas Tauber-Marshall created IMPALA-8783: -- Summary: Add Kerberos SPNEGO support to the http hs2 server Key: IMPALA-8783 URL: https://issues.apache.org/jira/browse/IMPALA-8783 Project: IMPALA Issue Type: Bug Components: Clients Affects Versions: Impala 3.3.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall IMPALA-8538 added support for http connections to the hs2 server along with LDAP auth support. We should also add support for Kerberos auth via SPNEGO -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8783) Add Kerberos SPNEGO support to the http hs2 server
Thomas Tauber-Marshall created IMPALA-8783: -- Summary: Add Kerberos SPNEGO support to the http hs2 server Key: IMPALA-8783 URL: https://issues.apache.org/jira/browse/IMPALA-8783 Project: IMPALA Issue Type: Bug Components: Clients Affects Versions: Impala 3.3.0 Reporter: Thomas Tauber-Marshall Assignee: Thomas Tauber-Marshall IMPALA-8538 added support for http connections to the hs2 server along with LDAP auth support. We should also add support for Kerberos auth via SPNEGO -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work started] (IMPALA-8696) Cmake dependencies for container build do not work correctly
[ https://issues.apache.org/jira/browse/IMPALA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-8696 started by Tim Armstrong. - > Cmake dependencies for container build do not work correctly > > > Key: IMPALA-8696 > URL: https://issues.apache.org/jira/browse/IMPALA-8696 > Project: IMPALA > Issue Type: Sub-task > Components: Infrastructure >Affects Versions: Impala 3.3.0 >Reporter: Tim Armstrong >Assignee: Tim Armstrong >Priority: Major > > I've noticed in many cases that containers are not reliably rebuilt when > inputs change, e.g. if the binaries are rebuilt. Investigate and fix this, > it's a major development headache. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-8622) Generate list of docker images generated by the build
[
https://issues.apache.org/jira/browse/IMPALA-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-8622 started by Tim Armstrong.
-
> Generate list of docker images generated by the build
> -
>
> Key: IMPALA-8622
> URL: https://issues.apache.org/jira/browse/IMPALA-8622
> Project: IMPALA
> Issue Type: Sub-task
> Components: Infrastructure
>Reporter: Tim Armstrong
>Assignee: Tim Armstrong
>Priority: Major
> Labels: docker
>
> Some consumers of the Impala docker images will want to publish them to a
> repo. Currently they would have to hardcode a list of the docker images
> produced by the Impala build to publish them, or do something brittle like
> "docker image ls" to deduce what images were built.
> It would be useful if the build produced a list of the images in some
> consumable format that would be updated (either manually or automatically)
> when the set of images changes. If it needs to be manually updated, we should
> have some kind of check to verify that it isn't stale.
> One option is to produce JSON, something like:
> {noformat}
> {
> “docker_images”: {
>“impala_base”: “$docker_registry/impala_base”,
>” impalad_coord_exec”: “$docker_registry/impalad_coord_exec”,
>
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5278) Add Kerberos HTTP SPNEGO authentication support to Impala web consoles
[ https://issues.apache.org/jira/browse/IMPALA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-5278. Resolution: Fixed Fix Version/s: Impala 3.3.0 commit 9f0cd9743a9c364d1eb42f29f67494298ed574ae Author: Todd Lipcon Date: Mon Jul 1 16:59:24 2019 -0700 Support SPNEGO for Impala webserver This ports over changes from kudu commit 1f291b77ef0868ac888a850678adc2d7cce65529 which implemented SPNEGO for the Kudu webserver. Unfortunately, thorough testing of this is difficult given that curl isn't currently in the toolchain. I was able to manually test this by adding a 'sleep(1000)' call into the newly added test case, then setting up $KRB5_CONFIG in my shell to point to the temporary KDC's environment, and using 'curl -u : --negotiate http://...' to authenticate. Strangely, using the version of curl on el7 didn't seem to work properly (perhaps an el7 curl bug) but using curl on my Ubuntu 18 laptop I was able to authenticate with SPNEGO. Change-Id: Ife2b04310e1571d231bf8ee1bcfd3b7afc2edd8f Reviewed-on: http://gerrit.cloudera.org:8080/13774 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add Kerberos HTTP SPNEGO authentication support to Impala web consoles > -- > > Key: IMPALA-5278 > URL: https://issues.apache.org/jira/browse/IMPALA-5278 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Clients >Reporter: Chris A >Priority: Major > Labels: kerberos, security, spnego, web-console, web-ui > Fix For: Impala 3.3.0 > > > https://issues.apache.org/jira/browse/HADOOP-7119 and > https://issues.apache.org/jira/browse/HBASE-5291 show SPNEGO being added to > web consoles and web UIs to provide security through Kerberos (as a single > sign-on solution) > According to current documentation > (https://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_kerberos.html), > Kerberos HTTP SPNEGO is not supported in Impala's Web UIs. Is there a way to > implement this in the Impala WebUIs? -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-5278) Add Kerberos HTTP SPNEGO authentication support to Impala web consoles
[ https://issues.apache.org/jira/browse/IMPALA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Tauber-Marshall resolved IMPALA-5278. Resolution: Fixed Fix Version/s: Impala 3.3.0 commit 9f0cd9743a9c364d1eb42f29f67494298ed574ae Author: Todd Lipcon Date: Mon Jul 1 16:59:24 2019 -0700 Support SPNEGO for Impala webserver This ports over changes from kudu commit 1f291b77ef0868ac888a850678adc2d7cce65529 which implemented SPNEGO for the Kudu webserver. Unfortunately, thorough testing of this is difficult given that curl isn't currently in the toolchain. I was able to manually test this by adding a 'sleep(1000)' call into the newly added test case, then setting up $KRB5_CONFIG in my shell to point to the temporary KDC's environment, and using 'curl -u : --negotiate http://...' to authenticate. Strangely, using the version of curl on el7 didn't seem to work properly (perhaps an el7 curl bug) but using curl on my Ubuntu 18 laptop I was able to authenticate with SPNEGO. Change-Id: Ife2b04310e1571d231bf8ee1bcfd3b7afc2edd8f Reviewed-on: http://gerrit.cloudera.org:8080/13774 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins > Add Kerberos HTTP SPNEGO authentication support to Impala web consoles > -- > > Key: IMPALA-5278 > URL: https://issues.apache.org/jira/browse/IMPALA-5278 > Project: IMPALA > Issue Type: Improvement > Components: Catalog, Clients >Reporter: Chris A >Priority: Major > Labels: kerberos, security, spnego, web-console, web-ui > Fix For: Impala 3.3.0 > > > https://issues.apache.org/jira/browse/HADOOP-7119 and > https://issues.apache.org/jira/browse/HBASE-5291 show SPNEGO being added to > web consoles and web UIs to provide security through Kerberos (as a single > sign-on solution) > According to current documentation > (https://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_kerberos.html), > Kerberos HTTP SPNEGO is not supported in Impala's Web UIs. Is there a way to > implement this in the Impala WebUIs? -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (IMPALA-7936) Enable better control over Parquet writing
[ https://issues.apache.org/jira/browse/IMPALA-7936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891141#comment-16891141 ] Zoltán Borók-Nagy commented on IMPALA-7936: --- Normally the users won't need to set these query options, but I think it would be better to have these documented anyway. > Enable better control over Parquet writing > -- > > Key: IMPALA-7936 > URL: https://issues.apache.org/jira/browse/IMPALA-7936 > Project: IMPALA > Issue Type: Improvement >Reporter: Zoltán Borók-Nagy >Assignee: Zoltán Borók-Nagy >Priority: Major > Fix For: Impala 3.3.0 > > > With the introduction of the Parquet page indexes it became desirable to have > more control over how Impala writes Parquet files. > These configuration options (probably implemented as query options) would be: > * enable/disable Parquet page index writing (currently we can do it with a > command-line argument) > * set page-size limits based on row count > * -Set truncation length for statistics about string values- (current > truncation length is 64, it is unlikely to have user data that needs longer > truncation than that) > They'd enable writing more complete tests for page filtering. They'd be also > useful for fine-tuning the page index for better performance. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8742) ScannerContext::eosr() should depends on scan_range->bytes_to_read() instead of scan_range->len()
[ https://issues.apache.org/jira/browse/IMPALA-8742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891133#comment-16891133 ] Zoltán Borók-Nagy commented on IMPALA-8742: --- Stream::bytes_left() also uses ScanRange::len() instead of bytes_to_read(). We don't hit this bug because in the Parquet scanner we issue Stream::GetBuffer() only in ReadPageHeader() and we track how many pages we need to read. There is a TODO in BaseScalarColumnReader::ReadDataPage() about adding a check for stream_->eosr(), maybe we should add that check as well after fixing this bug. I think I'll add some backend tests for the Stream class, unfortunately there isn't any yet. The other scanners don't issue sub-ranges. > ScannerContext::eosr() should depends on scan_range->bytes_to_read() instead > of scan_range->len() > - > > Key: IMPALA-8742 > URL: https://issues.apache.org/jira/browse/IMPALA-8742 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 3.2.0 >Reporter: Chen Wang >Assignee: Zoltán Borók-Nagy >Priority: Minor > > Normally, scan_range->len() equals to scan_range->bytes_to_read(). > But when SubRanges exists, only bytes_to_read was updated, > ScannerContext::GetBuffer would read entire scan_range->len() bytes, and I > think it would be a bug. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-8782) Impala errors can make profiles huge
Gabor Kaszab created IMPALA-8782:
Summary: Impala errors can make profiles huge
Key: IMPALA-8782
URL: https://issues.apache.org/jira/browse/IMPALA-8782
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.9.0
Reporter: Gabor Kaszab
We observed that in some cases Impala could flood the query profile with error
messages causing other systems that read the profiles crash (e.g. Cloudera
Manager) due to the size of the profile. Let's consider limiting the number of
errors we write into the profile.
In the particular case I saw the profile was full of these errors, one of this
for each row:
{code:java}
Error converting column: 4 to TIMESTAMP\nError converting column: 10 to
INT\nError converting column: 11 to INT\nError converting column: 13 to
INT\nError converting column: 15 to INT\nError converting column: 16 to
INT\nError converting column: 19 to INT\nError converting column: 20 to
INT\nError converting column: 21 to INT\nError converting column: 22 to
INT\nError converting column: 23 to INT\nError converting column: 24 to
INT\nError converting column: 26 to INT\nError converting column: 29 to
INT\nError converting column: 31 to INT\nError converting column: 35 to
INT\nError converting column: 36 to INT\nError converting column: 37 to
INT\nError converting column: 39 to INT\nError converting column: 40 to
INT\nError converting column: 41 to INT\nError converting column: 42 to
INT\nError converting column: 43 to INT\nError converting column: 44 to
INT\nError converting column: 45 to INT\nError converting column: 46 to
INT\nError converting column: 48 to INT\nError converting column: 50 to
INT\nError converting column: 51 to INT\nError converting column: 53 to
INT\nError converting column: 54 to INT\nError converting column: 59 to
INT\nError converting column: 60 to INT\nError converting column: 63 to
INT\nError converting column: 64 to INT\nError converting column: 65 to
INT\nError converting column: 69 to INT\nError converting column: 70 to
INT\nError converting column: 71 to INT\nError converting column: 72 to
INT\nError converting column: 73 to INT\nError converting column: 75 to
INT\nError converting column: 76 to INT\nError converting column: 77 to
INT\nError converting column: 78 to INT\nError converting column: 79 to
INT\nError converting column: 80 to INT\nError converting column: 81 to
INT\nError converting column: 82 to INT\nError converting column: 83 to
INT\nError converting column: 84 to INT\nError converting column: 85 to
INT\nError converting column: 87 to INT\nError converting column: 88 to
INT\nError converting column: 90 to INT\nError converting column: 91 to
INT\nError converting column: 95 to INT\nError converting column: 96 to
INT\nError converting column: 97 to INT\nError converting column: 99 to
INT\nError converting column: 102 to INT\nError converting column: 103 to
INT\nError converting column: 104 to INT\nError converting column: 106 to
INT\nError converting column: 107 to INT\nError converting column: 109 to
INT\nError converting column: 117 to INT\nError converting column: 119 to
INT\nError parsing row: file:
hdfs://nameserviceci/data/CTL/encrypt/db/ingest/raw/capm/file=interfaces/dt=20170626/interfaces.1498453200039.snappy,
before offset: 43276630\n
{code}
As a result, the uncompressed profile grew something like 700MB.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (IMPALA-8782) Impala errors can make profiles huge
Gabor Kaszab created IMPALA-8782:
Summary: Impala errors can make profiles huge
Key: IMPALA-8782
URL: https://issues.apache.org/jira/browse/IMPALA-8782
Project: IMPALA
Issue Type: Bug
Components: Backend
Affects Versions: Impala 2.9.0
Reporter: Gabor Kaszab
We observed that in some cases Impala could flood the query profile with error
messages causing other systems that read the profiles crash (e.g. Cloudera
Manager) due to the size of the profile. Let's consider limiting the number of
errors we write into the profile.
In the particular case I saw the profile was full of these errors, one of this
for each row:
{code:java}
Error converting column: 4 to TIMESTAMP\nError converting column: 10 to
INT\nError converting column: 11 to INT\nError converting column: 13 to
INT\nError converting column: 15 to INT\nError converting column: 16 to
INT\nError converting column: 19 to INT\nError converting column: 20 to
INT\nError converting column: 21 to INT\nError converting column: 22 to
INT\nError converting column: 23 to INT\nError converting column: 24 to
INT\nError converting column: 26 to INT\nError converting column: 29 to
INT\nError converting column: 31 to INT\nError converting column: 35 to
INT\nError converting column: 36 to INT\nError converting column: 37 to
INT\nError converting column: 39 to INT\nError converting column: 40 to
INT\nError converting column: 41 to INT\nError converting column: 42 to
INT\nError converting column: 43 to INT\nError converting column: 44 to
INT\nError converting column: 45 to INT\nError converting column: 46 to
INT\nError converting column: 48 to INT\nError converting column: 50 to
INT\nError converting column: 51 to INT\nError converting column: 53 to
INT\nError converting column: 54 to INT\nError converting column: 59 to
INT\nError converting column: 60 to INT\nError converting column: 63 to
INT\nError converting column: 64 to INT\nError converting column: 65 to
INT\nError converting column: 69 to INT\nError converting column: 70 to
INT\nError converting column: 71 to INT\nError converting column: 72 to
INT\nError converting column: 73 to INT\nError converting column: 75 to
INT\nError converting column: 76 to INT\nError converting column: 77 to
INT\nError converting column: 78 to INT\nError converting column: 79 to
INT\nError converting column: 80 to INT\nError converting column: 81 to
INT\nError converting column: 82 to INT\nError converting column: 83 to
INT\nError converting column: 84 to INT\nError converting column: 85 to
INT\nError converting column: 87 to INT\nError converting column: 88 to
INT\nError converting column: 90 to INT\nError converting column: 91 to
INT\nError converting column: 95 to INT\nError converting column: 96 to
INT\nError converting column: 97 to INT\nError converting column: 99 to
INT\nError converting column: 102 to INT\nError converting column: 103 to
INT\nError converting column: 104 to INT\nError converting column: 106 to
INT\nError converting column: 107 to INT\nError converting column: 109 to
INT\nError converting column: 117 to INT\nError converting column: 119 to
INT\nError parsing row: file:
hdfs://nameserviceci/data/CTL/encrypt/db/ingest/raw/capm/file=interfaces/dt=20170626/interfaces.1498453200039.snappy,
before offset: 43276630\n
{code}
As a result, the uncompressed profile grew something like 700MB.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-8378) test_cancel_select failed with Invalid query handle
[
https://issues.apache.org/jira/browse/IMPALA-8378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890958#comment-16890958
]
Quanlong Huang commented on IMPALA-8378:
Hit this too: [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/6721/]
> test_cancel_select failed with Invalid query handle
> ---
>
> Key: IMPALA-8378
> URL: https://issues.apache.org/jira/browse/IMPALA-8378
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 3.3.0
>Reporter: Gabor Kaszab
>Assignee: Andrew Sherman
>Priority: Critical
> Labels: broken-build
>
> Error Message
> {code:java}
> query_test/test_cancellation.py:241: in test_cancel_select
> self.execute_cancel_test(vector) query_test/test_cancellation.py:213: in
> execute_cancel_test assert 'Cancelled' in str(thread.fetch_results_error)
> E assert 'Cancelled' in "ImpalaBeeswaxException:\n INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: Invalid query handle:
> b64a91016fe343dd:fefaf600\n" E+ where "ImpalaBeeswaxException:\n
> INNER EXCEPTION: \n MESSAGE:
> Invalid query handle: b64a91016fe343dd:fefaf600\n" =
> str(ImpalaBeeswaxException()) E+where ImpalaBeeswaxException() =
> .fetch_results_error
> {code}
> Stacktrace
> {code:java}
> query_test/test_cancellation.py:241: in test_cancel_select
> self.execute_cancel_test(vector)
> query_test/test_cancellation.py:213: in execute_cancel_test
> assert 'Cancelled' in str(thread.fetch_results_error)
> E assert 'Cancelled' in "ImpalaBeeswaxException:\n INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: Invalid query handle:
> b64a91016fe343dd:fefaf600\n"
> E+ where "ImpalaBeeswaxException:\n INNER EXCEPTION: 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: Invalid query handle:
> b64a91016fe343dd:fefaf600\n" = str(ImpalaBeeswaxException())
> E+where ImpalaBeeswaxException() = 139721936148224)>.fetch_results_error
> {code}
> Standard error
> {code:java}
> SET
> client_identifier=query_test/test_cancellation.py::TestCancellationParallel::()::test_cancel_select[protocol:beeswax|table_format:rc/none|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_no;
> -- executing against localhost:21000
> use tpch_rc;
> -- 2019-03-30 23:15:53,293 INFO MainThread: Started query
> 5e42c7ec2df1febb:b25eff72
> SET
> client_identifier=query_test/test_cancellation.py::TestCancellationParallel::()::test_cancel_select[protocol:beeswax|table_format:rc/none|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_no;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=0;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET cpu_limit_s=10;
> SET debug_action=0:GETNEXT:WAIT|COORD_CANCEL_QUERY_FINSTANCES_RPC:FAIL;
> SET exec_single_node_rows_threshold=0;
> SET buffer_pool_limit=0;
> -- executing async: localhost:21000
> compute stats lineitem;
> -- 2019-03-30 23:15:53,359 INFO MainThread: Started query
> f54a0ecfa32d40b4:07510d7a
> SET
> client_identifier=query_test/test_cancellation.py::TestCancellationParallel::()::test_cancel_select[protocol:beeswax|table_format:rc/none|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_no;
> -- connecting to: localhost:21000
> -- fetching results from: object at 0x5d535d0>
> -- getting state for operation:
>
> -- canceling operation: object at 0x5d535d0>
> -- 2019-03-30 23:15:53,427 INFO Thread-5: Starting new HTTP connection
> (1): localhost
> -- closing query for operation handle:
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
