[jira] [Updated] (IMPALA-5509) Runtime filter : Extend runtime filter to support Dictionary values
[ https://issues.apache.org/jira/browse/IMPALA-5509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Quanlong Huang updated IMPALA-5509: --- Fix Version/s: Impala 4.1.0 > Runtime filter : Extend runtime filter to support Dictionary values > --- > > Key: IMPALA-5509 > URL: https://issues.apache.org/jira/browse/IMPALA-5509 > Project: IMPALA > Issue Type: Bug > Components: Backend >Affects Versions: Impala 2.9.0 >Reporter: Alan Choi >Assignee: Csaba Ringhofer >Priority: Major > Labels: performance, runtime-filters > Fix For: Impala 4.1.0 > > > For runtime filter on a single column, it can be run against the dictionary > values in Parquet to enable efficient block filtering. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13192) Impala Coordinator stuck and Full GC when execute query from nested temporary table.
LiuYuan created IMPALA-13192:
Summary: Impala Coordinator stuck and Full GC when execute query
from nested temporary table.
Key: IMPALA-13192
URL: https://issues.apache.org/jira/browse/IMPALA-13192
Project: IMPALA
Issue Type: Bug
Components: Frontend
Environment: impalad version 4.3.0-RELEASE RELEASE (build
14bb13e67e48742df72f9e1dd73be15ec7ba31bd)
Reporter: LiuYuan
1.Create a table as below:
{code:java}
CREATE TABLE trunck_info (
user_id BIGINT ,
truck_length DOUBLE,
length_type STRING,
point_km DOUBLE,
estimate_mileage DOUBLE,
dep_rate DOUBLE,
line_day_cnt_01 BIGINT,
line_ly_cnt_01 BIGINT,
line_day_cnt_30 BIGINT,
line_ly_cnt_30 BIGINT,
line_day_cnt_60 BIGINT,
line_ly_cnt_60 BIGINT,
num_all_60 BIGINT,
num_est_60 BIGINT,
num_est_order_60 BIGINT,
num_act_60 BIGINT,
num_inh_60 BIGINT,
num_all_30 BIGINT,
num_est_30 BIGINT,
num_est_order_30 BIGINT,
num_act_30 BIGINT,
num_inh_30 BIGINT,
conn_num_60 BIGINT,
conn_num_30 BIGINT,
hp_num_60 INT,
hp_num_30 INT,
bzj_num INT,
feidan8_num_60 BIGINT,
feidan8_num_30 INT,
ts_num_60 BIGINT,
ts_num_30 INT,
new_mile_point_60 BIGINT,
new_mile_point_30 BIGINT
)
WITH SERDEPROPERTIES ('serialization.format'='1')
STORED AS TEXTFILE {code}
2.Query from nested temporary table, we can see coordinator hung and full gc
{panel:title=hung.sql}
with t1
as
(
select user_id
,nvl(num_inh_60,0)+nvl(conn_num_60,0)+nvl(new_mile_point_60,0) as score_all
, nvl(conn_num_60,0)+nvl(new_mile_point_60,0) as
score_noinh
from trunck_info
)
,t2
as
(
select user_id
,score_noinh + score_inh as score_all
,score_noinh
from
(
select user_id
,score_noinh
,case when score_all >= 800 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 600 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 450 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 300 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all > 0 then if(score_all*0.5 >=
450,450,score_all*0.5)
end as score_inh
from t1
where score_noinh > 0
) a
)
,t3
as
(
select user_id
,score_noinh + score_inh as score_all
,score_noinh
from
(
select user_id
,score_noinh
,case when score_all >= 800 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 600 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 450 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 300 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all > 0 then if(score_all*0.5 >=
450,450,score_all*0.5)
end as score_inh
from t2
where score_noinh > 0
) a
)
,t4
as
(
select user_id
,score_noinh + score_inh as score_all
,score_noinh
from
(
select user_id
,score_noinh
,case when score_all >= 800 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 600 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 450 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 300 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all > 0 then if(score_all*0.5 >=
450,450,score_all*0.5)
end as score_inh
from t3
where score_noinh > 0
) a
)
,t5
as
(
select user_id
,score_noinh + score_inh as score_all
,score_noinh
from
(
select user_id
,score_noinh
,case when score_all >= 800 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 600 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 450 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 300 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all > 0 then if(score_all*0.5 >=
450,450,score_all*0.5)
end as score_inh
from t4
where score_noinh > 0
) a
)
,t6
as
(
select user_id
,score_noinh + score_inh as score_all
,score_noinh
from
(
select user_id
,score_noinh
,case when score_all >= 800 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 600 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 450 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all >= 300 then if(score_all*0.5 >=
450,450,score_all*0.5)
when score_all > 0 then if(score_all*0.5 >=
450,450,score_all*0.5)
end as score_inh
from t5
where score_noinh > 0
) a
)
,t7
as
(
select user_id
,score_n
[jira] [Created] (IMPALA-13193) RuntimeFilter on parquet dictionary should evaluate null values
Quanlong Huang created IMPALA-13193:
---
Summary: RuntimeFilter on parquet dictionary should evaluate null
values
Key: IMPALA-13193
URL: https://issues.apache.org/jira/browse/IMPALA-13193
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Quanlong Huang
IMPALA-10910, IMPALA-5509 introduces an optimization to evaluate runtime filter
on parquet dictionary values. If non of the values can pass the check, the
whole row group will be skipped. However, NULL values are not included in the
parquet dictionary. Runtime filters that accept NULL values might incorrectly
reject the row group if none of the dictionary values can pass the check.
Here are steps to reproduce the bug:
{code:sql}
create table parq_tbl (id bigint, name string) stored as parquet;
insert into parq_tbl values (0, "abc"), (1, NULL), (2, NULL), (3, "abc");
create table dim_tbl (name string);
insert into dim_tbl values (NULL);
select * from parq_tbl p join dim_tbl d
on COALESCE(p.name, '') = COALESCE(d.name, '');{code}
The SELECT query should return 2 rows but now it returns 0 rows.
A workaround is to disable this optimization:
{code:sql}
set PARQUET_DICTIONARY_RUNTIME_FILTER_ENTRY_LIMIT=0;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13193) RuntimeFilter on parquet dictionary should evaluate null values
[
https://issues.apache.org/jira/browse/IMPALA-13193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13193:
Affects Version/s: Impala 4.4.0
Impala 4.3.0
Impala 4.1.2
Impala 4.1.1
Impala 4.2.0
Impala 4.1.0
> RuntimeFilter on parquet dictionary should evaluate null values
> ---
>
> Key: IMPALA-13193
> URL: https://issues.apache.org/jira/browse/IMPALA-13193
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, Impala 4.1.2,
> Impala 4.3.0, Impala 4.4.0
>Reporter: Quanlong Huang
>Priority: Critical
>
> IMPALA-10910, IMPALA-5509 introduces an optimization to evaluate runtime
> filter on parquet dictionary values. If non of the values can pass the check,
> the whole row group will be skipped. However, NULL values are not included in
> the parquet dictionary. Runtime filters that accept NULL values might
> incorrectly reject the row group if none of the dictionary values can pass
> the check.
> Here are steps to reproduce the bug:
> {code:sql}
> create table parq_tbl (id bigint, name string) stored as parquet;
> insert into parq_tbl values (0, "abc"), (1, NULL), (2, NULL), (3, "abc");
> create table dim_tbl (name string);
> insert into dim_tbl values (NULL);
> select * from parq_tbl p join dim_tbl d
> on COALESCE(p.name, '') = COALESCE(d.name, '');{code}
> The SELECT query should return 2 rows but now it returns 0 rows.
> A workaround is to disable this optimization:
> {code:sql}
> set PARQUET_DICTIONARY_RUNTIME_FILTER_ENTRY_LIMIT=0;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13192) Impala Coordinator stuck and Full GC when execute query from nested temporary table.
[
https://issues.apache.org/jira/browse/IMPALA-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Quanlong Huang updated IMPALA-13192:
Priority: Critical (was: Major)
> Impala Coordinator stuck and Full GC when execute query from nested temporary
> table.
>
>
> Key: IMPALA-13192
> URL: https://issues.apache.org/jira/browse/IMPALA-13192
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Environment: impalad version 4.3.0-RELEASE RELEASE (build
> 14bb13e67e48742df72f9e1dd73be15ec7ba31bd)
>Reporter: LiuYuan
>Priority: Critical
>
> 1.Create a table as below:
>
> {code:java}
> CREATE TABLE trunck_info (
> user_id BIGINT ,
> truck_length DOUBLE,
> length_type STRING,
> point_km DOUBLE,
> estimate_mileage DOUBLE,
> dep_rate DOUBLE,
> line_day_cnt_01 BIGINT,
> line_ly_cnt_01 BIGINT,
> line_day_cnt_30 BIGINT,
> line_ly_cnt_30 BIGINT,
> line_day_cnt_60 BIGINT,
> line_ly_cnt_60 BIGINT,
> num_all_60 BIGINT,
> num_est_60 BIGINT,
> num_est_order_60 BIGINT,
> num_act_60 BIGINT,
> num_inh_60 BIGINT,
> num_all_30 BIGINT,
> num_est_30 BIGINT,
> num_est_order_30 BIGINT,
> num_act_30 BIGINT,
> num_inh_30 BIGINT,
> conn_num_60 BIGINT,
> conn_num_30 BIGINT,
> hp_num_60 INT,
> hp_num_30 INT,
> bzj_num INT,
> feidan8_num_60 BIGINT,
> feidan8_num_30 INT,
> ts_num_60 BIGINT,
> ts_num_30 INT,
> new_mile_point_60 BIGINT,
> new_mile_point_30 BIGINT
> )
> WITH SERDEPROPERTIES ('serialization.format'='1')
> STORED AS TEXTFILE {code}
>
> 2.Query from nested temporary table, we can see coordinator hung and full gc
>
>
>
> {panel:title=hung.sql}
> with t1
> as
> (
> select user_id
> ,nvl(num_inh_60,0)+nvl(conn_num_60,0)+nvl(new_mile_point_60,0) as
> score_all
> , nvl(conn_num_60,0)+nvl(new_mile_point_60,0) as
> score_noinh
> from trunck_info
> )
> ,t2
> as
> (
> select user_id
> ,score_noinh + score_inh as score_all
> ,score_noinh
> from
> (
> select user_id
> ,score_noinh
> ,case when score_all >= 800 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 600 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 450 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 300 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all > 0 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> end as score_inh
> from t1
> where score_noinh > 0
> ) a
> )
> ,t3
> as
> (
> select user_id
> ,score_noinh + score_inh as score_all
> ,score_noinh
> from
> (
> select user_id
> ,score_noinh
> ,case when score_all >= 800 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 600 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 450 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 300 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all > 0 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> end as score_inh
> from t2
> where score_noinh > 0
> ) a
> )
> ,t4
> as
> (
> select user_id
> ,score_noinh + score_inh as score_all
> ,score_noinh
> from
> (
> select user_id
> ,score_noinh
> ,case when score_all >= 800 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 600 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 450 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 300 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all > 0 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> end as score_inh
> from t3
> where score_noinh > 0
> ) a
> )
> ,t5
> as
> (
> select user_id
> ,score_noinh + score_inh as score_all
> ,score_noinh
> from
> (
> select user_id
> ,score_noinh
> ,case when score_all >= 800 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 600 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 450 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all >= 300 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> when score_all > 0 then if(score_all*0.5 >=
> 450,450,score_all*0.5)
> end as score_inh
> from t4
> where score_noinh > 0
> ) a
> )
> ,t6
> as
> (
> select user_id
> ,score_noinh + score
[jira] [Created] (IMPALA-13194) Fast-serialize position delete records
Zoltán Borók-Nagy created IMPALA-13194:
--
Summary: Fast-serialize position delete records
Key: IMPALA-13194
URL: https://issues.apache.org/jira/browse/IMPALA-13194
Project: IMPALA
Issue Type: Improvement
Components: Backend
Reporter: Zoltán Borók-Nagy
Currently the serialization of position delete records are very wasteful. The
records contain slots 'file_path' and 'pos'. And what we do during
serialization is the following.
# Write fixed-size tuple that have a StringValue and a BigInt slot (20 bytes
in total)
# We copy the StringValue's contents after the tuple.
# We convert the StringValue slot to be an offset to the string data
So we end up having something like this:
{noformat}
+-+++-+++-+
| StringValue | BigInt | File path | StringValue | BigInt | File path
| ... |
+-+++-+++-+
| ptr, len | 42 | /.../a.parquet | ptr, len | 43 |
/.../a.parquet | ... |
+-+++-+++-+
{noformat}
This is very redundant to store the file paths that way, and at the end we will
have a huge buffer that we need to compress and send over the network.
Moreover, we copy the file paths in memory twice:
# From input row batch to the KrpcDataStreamSender::Channel's temporary row
batch
# From the temporary row batch to the outbound row batch (during serialization)
The position delete files store the delete records in ascending order. This
means adjacent records mostly have the same file path. So we could just buffer
the position delete records up to the Channel's capacity, then serialize the
data in a more efficient way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13194) Fast-serialize position delete records
[
https://issues.apache.org/jira/browse/IMPALA-13194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-13194:
---
Description:
Currently the serialization of position delete records are very wasteful. The
records contain slots 'file_path' and 'pos'. And what we do during
serialization is the following.
# Write fixed-size tuple that have a StringValue and a BigInt slot (20 bytes
in total)
# We copy the StringValue's contents after the tuple.
# We convert the StringValue slot to be an offset to the string data
So we end up having something like this:
{noformat}
+-+++-+++-+
| StringValue | BigInt | File path| StringValue | BigInt | File path
| ... |
+-+++-+++-+
| ptr, len| 42 | /.../a.parquet | ptr, len| 43 | /.../a.parquet
| ... |
+-+++-+++-+
{noformat}
This is very redundant to store the file paths that way, and at the end we will
have a huge buffer that we need to compress and send over the network.
Moreover, we copy the file paths in memory twice:
# From input row batch to the KrpcDataStreamSender::Channel's temporary row
batch
# From the temporary row batch to the outbound row batch (during serialization)
The position delete files store the delete records in ascending order. This
means adjacent records mostly have the same file path. So we could just buffer
the position delete records up to the Channel's capacity, then serialize the
data in a more efficient way.
was:
Currently the serialization of position delete records are very wasteful. The
records contain slots 'file_path' and 'pos'. And what we do during
serialization is the following.
# Write fixed-size tuple that have a StringValue and a BigInt slot (20 bytes
in total)
# We copy the StringValue's contents after the tuple.
# We convert the StringValue slot to be an offset to the string data
So we end up having something like this:
{noformat}
+-+++-+++-+
| StringValue | BigInt | File path | StringValue | BigInt | File path
| ... |
+-+++-+++-+
| ptr, len | 42 | /.../a.parquet | ptr, len | 43 |
/.../a.parquet | ... |
+-+++-+++-+
{noformat}
This is very redundant to store the file paths that way, and at the end we will
have a huge buffer that we need to compress and send over the network.
Moreover, we copy the file paths in memory twice:
# From input row batch to the KrpcDataStreamSender::Channel's temporary row
batch
# From the temporary row batch to the outbound row batch (during serialization)
The position delete files store the delete records in ascending order. This
means adjacent records mostly have the same file path. So we could just buffer
the position delete records up to the Channel's capacity, then serialize the
data in a more efficient way.
> Fast-serialize position delete records
> --
>
> Key: IMPALA-13194
> URL: https://issues.apache.org/jira/browse/IMPALA-13194
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
>Reporter: Zoltán Borók-Nagy
>Priority: Major
> Labels: impala-iceberg
>
> Currently the serialization of position delete records are very wasteful. The
> records contain slots 'file_path' and 'pos'. And what we do during
> serialization is the following.
> # Write fixed-size tuple that have a StringValue and a BigInt slot (20 bytes
> in total)
> # We copy the StringValue's contents after the tuple.
> # We convert the StringValue slot to be an offset to the string data
> So we end up having something like this:
> {noformat}
> +-+++-+++-+
> | StringValue | BigInt | File path| StringValue | BigInt | File path
> | ... |
> +-+++-+++-+
> | ptr, len| 42 | /.../a.parquet | ptr, len| 43 |
> /.../a.parquet | ... |
> +-+++-+++-+
> {noformat}
> This is very redundant to store the file paths that way, and at the end we
> will have a huge buffer that we need to compress and send over the network.
> Moreover, we copy the file paths in memory twice:
> # From input row batch to the KrpcDataStreamSender::Channel's temporary row
> batch
> # From the temporary row batch to the outbound row batch (
