[jira] [Created] (IMPALA-13275) Analytic query leads to NullPointerException
Daniel Becker created IMPALA-13275:
--
Summary: Analytic query leads to NullPointerException
Key: IMPALA-13275
URL: https://issues.apache.org/jira/browse/IMPALA-13275
Project: IMPALA
Issue Type: Bug
Reporter: Daniel Becker
The following query leads to NullPointerException:
{code:java}
select row_no, i1, i2, small from (
select
arr.inner_struct1 i1,
arr.inner_struct2 i2,
arr.small,
row_number() over ( order by arr.inner_struct1.str) as row_no
from functional_parquet.collection_struct_mix t, t.arr_contains_nested_struct
arr ) res;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13272) Analytic function of collections can lead to crash
[
https://issues.apache.org/jira/browse/IMPALA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13272 started by Daniel Becker.
--
> Analytic function of collections can lead to crash
> --
>
> Key: IMPALA-13272
> URL: https://issues.apache.org/jira/browse/IMPALA-13272
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.4.0
>Reporter: Csaba Ringhofer
>Assignee: Daniel Becker
>Priority: Critical
>
> Using Impala's test data the following query leads to DCHECK in debug builds
> and may cause more subtle issues in RELEASE builds:
> {code}
> select
> row_no
> from (
> select
>arr.small,
>row_number() over (
> order by arr.inner_struct1.str) as row_no
> from functional_parquet.collection_struct_mix t,
> t.arr_contains_nested_struct arr
>) res
> {code}
> The following DCHECK is hit:
> {code}
> tuple.h:296 Check failed: offset != -1
> {code}
> The problem seems to be with arr.small, which is referenced in the inline
> view, but not used in the outer query - removing it from the inline view or
> adding it to the outer select leads to avoiding the bug. The problem seems
> related to materialization - offset==-1 means that the slot is not
> materialized, but the Parquet scanner still tries to materialize it.
> It is not clear yet which commit introduced the bug or whether this is a bug
> in the planner or the backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13272) Analytic function of collections can lead to crash
[
https://issues.apache.org/jira/browse/IMPALA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-13272:
---
Summary: Analytic function of collections can lead to crash (was:
Analyitic function of collections can lead to crash)
> Analytic function of collections can lead to crash
> --
>
> Key: IMPALA-13272
> URL: https://issues.apache.org/jira/browse/IMPALA-13272
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.4.0
>Reporter: Csaba Ringhofer
>Assignee: Daniel Becker
>Priority: Critical
>
> Using Impala's test data the following query leads to DCHECK in debug builds
> and may cause more subtle issues in RELEASE builds:
> {code}
> select
> row_no
> from (
> select
>arr.small,
>row_number() over (
> order by arr.inner_struct1.str) as row_no
> from functional_parquet.collection_struct_mix t,
> t.arr_contains_nested_struct arr
>) res
> {code}
> The following DCHECK is hit:
> {code}
> tuple.h:296 Check failed: offset != -1
> {code}
> The problem seems to be with arr.small, which is referenced in the inline
> view, but not used in the outer query - removing it from the inline view or
> adding it to the outer select leads to avoiding the bug. The problem seems
> related to materialization - offset==-1 means that the slot is not
> materialized, but the Parquet scanner still tries to materialize it.
> It is not clear yet which commit introduced the bug or whether this is a bug
> in the planner or the backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-13272) Analyitic function of collections can lead to crash
[
https://issues.apache.org/jira/browse/IMPALA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17871107#comment-17871107
]
Daniel Becker commented on IMPALA-13272:
The bug is present before IMPALA-12159. The repro query didn't fail because the
query contains var-len types within the array. Therefore, before those were
allowed, the bug was "accidentally" avoided. If we use a different query
without var-len types in the array, we can reproduce the bug also before
IMPALA-12159.
>From Hive:
{code:java}
create table arr_fixed (arr_contains_nested_struct ARRAY, inner_struct2: STRUCT, small:
SMALLINT>> ) stored as parquet;{code}
{code:java}
insert into arr_fixed values (
array(named_struct("inner_struct1", named_struct("b", 10L, "l", 0),
"inner_struct2", named_struct("b", 100L, "l", 2), "small",
2S), NULL,
named_struct("inner_struct1", named_struct("b", NULL, "l", 5),
"inner_struct2", named_struct("b", 1000L, "l", 8), "small",
20S)));{code}
Then from Impala:
{code:java}
select
row_no
from (
select
arr.small,
row_number() over (
order by arr.inner_struct1.b) as row_no
from arr_fixed t, t.arr_contains_nested_struct arr
) res;{code}
It produces the same error, so we should say the bug was introduced by
IMPALA-12019: Support ORDER BY for collections of fixed length types in select
list.
> Analyitic function of collections can lead to crash
> ---
>
> Key: IMPALA-13272
> URL: https://issues.apache.org/jira/browse/IMPALA-13272
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.4.0
>Reporter: Csaba Ringhofer
>Assignee: Daniel Becker
>Priority: Critical
>
> Using Impala's test data the following query leads to DCHECK in debug builds
> and may cause more subtle issues in RELEASE builds:
> {code}
> select
> row_no
> from (
> select
>arr.small,
>row_number() over (
> order by arr.inner_struct1.str) as row_no
> from functional_parquet.collection_struct_mix t,
> t.arr_contains_nested_struct arr
>) res
> {code}
> The following DCHECK is hit:
> {code}
> tuple.h:296 Check failed: offset != -1
> {code}
> The problem seems to be with arr.small, which is referenced in the inline
> view, but not used in the outer query - removing it from the inline view or
> adding it to the outer select leads to avoiding the bug. The problem seems
> related to materialization - offset==-1 means that the slot is not
> materialized, but the Parquet scanner still tries to materialize it.
> It is not clear yet which commit introduced the bug or whether this is a bug
> in the planner or the backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13272) Analyitic function of collections can lead to crash
[
https://issues.apache.org/jira/browse/IMPALA-13272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker reassigned IMPALA-13272:
--
Assignee: Daniel Becker
> Analyitic function of collections can lead to crash
> ---
>
> Key: IMPALA-13272
> URL: https://issues.apache.org/jira/browse/IMPALA-13272
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.4.0
>Reporter: Csaba Ringhofer
>Assignee: Daniel Becker
>Priority: Critical
>
> Using Impala's test data the following query leads to DCHECK in debug builds
> and may cause more subtle issues in RELEASE builds:
> {code}
> select
> row_no
> from (
> select
>arr.small,
>row_number() over (
> order by arr.inner_struct1.str) as row_no
> from functional_parquet.collection_struct_mix t,
> t.arr_contains_nested_struct arr
>) res
> {code}
> The following DCHECK is hit:
> {code}
> tuple.h:296 Check failed: offset != -1
> {code}
> The problem seems to be with arr.small, which is referenced in the inline
> view, but not used in the outer query - removing it from the inline view or
> adding it to the outer select leads to avoiding the bug. The problem seems
> related to materialization - offset==-1 means that the slot is not
> materialized, but the Parquet scanner still tries to materialize it.
> It is not clear yet which commit introduced the bug or whether this is a bug
> in the planner or the backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-11831) Error with zipping unnest if the array is also selected without unnesting
[
https://issues.apache.org/jira/browse/IMPALA-11831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker reassigned IMPALA-11831:
--
Assignee: (was: Daniel Becker)
> Error with zipping unnest if the array is also selected without unnesting
> -
>
> Key: IMPALA-11831
> URL: https://issues.apache.org/jira/browse/IMPALA-11831
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Priority: Major
>
> The following query returns an error:
>
> {code:java}
> select int_array, unnest(int_array) from complextypestbl;
> ERROR: IllegalStateException: null
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13247) Support Reading Puffin files for the current snapshot
[ https://issues.apache.org/jira/browse/IMPALA-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-13247: --- Labels: impala-iceberg (was: ) > Support Reading Puffin files for the current snapshot > - > > Key: IMPALA-13247 > URL: https://issues.apache.org/jira/browse/IMPALA-13247 > Project: IMPALA > Issue Type: Sub-task > Components: Frontend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13247) Support Reading Puffin files for the current snapshot
Daniel Becker created IMPALA-13247: -- Summary: Support Reading Puffin files for the current snapshot Key: IMPALA-13247 URL: https://issues.apache.org/jira/browse/IMPALA-13247 Project: IMPALA Issue Type: Sub-task Components: Frontend Reporter: Daniel Becker Assignee: Daniel Becker -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13041) Support Reading and Writing Puffin File Stats for Iceberg Tables
[ https://issues.apache.org/jira/browse/IMPALA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker reassigned IMPALA-13041: -- Assignee: Daniel Becker > Support Reading and Writing Puffin File Stats for Iceberg Tables > > > Key: IMPALA-13041 > URL: https://issues.apache.org/jira/browse/IMPALA-13041 > Project: IMPALA > Issue Type: Improvement > Components: Backend, Frontend >Reporter: Manish Maheshwari >Assignee: Daniel Becker >Priority: Major > Labels: catalog-2024, impala-iceberg > > Puffin File format is an iceberg upstream spec to support stats for iceberg > tables. These stats cannot be both read and used for query planning and > written by Impala today. We want to extend support in Impala to do the below > - > # Read stats from Puffin files > # Write stats to puffin files during load/insert/update/delete commands (as > applicable) > # Modify compute stats command for iceberg tables to compute stats and store > them in Puffin files > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12604) Hit DCHECK in raw-value after IMPALA-12159
[
https://issues.apache.org/jira/browse/IMPALA-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17866656#comment-17866656
]
Daniel Becker commented on IMPALA-12604:
We haven't seen this error for a while now, other changes in the area have
probably fixed this.
> Hit DCHECK in raw-value after IMPALA-12159
> --
>
> Key: IMPALA-12604
> URL: https://issues.apache.org/jira/browse/IMPALA-12604
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Peter Rozsa
>Assignee: Daniel Becker
>Priority: Major
>
> After IMPALA-12159, core test suite hits a DCHECK in raw-value.cc
> Log entry:
> {code:java}
> F1207 01:12:57.681260 17760 raw-value.cc:516]
> b44210f83bc16917:55b6589e0001] Check failed: false Unknown type:
> STRUCT{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12604) Hit DCHECK in raw-value after IMPALA-12159
[
https://issues.apache.org/jira/browse/IMPALA-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-12604.
Resolution: Fixed
> Hit DCHECK in raw-value after IMPALA-12159
> --
>
> Key: IMPALA-12604
> URL: https://issues.apache.org/jira/browse/IMPALA-12604
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Peter Rozsa
>Assignee: Daniel Becker
>Priority: Major
>
> After IMPALA-12159, core test suite hits a DCHECK in raw-value.cc
> Log entry:
> {code:java}
> F1207 01:12:57.681260 17760 raw-value.cc:516]
> b44210f83bc16917:55b6589e0001] Check failed: false Unknown type:
> STRUCT{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13210) Implement Parquet Byte Stream Split encoding in Impala
[ https://issues.apache.org/jira/browse/IMPALA-13210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker reassigned IMPALA-13210: -- Assignee: Myktybek (was: Daniel Becker) > Implement Parquet Byte Stream Split encoding in Impala > -- > > Key: IMPALA-13210 > URL: https://issues.apache.org/jira/browse/IMPALA-13210 > Project: IMPALA > Issue Type: Improvement >Reporter: Daniel Becker >Assignee: Myktybek >Priority: Major > > Currently Impala can't read or write files with Parquet's Byte Stream Split > encoding (see > https://github.com/apache/parquet-format/blob/master/Encodings.md#byte-stream-split-byte_stream_split–9). > This Jira tracks the work on adding support for it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13210) Implement Parquet Byte Stream Split encoding in Impala
Daniel Becker created IMPALA-13210: -- Summary: Implement Parquet Byte Stream Split encoding in Impala Key: IMPALA-13210 URL: https://issues.apache.org/jira/browse/IMPALA-13210 Project: IMPALA Issue Type: Improvement Reporter: Daniel Becker Assignee: Daniel Becker Currently Impala can't read or write files with Parquet's Byte Stream Split encoding (see https://github.com/apache/parquet-format/blob/master/Encodings.md#byte-stream-split-byte_stream_split–9). This Jira tracks the work on adding support for it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13168) Add README file for setting up Trino
[ https://issues.apache.org/jira/browse/IMPALA-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-13168 started by Daniel Becker. -- > Add README file for setting up Trino > > > Key: IMPALA-13168 > URL: https://issues.apache.org/jira/browse/IMPALA-13168 > Project: IMPALA > Issue Type: Improvement >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > The Impala repository contains scripts that make it easy to set up Trino in > the development environment. We should add a README file that describes how > they can be used. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13168) Add README file for setting up Trino
Daniel Becker created IMPALA-13168: -- Summary: Add README file for setting up Trino Key: IMPALA-13168 URL: https://issues.apache.org/jira/browse/IMPALA-13168 Project: IMPALA Issue Type: Improvement Reporter: Daniel Becker Assignee: Daniel Becker The Impala repository contains scripts that make it easy to set up Trino in the development environment. We should add a README file that describes how they can be used. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13150) Possible buffer overflow in StringVal::CopyFrom()
[
https://issues.apache.org/jira/browse/IMPALA-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-13150.
Resolution: Fixed
> Possible buffer overflow in StringVal::CopyFrom()
> -
>
> Key: IMPALA-13150
> URL: https://issues.apache.org/jira/browse/IMPALA-13150
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> In {{{}StringVal::CopyFrom(){}}}, we take the 'len' parameter as a
> {{{}size_t{}}}, which is usually a 64-bit unsigned integer. We pass it to the
> constructor of {{{}StringVal{}}}, which takes it as an {{{}int{}}}, which is
> usually a 32-bit signed integer. The constructor then allocates memory for
> the length using the {{int}} value, but back in {{{}CopyFrom(){}}}, we copy
> the buffer with the {{size_t}} length. If {{size_t}} is indeed 64 bits and
> {{int}} is 32 bits, and the value is truncated, we may copy more bytes that
> what we have allocated the destination for. See
> https://github.com/apache/impala/blob/ce8078204e5995277f79e226e26fe8b9eaca408b/be/src/udf/udf.cc#L546
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13150) Possible buffer overflow in StringVal::CopyFrom()
[
https://issues.apache.org/jira/browse/IMPALA-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13150 started by Daniel Becker.
--
> Possible buffer overflow in StringVal::CopyFrom()
> -
>
> Key: IMPALA-13150
> URL: https://issues.apache.org/jira/browse/IMPALA-13150
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> In {{{}StringVal::CopyFrom(){}}}, we take the 'len' parameter as a
> {{{}size_t{}}}, which is usually a 64-bit unsigned integer. We pass it to the
> constructor of {{{}StringVal{}}}, which takes it as an {{{}int{}}}, which is
> usually a 32-bit signed integer. The constructor then allocates memory for
> the length using the {{int}} value, but back in {{{}CopyFrom(){}}}, we copy
> the buffer with the {{size_t}} length. If {{size_t}} is indeed 64 bits and
> {{int}} is 32 bits, and the value is truncated, we may copy more bytes that
> what we have allocated the destination for. See
> https://github.com/apache/impala/blob/ce8078204e5995277f79e226e26fe8b9eaca408b/be/src/udf/udf.cc#L546
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13150) Possible buffer overflow in StringVal::CopyFrom()
[
https://issues.apache.org/jira/browse/IMPALA-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-13150:
---
Summary: Possible buffer overflow in StringVal::CopyFrom() (was: Possible
buffer overflow in StringVal)
> Possible buffer overflow in StringVal::CopyFrom()
> -
>
> Key: IMPALA-13150
> URL: https://issues.apache.org/jira/browse/IMPALA-13150
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> In {{{}StringVal::CopyFrom(){}}}, we take the 'len' parameter as a
> {{{}size_t{}}}, which is usually a 64-bit unsigned integer. We pass it to the
> constructor of {{{}StringVal{}}}, which takes it as an {{{}int{}}}, which is
> usually a 32-bit signed integer. The constructor then allocates memory for
> the length using the {{int}} value, but back in {{{}CopyFrom(){}}}, we copy
> the buffer with the {{size_t}} length. If {{size_t}} is indeed 64 bits and
> {{int}} is 32 bits, and the value is truncated, we may copy more bytes that
> what we have allocated the destination for. See
> https://github.com/apache/impala/blob/ce8078204e5995277f79e226e26fe8b9eaca408b/be/src/udf/udf.cc#L546
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13150) Possible buffer overflow in StringVal
Daniel Becker created IMPALA-13150:
--
Summary: Possible buffer overflow in StringVal
Key: IMPALA-13150
URL: https://issues.apache.org/jira/browse/IMPALA-13150
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Daniel Becker
Assignee: Daniel Becker
In {{{}StringVal::CopyFrom(){}}}, we take the 'len' parameter as a
{{{}size_t{}}}, which is usually a 64-bit unsigned integer. We pass it to the
constructor of {{{}StringVal{}}}, which takes it as an {{{}int{}}}, which is
usually a 32-bit signed integer. The constructor then allocates memory for the
length using the {{int}} value, but back in {{{}CopyFrom(){}}}, we copy the
buffer with the {{size_t}} length. If {{size_t}} is indeed 64 bits and {{int}}
is 32 bits, and the value is truncated, we may copy more bytes that what we
have allocated the destination for. See
https://github.com/apache/impala/blob/ce8078204e5995277f79e226e26fe8b9eaca408b/be/src/udf/udf.cc#L546
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13085) Add warning and NULL out DECIMAL values in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-13085. Resolution: Fixed > Add warning and NULL out DECIMAL values in Iceberg metadata tables > -- > > Key: IMPALA-13085 > URL: https://issues.apache.org/jira/browse/IMPALA-13085 > Project: IMPALA > Issue Type: Bug >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > IMPALA-13080 is about adding support for DECIMAL values in Iceberg metadata > tables. Until it is done, we should NULL out the values and issue a warning > instead of running on a DCHECK and crashing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13091) query_test.test_iceberg.TestIcebergV2Table.test_metadata_tables fails on an expected constant
[
https://issues.apache.org/jira/browse/IMPALA-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-13091.
Resolution: Fixed
> query_test.test_iceberg.TestIcebergV2Table.test_metadata_tables fails on an
> expected constant
> -
>
> Key: IMPALA-13091
> URL: https://issues.apache.org/jira/browse/IMPALA-13091
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.5.0
>Reporter: Laszlo Gaal
>Assignee: Daniel Becker
>Priority: Critical
> Labels: impala-iceberg
>
> This fails in various sanitizer builds (ASAN, UBSAN):
> Failure report:{code}
> query_test/test_iceberg.py:1527: in test_metadata_tables
> '$OVERWRITE_SNAPSHOT_TS': str(overwrite_snapshot_ts.data[0])})
> common/impala_test_suite.py:820: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:627: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:520: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:313: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E
> 0,regex:'.*\.parquet','PARQUET',0,3,3648,'{1:32,2:63,3:71,4:43,5:55,6:47,7:39,8:58,9:47,13:63,14:96,15:75,16:78}','{1:3,2:3,3:3,4:3,5:3,6:3,7:3,8:3,9:3,13:3,14:6,15:6,16:6}','{1:1,2:0,3:0,4:0,5:0,6:1,7:1,8:1,9:1,13:0,14:0,15:0,16:0}','{16:0,4:1,5:1,14:0}','{1:"AA==",2:"AQ==",3:"9v8=",4:"/+ZbLw==",5:"MAWO5C7/O6s=",6:"AFgLImsYBgA=",7:"kU0AAA==",8:"QSBzdHJpbmc=",9:"YmluMQ==",13:"av///w==",14:"fcOUJa1JwtQ=",16:"Pw=="}','{1:"AQ==",2:"BQ==",3:"lgA=",4:"qV/jWA==",5:"fcOUJa1JwlQ=",6:"AMhZw6A3BgA=",7:"Hk8AAA==",8:"U29tZSBzdHJpbmc=",9:"YmluMg==",13:"Cg==",14:"NEA=",16:"AAB6RA=="}','NULL','[4]','NULL',0,'{"arr.element":{"column_size":96,"value_count":6,"null_value_count":0,"nan_value_count":0,"lower_bound":-2e+100,"upper_bound":20},"b":{"column_size":32,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":false,"upper_bound":true},"bn":{"column_size":47,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"YmluMQ==","upper_bound":"YmluMg=="},"d":{"column_size":55,"value_count":3,"null_value_count":0,"nan_value_count":1,"lower_bound":-2e-100,"upper_bound":2e+100},"dt":{"column_size":39,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"2024-05-14","upper_bound":"2025-06-15"},"f":{"column_size":43,"value_count":3,"null_value_count":0,"nan_value_count":1,"lower_bound":2.00026702864e-10,"upper_bound":199973982208},"i":{"column_size":63,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":1,"upper_bound":5},"l":{"column_size":71,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":-10,"upper_bound":150},"mp.key":{"column_size":75,"value_count":6,"null_value_count":0,"nan_value_count":null,"lower_bound":null,"upper_bound":null},"mp.value":{"column_size":78,"value_count":6,"null_value_count":0,"nan_value_count":0,"lower_bound":0.5,"upper_bound":1000},"s":{"column_size":58,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"A
> string","upper_bound":"Some
> string"},"strct.i":{"column_size":63,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":-150,"upper_bound":10},"ts":{"column_size":47,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"2024-05-14
> 14:51:12","upper_bound":"2025-06-15 18:51:12"}}' !=
>
[jira] [Work started] (IMPALA-13091) query_test.test_iceberg.TestIcebergV2Table.test_metadata_tables fails on an expected constant
[
https://issues.apache.org/jira/browse/IMPALA-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13091 started by Daniel Becker.
--
> query_test.test_iceberg.TestIcebergV2Table.test_metadata_tables fails on an
> expected constant
> -
>
> Key: IMPALA-13091
> URL: https://issues.apache.org/jira/browse/IMPALA-13091
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.5.0
>Reporter: Laszlo Gaal
>Assignee: Daniel Becker
>Priority: Critical
> Labels: impala-iceberg
>
> This fails in various sanitizer builds (ASAN, UBSAN):
> Failure report:{code}
> query_test/test_iceberg.py:1527: in test_metadata_tables
> '$OVERWRITE_SNAPSHOT_TS': str(overwrite_snapshot_ts.data[0])})
> common/impala_test_suite.py:820: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:627: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:520: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:313: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E
> 0,regex:'.*\.parquet','PARQUET',0,3,3648,'{1:32,2:63,3:71,4:43,5:55,6:47,7:39,8:58,9:47,13:63,14:96,15:75,16:78}','{1:3,2:3,3:3,4:3,5:3,6:3,7:3,8:3,9:3,13:3,14:6,15:6,16:6}','{1:1,2:0,3:0,4:0,5:0,6:1,7:1,8:1,9:1,13:0,14:0,15:0,16:0}','{16:0,4:1,5:1,14:0}','{1:"AA==",2:"AQ==",3:"9v8=",4:"/+ZbLw==",5:"MAWO5C7/O6s=",6:"AFgLImsYBgA=",7:"kU0AAA==",8:"QSBzdHJpbmc=",9:"YmluMQ==",13:"av///w==",14:"fcOUJa1JwtQ=",16:"Pw=="}','{1:"AQ==",2:"BQ==",3:"lgA=",4:"qV/jWA==",5:"fcOUJa1JwlQ=",6:"AMhZw6A3BgA=",7:"Hk8AAA==",8:"U29tZSBzdHJpbmc=",9:"YmluMg==",13:"Cg==",14:"NEA=",16:"AAB6RA=="}','NULL','[4]','NULL',0,'{"arr.element":{"column_size":96,"value_count":6,"null_value_count":0,"nan_value_count":0,"lower_bound":-2e+100,"upper_bound":20},"b":{"column_size":32,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":false,"upper_bound":true},"bn":{"column_size":47,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"YmluMQ==","upper_bound":"YmluMg=="},"d":{"column_size":55,"value_count":3,"null_value_count":0,"nan_value_count":1,"lower_bound":-2e-100,"upper_bound":2e+100},"dt":{"column_size":39,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"2024-05-14","upper_bound":"2025-06-15"},"f":{"column_size":43,"value_count":3,"null_value_count":0,"nan_value_count":1,"lower_bound":2.00026702864e-10,"upper_bound":199973982208},"i":{"column_size":63,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":1,"upper_bound":5},"l":{"column_size":71,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":-10,"upper_bound":150},"mp.key":{"column_size":75,"value_count":6,"null_value_count":0,"nan_value_count":null,"lower_bound":null,"upper_bound":null},"mp.value":{"column_size":78,"value_count":6,"null_value_count":0,"nan_value_count":0,"lower_bound":0.5,"upper_bound":1000},"s":{"column_size":58,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"A
> string","upper_bound":"Some
> string"},"strct.i":{"column_size":63,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":-150,"upper_bound":10},"ts":{"column_size":47,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"2024-05-14
> 14:51:12","upper_bound":"2025-06-15 18:51:12"}}' !=
>
[jira] [Updated] (IMPALA-13091) query_test.test_iceberg.TestIcebergV2Table.test_metadata_tables fails on an expected constant
[
https://issues.apache.org/jira/browse/IMPALA-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-13091:
---
Labels: impala-iceberg (was: )
> query_test.test_iceberg.TestIcebergV2Table.test_metadata_tables fails on an
> expected constant
> -
>
> Key: IMPALA-13091
> URL: https://issues.apache.org/jira/browse/IMPALA-13091
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.5.0
>Reporter: Laszlo Gaal
>Assignee: Daniel Becker
>Priority: Critical
> Labels: impala-iceberg
>
> This fails in various sanitizer builds (ASAN, UBSAN):
> Failure report:{code}
> query_test/test_iceberg.py:1527: in test_metadata_tables
> '$OVERWRITE_SNAPSHOT_TS': str(overwrite_snapshot_ts.data[0])})
> common/impala_test_suite.py:820: in run_test_case
> self.__verify_results_and_errors(vector, test_section, result, use_db)
> common/impala_test_suite.py:627: in __verify_results_and_errors
> replace_filenames_with_placeholder)
> common/test_result_verifier.py:520: in verify_raw_results
> VERIFIER_MAP[verifier](expected, actual)
> common/test_result_verifier.py:313: in verify_query_result_is_equal
> assert expected_results == actual_results
> E assert Comparing QueryTestResults (expected vs actual):
> E
> 0,regex:'.*\.parquet','PARQUET',0,3,3648,'{1:32,2:63,3:71,4:43,5:55,6:47,7:39,8:58,9:47,13:63,14:96,15:75,16:78}','{1:3,2:3,3:3,4:3,5:3,6:3,7:3,8:3,9:3,13:3,14:6,15:6,16:6}','{1:1,2:0,3:0,4:0,5:0,6:1,7:1,8:1,9:1,13:0,14:0,15:0,16:0}','{16:0,4:1,5:1,14:0}','{1:"AA==",2:"AQ==",3:"9v8=",4:"/+ZbLw==",5:"MAWO5C7/O6s=",6:"AFgLImsYBgA=",7:"kU0AAA==",8:"QSBzdHJpbmc=",9:"YmluMQ==",13:"av///w==",14:"fcOUJa1JwtQ=",16:"Pw=="}','{1:"AQ==",2:"BQ==",3:"lgA=",4:"qV/jWA==",5:"fcOUJa1JwlQ=",6:"AMhZw6A3BgA=",7:"Hk8AAA==",8:"U29tZSBzdHJpbmc=",9:"YmluMg==",13:"Cg==",14:"NEA=",16:"AAB6RA=="}','NULL','[4]','NULL',0,'{"arr.element":{"column_size":96,"value_count":6,"null_value_count":0,"nan_value_count":0,"lower_bound":-2e+100,"upper_bound":20},"b":{"column_size":32,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":false,"upper_bound":true},"bn":{"column_size":47,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"YmluMQ==","upper_bound":"YmluMg=="},"d":{"column_size":55,"value_count":3,"null_value_count":0,"nan_value_count":1,"lower_bound":-2e-100,"upper_bound":2e+100},"dt":{"column_size":39,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"2024-05-14","upper_bound":"2025-06-15"},"f":{"column_size":43,"value_count":3,"null_value_count":0,"nan_value_count":1,"lower_bound":2.00026702864e-10,"upper_bound":199973982208},"i":{"column_size":63,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":1,"upper_bound":5},"l":{"column_size":71,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":-10,"upper_bound":150},"mp.key":{"column_size":75,"value_count":6,"null_value_count":0,"nan_value_count":null,"lower_bound":null,"upper_bound":null},"mp.value":{"column_size":78,"value_count":6,"null_value_count":0,"nan_value_count":0,"lower_bound":0.5,"upper_bound":1000},"s":{"column_size":58,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"A
> string","upper_bound":"Some
> string"},"strct.i":{"column_size":63,"value_count":3,"null_value_count":0,"nan_value_count":null,"lower_bound":-150,"upper_bound":10},"ts":{"column_size":47,"value_count":3,"null_value_count":1,"nan_value_count":null,"lower_bound":"2024-05-14
> 14:51:12","upper_bound":"2025-06-15 18:51:12"}}' !=
>
[jira] [Resolved] (IMPALA-13078) Unsupported column type: 8 error during query on `files` metadata table
[ https://issues.apache.org/jira/browse/IMPALA-13078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-13078. Resolution: Fixed > Unsupported column type: 8 error during query on `files` metadata table > --- > > Key: IMPALA-13078 > URL: https://issues.apache.org/jira/browse/IMPALA-13078 > Project: IMPALA > Issue Type: Bug > Components: be >Affects Versions: Impala 4.3.0 >Reporter: Peter Rozsa >Assignee: Daniel Becker >Priority: Minor > Labels: iceberg, impala-iceberg > > Query on metadata table `files` fails with the following error message: > F0514 13:50:15.044785 142755 iceberg-row-reader.cc:138] > 4c4ac446b97bbadc:d4bf9495] Check failed: false Unsupported column > type: 8 > Any table that contains double/float column could be used to reproduce the > issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13079) Add support for FLOAT/DOUBLE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-13079. Resolution: Fixed > Add support for FLOAT/DOUBLE in Iceberg metadata tables > --- > > Key: IMPALA-13079 > URL: https://issues.apache.org/jira/browse/IMPALA-13079 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-13080) Add support for DECIMAL in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker reassigned IMPALA-13080: -- Assignee: (was: Daniel Becker) > Add support for DECIMAL in Iceberg metadata tables > -- > > Key: IMPALA-13080 > URL: https://issues.apache.org/jira/browse/IMPALA-13080 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Daniel Becker >Priority: Major > Labels: impala-iceberg, ramp-up > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13085) Add warning and NULL out DECIMAL values in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-13085 started by Daniel Becker. -- > Add warning and NULL out DECIMAL values in Iceberg metadata tables > -- > > Key: IMPALA-13085 > URL: https://issues.apache.org/jira/browse/IMPALA-13085 > Project: IMPALA > Issue Type: Bug >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > IMPALA-13080 is about adding support for DECIMAL values in Iceberg metadata > tables. Until it is done, we should NULL out the values and issue a warning > instead of running on a DCHECK and crashing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13084) Add support for DECIMAL in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-13084: --- Parent: (was: IMPALA-10947) Issue Type: Bug (was: Sub-task) > Add support for DECIMAL in Iceberg metadata tables > -- > > Key: IMPALA-13084 > URL: https://issues.apache.org/jira/browse/IMPALA-13084 > Project: IMPALA > Issue Type: Bug > Components: Backend >Reporter: Daniel Becker >Priority: Major > Labels: impala-iceberg, ramp-up > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13080) Add support for DECIMAL in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-13080: --- Labels: impala-iceberg ramp-up (was: ) > Add support for DECIMAL in Iceberg metadata tables > -- > > Key: IMPALA-13080 > URL: https://issues.apache.org/jira/browse/IMPALA-13080 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg, ramp-up > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13084) Add support for DECIMAL in Iceberg metadata tables
Daniel Becker created IMPALA-13084: -- Summary: Add support for DECIMAL in Iceberg metadata tables Key: IMPALA-13084 URL: https://issues.apache.org/jira/browse/IMPALA-13084 Project: IMPALA Issue Type: Sub-task Components: Backend Reporter: Daniel Becker -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Closed] (IMPALA-13084) Add support for DECIMAL in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker closed IMPALA-13084. -- Resolution: Duplicate > Add support for DECIMAL in Iceberg metadata tables > -- > > Key: IMPALA-13084 > URL: https://issues.apache.org/jira/browse/IMPALA-13084 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Daniel Becker >Priority: Major > Labels: impala-iceberg, ramp-up > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13078) Unsupported column type: 8 error during query on `files` metadata table
[ https://issues.apache.org/jira/browse/IMPALA-13078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-13078: --- Labels: iceberg impala-iceberg (was: iceberg) > Unsupported column type: 8 error during query on `files` metadata table > --- > > Key: IMPALA-13078 > URL: https://issues.apache.org/jira/browse/IMPALA-13078 > Project: IMPALA > Issue Type: Bug > Components: be >Affects Versions: Impala 4.3.0 >Reporter: Peter Rozsa >Assignee: Daniel Becker >Priority: Minor > Labels: iceberg, impala-iceberg > > Query on metadata table `files` fails with the following error message: > F0514 13:50:15.044785 142755 iceberg-row-reader.cc:138] > 4c4ac446b97bbadc:d4bf9495] Check failed: false Unsupported column > type: 8 > Any table that contains double/float column could be used to reproduce the > issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13079) Add support for FLOAT/DOUBLE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-13079: --- Labels: impala-iceberg (was: ) > Add support for FLOAT/DOUBLE in Iceberg metadata tables > --- > > Key: IMPALA-13079 > URL: https://issues.apache.org/jira/browse/IMPALA-13079 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13080) Add support for DECIMAL in Iceberg metadata tables
Daniel Becker created IMPALA-13080: -- Summary: Add support for DECIMAL in Iceberg metadata tables Key: IMPALA-13080 URL: https://issues.apache.org/jira/browse/IMPALA-13080 Project: IMPALA Issue Type: Sub-task Components: Backend Reporter: Daniel Becker Assignee: Daniel Becker -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13079) Add support for FLOAT/DOUBLE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-13079 started by Daniel Becker. -- > Add support for FLOAT/DOUBLE in Iceberg metadata tables > --- > > Key: IMPALA-13079 > URL: https://issues.apache.org/jira/browse/IMPALA-13079 > Project: IMPALA > Issue Type: Sub-task > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13079) Add support for FLOAT/DOUBLE in Iceberg metadata tables
Daniel Becker created IMPALA-13079: -- Summary: Add support for FLOAT/DOUBLE in Iceberg metadata tables Key: IMPALA-13079 URL: https://issues.apache.org/jira/browse/IMPALA-13079 Project: IMPALA Issue Type: Sub-task Components: Backend Reporter: Daniel Becker Assignee: Daniel Becker -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13036) Document Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-13036. Resolution: Fixed > Document Iceberg metadata tables > > > Key: IMPALA-13036 > URL: https://issues.apache.org/jira/browse/IMPALA-13036 > Project: IMPALA > Issue Type: Documentation >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala now supports displaying Iceberg metadata tables, we should document > this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13035) Querying metadata tables from non-Iceberg tables throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/IMPALA-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-13035.
Resolution: Fixed
> Querying metadata tables from non-Iceberg tables throws
> IllegalArgumentException
>
>
> Key: IMPALA-13035
> URL: https://issues.apache.org/jira/browse/IMPALA-13035
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.3.0
>Reporter: Peter Rozsa
>Assignee: Daniel Becker
>Priority: Minor
> Labels: impala-iceberg
>
> If a query targets an Iceberg metadata table like default.xy.`files` and the
> xy table is not an Iceberg table then the analyzer throws
> IllegalArgumentException.
> The main concern is that IcebergMetadataTable.java:isIcebergMetadataTable is
> called before it's validated that the table is indeed an IcebergTable.
> Example:
> {code:java}
> create table xy(a int);
> select * from default.xy.`files`;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13036) Document Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-13036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-13036 started by Daniel Becker. -- > Document Iceberg metadata tables > > > Key: IMPALA-13036 > URL: https://issues.apache.org/jira/browse/IMPALA-13036 > Project: IMPALA > Issue Type: Documentation >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala now supports displaying Iceberg metadata tables, we should document > this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12651) Add support to BINARY type Iceberg Metadata table columns
[ https://issues.apache.org/jira/browse/IMPALA-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-12651. Resolution: Implemented > Add support to BINARY type Iceberg Metadata table columns > - > > Key: IMPALA-12651 > URL: https://issues.apache.org/jira/browse/IMPALA-12651 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Affects Versions: Impala 4.4.0 >Reporter: Tamas Mate >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala should be able to read BINARY type columns from Iceberg metadata > tables as strings, additionally this should be allowed when reading these > types from complex types. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12973) Add support for BINARY in complex types in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-12973. Resolution: Fixed > Add support for BINARY in complex types in Iceberg metadata tables > -- > > Key: IMPALA-12973 > URL: https://issues.apache.org/jira/browse/IMPALA-12973 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > IMPALA-12899 added a temporary workaround to allow queries that contain > binary fields in complex types to succeed by NULLing out these binary fields. > We should now implement displaying them with base64 encoding. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13035) Querying metadata tables from non-Iceberg tables throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/IMPALA-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13035 started by Daniel Becker.
--
> Querying metadata tables from non-Iceberg tables throws
> IllegalArgumentException
>
>
> Key: IMPALA-13035
> URL: https://issues.apache.org/jira/browse/IMPALA-13035
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.3.0
>Reporter: Peter Rozsa
>Assignee: Daniel Becker
>Priority: Minor
> Labels: impala-iceberg
>
> If a query targets an Iceberg metadata table like default.xy.`files` and the
> xy table is not an Iceberg table then the analyzer throws
> IllegalArgumentException.
> The main concern is that IcebergMetadataTable.java:isIcebergMetadataTable is
> called before it's validated that the table is indeed an IcebergTable.
> Example:
> {code:java}
> create table xy(a int);
> select * from default.xy.`files`;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13036) Document Iceberg metadata tables
Daniel Becker created IMPALA-13036: -- Summary: Document Iceberg metadata tables Key: IMPALA-13036 URL: https://issues.apache.org/jira/browse/IMPALA-13036 Project: IMPALA Issue Type: Documentation Reporter: Daniel Becker Assignee: Daniel Becker Impala now supports displaying Iceberg metadata tables, we should document this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-13035) Querying metadata tables from non-Iceberg tables throws IllegalArgumentException
[
https://issues.apache.org/jira/browse/IMPALA-13035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-13035:
---
Labels: impala-iceberg (was: )
> Querying metadata tables from non-Iceberg tables throws
> IllegalArgumentException
>
>
> Key: IMPALA-13035
> URL: https://issues.apache.org/jira/browse/IMPALA-13035
> Project: IMPALA
> Issue Type: Bug
>Affects Versions: Impala 4.3.0
>Reporter: Peter Rozsa
>Assignee: Daniel Becker
>Priority: Minor
> Labels: impala-iceberg
>
> If a query targets an Iceberg metadata table like default.xy.`files` and the
> xy table is not an Iceberg table then the analyzer throws
> IllegalArgumentException.
> The main concern is that IcebergMetadataTable.java:isIcebergMetadataTable is
> called before it's validated that the table is indeed an IcebergTable.
> Example:
> {code:java}
> create table xy(a int);
> select * from default.xy.`files`;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-13008) test_metadata_tables failed in Ubuntu 20 build
[
https://issues.apache.org/jira/browse/IMPALA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-13008.
Resolution: Fixed
> test_metadata_tables failed in Ubuntu 20 build
> --
>
> Key: IMPALA-13008
> URL: https://issues.apache.org/jira/browse/IMPALA-13008
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Daniel Becker
>Priority: Major
> Labels: impala-iceberg
>
> test_metadata_tables failed in an Ubuntu 20 release test build:
> *
> https://jenkins.impala.io/job/parallel-all-tests-ub2004/1059/artifact/https_%5E%5Ejenkins.impala.io%5Ejob%5Eubuntu-20.04-dockerised-tests%5E1642%5E.log
> *
> https://jenkins.impala.io/job/parallel-all-tests-ub2004/1059/artifact/https_%5E%5Ejenkins.impala.io%5Ejob%5Eubuntu-20.04-from-scratch%5E2363%5E.log
> h2. Error
> {noformat}
> E assert Comparing QueryTestResults (expected vs actual):
> E
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"351","changed-partition-count":"1","total-records":"1","total-files-size":"351","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> !=
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"350","changed-partition-count":"1","total-records":"1","total-files-size":"350","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> E
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"351","changed-partition-count":"1","total-records":"2","total-files-size":"702","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> !=
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"350","changed-partition-count":"1","total-records":"2","total-files-size":"700","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> E
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"351","changed-partition-count":"1","total-records":"3","total-files-size":"1053","total-data-files":"3","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> !=
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"350","changed-partition-count":"1","total-records":"3","total-files-size":"1050","total-data-files":"3","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> E
> row_regex:'overwrite',true,'{"added-position-delete-files":"1","added-delete-files":"1","added-files-size":"[1-9][0-9]*","added-position-deletes":"1","changed-partition-count":"1","total-records":"3","total-files-size":"[1-9][0-9]*","total-data-files":"3","total-delete-files":"1","total-position-deletes":"1","total-equality-deletes":"0"}'
> ==
> 'overwrite',true,'{"added-position-delete-files":"1","added-delete-files":"1","added-files-size":"1551","added-position-deletes":"1","changed-partition-count":"1","total-records":"3","total-files-size":"2601","total-data-files":"3","total-delete-files":"1","total-position-deletes":"1","total-equality-deletes":"0"}'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-13008) test_metadata_tables failed in Ubuntu 20 build
[
https://issues.apache.org/jira/browse/IMPALA-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-13008 started by Daniel Becker.
--
> test_metadata_tables failed in Ubuntu 20 build
> --
>
> Key: IMPALA-13008
> URL: https://issues.apache.org/jira/browse/IMPALA-13008
> Project: IMPALA
> Issue Type: Bug
>Reporter: Zoltán Borók-Nagy
>Assignee: Daniel Becker
>Priority: Major
> Labels: impala-iceberg
>
> test_metadata_tables failed in an Ubuntu 20 release test build:
> *
> https://jenkins.impala.io/job/parallel-all-tests-ub2004/1059/artifact/https_%5E%5Ejenkins.impala.io%5Ejob%5Eubuntu-20.04-dockerised-tests%5E1642%5E.log
> *
> https://jenkins.impala.io/job/parallel-all-tests-ub2004/1059/artifact/https_%5E%5Ejenkins.impala.io%5Ejob%5Eubuntu-20.04-from-scratch%5E2363%5E.log
> h2. Error
> {noformat}
> E assert Comparing QueryTestResults (expected vs actual):
> E
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"351","changed-partition-count":"1","total-records":"1","total-files-size":"351","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> !=
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"350","changed-partition-count":"1","total-records":"1","total-files-size":"350","total-data-files":"1","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> E
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"351","changed-partition-count":"1","total-records":"2","total-files-size":"702","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> !=
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"350","changed-partition-count":"1","total-records":"2","total-files-size":"700","total-data-files":"2","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> E
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"351","changed-partition-count":"1","total-records":"3","total-files-size":"1053","total-data-files":"3","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> !=
> 'append',true,'{"added-data-files":"1","added-records":"1","added-files-size":"350","changed-partition-count":"1","total-records":"3","total-files-size":"1050","total-data-files":"3","total-delete-files":"0","total-position-deletes":"0","total-equality-deletes":"0"}'
> E
> row_regex:'overwrite',true,'{"added-position-delete-files":"1","added-delete-files":"1","added-files-size":"[1-9][0-9]*","added-position-deletes":"1","changed-partition-count":"1","total-records":"3","total-files-size":"[1-9][0-9]*","total-data-files":"3","total-delete-files":"1","total-position-deletes":"1","total-equality-deletes":"0"}'
> ==
> 'overwrite',true,'{"added-position-delete-files":"1","added-delete-files":"1","added-files-size":"1551","added-position-deletes":"1","changed-partition-count":"1","total-records":"3","total-files-size":"2601","total-data-files":"3","total-delete-files":"1","total-position-deletes":"1","total-equality-deletes":"0"}'
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-13006) Some Iceberg test tables are not restricted to Parquet
Daniel Becker created IMPALA-13006:
--
Summary: Some Iceberg test tables are not restricted to Parquet
Key: IMPALA-13006
URL: https://issues.apache.org/jira/browse/IMPALA-13006
Project: IMPALA
Issue Type: Bug
Reporter: Daniel Becker
Assignee: Daniel Becker
Our Iceberg test tables/views are restricted to the Parquet file format in
functional/schema_constraints.csv except for the following two:
{code:java}
iceberg_query_metadata
iceberg_view{code}
This is not intentional, so we should add the constraint for these tables too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12996) Add support for DATE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-12996. Resolution: Fixed > Add support for DATE in Iceberg metadata tables > --- > > Key: IMPALA-12996 > URL: https://issues.apache.org/jira/browse/IMPALA-12996 > Project: IMPALA > Issue Type: Sub-task >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-11991) Iceberg Metadata querying with time travel
[ https://issues.apache.org/jira/browse/IMPALA-11991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-11991: --- Labels: impala-iceberg (was: ) > Iceberg Metadata querying with time travel > -- > > Key: IMPALA-11991 > URL: https://issues.apache.org/jira/browse/IMPALA-11991 > Project: IMPALA > Issue Type: Sub-task >Reporter: Tamas Mate >Priority: Major > Labels: impala-iceberg > > The initial version of metadtata querying does not support time travels, we > support it with metadata tables as well. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12973) Add support for BINARY in complex types in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-12973: --- Labels: impala-iceberg (was: ) > Add support for BINARY in complex types in Iceberg metadata tables > -- > > Key: IMPALA-12973 > URL: https://issues.apache.org/jira/browse/IMPALA-12973 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > IMPALA-12899 added a temporary workaround to allow queries that contain > binary fields in complex types to succeed by NULLing out these binary fields. > We should now implement displaying them with base64 encoding. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-11908) Iceberg Metadata querying parser change
[ https://issues.apache.org/jira/browse/IMPALA-11908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-11908: --- Labels: impala-iceberg (was: ) > Iceberg Metadata querying parser change > --- > > Key: IMPALA-11908 > URL: https://issues.apache.org/jira/browse/IMPALA-11908 > Project: IMPALA > Issue Type: Sub-task >Reporter: Tamas Mate >Assignee: Tamas Mate >Priority: Major > Labels: impala-iceberg > Fix For: Impala 4.3.0 > > > As a first step the parser should accept > database_name.table_name.metadata_table_name referrals to metadata tables. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12996) Add support for DATE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-12996: --- Labels: impala-iceberg (was: ) > Add support for DATE in Iceberg metadata tables > --- > > Key: IMPALA-12996 > URL: https://issues.apache.org/jira/browse/IMPALA-12996 > Project: IMPALA > Issue Type: Sub-task >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12996) Add support for DATE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12996 started by Daniel Becker. -- > Add support for DATE in Iceberg metadata tables > --- > > Key: IMPALA-12996 > URL: https://issues.apache.org/jira/browse/IMPALA-12996 > Project: IMPALA > Issue Type: Sub-task >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12996) Add support for DATE in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker reassigned IMPALA-12996: -- Assignee: Daniel Becker > Add support for DATE in Iceberg metadata tables > --- > > Key: IMPALA-12996 > URL: https://issues.apache.org/jira/browse/IMPALA-12996 > Project: IMPALA > Issue Type: Sub-task >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12986) Base64Encode fails if the 'out_len' output parameter is passed with certain values
[
https://issues.apache.org/jira/browse/IMPALA-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-12986.
Resolution: Fixed
> Base64Encode fails if the 'out_len' output parameter is passed with certain
> values
> --
>
> Key: IMPALA-12986
> URL: https://issues.apache.org/jira/browse/IMPALA-12986
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> The Base64Encode function in coding-util.h with signature
> {code:java}
> bool Base64Encode(const char* in, int64_t in_len, int64_t out_max, char* out,
> int64_t* out_len);{code}
> fails if '*out_len', when passed to the function, contains a negative value
> or a value that does not fit in a 32 bit integer.
> Internally we use the
>
> {code:java}
> int sasl_encode64(const char *in, unsigned inlen, char *out, unsigned outmax,
> unsigned *outlen);{code}
>
> function and explicitly cast 'out_len' to 'unsigned*'.
> The success of this function shouldn't depend on the value of '*out_len'
> because it is an output parameter, so we should set '*out_len' to zero before
> passing it to {{{}sasl_encode64(){}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12996) Add support for DATE in Iceberg metadata tables
Daniel Becker created IMPALA-12996: -- Summary: Add support for DATE in Iceberg metadata tables Key: IMPALA-12996 URL: https://issues.apache.org/jira/browse/IMPALA-12996 Project: IMPALA Issue Type: Sub-task Reporter: Daniel Becker -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12986) Base64Encode fails if the 'out_len' output parameter is passed with certain values
[
https://issues.apache.org/jira/browse/IMPALA-12986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12986 started by Daniel Becker.
--
> Base64Encode fails if the 'out_len' output parameter is passed with certain
> values
> --
>
> Key: IMPALA-12986
> URL: https://issues.apache.org/jira/browse/IMPALA-12986
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> The Base64Encode function in coding-util.h with signature
> {code:java}
> bool Base64Encode(const char* in, int64_t in_len, int64_t out_max, char* out,
> int64_t* out_len);{code}
> fails if '*out_len', when passed to the function, contains a negative value
> or a value that does not fit in a 32 bit integer.
> Internally we use the
>
> {code:java}
> int sasl_encode64(const char *in, unsigned inlen, char *out, unsigned outmax,
> unsigned *outlen);{code}
>
> function and explicitly cast 'out_len' to 'unsigned*'.
> The success of this function shouldn't depend on the value of '*out_len'
> because it is an output parameter, so we should set '*out_len' to zero before
> passing it to {{{}sasl_encode64(){}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12986) Base64Encode fails if the 'out_len' output parameter is passed with certain values
Daniel Becker created IMPALA-12986:
--
Summary: Base64Encode fails if the 'out_len' output parameter is
passed with certain values
Key: IMPALA-12986
URL: https://issues.apache.org/jira/browse/IMPALA-12986
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Daniel Becker
Assignee: Daniel Becker
The Base64Encode function in coding-util.h with signature
{code:java}
bool Base64Encode(const char* in, int64_t in_len, int64_t out_max, char* out,
int64_t* out_len);{code}
fails if '*out_len', when passed to the function, contains a negative value or
a value that does not fit in a 32 bit integer.
Internally we use the
{code:java}
int sasl_encode64(const char *in, unsigned inlen, char *out, unsigned outmax,
unsigned *outlen);{code}
function and explicitly cast 'out_len' to 'unsigned*'.
The success of this function shouldn't depend on the value of '*out_len'
because it is an output parameter, so we should set '*out_len' to zero before
passing it to {{{}sasl_encode64(){}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12651) Add support to BINARY type Iceberg Metadata table columns
[ https://issues.apache.org/jira/browse/IMPALA-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12651 started by Daniel Becker. -- > Add support to BINARY type Iceberg Metadata table columns > - > > Key: IMPALA-12651 > URL: https://issues.apache.org/jira/browse/IMPALA-12651 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Affects Versions: Impala 4.4.0 >Reporter: Tamas Mate >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala should be able to read BINARY type columns from Iceberg metadata > tables as strings, additionally this should be allowed when reading these > types from complex types. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12651) Add support to BINARY type Iceberg Metadata table columns
[ https://issues.apache.org/jira/browse/IMPALA-12651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker reassigned IMPALA-12651: -- Assignee: Daniel Becker > Add support to BINARY type Iceberg Metadata table columns > - > > Key: IMPALA-12651 > URL: https://issues.apache.org/jira/browse/IMPALA-12651 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Affects Versions: Impala 4.4.0 >Reporter: Tamas Mate >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > Impala should be able to read BINARY type columns from Iceberg metadata > tables as strings, additionally this should be allowed when reading these > types from complex types. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12973) Add support for BINARY in complex types in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-12973: --- Description: IMPALA-12899 added a temporary workaround to allow queries that contain binary fields in complex types to succeed by NULLing out these binary fields. We should now implement displaying them with base64 encoding. > Add support for BINARY in complex types in Iceberg metadata tables > -- > > Key: IMPALA-12973 > URL: https://issues.apache.org/jira/browse/IMPALA-12973 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > IMPALA-12899 added a temporary workaround to allow queries that contain > binary fields in complex types to succeed by NULLing out these binary fields. > We should now implement displaying them with base64 encoding. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12973) Add support for BINARY in complex types in Iceberg metadata tables
[ https://issues.apache.org/jira/browse/IMPALA-12973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12973 started by Daniel Becker. -- > Add support for BINARY in complex types in Iceberg metadata tables > -- > > Key: IMPALA-12973 > URL: https://issues.apache.org/jira/browse/IMPALA-12973 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12973) Add support for BINARY in complex types in Iceberg metadata tables
Daniel Becker created IMPALA-12973: -- Summary: Add support for BINARY in complex types in Iceberg metadata tables Key: IMPALA-12973 URL: https://issues.apache.org/jira/browse/IMPALA-12973 Project: IMPALA Issue Type: Sub-task Components: Backend, Frontend Reporter: Daniel Becker Assignee: Daniel Becker -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12926) Refactor BINARY type handling in the backend
[ https://issues.apache.org/jira/browse/IMPALA-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833937#comment-17833937 ] Daniel Becker commented on IMPALA-12926: https://gerrit.cloudera.org/#/c/21157/ > Refactor BINARY type handling in the backend > > > Key: IMPALA-12926 > URL: https://issues.apache.org/jira/browse/IMPALA-12926 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > Currently the STRING and BINARY types are not distinguished in most of the > backend. In contrast to the frontend, PrimitiveType::TYPE_BINARY is not used > there at all, TYPE_STRING being used instead. This is to ensure that > everything that works for STRING also works for BINARY. So far only file > readers and writers have had to handle them differently, and they have access > to ColumnDescriptors which contain AuxColumnType fields that differentiate > these two types. > However, only top-level columns have ColumnDescriptors. Adding support or > BINARYs within complex types (see IMPALA-11491 and IMPALA-12651) necessitates > adding type information about STRING vs BINARY to embedded fields as well. > Using PrimitiveType::TYPE_BINARY would probably be the cleanest solution but > it would affect huge parts of the code as TYPE_BINARY would have to be added > to hundreds of switch statements and this would be error prone. > Instead, we should introduce a new field in ColumnType: 'is_binary', which is > true if the type is a BINARY and false otherwise. We keep using TYPE_STRING > as the PrimitiveType of the ColumnType for BINARYs. This way full type > information is present in ColumnType but code that does not differentiate > between STRING and BINARY will continue to work for BINARY. > With this change, AuxColumnType is no longer needed and should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12609) Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables
[
https://issues.apache.org/jira/browse/IMPALA-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12609:
---
Summary: Implement SHOW METADATA TABLES IN statement to list Iceberg
Metadata tables (was: Implement SHOW TABLES IN statement to list Iceberg
Metadata tables)
> Implement SHOW METADATA TABLES IN statement to list Iceberg Metadata tables
> ---
>
> Key: IMPALA-12609
> URL: https://issues.apache.org/jira/browse/IMPALA-12609
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Affects Versions: Impala 4.4.0
>Reporter: Tamas Mate
>Assignee: Daniel Becker
>Priority: Minor
> Labels: impala-iceberg
>
> {{SHOW TABLES IN}} statement could be used to list all the available metadata
> tables of an Iceberg table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12967) Testcase fails at test_migrated_table_field_id_resolution due to "Table does not exist"
[
https://issues.apache.org/jira/browse/IMPALA-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker reassigned IMPALA-12967:
--
Assignee: Quanlong Huang (was: Daniel Becker)
> Testcase fails at test_migrated_table_field_id_resolution due to "Table does
> not exist"
> ---
>
> Key: IMPALA-12967
> URL: https://issues.apache.org/jira/browse/IMPALA-12967
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Yida Wu
>Assignee: Quanlong Huang
>Priority: Major
> Labels: broken-build
>
> Testcase test_migrated_table_field_id_resolution fails at exhaustive release
> build with following messages:
> *Regression*
> {code:java}
> query_test.test_iceberg.TestIcebergTable.test_migrated_table_field_id_resolution[protocol:
> beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] (from pytest)
> {code}
> *Error Message*
> {code:java}
> query_test/test_iceberg.py:266: in test_migrated_table_field_id_resolution
> "iceberg_migrated_alter_test_orc", "orc") common/file_utils.py:68: in
> create_iceberg_table_from_directory file_format))
> common/impala_connection.py:215: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:384: in __execute_query
> self.wait_for_finished(handle) beeswax/impala_beeswax.py:405: in
> wait_for_finished raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore: E CAUSED BY: IcebergTableLoadingException: Table does not exist
> at location:
> hdfs://localhost:20500/test-warehouse/iceberg_migrated_alter_test_orc
> Stacktrace
> query_test/test_iceberg.py:266: in test_migrated_table_field_id_resolution
> "iceberg_migrated_alter_test_orc", "orc")
> common/file_utils.py:68: in create_iceberg_table_from_directory
> file_format))
> common/impala_connection.py:215: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:384: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:405: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:ImpalaRuntimeException: Error making 'createTable' RPC to
> Hive Metastore:
> E CAUSED BY: IcebergTableLoadingException: Table does not exist at
> location:
> hdfs://localhost:20500/test-warehouse/iceberg_migrated_alter_test_orc
> {code}
> *Standard Error*
> {code:java}
> SET
> client_identifier=query_test/test_iceberg.py::TestIcebergTable::()::test_migrated_table_field_id_resolution[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':True;'abort_on_error':1;'exec_single_;
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_migrated_table_field_id_resolution_b59d79db`
> CASCADE;
> -- 2024-04-02 00:56:55,137 INFO MainThread: Started query
> f34399a8b7cddd67:031a3b96
> SET
> client_identifier=query_test/test_iceberg.py::TestIcebergTable::()::test_migrated_table_field_id_resolution[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':True;'abort_on_error':1;'exec_single_;
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_migrated_table_field_id_resolution_b59d79db`;
> -- 2024-04-02 00:56:57,302 INFO MainThread: Started query
> 94465af69907eac5:e33f17e0
> -- 2024-04-02 00:56:57,353 INFO MainThread: Created database
> "test_migrated_table_field_id_resolution_b59d79db" for test ID
> "query_test/test_iceberg.py::TestIcebergTable::()::test_migrated_table_field_id_resolution[protocol:
> beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]"
> Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/data/jenkins/workspace/impala-asf-master-exhaustive-release/repos/Impala/fe/target/dependency/jamm-0.4.0.jar
> Picked up JAVA_TOOL_OPTIONS:
>
[jira] [Commented] (IMPALA-12967) Testcase fails at test_migrated_table_field_id_resolution due to "Table does not exist"
[
https://issues.apache.org/jira/browse/IMPALA-12967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17833463#comment-17833463
]
Daniel Becker commented on IMPALA-12967:
This also happened before IMPALA-12809, and looks a bit like IMPALA-12266,
[~stigahuang] could you check it?
Thanks
> Testcase fails at test_migrated_table_field_id_resolution due to "Table does
> not exist"
> ---
>
> Key: IMPALA-12967
> URL: https://issues.apache.org/jira/browse/IMPALA-12967
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Reporter: Yida Wu
>Assignee: Daniel Becker
>Priority: Major
> Labels: broken-build
>
> Testcase test_migrated_table_field_id_resolution fails at exhaustive release
> build with following messages:
> *Regression*
> {code:java}
> query_test.test_iceberg.TestIcebergTable.test_migrated_table_field_id_resolution[protocol:
> beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none] (from pytest)
> {code}
> *Error Message*
> {code:java}
> query_test/test_iceberg.py:266: in test_migrated_table_field_id_resolution
> "iceberg_migrated_alter_test_orc", "orc") common/file_utils.py:68: in
> create_iceberg_table_from_directory file_format))
> common/impala_connection.py:215: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute handle =
> self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:384: in __execute_query
> self.wait_for_finished(handle) beeswax/impala_beeswax.py:405: in
> wait_for_finished raise ImpalaBeeswaxException("Query aborted:" +
> error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E
> Query aborted:ImpalaRuntimeException: Error making 'createTable' RPC to Hive
> Metastore: E CAUSED BY: IcebergTableLoadingException: Table does not exist
> at location:
> hdfs://localhost:20500/test-warehouse/iceberg_migrated_alter_test_orc
> Stacktrace
> query_test/test_iceberg.py:266: in test_migrated_table_field_id_resolution
> "iceberg_migrated_alter_test_orc", "orc")
> common/file_utils.py:68: in create_iceberg_table_from_directory
> file_format))
> common/impala_connection.py:215: in execute
> fetch_profile_after_close=fetch_profile_after_close)
> beeswax/impala_beeswax.py:191: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:384: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:405: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> EQuery aborted:ImpalaRuntimeException: Error making 'createTable' RPC to
> Hive Metastore:
> E CAUSED BY: IcebergTableLoadingException: Table does not exist at
> location:
> hdfs://localhost:20500/test-warehouse/iceberg_migrated_alter_test_orc
> {code}
> *Standard Error*
> {code:java}
> SET
> client_identifier=query_test/test_iceberg.py::TestIcebergTable::()::test_migrated_table_field_id_resolution[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':True;'abort_on_error':1;'exec_single_;
> SET sync_ddl=False;
> -- executing against localhost:21000
> DROP DATABASE IF EXISTS `test_migrated_table_field_id_resolution_b59d79db`
> CASCADE;
> -- 2024-04-02 00:56:55,137 INFO MainThread: Started query
> f34399a8b7cddd67:031a3b96
> SET
> client_identifier=query_test/test_iceberg.py::TestIcebergTable::()::test_migrated_table_field_id_resolution[protocol:beeswax|exec_option:{'test_replan':1;'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':True;'abort_on_error':1;'exec_single_;
> SET sync_ddl=False;
> -- executing against localhost:21000
> CREATE DATABASE `test_migrated_table_field_id_resolution_b59d79db`;
> -- 2024-04-02 00:56:57,302 INFO MainThread: Started query
> 94465af69907eac5:e33f17e0
> -- 2024-04-02 00:56:57,353 INFO MainThread: Created database
> "test_migrated_table_field_id_resolution_b59d79db" for test ID
> "query_test/test_iceberg.py::TestIcebergTable::()::test_migrated_table_field_id_resolution[protocol:
> beeswax | exec_option: {'test_replan': 1, 'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 0, 'disable_codegen': True,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> parquet/none]"
> Picked up JAVA_TOOL_OPTIONS:
>
[jira] [Updated] (IMPALA-12966) Investigate expected error message in AuthorizationStmtTest.testShow after IMPALA-12609
[ https://issues.apache.org/jira/browse/IMPALA-12966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-12966: --- Labels: impala-iceberg (was: ) > Investigate expected error message in AuthorizationStmtTest.testShow after > IMPALA-12609 > --- > > Key: IMPALA-12966 > URL: https://issues.apache.org/jira/browse/IMPALA-12966 > Project: IMPALA > Issue Type: Improvement >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > In the commit belonging to IMPALA-12609, in one of the test calls in > AuthorizationStmtTest.testShow() the error message kept varying between > "functional_parquet.iceberg_query_metadata" and "functional_parquet.*.*". In > the end the expected error message was limited to "functional_parquet" > because it fit both, but we should find out why it kept changing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12966) Investigate expected error message in AuthorizationStmtTest.testShow after IMPALA-12609
Daniel Becker created IMPALA-12966: -- Summary: Investigate expected error message in AuthorizationStmtTest.testShow after IMPALA-12609 Key: IMPALA-12966 URL: https://issues.apache.org/jira/browse/IMPALA-12966 Project: IMPALA Issue Type: Improvement Reporter: Daniel Becker Assignee: Daniel Becker In the commit belonging to IMPALA-12609, in one of the test calls in AuthorizationStmtTest.testShow() the error message kept varying between "functional_parquet.iceberg_query_metadata" and "functional_parquet.*.*". In the end the expected error message was limited to "functional_parquet" because it fit both, but we should find out why it kept changing. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12609) Implement SHOW TABLES IN statement to list Iceberg Metadata tables
[
https://issues.apache.org/jira/browse/IMPALA-12609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-12609.
Resolution: Fixed
> Implement SHOW TABLES IN statement to list Iceberg Metadata tables
> --
>
> Key: IMPALA-12609
> URL: https://issues.apache.org/jira/browse/IMPALA-12609
> Project: IMPALA
> Issue Type: Sub-task
> Components: Frontend
>Affects Versions: Impala 4.4.0
>Reporter: Tamas Mate
>Assignee: Daniel Becker
>Priority: Minor
> Labels: impala-iceberg
>
> {{SHOW TABLES IN}} statement could be used to list all the available metadata
> tables of an Iceberg table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12809) Iceberg metadata table scanner should always be scheduled to the coordinator
[
https://issues.apache.org/jira/browse/IMPALA-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-12809.
Resolution: Fixed
> Iceberg metadata table scanner should always be scheduled to the coordinator
>
>
> Key: IMPALA-12809
> URL: https://issues.apache.org/jira/browse/IMPALA-12809
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Tamas Mate
>Assignee: Daniel Becker
>Priority: Major
> Labels: impala-iceberg
>
> On larger clusters the Iceberg metadata scanner can be scheduled to
> executors, for example during a join. The fragment in this case will fail a
> precondition check, because either the frontend_ object will not be present
> or the table. Setting {{exec_at_coord}} to true is not enough and these
> fragments should be scheduled to the {{{}coord_only_executor_group{}}}.
> Additionally, setting NUM_NODES=1 should be a viable workaround.
> Reproducible with the following local dev Impala cluster:
> {{./bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=1
> --use_exclusive_coordinators}}
> and query:
> {{select count(b.parent_id) from
> functional_parquet.iceberg_query_metadata.history a}}
> {{join functional_parquet.iceberg_query_metadata.history b on a.snapshot_id =
> b.snapshot_id;}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12926) Refactor BINARY type handling in the backend
[ https://issues.apache.org/jira/browse/IMPALA-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-12926. Resolution: Fixed > Refactor BINARY type handling in the backend > > > Key: IMPALA-12926 > URL: https://issues.apache.org/jira/browse/IMPALA-12926 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > Currently the STRING and BINARY types are not distinguished in most of the > backend. In contrast to the frontend, PrimitiveType::TYPE_BINARY is not used > there at all, TYPE_STRING being used instead. This is to ensure that > everything that works for STRING also works for BINARY. So far only file > readers and writers have had to handle them differently, and they have access > to ColumnDescriptors which contain AuxColumnType fields that differentiate > these two types. > However, only top-level columns have ColumnDescriptors. Adding support or > BINARYs within complex types (see IMPALA-11491 and IMPALA-12651) necessitates > adding type information about STRING vs BINARY to embedded fields as well. > Using PrimitiveType::TYPE_BINARY would probably be the cleanest solution but > it would affect huge parts of the code as TYPE_BINARY would have to be added > to hundreds of switch statements and this would be error prone. > Instead, we should introduce a new field in ColumnType: 'is_binary', which is > true if the type is a BINARY and false otherwise. We keep using TYPE_STRING > as the PrimitiveType of the ColumnType for BINARYs. This way full type > information is present in ColumnType but code that does not differentiate > between STRING and BINARY will continue to work for BINARY. > With this change, AuxColumnType is no longer needed and should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12931) TestEventProcessingCustomConfigs.test_no_hms_event_incremental_refresh_transactional_table fails in some builds
Daniel Becker created IMPALA-12931:
--
Summary:
TestEventProcessingCustomConfigs.test_no_hms_event_incremental_refresh_transactional_table
fails in some builds
Key: IMPALA-12931
URL: https://issues.apache.org/jira/browse/IMPALA-12931
Project: IMPALA
Issue Type: Bug
Reporter: Daniel Becker
Assignee: Csaba Ringhofer
TestEventProcessingCustomConfigs.test_no_hms_event_incremental_refresh_transactional_table
fails in some builds:
h3. Error Message
{code:java}
assert [] == ['1', '2', '3'] Right contains more items, first extra item: '1'
Full diff: - [] + ['1', '2', '3']{code}
h3. Stacktrace
{code:java}
custom_cluster/test_events_custom_configs.py:1246: in
test_no_hms_event_incremental_refresh_transactional_table
assert results.data == ["1", "2", "3"]
E assert [] == ['1', '2', '3']
E Right contains more items, first extra item: '1'
E Full diff:
E - []
E + ['1', '2', '3']{code}
h3. Standard Error
{code:java}
-- 2024-03-19 21:11:27,574 INFO MainThread: Starting cluster with command:
/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/bin/start-impala-cluster.py
'--state_store_args=--statestore_update_frequency_ms=50
--statestore_priority_update_frequency_ms=50
--statestore_heartbeat_frequency_ms=50' --cluster_size=3 --num_coordinators=3
--log_dir=/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests
--log_level=1 '--state_store_args=None '
'--catalogd_args=--hms_event_incremental_refresh_transactional_table=false '
--impalad_args=--default_query_options=
21:11:28 MainThread: Found 0 impalad/0 statestored/0 catalogd process(es)
21:11:28 MainThread: Starting State Store logging to
/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/statestored.INFO
21:11:28 MainThread: Starting Catalog Service logging to
/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/catalogd.INFO
21:11:28 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/impalad.INFO
21:11:28 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/impalad_node1.INFO
21:11:28 MainThread: Starting Impala Daemon logging to
/data/jenkins/workspace/impala-cdw-master-staging-exhaustive-redhat8-shard1/repos/Impala/logs/custom_cluster_tests/impalad_node2.INFO
21:11:31 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:31 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:31 MainThread: Getting num_known_live_backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
21:11:31 MainThread: 'backends'
21:11:31 MainThread: Waiting for num_known_live_backends=3. Current value: None
21:11:32 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:32 MainThread: Getting num_known_live_backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
21:11:32 MainThread: Waiting for num_known_live_backends=3. Current value: 0
21:11:33 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:33 MainThread: Getting num_known_live_backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
21:11:33 MainThread: Waiting for num_known_live_backends=3. Current value: 0
21:11:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:34 MainThread: Getting num_known_live_backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25000
21:11:34 MainThread: num_known_live_backends has reached value: 3
21:11:34 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:34 MainThread: Getting num_known_live_backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25001
21:11:34 MainThread: num_known_live_backends has reached value: 3
21:11:35 MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
21:11:35 MainThread: Getting num_known_live_backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25002
21:11:35 MainThread: num_known_live_backends has reached value: 3
21:11:35 MainThread: Impala Cluster Running with 3 nodes (3 coordinators, 3
executors).
-- 2024-03-19 21:11:35,965 DEBUG MainThread: Found 3 impalad/1 statestored/1
catalogd process(es)
-- 2024-03-19 21:11:35,965 INFO MainThread: Getting metric:
statestore.live-backends from
impala-ec2-redhat86-m6i-4xlarge-ondemand-0866.vpc.cloudera.com:25010
-- 2024-03-19 21:11:35,968 INFO MainThread: Metric
'statestore.live-backends' has reached desired value: 4
-- 2024-03-19
[jira] [Created] (IMPALA-12930) TestExtDataSources.test_jdbc_data_source fails in some builds
Daniel Becker created IMPALA-12930:
--
Summary: TestExtDataSources.test_jdbc_data_source fails in some
builds
Key: IMPALA-12930
URL: https://issues.apache.org/jira/browse/IMPALA-12930
Project: IMPALA
Issue Type: Bug
Reporter: Daniel Becker
Assignee: Wenzhe Zhou
TestExtDataSources.test_jdbc_data_source fails in some of our builds.
h3. Stacktrace
{code:java}
query_test/test_ext_data_sources.py:75: in test_jdbc_data_source
self.run_test_case('QueryTest/jdbc-data-source', vector,
use_db=unique_database)
common/impala_test_suite.py:731: in run_test_case
self.__verify_exceptions(test_section['CATCH'], str(e), use_db)
common/impala_test_suite.py:551: in __verify_exceptions
(expected_str, actual_str)
E AssertionError: Unexpected exception string. Expected: row_regex:.*Caught
exception.*(FATAL: password authentication failed for user "UNKNOWN")
E Not found in actual: ImpalaBeeswaxException: Query aborted:Caught exception
while trying to execute query:Cannot create PoolableConnectionFactory (FATAL:
role "UNKNOWN" does not exist){code}
The parts of the test that failed were added by IMPALA-12802. [~wzhou] could
you take a look?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12926) Refactor BINARY type handling in the backend
[ https://issues.apache.org/jira/browse/IMPALA-12926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-12926: --- Summary: Refactor BINARY type handling in the backend (was: Remove AuxColumnType and add 'is_binary' field to ColumnType) > Refactor BINARY type handling in the backend > > > Key: IMPALA-12926 > URL: https://issues.apache.org/jira/browse/IMPALA-12926 > Project: IMPALA > Issue Type: Improvement > Components: Backend >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > Currently the STRING and BINARY types are not distinguished in most of the > backend. In contrast to the frontend, PrimitiveType::TYPE_BINARY is not used > there at all, TYPE_STRING being used instead. This is to ensure that > everything that works for STRING also works for BINARY. So far only file > readers and writers have had to handle them differently, and they have access > to ColumnDescriptors which contain AuxColumnType fields that differentiate > these two types. > However, only top-level columns have ColumnDescriptors. Adding support or > BINARYs within complex types (see IMPALA-11491 and IMPALA-12651) necessitates > adding type information about STRING vs BINARY to embedded fields as well. > Using PrimitiveType::TYPE_BINARY would probably be the cleanest solution but > it would affect huge parts of the code as TYPE_BINARY would have to be added > to hundreds of switch statements and this would be error prone. > Instead, we should introduce a new field in ColumnType: 'is_binary', which is > true if the type is a BINARY and false otherwise. We keep using TYPE_STRING > as the PrimitiveType of the ColumnType for BINARYs. This way full type > information is present in ColumnType but code that does not differentiate > between STRING and BINARY will continue to work for BINARY. > With this change, AuxColumnType is no longer needed and should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12926) Remove AuxColumnType and add 'is_binary' field to ColumnType
Daniel Becker created IMPALA-12926: -- Summary: Remove AuxColumnType and add 'is_binary' field to ColumnType Key: IMPALA-12926 URL: https://issues.apache.org/jira/browse/IMPALA-12926 Project: IMPALA Issue Type: Improvement Components: Backend Reporter: Daniel Becker Assignee: Daniel Becker Currently the STRING and BINARY types are not distinguished in most of the backend. In contrast to the frontend, PrimitiveType::TYPE_BINARY is not used there at all, TYPE_STRING being used instead. This is to ensure that everything that works for STRING also works for BINARY. So far only file readers and writers have had to handle them differently, and they have access to ColumnDescriptors which contain AuxColumnType fields that differentiate these two types. However, only top-level columns have ColumnDescriptors. Adding support or BINARYs within complex types (see IMPALA-11491 and IMPALA-12651) necessitates adding type information about STRING vs BINARY to embedded fields as well. Using PrimitiveType::TYPE_BINARY would probably be the cleanest solution but it would affect huge parts of the code as TYPE_BINARY would have to be added to hundreds of switch statements and this would be error prone. Instead, we should introduce a new field in ColumnType: 'is_binary', which is true if the type is a BINARY and false otherwise. We keep using TYPE_STRING as the PrimitiveType of the ColumnType for BINARYs. This way full type information is present in ColumnType but code that does not differentiate between STRING and BINARY will continue to work for BINARY. With this change, AuxColumnType is no longer needed and should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12916) TestEventProcessingError.test_event_processor_error_global_invalidate fails in some builds
[
https://issues.apache.org/jira/browse/IMPALA-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12916:
---
Issue Type: Bug (was: Improvement)
> TestEventProcessingError.test_event_processor_error_global_invalidate fails
> in some builds
> --
>
> Key: IMPALA-12916
> URL: https://issues.apache.org/jira/browse/IMPALA-12916
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Venugopal Reddy K
>Priority: Blocker
> Labels: broken-build
>
> TestEventProcessingError.test_event_processor_error_global_invalidate failed
> in one of our builds.
> {code:java}
> custom_cluster/test_event_processing_error.py:333: in
> test_event_processor_error_global_invalidate
> EventProcessorUtils.wait_for_event_processing(self)
> util/event_processor_utils.py:90: in wait_for_event_processing
> impala_cluster, timeout)
> util/event_processor_utils.py:102: in wait_for_event_processing_impl
> EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
> util/event_processor_utils.py:65: in wait_for_synced_event_id
> .format(status, error_msg))
> E Exception: Event processor is not working. Status: ERROR. Error msg:
> 2024-03-16T11:02:19.985
> E Unexpected exception received while processing event
> E java.lang.RuntimeException: Event processing failed due to error injection
> E at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
> E at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
> E at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
> E at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
> E at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> E at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> E at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> E at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> E at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> E at java.lang.Thread.run(Thread.java:750)
> E
> E Event id: 61885
> E Event Type: ADD_PARTITION
> E Event time: 1710612138
> E Database name: test_event_processor_error_global_invalidate_d325a3c5
> E Table name: hive_table_global_invalidate
> E Event message:
> H4sIAO1WS3PaMBD+Lz4TY5nwnOkhnSRt2jTJTLiFjEfYCyiRLVcStIThv3clmdQGA5lMb80FxH770D70LStPgVyA9Aaenkk20YNmk4uY8plQetAPei2vYVVYDHeSZTHLKUdllCZjYwRKR7CATEe5FDEoJWQEUuLnlIsx5RHLFpSzhGqIklbYpq24jcaajjmg/YwtILI/dvU3asNlblR/nN2cfbk4j4Znn68vNtjt+OmbEhnCq5FHRt4Av5SWeBgd9D3y1o2RF1YN3pWM89Ta9XQyx7o59NShrGUCki4JOiQkrcBA7TJkJZ0dSddJJMTuUOTJlcbDQwE0SGO3BKw2T5ah4Xr9WAPNkomqjEAYtIOgafP5RSXMBGbVTGlGp5A031UwPxk3j7dmq6BCTn2a03gG/owmQuR+SnMJiT+E3/oqy+f6UsiU6kq99xub+P5P7jPhf8Xj1TQTEr7D0ni7nestd0WP9KS2RSekrkfhm66AE5JA6HP6svSv8eOepTmHe5Dn1TJgsq7V1l9j821bjj4Ylu6FaiYyf1Lce2BnAXu8Nm56WxPjzAM7Af3aaXIYCQ7GDxorN0WEHBnRcsBwC7OHAtvO12HVeDYjElaaUpvl/nexBCqPPo3+wdw7xi2n+ABSkbAJgyTSLAVX+c0Tb1sDLWmmaGz6g5OOLyUHqRmoIiI2UEci48u/ysw8KeP8POHDPV7H8/gZNMumEfK3Qt9WKbRY9V7jpYWeIHvGaLtXcpQl52AHplTcTVUq1OvqRnYeBenu0m+h26vwnxUVxWWd040o7JRFbUL667WheUxfaZrmJebsNbycYgnN9RX2ZmUbiksgDMJTzKEEXzOl3YZ48PbOJTHh0XTk1XHiv1sLx2mvflP0Dm6K/2QvNE2PP7k2fWyIjw3xng3RPV6sN/Bvb8OT/b2MF5S5LEQiK3PSJePgaOtVdENT8x/3dcSR+CaF1iPG+gO2JHHipwsAAA==
> {code}
> This test was added by IMPALA-12832, [~VenuReddy] could one of you take a
> look?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12917) Several tests in TestEventProcessingError fail
Daniel Becker created IMPALA-12917:
--
Summary: Several tests in TestEventProcessingError fail
Key: IMPALA-12917
URL: https://issues.apache.org/jira/browse/IMPALA-12917
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Daniel Becker
Assignee: Venugopal Reddy K
The failing tests are
TestEventProcessingError.test_event_processor_error_alter_partition
TestEventProcessingError.test_event_processor_error_alter_partitions
TestEventProcessingError.test_event_processor_error_commit_compaction_event
TestEventProcessingError.test_event_processor_error_commit_txn
TestEventProcessingError.test_event_processor_error_stress_test
Stacktrace:
{code:java}
E Error: Error while compiling statement: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask.
java.lang.NullPointerException
E at org.apache.tez.client.TezClient.cleanStagingDir(TezClient.java:424)
E at org.apache.tez.client.TezClient.start(TezClient.java:413)
E at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:556)
E at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:387)
E at
org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:302)
E at
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:106)
E at
org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:468)
E at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:227)
E at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
E at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
E at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:356)
E at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:329)
E at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
E at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:107)
E at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:809)
E at org.apache.hadoop.hive.ql.Driver.run(Driver.java:546)
E at org.apache.hadoop.hive.ql.Driver.run(Driver.java:540)
E at
org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:190)
E at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:235)
E at
org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:92)
E at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:340)
E at java.security.AccessController.doPrivileged(Native Method)
E at javax.security.auth.Subject.doAs(Subject.java:422)
E at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
E at
org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:360)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.run(FutureTask.java:266)
E at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:748) (state=08S01,code=1)
{code}
These tests were introduced by IMPALA-12832, [~VenuReddy] could you take a look?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12916) TestEventProcessingError.test_event_processor_error_global_invalidate fails in some builds
[
https://issues.apache.org/jira/browse/IMPALA-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12916:
---
Description:
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
E at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:750)
E
E Event id: 61885
E Event Type: ADD_PARTITION
E Event time: 1710612138
E Database name: test_event_processor_error_global_invalidate_d325a3c5
E Table name: hive_table_global_invalidate
E Event message:
H4sIAO1WS3PaMBD+Lz4TY5nwnOkhnSRt2jTJTLiFjEfYCyiRLVcStIThv3clmdQGA5lMb80FxH770D70LStPgVyA9Aaenkk20YNmk4uY8plQetAPei2vYVVYDHeSZTHLKUdllCZjYwRKR7CATEe5FDEoJWQEUuLnlIsx5RHLFpSzhGqIklbYpq24jcaajjmg/YwtILI/dvU3asNlblR/nN2cfbk4j4Znn68vNtjt+OmbEhnCq5FHRt4Av5SWeBgd9D3y1o2RF1YN3pWM89Ta9XQyx7o59NShrGUCki4JOiQkrcBA7TJkJZ0dSddJJMTuUOTJlcbDQwE0SGO3BKw2T5ah4Xr9WAPNkomqjEAYtIOgafP5RSXMBGbVTGlGp5A031UwPxk3j7dmq6BCTn2a03gG/owmQuR+SnMJiT+E3/oqy+f6UsiU6kq99xub+P5P7jPhf8Xj1TQTEr7D0ni7nestd0WP9KS2RSekrkfhm66AE5JA6HP6svSv8eOepTmHe5Dn1TJgsq7V1l9j821bjj4Ylu6FaiYyf1Lce2BnAXu8Nm56WxPjzAM7Af3aaXIYCQ7GDxorN0WEHBnRcsBwC7OHAtvO12HVeDYjElaaUpvl/nexBCqPPo3+wdw7xi2n+ABSkbAJgyTSLAVX+c0Tb1sDLWmmaGz6g5OOLyUHqRmoIiI2UEci48u/ysw8KeP8POHDPV7H8/gZNMumEfK3Qt9WKbRY9V7jpYWeIHvGaLtXcpQl52AHplTcTVUq1OvqRnYeBenu0m+h26vwnxUVxWWd040o7JRFbUL667WheUxfaZrmJebsNbycYgnN9RX2ZmUbiksgDMJTzKEEXzOl3YZ48PbOJTHh0XTk1XHiv1sLx2mvflP0Dm6K/2QvNE2PP7k2fWyIjw3xng3RPV6sN/Bvb8OT/b2MF5S5LEQiK3PSJePgaOtVdENT8x/3dcSR+CaF1iPG+gO2JHHipwsAAA==
{code}
This test was added by IMPALA-12832, [~VenuReddy] could one of you take a look?
was:
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
[jira] [Assigned] (IMPALA-12916) TestEventProcessingError.test_event_processor_error_global_invalidate fails in some builds
[
https://issues.apache.org/jira/browse/IMPALA-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker reassigned IMPALA-12916:
--
Assignee: Venugopal Reddy K
> TestEventProcessingError.test_event_processor_error_global_invalidate fails
> in some builds
> --
>
> Key: IMPALA-12916
> URL: https://issues.apache.org/jira/browse/IMPALA-12916
> Project: IMPALA
> Issue Type: Improvement
>Reporter: Daniel Becker
>Assignee: Venugopal Reddy K
>Priority: Blocker
> Labels: broken-build
>
> TestEventProcessingError.test_event_processor_error_global_invalidate failed
> in one of our builds.
> {code:java}
> custom_cluster/test_event_processing_error.py:333: in
> test_event_processor_error_global_invalidate
> EventProcessorUtils.wait_for_event_processing(self)
> util/event_processor_utils.py:90: in wait_for_event_processing
> impala_cluster, timeout)
> util/event_processor_utils.py:102: in wait_for_event_processing_impl
> EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
> util/event_processor_utils.py:65: in wait_for_synced_event_id
> .format(status, error_msg))
> E Exception: Event processor is not working. Status: ERROR. Error msg:
> 2024-03-16T11:02:19.985
> E Unexpected exception received while processing event
> E java.lang.RuntimeException: Event processing failed due to error injection
> E at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
> E at
> org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
> E at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
> E at
> org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
> E at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> E at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> E at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> E at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> E at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> E at java.lang.Thread.run(Thread.java:750)
> E
> E Event id: 61885
> E Event Type: ADD_PARTITION
> E Event time: 1710612138
> E Database name: test_event_processor_error_global_invalidate_d325a3c5
> E Table name: hive_table_global_invalidate
> E Event message:
> H4sIAO1WS3PaMBD+Lz4TY5nwnOkhnSRt2jTJTLiFjEfYCyiRLVcStIThv3clmdQGA5lMb80FxH770D70LStPgVyA9Aaenkk20YNmk4uY8plQetAPei2vYVVYDHeSZTHLKUdllCZjYwRKR7CATEe5FDEoJWQEUuLnlIsx5RHLFpSzhGqIklbYpq24jcaajjmg/YwtILI/dvU3asNlblR/nN2cfbk4j4Znn68vNtjt+OmbEhnCq5FHRt4Av5SWeBgd9D3y1o2RF1YN3pWM89Ta9XQyx7o59NShrGUCki4JOiQkrcBA7TJkJZ0dSddJJMTuUOTJlcbDQwE0SGO3BKw2T5ah4Xr9WAPNkomqjEAYtIOgafP5RSXMBGbVTGlGp5A031UwPxk3j7dmq6BCTn2a03gG/owmQuR+SnMJiT+E3/oqy+f6UsiU6kq99xub+P5P7jPhf8Xj1TQTEr7D0ni7nestd0WP9KS2RSekrkfhm66AE5JA6HP6svSv8eOepTmHe5Dn1TJgsq7V1l9j821bjj4Ylu6FaiYyf1Lce2BnAXu8Nm56WxPjzAM7Af3aaXIYCQ7GDxorN0WEHBnRcsBwC7OHAtvO12HVeDYjElaaUpvl/nexBCqPPo3+wdw7xi2n+ABSkbAJgyTSLAVX+c0Tb1sDLWmmaGz6g5OOLyUHqRmoIiI2UEci48u/ysw8KeP8POHDPV7H8/gZNMumEfK3Qt9WKbRY9V7jpYWeIHvGaLtXcpQl52AHplTcTVUq1OvqRnYeBenu0m+h26vwnxUVxWWd040o7JRFbUL667WheUxfaZrmJebsNbycYgnN9RX2ZmUbiksgDMJTzKEEXzOl3YZ48PbOJTHh0XTk1XHiv1sLx2mvflP0Dm6K/2QvNE2PP7k2fWyIjw3xng3RPV6sN/Bvb8OT/b2MF5S5LEQiK3PSJePgaOtVdENT8x/3dcSR+CaF1iPG+gO2JHHipwsAAA==
> {code}
> This test was added by IMPALA-12832, [~VenuReddy] could one of you take a
> look?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12916) TestEventProcessingError.test_event_processor_error_global_invalidate fails in some builds
[
https://issues.apache.org/jira/browse/IMPALA-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12916:
---
Description:
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
E at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:750)
E
E Event id: 61885
E Event Type: ADD_PARTITION
E Event time: 1710612138
E Database name: test_event_processor_error_global_invalidate_d325a3c5
E Table name: hive_table_global_invalidate
E Event message:
H4sIAO1WS3PaMBD+Lz4TY5nwnOkhnSRt2jTJTLiFjEfYCyiRLVcStIThv3clmdQGA5lMb80FxH770D70LStPgVyA9Aaenkk20YNmk4uY8plQetAPei2vYVVYDHeSZTHLKUdllCZjYwRKR7CATEe5FDEoJWQEUuLnlIsx5RHLFpSzhGqIklbYpq24jcaajjmg/YwtILI/dvU3asNlblR/nN2cfbk4j4Znn68vNtjt+OmbEhnCq5FHRt4Av5SWeBgd9D3y1o2RF1YN3pWM89Ta9XQyx7o59NShrGUCki4JOiQkrcBA7TJkJZ0dSddJJMTuUOTJlcbDQwE0SGO3BKw2T5ah4Xr9WAPNkomqjEAYtIOgafP5RSXMBGbVTGlGp5A031UwPxk3j7dmq6BCTn2a03gG/owmQuR+SnMJiT+E3/oqy+f6UsiU6kq99xub+P5P7jPhf8Xj1TQTEr7D0ni7nestd0WP9KS2RSekrkfhm66AE5JA6HP6svSv8eOepTmHe5Dn1TJgsq7V1l9j821bjj4Ylu6FaiYyf1Lce2BnAXu8Nm56WxPjzAM7Af3aaXIYCQ7GDxorN0WEHBnRcsBwC7OHAtvO12HVeDYjElaaUpvl/nexBCqPPo3+wdw7xi2n+ABSkbAJgyTSLAVX+c0Tb1sDLWmmaGz6g5OOLyUHqRmoIiI2UEci48u/ysw8KeP8POHDPV7H8/gZNMumEfK3Qt9WKbRY9V7jpYWeIHvGaLtXcpQl52AHplTcTVUq1OvqRnYeBenu0m+h26vwnxUVxWWd040o7JRFbUL667WheUxfaZrmJebsNbycYgnN9RX2ZmUbiksgDMJTzKEEXzOl3YZ48PbOJTHh0XTk1XHiv1sLx2mvflP0Dm6K/2QvNE2PP7k2fWyIjw3xng3RPV6sN/Bvb8OT/b2MF5S5LEQiK3PSJePgaOtVdENT8x/3dcSR+CaF1iPG+gO2JHHipwsAAA==
{code}
This test was added by IMPALA-12832, @Reddy could one of you take a look?
was:
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
[jira] [Updated] (IMPALA-12916) TestEventProcessingError.test_event_processor_error_global_invalidate fails in some builds
[
https://issues.apache.org/jira/browse/IMPALA-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12916:
---
Description:
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
E at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:750)
E
E Event id: 61885
E Event Type: ADD_PARTITION
E Event time: 1710612138
E Database name: test_event_processor_error_global_invalidate_d325a3c5
E Table name: hive_table_global_invalidate
E Event message:
H4sIAO1WS3PaMBD+Lz4TY5nwnOkhnSRt2jTJTLiFjEfYCyiRLVcStIThv3clmdQGA5lMb80FxH770D70LStPgVyA9Aaenkk20YNmk4uY8plQetAPei2vYVVYDHeSZTHLKUdllCZjYwRKR7CATEe5FDEoJWQEUuLnlIsx5RHLFpSzhGqIklbYpq24jcaajjmg/YwtILI/dvU3asNlblR/nN2cfbk4j4Znn68vNtjt+OmbEhnCq5FHRt4Av5SWeBgd9D3y1o2RF1YN3pWM89Ta9XQyx7o59NShrGUCki4JOiQkrcBA7TJkJZ0dSddJJMTuUOTJlcbDQwE0SGO3BKw2T5ah4Xr9WAPNkomqjEAYtIOgafP5RSXMBGbVTGlGp5A031UwPxk3j7dmq6BCTn2a03gG/owmQuR+SnMJiT+E3/oqy+f6UsiU6kq99xub+P5P7jPhf8Xj1TQTEr7D0ni7nestd0WP9KS2RSekrkfhm66AE5JA6HP6svSv8eOepTmHe5Dn1TJgsq7V1l9j821bjj4Ylu6FaiYyf1Lce2BnAXu8Nm56WxPjzAM7Af3aaXIYCQ7GDxorN0WEHBnRcsBwC7OHAtvO12HVeDYjElaaUpvl/nexBCqPPo3+wdw7xi2n+ABSkbAJgyTSLAVX+c0Tb1sDLWmmaGz6g5OOLyUHqRmoIiI2UEci48u/ysw8KeP8POHDPV7H8/gZNMumEfK3Qt9WKbRY9V7jpYWeIHvGaLtXcpQl52AHplTcTVUq1OvqRnYeBenu0m+h26vwnxUVxWWd040o7JRFbUL667WheUxfaZrmJebsNbycYgnN9RX2ZmUbiksgDMJTzKEEXzOl3YZ48PbOJTHh0XTk1XHiv1sLx2mvflP0Dm6K/2QvNE2PP7k2fWyIjw3xng3RPV6sN/Bvb8OT/b2MF5S5LEQiK3PSJePgaOtVdENT8x/3dcSR+CaF1iPG+gO2JHHipwsAAA==
{code}
This test was added by IMPALA-12832, Venugopal Reddy Kondreddy could one of you
take a look?
was:
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
[jira] [Created] (IMPALA-12916) TestEventProcessingError.test_event_processor_error_global_invalidate fails in some builds
Daniel Becker created IMPALA-12916:
--
Summary:
TestEventProcessingError.test_event_processor_error_global_invalidate fails in
some builds
Key: IMPALA-12916
URL: https://issues.apache.org/jira/browse/IMPALA-12916
Project: IMPALA
Issue Type: Improvement
Reporter: Daniel Becker
TestEventProcessingError.test_event_processor_error_global_invalidate failed in
one of our builds.
{code:java}
custom_cluster/test_event_processing_error.py:333: in
test_event_processor_error_global_invalidate
EventProcessorUtils.wait_for_event_processing(self)
util/event_processor_utils.py:90: in wait_for_event_processing
impala_cluster, timeout)
util/event_processor_utils.py:102: in wait_for_event_processing_impl
EventProcessorUtils.wait_for_synced_event_id(timeout, current_event_id)
util/event_processor_utils.py:65: in wait_for_synced_event_id
.format(status, error_msg))
E Exception: Event processor is not working. Status: ERROR. Error msg:
2024-03-16T11:02:19.985
E Unexpected exception received while processing event
E java.lang.RuntimeException: Event processing failed due to error injection
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.injectErrorIfNeeded(MetastoreEvents.java:667)
E at
org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:694)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1188)
E at
org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:973)
E at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
E at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
E at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
E at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
E at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
E at java.lang.Thread.run(Thread.java:750)
E
E Event id: 61885
E Event Type: ADD_PARTITION
E Event time: 1710612138
E Database name: test_event_processor_error_global_invalidate_d325a3c5
E Table name: hive_table_global_invalidate
E Event message:
H4sIAO1WS3PaMBD+Lz4TY5nwnOkhnSRt2jTJTLiFjEfYCyiRLVcStIThv3clmdQGA5lMb80FxH770D70LStPgVyA9Aaenkk20YNmk4uY8plQetAPei2vYVVYDHeSZTHLKUdllCZjYwRKR7CATEe5FDEoJWQEUuLnlIsx5RHLFpSzhGqIklbYpq24jcaajjmg/YwtILI/dvU3asNlblR/nN2cfbk4j4Znn68vNtjt+OmbEhnCq5FHRt4Av5SWeBgd9D3y1o2RF1YN3pWM89Ta9XQyx7o59NShrGUCki4JOiQkrcBA7TJkJZ0dSddJJMTuUOTJlcbDQwE0SGO3BKw2T5ah4Xr9WAPNkomqjEAYtIOgafP5RSXMBGbVTGlGp5A031UwPxk3j7dmq6BCTn2a03gG/owmQuR+SnMJiT+E3/oqy+f6UsiU6kq99xub+P5P7jPhf8Xj1TQTEr7D0ni7nestd0WP9KS2RSekrkfhm66AE5JA6HP6svSv8eOepTmHe5Dn1TJgsq7V1l9j821bjj4Ylu6FaiYyf1Lce2BnAXu8Nm56WxPjzAM7Af3aaXIYCQ7GDxorN0WEHBnRcsBwC7OHAtvO12HVeDYjElaaUpvl/nexBCqPPo3+wdw7xi2n+ABSkbAJgyTSLAVX+c0Tb1sDLWmmaGz6g5OOLyUHqRmoIiI2UEci48u/ysw8KeP8POHDPV7H8/gZNMumEfK3Qt9WKbRY9V7jpYWeIHvGaLtXcpQl52AHplTcTVUq1OvqRnYeBenu0m+h26vwnxUVxWWd040o7JRFbUL667WheUxfaZrmJebsNbycYgnN9RX2ZmUbiksgDMJTzKEEXzOl3YZ48PbOJTHh0XTk1XHiv1sLx2mvflP0Dm6K/2QvNE2PP7k2fWyIjw3xng3RPV6sN/Bvb8OT/b2MF5S5LEQiK3PSJePgaOtVdENT8x/3dcSR+CaF1iPG+gO2JHHipwsAAA==
{code}
This test was added by IMPALA-12832, reviewed by [~csringhofer] and
[~stigahuang]. Could one of you take a look?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Comment Edited] (IMPALA-11491) Support BINARY nested in complex types in select list
[ https://issues.apache.org/jira/browse/IMPALA-11491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827488#comment-17827488 ] Daniel Becker edited comment on IMPALA-11491 at 3/15/24 12:27 PM: -- In the backend the distinction between STRING and BINARY values is not handled in the normal PrimitiveType, both are TYPE_STRING. Differentiation is possible through AuxColumnType objects which are found in ColumnDescriptors. However, as far as I can see, nested fields do not have ColumnDescriptors. If/when we want to support nested BINARY fields we should consider removing AuxColumnType and using PrimitiveType::TYPE_BINARY for BINARY fields. In my opinion it is cleaner and and more straightforward, although it affects many places in the code. Alternatively we could make AuxColumnType a member of ColumnType so it is available everywhere but existing code that handles strings (and binaries where they should be handled identically) does not have to be changed. [~csringhofer] what do you think? was (Author: daniel.becker): In the backend the distinction between STRING and BINARY values is not handled in the normal PrimitiveType, both are TYPE_STRING. Differentiation is possible through AuxColumnType objects which are found in ColumnDescriptors. However, as far as I can see, nested fields do not have ColumnDescriptors. If/when we want to support nested BINARY fields we should consider removing AuxColumnType and using PrimitiveType::TYPE_BINARY for BINARY fields. In my opinion it is cleaner and and more straightforward, although it affects many places in the code. Alternatively we could make AuxColumnType a member of ColumnType so it is available everywhere but existing code that handles strings (and binaries where they should be handled identically) does not have to be changed. > Support BINARY nested in complex types in select list > - > > Key: IMPALA-11491 > URL: https://issues.apache.org/jira/browse/IMPALA-11491 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Reporter: Csaba Ringhofer >Priority: Major > > The question is how the print BINARY types when the parent complex type is > printed to JSON. We can't rely on the existing Hive behavior, as it turned > out to be buggy: > HIVE-26454 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12809) Iceberg metadata table scanner should always be scheduled to the coordinator
[
https://issues.apache.org/jira/browse/IMPALA-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12809:
---
Summary: Iceberg metadata table scanner should always be scheduled to the
coordinator (was: Iceberg metadata table scanner can be scheduled to executors)
> Iceberg metadata table scanner should always be scheduled to the coordinator
>
>
> Key: IMPALA-12809
> URL: https://issues.apache.org/jira/browse/IMPALA-12809
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Tamas Mate
>Assignee: Daniel Becker
>Priority: Major
> Labels: impala-iceberg
>
> On larger clusters the Iceberg metadata scanner can be scheduled to
> executors, for example during a join. The fragment in this case will fail a
> precondition check, because either the frontend_ object will not be present
> or the table. Setting {{exec_at_coord}} to true is not enough and these
> fragments should be scheduled to the {{{}coord_only_executor_group{}}}.
> Additionally, setting NUM_NODES=1 should be a viable workaround.
> Reproducible with the following local dev Impala cluster:
> {{./bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=1
> --use_exclusive_coordinators}}
> and query:
> {{select count(b.parent_id) from
> functional_parquet.iceberg_query_metadata.history a}}
> {{join functional_parquet.iceberg_query_metadata.history b on a.snapshot_id =
> b.snapshot_id;}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12899) Temporary workaround for BINARY in complex types
[ https://issues.apache.org/jira/browse/IMPALA-12899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-12899: --- Parent: IMPALA-10947 Issue Type: Sub-task (was: Improvement) > Temporary workaround for BINARY in complex types > > > Key: IMPALA-12899 > URL: https://issues.apache.org/jira/browse/IMPALA-12899 > Project: IMPALA > Issue Type: Sub-task >Reporter: Daniel Becker >Assignee: Daniel Becker >Priority: Major > > The BINARY type is currently not supported inside complex types and a > cross-component decision is probably needed to support it (see IMPALA-11491). > We would like to enable EXPAND_COMPLEX_TYPES for Iceberg metadata tables > (IMPALA-12612), which requires that queries with BINARY inside complex types > don't fail. Enabling EXPAND_COMPLEX_TYPES is a more prioritised issue than > IMPALA-11491, so we should come up with a temporary solution, e.g. NULLing > BINARY values in complex types and logging a warning, or setting these BINARY > values to a warning string. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12899) Temporary workaround for BINARY in complex types
Daniel Becker created IMPALA-12899: -- Summary: Temporary workaround for BINARY in complex types Key: IMPALA-12899 URL: https://issues.apache.org/jira/browse/IMPALA-12899 Project: IMPALA Issue Type: Improvement Reporter: Daniel Becker Assignee: Daniel Becker The BINARY type is currently not supported inside complex types and a cross-component decision is probably needed to support it (see IMPALA-11491). We would like to enable EXPAND_COMPLEX_TYPES for Iceberg metadata tables (IMPALA-12612), which requires that queries with BINARY inside complex types don't fail. Enabling EXPAND_COMPLEX_TYPES is a more prioritised issue than IMPALA-11491, so we should come up with a temporary solution, e.g. NULLing BINARY values in complex types and logging a warning, or setting these BINARY values to a warning string. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12809) Iceberg metadata table scanner can be scheduled to executors
[
https://issues.apache.org/jira/browse/IMPALA-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on IMPALA-12809 started by Daniel Becker.
--
> Iceberg metadata table scanner can be scheduled to executors
>
>
> Key: IMPALA-12809
> URL: https://issues.apache.org/jira/browse/IMPALA-12809
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Tamas Mate
>Assignee: Daniel Becker
>Priority: Major
> Labels: impala-iceberg
>
> On larger clusters the Iceberg metadata scanner can be scheduled to
> executors, for example during a join. The fragment in this case will fail a
> precondition check, because either the frontend_ object will not be present
> or the table. Setting {{exec_at_coord}} to true is not enough and these
> fragments should be scheduled to the {{{}coord_only_executor_group{}}}.
> Additionally, setting NUM_NODES=1 should be a viable workaround.
> Reproducible with the following local dev Impala cluster:
> {{./bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=1
> --use_exclusive_coordinators}}
> and query:
> {{select count(b.parent_id) from
> functional_parquet.iceberg_query_metadata.history a}}
> {{join functional_parquet.iceberg_query_metadata.history b on a.snapshot_id =
> b.snapshot_id;}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Assigned] (IMPALA-12809) Iceberg metadata table scanner can be scheduled to executors
[
https://issues.apache.org/jira/browse/IMPALA-12809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker reassigned IMPALA-12809:
--
Assignee: Daniel Becker (was: Gabor Kaszab)
> Iceberg metadata table scanner can be scheduled to executors
>
>
> Key: IMPALA-12809
> URL: https://issues.apache.org/jira/browse/IMPALA-12809
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 4.4.0
>Reporter: Tamas Mate
>Assignee: Daniel Becker
>Priority: Major
> Labels: impala-iceberg
>
> On larger clusters the Iceberg metadata scanner can be scheduled to
> executors, for example during a join. The fragment in this case will fail a
> precondition check, because either the frontend_ object will not be present
> or the table. Setting {{exec_at_coord}} to true is not enough and these
> fragments should be scheduled to the {{{}coord_only_executor_group{}}}.
> Additionally, setting NUM_NODES=1 should be a viable workaround.
> Reproducible with the following local dev Impala cluster:
> {{./bin/start-impala-cluster.py --cluster_size=3 --num_coordinators=1
> --use_exclusive_coordinators}}
> and query:
> {{select count(b.parent_id) from
> functional_parquet.iceberg_query_metadata.history a}}
> {{join functional_parquet.iceberg_query_metadata.history b on a.snapshot_id =
> b.snapshot_id;}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12845) Crash with DESCRIBE on a complex type from an Iceberg table
[
https://issues.apache.org/jira/browse/IMPALA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker resolved IMPALA-12845.
Resolution: Fixed
> Crash with DESCRIBE on a complex type from an Iceberg table
> ---
>
> Key: IMPALA-12845
> URL: https://issues.apache.org/jira/browse/IMPALA-12845
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Critical
>
> A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
> In Hive:
> {code:java}
> create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
> In Impala:
> {code:java}
> describe ice_struct.s
> Caught exception TSocket read 0 bytes, type= 'thrift.transport.TTransport.TTransportException'> in FetchResults.
> Error communicating with impalad: TSocket read 0 bytes{code}
> The logs in impalad.FATAL indicate that we hit a DCHECK:
> {code:java}
> F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col
> == metadata_.columns.size() (3 vs. 4){code}
>
> Note that this also happens with Iceberg metadata tables, for example:
> {code:java}
> describe
> functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
> With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-11475) Codegen WriteSlot for BINARY columns in TextConverter
[ https://issues.apache.org/jira/browse/IMPALA-11475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker updated IMPALA-11475: --- Summary: Codegen WriteSlot for BINARY columns in TextConverter (was: Codegen WriteSlot for BINARY columns) > Codegen WriteSlot for BINARY columns in TextConverter > - > > Key: IMPALA-11475 > URL: https://issues.apache.org/jira/browse/IMPALA-11475 > Project: IMPALA > Issue Type: Sub-task >Reporter: Csaba Ringhofer >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Work started] (IMPALA-12611) Add support to MAP type Iceberg Metadata table columns
[ https://issues.apache.org/jira/browse/IMPALA-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on IMPALA-12611 started by Daniel Becker. -- > Add support to MAP type Iceberg Metadata table columns > -- > > Key: IMPALA-12611 > URL: https://issues.apache.org/jira/browse/IMPALA-12611 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Affects Versions: Impala 4.4.0 >Reporter: Tamas Mate >Assignee: Daniel Becker >Priority: Major > Labels: impala-iceberg > > MAP type columns are currently filled with NULLs, we should populate them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Resolved] (IMPALA-12610) Add support to ARRAY type Iceberg Metadata table columns
[ https://issues.apache.org/jira/browse/IMPALA-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Becker resolved IMPALA-12610. Resolution: Fixed > Add support to ARRAY type Iceberg Metadata table columns > > > Key: IMPALA-12610 > URL: https://issues.apache.org/jira/browse/IMPALA-12610 > Project: IMPALA > Issue Type: Sub-task > Components: Backend, Frontend >Affects Versions: Impala 4.4.0 >Reporter: Tamas Mate >Assignee: Tamas Mate >Priority: Major > Labels: impala-iceberg > > ARRAY type columns are currently filled with NULLs, we should populate them. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12845) Crash with DESCRIBE on a complex type from an Iceberg table
[
https://issues.apache.org/jira/browse/IMPALA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823582#comment-17823582
]
Daniel Becker commented on IMPALA-12845:
I checked and in release mode it doesn't crash Impala, though it confuses
impala-shell.
> Crash with DESCRIBE on a complex type from an Iceberg table
> ---
>
> Key: IMPALA-12845
> URL: https://issues.apache.org/jira/browse/IMPALA-12845
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Critical
>
> A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
> In Hive:
> {code:java}
> create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
> In Impala:
> {code:java}
> describe ice_struct.s
> Caught exception TSocket read 0 bytes, type= 'thrift.transport.TTransport.TTransportException'> in FetchResults.
> Error communicating with impalad: TSocket read 0 bytes{code}
> The logs in impalad.FATAL indicate that we hit a DCHECK:
> {code:java}
> F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col
> == metadata_.columns.size() (3 vs. 4){code}
>
> Note that this also happens with Iceberg metadata tables, for example:
> {code:java}
> describe
> functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
> With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12845) Crash with DESCRIBE on a complex type from an Iceberg table
[
https://issues.apache.org/jira/browse/IMPALA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823560#comment-17823560
]
Daniel Becker commented on IMPALA-12845:
Hi [~laszlog] it is only related because it affects Iceberg metadata tables, I
actually noticed this bug when playing with metadata tables.
??a completely innocent user action can bring down an Impala cluster??
It may not be that bad because the crash is because of a DCHECK. I can check
what happens in release mode.
Anyway, this is under review: https://gerrit.cloudera.org/#/c/21105/
> Crash with DESCRIBE on a complex type from an Iceberg table
> ---
>
> Key: IMPALA-12845
> URL: https://issues.apache.org/jira/browse/IMPALA-12845
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Critical
>
> A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
> In Hive:
> {code:java}
> create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
> In Impala:
> {code:java}
> describe ice_struct.s
> Caught exception TSocket read 0 bytes, type= 'thrift.transport.TTransport.TTransportException'> in FetchResults.
> Error communicating with impalad: TSocket read 0 bytes{code}
> The logs in impalad.FATAL indicate that we hit a DCHECK:
> {code:java}
> F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col
> == metadata_.columns.size() (3 vs. 4){code}
>
> Note that this also happens with Iceberg metadata tables, for example:
> {code:java}
> describe
> functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
> With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12616) test_restart_catalogd_while_handling_rpc_response* tests fail not reaching expected states
[
https://issues.apache.org/jira/browse/IMPALA-12616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823182#comment-17823182
]
Daniel Becker commented on IMPALA-12616:
Working on it.
> test_restart_catalogd_while_handling_rpc_response* tests fail not reaching
> expected states
> --
>
> Key: IMPALA-12616
> URL: https://issues.apache.org/jira/browse/IMPALA-12616
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
>Affects Versions: Impala 1.4.2
>Reporter: Andrew Sherman
>Assignee: Daniel Becker
>Priority: Critical
>
> There are failures in both
> custom_cluster.test_restart_services.TestRestart.test_restart_catalogd_while_handling_rpc_response_with_timeout
> and
> custom_cluster.test_restart_services.TestRestart.test_restart_catalogd_while_handling_rpc_response_with_max_iters,
> both look the same:
> {code:java}
> custom_cluster/test_restart_services.py:232: in
> test_restart_catalogd_while_handling_rpc_response_with_timeout
> self.wait_for_state(handle, self.client.QUERY_STATES["FINISHED"],
> max_wait_time)
> common/impala_test_suite.py:1181: in wait_for_state
> self.wait_for_any_state(handle, [expected_state], timeout, client)
> common/impala_test_suite.py:1199: in wait_for_any_state
> raise Timeout(timeout_msg)
> E Timeout: query '6a4e0bad9b511ccf:bf93de68' did not reach one of
> the expected states [4], last known state 5
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Updated] (IMPALA-12845) Crash with DESCRIBE on a complex type from an Iceberg table
[
https://issues.apache.org/jira/browse/IMPALA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Becker updated IMPALA-12845:
---
Summary: Crash with DESCRIBE on a complex type from an Iceberg table (was:
Crash with DESCRIBE on a struct from an Iceberg table)
> Crash with DESCRIBE on a complex type from an Iceberg table
> ---
>
> Key: IMPALA-12845
> URL: https://issues.apache.org/jira/browse/IMPALA-12845
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
> In Hive:
> {code:java}
> create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
> In Impala:
> {code:java}
> describe ice_struct.s
> Caught exception TSocket read 0 bytes, type= 'thrift.transport.TTransport.TTransportException'> in FetchResults.
> Error communicating with impalad: TSocket read 0 bytes{code}
> The logs in impalad.FATAL indicate that we hit a DCHECK:
> {code:java}
> F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col
> == metadata_.columns.size() (3 vs. 4){code}
>
> Note that this also happens with Iceberg metadata tables, for example:
> {code:java}
> describe
> functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
> With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12845) Crash with DESCRIBE on a struct from an Iceberg table
[
https://issues.apache.org/jira/browse/IMPALA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822645#comment-17822645
]
Daniel Becker commented on IMPALA-12845:
The problem is that for Iceberg tables, the DESCRIBE statement returns four
columns: {{name, type, comment and nullable}} (only Iceberg and Kudu tables
have '{{{}nullable{}}}'). The describe statement for complex types only returns
the first three columns as they are always nullable.
For non-Iceberg non-Kudu tables and complex types see
[https://github.com/apache/impala/blob/30fbcc94ea08bb0a5eb58d68bcce9c2a0eb9750e/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java#L269|https://github.com/apache/impala/blob/30fbcc94ea08bb0a5eb58d68bcce9c2a0eb9750e/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java#L269,],
for Iceberg tables see
[https://github.com/apache/impala/blob/30fbcc94ea08bb0a5eb58d68bcce9c2a0eb9750e/fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java#L334.]
The DCHECK in HS2ColumnarResultSet::AddOneRow() expects the number of columns
to be the same from the DESCRIBE statement response and the {{metadata_}}
field, for which the columns are ultimately set here:
[https://github.com/apache/impala/blob/2f14fd29c0b47fc2c170a7f0eb1cecaf6b9704f4/fe/src/main/java/org/apache/impala/service/Frontend.java#L661]
I'll try to find a way to skip adding "nullable" if the target is a complex
type instead of a table.
> Crash with DESCRIBE on a struct from an Iceberg table
> -
>
> Key: IMPALA-12845
> URL: https://issues.apache.org/jira/browse/IMPALA-12845
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
> In Hive:
> {code:java}
> create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
> In Impala:
> {code:java}
> describe ice_struct.s
> Caught exception TSocket read 0 bytes, type= 'thrift.transport.TTransport.TTransportException'> in FetchResults.
> Error communicating with impalad: TSocket read 0 bytes{code}
> The logs in impalad.FATAL indicate that we hit a DCHECK:
> {code:java}
> F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col
> == metadata_.columns.size() (3 vs. 4){code}
>
> Note that this also happens with Iceberg metadata tables, for example:
> {code:java}
> describe
> functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
> With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-12845) Crash with DESCRIBE on a struct from an Iceberg table
[
https://issues.apache.org/jira/browse/IMPALA-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17821179#comment-17821179
]
Daniel Becker commented on IMPALA-12845:
It also happens with collections:
Array:
In Hive:
{code:java}
create table ice_arr (id INT, a ARRAY) stored by iceberg;{code}
In Impala:
{code:java}
describe ice_arr.a;{code}
Map:
In Hive:
{code:java}
create table ice_map (id INT, m MAP) stored by iceberg;{code}
In Impala:
{code:java}
describe ice_map.m;{code}
> Crash with DESCRIBE on a struct from an Iceberg table
> -
>
> Key: IMPALA-12845
> URL: https://issues.apache.org/jira/browse/IMPALA-12845
> Project: IMPALA
> Issue Type: Bug
>Reporter: Daniel Becker
>Assignee: Daniel Becker
>Priority: Major
>
> A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
> In Hive:
> {code:java}
> create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
> In Impala:
> {code:java}
> describe ice_struct.s
> Caught exception TSocket read 0 bytes, type= 'thrift.transport.TTransport.TTransportException'> in FetchResults.
> Error communicating with impalad: TSocket read 0 bytes{code}
> The logs in impalad.FATAL indicate that we hit a DCHECK:
> {code:java}
> F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col
> == metadata_.columns.size() (3 vs. 4){code}
>
> Note that this also happens with Iceberg metadata tables, for example:
> {code:java}
> describe
> functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
> With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Created] (IMPALA-12845) Crash with DESCRIBE on a struct from an Iceberg table
Daniel Becker created IMPALA-12845:
--
Summary: Crash with DESCRIBE on a struct from an Iceberg table
Key: IMPALA-12845
URL: https://issues.apache.org/jira/browse/IMPALA-12845
Project: IMPALA
Issue Type: Bug
Reporter: Daniel Becker
Assignee: Daniel Becker
A DESCRIBE statement on a struct contained in an Iceberg table crashes Impala:
In Hive:
{code:java}
create table ice_struct (id INT, s STRUCT) stored by iceberg;{code}
In Impala:
{code:java}
describe ice_struct.s
Caught exception TSocket read 0 bytes, type= in FetchResults.
Error communicating with impalad: TSocket read 0 bytes{code}
The logs in impalad.FATAL indicate that we hit a DCHECK:
{code:java}
F0227 11:53:49.936864 680819 query-result-set.cc:386] Check failed: num_col ==
metadata_.columns.size() (3 vs. 4){code}
Note that this also happens with Iceberg metadata tables, for example:
{code:java}
describe
functional_parquet.iceberg_query_metadata.entries.readable_metrics;{code}
With non-Iceberg tables there is no error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
