[
https://issues.apache.org/jira/browse/FLINK-40236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolaus Schuetz updated FLINK-40236:
-------------------------------------
Description:
[_infer_type|https://github.com/apache/flink/blob/release-2.3.0/flink-python/pyflink/table/types.py#L1494]
in *flink-python/pyflink/table/types.py* infers a list's element type from
obj[0] instead of the first non-None element it scans for. When the first
element is None, the
array element type collapses to NULL, so schema inference becomes
order-dependent:
{code}
_infer_type([1, None]) -> ArrayType(BigIntType) # correct
_infer_type([None, 1]) -> ArrayType(NullType) # wrong; should be BigIntType
{code}
The dict branch just above handles this correctly (it infers from the found
non-None value). from_elements([Row(c=[None, 1])]) without an explicit schema
infers ARRAY<NULL>, which
then typically errors downstream.
Fix: infer from the scanned non-None element v rather than obj[0]. Verified
against the released apache-flink 2.3.0 wheel: the array element type is
NullType before the fix and
BigIntType after.
was:
_infer_type in flink-python/pyflink/table/types.py infers a list's element type
from obj[0] instead of the first non-None element it scans for. When the first
element is None, the
array element type collapses to NULL, so schema inference becomes
order-dependent:
{code}
_infer_type([1, None]) -> ArrayType(BigIntType) # correct
_infer_type([None, 1]) -> ArrayType(NullType) # wrong; should be BigIntType
{code}
The dict branch just above handles this correctly (it infers from the found
non-None value). from_elements([Row(c=[None, 1])]) without an explicit schema
infers ARRAY<NULL>, which
then typically errors downstream.
Fix: infer from the scanned non-None element v rather than obj[0]. Verified
against the released apache-flink 2.3.0 wheel: the array element type is
NullType before the fix and
BigIntType after.
> _infer_type infers ARRAY<NULL> for a list whose first element is None
> ---------------------------------------------------------------------
>
> Key: FLINK-40236
> URL: https://issues.apache.org/jira/browse/FLINK-40236
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Reporter: Nikolaus Schuetz
> Priority: Minor
> Labels: pull-request-available
>
> [_infer_type|https://github.com/apache/flink/blob/release-2.3.0/flink-python/pyflink/table/types.py#L1494]
> in *flink-python/pyflink/table/types.py* infers a list's element type from
> obj[0] instead of the first non-None element it scans for. When the first
> element is None, the
> array element type collapses to NULL, so schema inference becomes
> order-dependent:
> {code}
> _infer_type([1, None]) -> ArrayType(BigIntType) # correct
> _infer_type([None, 1]) -> ArrayType(NullType) # wrong; should be
> BigIntType
> {code}
> The dict branch just above handles this correctly (it infers from the found
> non-None value). from_elements([Row(c=[None, 1])]) without an explicit schema
> infers ARRAY<NULL>, which
> then typically errors downstream.
> Fix: infer from the scanned non-None element v rather than obj[0]. Verified
> against the released apache-flink 2.3.0 wheel: the array element type is
> NullType before the fix and
> BigIntType after.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
