[
https://issues.apache.org/jira/browse/SPARK-44634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giambattista Bloisi updated SPARK-44634:
----------------------------------------
Description:
Hi,
while upgrading a project from spark 2.4.0 to 3.4.1 version, I have
encountered the same problem described in [java - Encoders.bean attempts to
check the validity of a return type considering its generic type and not its
concrete class, with Spark 3.4.0 - Stack
Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when
the passed class contains a field whose type is a nested bean with type
arguments:
{code:java}
class A<T> {
T value;
// value getter and setter
}
class B {
A<String> stringHolder;
// stringHolder getter and setter
}
Encoders.bean(B.class); // throws "SparkUnsupportedOperationException:
[ENCODER_NOT_FOUND]..."{code}
It looks like this is a regression introduced with [SPARK-42093 SQL Move
JavaTypeInference to
AgnosticEncoders|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b]
while getting rid of TypeToken, that somehow managed that case.
was:
Hi,
while upgrading a project from spark 2.4.0 to 3.4.1 version, I have
encountered the same problem described in [java - Encoders.bean attempts to
check the validity of a return type considering its generic type and not its
concrete class, with Spark 3.4.0 - Stack
Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
Put it short, starting from Spark 3.4.x Encoders.bean throws an exception when
the passed class contains a field whose type is a nested bean with type
arguments:
{code:java}
class A<T> {
T value;
// value getter and setter
}
class B {
A<String> stringHolder;
// stringHolder getter and setter
}
Encoders.bean(B.class); // throws "SparkUnsupportedOperationException:
[ENCODER_NOT_FOUND]..."{code}
It looks like this is a regression introduced with [SPARK-42093 SQL Move
JavaTypeInference to
AgnosticEncoders|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b]
while getting rid of TypeToken, that somehow managed that case.
I'm going to submit a PR to re-enable this functionality.
> Encoders.bean does no longer support nested beans with type arguments
> ---------------------------------------------------------------------
>
> Key: SPARK-44634
> URL: https://issues.apache.org/jira/browse/SPARK-44634
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.1, 3.5.0, 4.0.0
> Reporter: Giambattista Bloisi
> Priority: Major
>
> Hi,
> while upgrading a project from spark 2.4.0 to 3.4.1 version, I have
> encountered the same problem described in [java - Encoders.bean attempts to
> check the validity of a return type considering its generic type and not its
> concrete class, with Spark 3.4.0 - Stack
> Overflow|https://stackoverflow.com/questions/76045255/encoders-bean-attempts-to-check-the-validity-of-a-return-type-considering-its-ge].
> Put it short, starting from Spark 3.4.x Encoders.bean throws an exception
> when the passed class contains a field whose type is a nested bean with type
> arguments:
>
> {code:java}
> class A<T> {
> T value;
> // value getter and setter
> }
> class B {
> A<String> stringHolder;
> // stringHolder getter and setter
> }
> Encoders.bean(B.class); // throws "SparkUnsupportedOperationException:
> [ENCODER_NOT_FOUND]..."{code}
>
>
> It looks like this is a regression introduced with [SPARK-42093 SQL Move
> JavaTypeInference to
> AgnosticEncoders|https://github.com/apache/spark/commit/18672003513d5a4aa610b6b94dbbc15c33185d3#diff-1191737b908340a2f4c22b71b1c40ebaa0da9d8b40c958089c346a3bda26943b]
> while getting rid of TypeToken, that somehow managed that case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
