[
https://issues.apache.org/jira/browse/SPARK-41344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17641666#comment-17641666
]
Kevin Cheung edited comment on SPARK-41344 at 12/1/22 4:57 AM:
---------------------------------------------------------------
[~hyukjin.kwon] I currently don't have the bandwidth for this at the moment. I
noticed this issue when I was upgrading a connector to spark 3.3. So I did the
investigation and concluded that this looks like a design issue since users
will never know what's the "real" error should they face NoSuchTableException,
NoSuchDatabaseException, NoSuchNamespaceException.
Hopefully someone who's familiar or has extra time can investigate the fix
further.
was (Author: kecheung):
[~hyukjin.kwon] I currently don't have the bandwidth for this at the moment. I
noticed this issue when I was upgrading a connector to spark 3.3. So I did the
investigation and concluded that this looks like a design issue since users
will never know what's the "real" error should they face NoSuchTableException,
NoSuchDatabaseException, NoSuchNamespaceException.
> Reading V2 datasource masks underlying error
> --------------------------------------------
>
> Key: SPARK-41344
> URL: https://issues.apache.org/jira/browse/SPARK-41344
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0, 3.3.1, 3.4.0
> Reporter: Kevin Cheung
> Priority: Major
>
> In Spark 3.3,
> # DataSourceV2Utils, the loadV2Source calls:
> {*}(CatalogV2Util.loadTable(catalog, ident, timeTravel).get{*},
> Some(catalog), Some(ident)).
> # CatalogV2Util.scala, when it tries to *loadTable(x,x,x)* and it fails with
> any of these exceptions NoSuchTableException, NoSuchDatabaseException,
> NoSuchNamespaceException, it would return None
> # Coming back to DataSourceV2Utils, None was previously returned and calling
> None.get results in a cryptic error technically "correct", but the *original
> exception NoSuchTableException, NoSuchDatabaseException,
> NoSuchNamespaceException are thrown away.*
>
> *Ask:*
> Retain the original error and propagate this to the user. Prior to Spark 3.3,
> the *original error* was shown and this seems like a design flaw.
>
> *Sample user facing error*
> None.get
> java.util.NoSuchElementException: None.get
> at scala.None$.get(Option.scala:529)
> at scala.None$.get(Option.scala:527)
> at
> org.apache.spark.sql.execution.datasources.v2.DataSourceV2Utils$.loadV2Source(DataSourceV2Utils.scala:129)
> at
> org.apache.spark.sql.DataFrameReader.$anonfun$load$1(DataFrameReader.scala:209)
> at scala.Option.flatMap(Option.scala:271)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:207)
> at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:171)
>
> *DataSourceV2Utils.scala - CatalogV2Util.loadTable(x,x,x).get*
> [https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Utils.scala#L137]
> *CatalogV2Util.scala - Option(catalog.asTableCatalog.loadTable(ident))*
> {*}{{*}}[https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L341]
> *CatalogV2Util.scala - catching the exceptions*
> [https://github.com/apache/spark/blob/7fd654c0142ab9e4002882da4e65d3b25bebd26c/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogV2Util.scala#L344]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
