I agree. If the user configures an invalid catalog, it should fail and propagate the exception. Running with a catalog other than the one the user requested is incorrect.
On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer <russell.spit...@gmail.com> wrote: > I was convinced that we should probably just fail, but if that is too much > of a change, then logging the exception is also acceptable. > > On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Hi devs, >> >> I got another report regarding configuring v2 session catalog, when Spark >> fails to instantiate the configured catalog. For now, it just simply logs >> error message without exception information, and silently uses the default >> session catalog. >> >> >> https://github.com/apache/spark/blob/3819d39607392aa968595e3d97b84fedf83d08d9/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala#L75-L95 >> >> IMO, as the user intentionally provides the session catalog, it shouldn't >> fail back and just throw the exception. Otherwise (if we still want to do >> the failback), we need to add the exception information in the error log >> message at least. >> >> Would like to hear the voices. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> > -- Ryan Blue Software Engineer Netflix
