+1 to fail fast. Thanks for reporting this, Jungtaek! On Mon, Oct 26, 2020 at 8:36 AM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote:
> Yeah I'm in favor of fast-fail if things are not working out as end users > intended. Spark should only fail back when it doesn't make any difference > but only some sort of performance. (like whole stage codegen) This fail > back brings behavioral differences, which should be considered as a bug. > > I'll file an issue and raise a PR sooner. Thanks for providing voices! > > On Sat, Oct 24, 2020 at 2:03 AM Ryan Blue <rb...@netflix.com> wrote: > >> I agree. If the user configures an invalid catalog, it should fail and >> propagate the exception. Running with a catalog other than the one the user >> requested is incorrect. >> >> On Fri, Oct 23, 2020 at 5:24 AM Russell Spitzer < >> russell.spit...@gmail.com> wrote: >> >>> I was convinced that we should probably just fail, but if that is too >>> much of a change, then logging the exception is also acceptable. >>> >>> On Thu, Oct 22, 2020, 10:32 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> >>>> Hi devs, >>>> >>>> I got another report regarding configuring v2 session catalog, when >>>> Spark fails to instantiate the configured catalog. For now, it just simply >>>> logs error message without exception information, and silently uses the >>>> default session catalog. >>>> >>>> >>>> https://github.com/apache/spark/blob/3819d39607392aa968595e3d97b84fedf83d08d9/sql/catalyst/src/main/scala/org/apache/spark/sql/connector/catalog/CatalogManager.scala#L75-L95 >>>> >>>> IMO, as the user intentionally provides the session catalog, it >>>> shouldn't fail back and just throw the exception. Otherwise (if we still >>>> want to do the failback), we need to add the exception information in the >>>> error log message at least. >>>> >>>> Would like to hear the voices. >>>> >>>> Thanks, >>>> Jungtaek Lim (HeartSaVioR) >>>> >>> >> >> -- >> Ryan Blue >> Software Engineer >> Netflix >> >
