No, you can't use the SparkSession from within a function executed by Spark tasks.
On Wed, Sep 30, 2020 at 7:29 AM Lakshmi Nivedita <klnived...@gmail.com> wrote: > Here is a spark udf structure as an example > > Def sampl_fn(x): > Spark.sql(“select count(Id) from sample Where Id = x ”) > > > Spark.udf.register(“sample_fn”, sample_fn) > > Spark.sql(“select id, sampl_fn(Id) from example”) > > Advance Thanks for the help > -- > k.Lakshmi Nivedita > >
