It would be a @dev internal api I think
If we wanted to go extremely general with post session init, it could be
added to SparkExtensions
def postSessionInit(session: SparkSession) : Unit
Which would allow you to do just about anything after sessionState was done
initialized.
Or if we specifically wanted to allow just functions
def injectFunction(name: String, function: Seq[Expression] => [Expression])
{
sparkSession.registerFunction(name, function) // Or add to a buffer which
is registered later
}
On Thu, Sep 27, 2018 at 1:16 PM Reynold Xin <r...@databricks.com> wrote:
> Thoughts on how the api would look like?
>
> On Thu, Sep 27, 2018 at 11:13 AM Russell Spitzer <
> russell.spit...@gmail.com> wrote:
>
>> While that's easy for some users, we basically want to load up some
>> functions by default into all session catalogues regardless of who made
>> them. We do this with certain rules and strategies using the
>> SparkExtensions, so all apps that run through our submit scripts get a
>> config parameter added and it's transparent to the user. I think we'll
>> probably have to do some forks (at least for the CliDriver), the
>> thriftserver has a bunch of code which doesn't run under "startWithContext"
>> so we may have an issue there as well.
>>
>>
>>
>> On Wed, Sep 26, 2018, 6:21 PM Mark Hamstra <m...@clearstorydata.com>
>> wrote:
>>
>>> You're talking about users starting Thriftserver or SqlShell from the
>>> command line, right? It's much easier if you are starting a Thriftserver
>>> programmatically so that you can register functions when initializing a
>>> SparkContext and then HiveThriftServer2.startWithContext using that
>>> context.
>>>
>>> On Wed, Sep 26, 2018 at 3:30 PM Russell Spitzer <
>>> russell.spit...@gmail.com> wrote:
>>>
>>>> I've been looking recently on possible avenues to load new functions
>>>> into the Thriftserver and SqlShell at launch time. I basically want to
>>>> preload a set of functions in addition to those already present in the
>>>> Spark Code. I'm not sure there is at present a way to do this and I was
>>>> wondering if anyone had any ideas.
>>>>
>>>> I would basically want to make it so that any user launching either of
>>>> these tools would automatically have access to some custom functions. In
>>>> the SparkShell I can do this by adding additional lines to the init section
>>>> but I think It would be nice if we could pass in a parameter which would
>>>> point to a class with a list of additional functions to add to all new
>>>> session states.
>>>>
>>>> An interface like Spark Sessions Extensions but instead of running
>>>> during Session Init, it would run after session init has completed.
>>>>
>>>> Thanks for your time and I would be glad to hear any opinions or ideas
>>>> on this,
>>>>
>>> --
> --
> excuse the brevity and lower case due to wrist injury
>
