Hi,
I'm a long-time listener, first-time committer to spark, so this is
good to get my feet wet. I'm particularly interested in SPARK-23836,
which is an itch I may want to dive into and scratch myself in the
next month or so since it's pretty painful for our use-case.
Thanks!
Andrew
On Mon, Aug
Sure, I don't think you should wait on that being merged in. If you want to
take the JIRA go ahead (although if you're already familiar with the Spark
code base it might make sense to leave it as a starter issue for someone
who is just getting started).
On Mon, Aug 27, 2018 at 12:18 PM Andrew
Hi Holden,
I'm agnostic to the approach (though it seems cleaner to have an
explicit API for it). If you would like, I can take that JIRA and
implement it (should be a 3-line function).
Cheers
Andrew
On Mon, Aug 27, 2018 at 2:14 PM, Holden Karau wrote:
> Seems reasonable. We should probably
Seems reasonable. We should probably add `getActiveSession` to the PySpark
API (filed a starter JIRA https://issues.apache.org/jira/browse/SPARK-25255
)
On Mon, Aug 27, 2018 at 12:09 PM Andrew Melo wrote:
> Hello Sean, others -
>
> Just to confirm, is it OK for client applications to access
>
Hello Sean, others -
Just to confirm, is it OK for client applications to access
SparkContext._active_spark_context, if it wraps the accesses in `with
SparkContext._lock:`?
If that's acceptable to Spark, I'll implement the modifications in the
Jupyter extensions.
thanks!
Andrew
On Tue, Aug 7,
Hi Sean,
On Tue, Aug 7, 2018 at 5:44 PM, Sean Owen wrote:
> Ah, python. How about SparkContext._active_spark_context then?
Ah yes, that looks like the right member, but I'm a bit wary about
depending on functionality of objects with leading underscores. I
assumed that was "private" and subject
Ah, python. How about SparkContext._active_spark_context then?
On Tue, Aug 7, 2018 at 5:34 PM Andrew Melo wrote:
> Hi Sean,
>
> On Tue, Aug 7, 2018 at 5:16 PM, Sean Owen wrote:
> > Is SparkSession.getActiveSession what you're looking for?
>
> Perhaps -- though there's not a corresponding
Hi Sean,
On Tue, Aug 7, 2018 at 5:16 PM, Sean Owen wrote:
> Is SparkSession.getActiveSession what you're looking for?
Perhaps -- though there's not a corresponding python function, and I'm
not exactly sure how to call the scala getActiveSession without first
instantiating the python version and
Is SparkSession.getActiveSession what you're looking for?
On Tue, Aug 7, 2018 at 5:11 PM Andrew Melo wrote:
> Hello,
>
> One pain point with various Jupyter extensions [1][2] that provide
> visual feedback about running spark processes is the lack of a public
> API to introspect the web URL.
Hello,
One pain point with various Jupyter extensions [1][2] that provide
visual feedback about running spark processes is the lack of a public
API to introspect the web URL. The notebook server needs to know the
URL to find information about the current SparkContext.
Simply looking for
10 matches
Mail list logo
