You can call `SparkContext#addSparkListener` with a listener that
implements `onApplicationEnd`.
On Tue, May 14, 2019 at 1:51 AM Nasrulla Khan Haris
wrote:
> HI All,
>
>
>
> Is there a API for sparkContext where we can add our custom code before
> stopping sparkcontext ?
&
HI All,
Is there a API for sparkContext where we can add our custom code before
stopping sparkcontext ?
Appreciate your help.
Thanks,
Nasrulla
Didn't notice that I can pass comma separated path in the existing API
(SparkContext#textFile). So no necessary for new api. Thanks all.
On Thu, Nov 12, 2015 at 10:24 AM, Jeff Zhang <zjf...@gmail.com> wrote:
> Hi Pradeep
>
> ≥≥≥ Looks like what I was suggesting doesn't work. :
ob
>
> On 11 November 2015 at 01:20, Jeff Zhang <zjf...@gmail.com> wrote:
>
>> Although user can use the hdfs glob syntax to support multiple inputs.
>> But sometimes, it is not convenient to do that. Not sure why there's no api
>> of SparkContext#textFiles. It should be easy to
ards,
--Jakob
On 11 November 2015 at 01:20, Jeff Zhang <zjf...@gmail.com> wrote:
> Although user can use the hdfs glob syntax to support multiple inputs. But
> sometimes, it is not convenient to do that. Not sure why there's no api
> of SparkContext#textFiles. It should be easy to i
Although user can use the hdfs glob syntax to support multiple inputs. But
sometimes, it is not convenient to do that. Not sure why there's no api
of SparkContext#textFiles. It should be easy to implement that. I'd love to
create a ticket and contribute for that if there's no other consideration
Hi Pradeep
≥≥≥ Looks like what I was suggesting doesn't work. :/
I guess you mean put comma separated path into one string and pass it
to existing API (SparkContext#textFile). It should not work. I suggest to
create new api SparkContext#textFiles to accept an array of string. I have
already
>> On 11 November 2015 at 01:20, Jeff Zhang <zjf...@gmail.com> wrote:
>>
>>> Although user can use the hdfs glob syntax to support multiple inputs.
>>> But sometimes, it is not convenient to do that. Not sure why there's no api
>>> of SparkContext#text
> --Jakob
>
> On 11 November 2015 at 01:20, Jeff Zhang <zjf...@gmail.com> wrote:
>
>> Although user can use the hdfs glob syntax to support multiple inputs.
>> But sometimes, it is not convenient to do that. Not sure why there's no api
>> of SparkContex
t;>> val rdd = lines1 union lines2
>>>>
>>>> regards,
>>>> --Jakob
>>>>
>>>> On 11 November 2015 at 01:20, Jeff Zhang <zjf...@gmail.com> wrote:
>>>>
>>>>> Although user can use the hdfs glob sy
10 matches
Mail list logo
