Hey there,
Creating a new SparkContext on workers will not work, only the driver is
allowed to own a SparkContext. Are you trying to distribute your model to
workers so you can create a distributed scoring service? If so, it may be
worth looking into taking your models outside of a SparkContext an
gt;>>>
>>>>>
>>>>> On Sat, Feb 4, 2017 at 10:24 PM, Debasish Das <
>>>>> debasish.da...@gmail.com> wrote:
>>>>>
>>>>>> I am not sure why I will use pipeline to do scoring...idea is to
>>>>>>
ks,
> Asher Krim
> Senior Software Engineer
>
> On Fri, Feb 3, 2017 at 11:53 AM, Hollin Wilkins wrote:
>
>> Hey Aseem,
>>
>> We have built pipelines that execute several string indexers, one hot
>> encoders, scaling, and a random forest or linear regress
mllib library?
>>
>> On Thu, Feb 2, 2017 at 10:12 PM, Hollin Wilkins
>> wrote:
>>
>>> Hey everyone,
>>>
>>>
>>> Some of you may have seen Mikhail and I talk at Spark/Hadoop Summits
>>> about MLeap and how you can use it to build
Hey everyone,
Some of you may have seen Mikhail and I talk at Spark/Hadoop Summits about
MLeap and how you can use it to build production services from your
Spark-trained ML pipelines. MLeap is an open-source technology that allows
Data Scientists and Engineers to deploy Spark-trained ML Pipeline
Hey Aseem,
If you are looking for a full-featured library to execute Spark ML
pipelines outside of Spark, take a look at MLeap:
https://github.com/combust/mleap
Not only does it support transforming single instances of a feature vector,
but you can execute your entire ML pipeline including featur
Hey,
You could also take a look at MLeap, which provides a runtime for any Spark
transformer and does not have any dependencies on a SparkContext or Spark
libraries (excepting MLlib-local for linear algebra).
https://github.com/combust/mleap
On Tue, Jan 31, 2017 at 2:33 AM, Aseem Bansal wrote: