Hey there,
Creating a new SparkContext on workers will not work, only the driver is
allowed to own a SparkContext. Are you trying to distribute your model to
workers so you can create a distributed scoring service? If so, it may be
worth looking into taking your models outside of a SparkContext
>
>>>>>> I am not sure why I will use pipeline to do scoring...idea is to
>>>>>> build a model, use model ser/deser feature to put it in the row or column
>>>>>> store of choice and provide a api access to the model...we support these
>>>&
>
> Thanks,
> Asher Krim
> Senior Software Engineer
>
> On Fri, Feb 3, 2017 at 11:53 AM, Hollin Wilkins <hol...@combust.ml> wrote:
>
>> Hey Aseem,
>>
>> We have built pipelines that execute several string indexers, one hot
>> encoders, scaling, a
f few milliseconds (<
>> 10 ms) like the old mllib library?
>>
>> On Thu, Feb 2, 2017 at 10:12 PM, Hollin Wilkins <hol...@combust.ml>
>> wrote:
>>
>>> Hey everyone,
>>>
>>>
>>> Some of you may have seen Mikhail and I t
Hey everyone,
Some of you may have seen Mikhail and I talk at Spark/Hadoop Summits about
MLeap and how you can use it to build production services from your
Spark-trained ML pipelines. MLeap is an open-source technology that allows
Data Scientists and Engineers to deploy Spark-trained ML
Hey Aseem,
If you are looking for a full-featured library to execute Spark ML
pipelines outside of Spark, take a look at MLeap:
https://github.com/combust/mleap
Not only does it support transforming single instances of a feature vector,
but you can execute your entire ML pipeline including
Hey,
You could also take a look at MLeap, which provides a runtime for any Spark
transformer and does not have any dependencies on a SparkContext or Spark
libraries (excepting MLlib-local for linear algebra).
https://github.com/combust/mleap
On Tue, Jan 31, 2017 at 2:33 AM, Aseem Bansal
