Re: Query about Spark

2020-09-07 Thread Ankur Das
Thanks a lot. On Mon, Sep 7, 2020 at 8:36 PM ☼ R Nair wrote: > Please read this as well, thanks > > Disclaimer: it's my article. > > > https://medium.com/@ravishankar.nair/online-and-batch-based-ml-execution-from-same-python-code-preserving-pre-and-post-transformation-ea7ebc27f50f?sk=c33bcf1d6c2

Re: Query about Spark

2020-09-07 Thread ☼ R Nair
Please read this as well, thanks Disclaimer: it's my article. https://medium.com/@ravishankar.nair/online-and-batch-based-ml-execution-from-same-python-code-preserving-pre-and-post-transformation-ea7ebc27f50f?sk=c33bcf1d6c28b562b7bd36fa39809294 Best, Ravion On Mon, Sep 7, 2020, 8:29 AM Enrico M

Re: Query about Spark

2020-09-07 Thread Enrico Minack
You could use Horovod to distribute your ML algorithm on a cluster, while Horovod also supports Spark clusters. Enrico Am 06.09.20 um 15:30 schrieb Ankur Das: Good Evening Sir/Madam, Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed e

Re: Query about Spark

2020-09-06 Thread Ankur Das
Thanks, I'll check it out. On Sun, Sep 6, 2020 at 7:15 PM ☼ R Nair wrote: > Or use MLFlow's PySpark UDF. First create a mlflow.pyfunc. > > Best, Ravion > > On Sun, Sep 6, 2020, 9:43 AM ☼ R Nair wrote: > >> Question is not clear..use accumulators, if I took it correctly. >> >> Best, Ravion >> >>

Re: Query about Spark

2020-09-06 Thread ☼ R Nair
Or use MLFlow's PySpark UDF. First create a mlflow.pyfunc. Best, Ravion On Sun, Sep 6, 2020, 9:43 AM ☼ R Nair wrote: > Question is not clear..use accumulators, if I took it correctly. > > Best, Ravion > > On Sun, Sep 6, 2020, 9:41 AM Ankur Das wrote: > >> >> Good Evening Sir/Madam, >> Hope you

Re: Query about Spark

2020-09-06 Thread ☼ R Nair
Question is not clear..use accumulators, if I took it correctly. Best, Ravion On Sun, Sep 6, 2020, 9:41 AM Ankur Das wrote: > > Good Evening Sir/Madam, > Hope you are doing well, I am experimenting on some ML techniques where I > need to test it on a distributed environment. > For example a par

Query about Spark

2020-09-06 Thread Ankur Das
Good Evening Sir/Madam, Hope you are doing well, I am experimenting on some ML techniques where I need to test it on a distributed environment. For example a particular algorithm I want to run it on different nodes at the same time and collect the results at the end in one single node or the parent