Hi,
Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.
What could be the reason for showi
You could use Horovod to distribute your ML algorithm on a cluster,
while Horovod also supports Spark clusters.
Enrico
Am 06.09.20 um 15:30 schrieb Ankur Das:
Good Evening Sir/Madam,
Hope you are doing well, I am experimenting on some ML techniques
where I need to test it on a distributed e
Please read this as well, thanks
Disclaimer: it's my article.
https://medium.com/@ravishankar.nair/online-and-batch-based-ml-execution-from-same-python-code-preserving-pre-and-post-transformation-ea7ebc27f50f?sk=c33bcf1d6c28b562b7bd36fa39809294
Best, Ravion
On Mon, Sep 7, 2020, 8:29 AM Enrico M
Hi,
Using structured spark streaming and sink the data into ElasticSearch.
In the stats emit for each batch the "numOutputRows" showing -1 for
ElasticSearch sink always
whereas when i see other sinks like Kafka it shows either 0 or some values
when it emit data.
What could be the reason for showi
I don't know about ES sink. The availability of "numOutputRows" depends on
the API version the sink is implementing (DSv1 vs DSv2), so you may be
better to ask a question to the author of ES sink and confirm the case.
On Tue, Sep 8, 2020 at 5:15 AM jainshasha wrote:
> Hi,
>
> Using structured sp
Thanks Jungtaek Lim-2 for replying.
May i knw the reference of the API version for sink for both types (DSv1 and
DSv2) in code ?
Where could i see it ? Under what module of spark code ?
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
-
Thanks a lot.
On Mon, Sep 7, 2020 at 8:36 PM ☼ R Nair wrote:
> Please read this as well, thanks
>
> Disclaimer: it's my article.
>
>
> https://medium.com/@ravishankar.nair/online-and-batch-based-ml-execution-from-same-python-code-preserving-pre-and-post-transformation-ea7ebc27f50f?sk=c33bcf1d6c2
