I am currently using mesos as a big data backend for spark, cassandra,
kafka and elasticsearch but I cannot find a good overall design regarding
service discovery. I explain:
Generally, the service discovery is managed by a HAproxy instance on each
node which redirect trafic from service ports to r
I would say that service discovery is only for those services that don't
have a built in method for discovery. When I run Elastic Search, I specify
the port range I can start elastic search in, and let it run. If the port
is taken, it tries a different one (I am using the Elastic Search for Yarn
pa
Can you confirm what I understand ? Spark will connect to Elasticsearch
through the service port (means HApoxy) and then will get direct IP/ports
for the topology?
2015-12-30 19:06 GMT+01:00 John Omernik :
> I would say that service discovery is only for those services that don't
> have a built i
So, no, I don't have Elastic Search in HA Proxy. For each instance of
Elastic Search I have, I specify the ports to user (a range) Now, Spark
can't do service discovery of Elastic serach the way Elastic Search can, so
that could be a challenge. That said, each ES node can be connected to
directly,
What about specifying all non-local instances as "backup" in haproxy.cfg?
This way haproxy would only direct traffic to the local instance as long as
the local instance is alive.
For example, if you plan to use the haproxy-marathon-bridge script, you can
modify this line to achieve that:
https://g
Good idea to get data locality for non distributed apps but spark driver
will distribute info to workers so it may result in all workers connecting
to instance on the same node as the driver.
I will do some test...
Le 31 déc. 2015 1:26 AM, "Shuai Lin" a écrit :
> What about specifying all non-l
6 matches
Mail list logo
