Thanks,
So the way I understand it, thick client will use the affinitly key to send
data to the right node, and hence will split the traiffic between all the
nodes, the thin client will just send the data to one node, and that node
will be responsible to send it to the actual node that owns the 'shard'?
I keep getting the following error when using the Spark driver, the driver
keeps writing, but very slowly. Any idea what is causing the error, or how
to fix it?
Cheers,
Eugene
"
[15:04:58,030][SEVERE][data-streamer-stripe-10-#43%Server%][DataStreamProcessor]
Failed to respond to node [nodeId=78af5d88-cbfa-4529-aaee-ff4982985cdf,
res=DataStreamerResponse [reqId=192, forceLocDep=true]]
class org.apache.ignite.IgniteCheckedException: Failed to send message
(node may have left the grid or TCP connection cannot be established due to
firewall issues) [node=ZookeeperClusterNode
[id=78af5d88-cbfa-4529-aaee-ff4982985cdf, addrs=[127.0.0.1], order=377,
loc=false, client=true], topic=T1 [topic=TOPIC_DATASTREAM,
id=b8d675c6561-78af5d88-cbfa-4529-aaee-ff4982985cdf],
msg=DataStreamerResponse [reqId=192, forceLocDep=true], policy=9]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1651)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToCustomTopic(GridIoManager.java:1703)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToCustomTopic(GridIoManager.java:1673)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.sendResponse(DataStreamProcessor.java:440)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:402)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:305)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:60)
at
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:90)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send
message to remote node: ZookeeperClusterNode
[id=78af5d88-cbfa-4529-aaee-ff4982985cdf, addrs=[127.0.0.1], order=377,
loc=false, client=true]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2718)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2651)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1643)
... 13 more
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to
connect to node (is node still alive?). Make sure that each ComputeTask and
cache Transaction has a timeout set in order to prevent parties from
waiting forever in case of network issues
[nodeId=78af5d88-cbfa-4529-aaee-ff4982985cdf, addrs=[/127.0.0.1:47101]]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3422)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2958)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2841)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2692)
... 15 more
Suppressed: class org.apache.ignite.IgniteCheckedException: Failed
to connect to address [addr=/127.0.0.1:47101, err=Connection refused]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3425)
... 18 more
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:111)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3262)
... 18 more
"
On Tue, Aug 14, 2018 at 4:39 PM, akurbanov <antkr....@gmail.com> wrote:
> Hi,
>
> Spark integration was implemented before java thin client was released and
> thick client performs better than thin one in general. Is your question
> related to existence of benchmarks for thin vs thick clients in Spark
> integration or just a comparison of these two options?
>
> Thin clients' functionality is limited compared to thick client, also it
> generally should be a bit slower as it is communicates not with whole
> cluster, but only with a single node and is not partition-aware. This
> introduces additional network costs which may affect performance compared
> to
> thick client in the simplest and ideal conditions where network transfer is
> a major part of workload.
>
> However this performance decrease may be completely irrelevant depending on
> use case and workload, so you should always measure peformance and do
> benchmarks for a specific use case and make a decision which option suits
> your needs more.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
