Hello, thanks for the advice, I think the behavior you describe explains why Trident becomes slow when using multiple workers. This is a pity because as the documentation states you do not have direct influence on how the batch size is set by Trident. However, to me it still remains unclear why it becomes THAT slow (1/10 of the speed with a single worker setup). I should probably add the all servers of the cluster we run are located in the same place and have a ping of 1ms or less(!) between them.
I was thinking of using Kafka/Redis or a similar pub/sub system, however, to me it seemed intriguing if I could just use the internal DRPC server without implementing another new technology. I have not given up completely yet: experimenting more I found that when I increase the parallelism IN TOPOLOGIES DOING THE DRPC EXECUTES the throughput increases up to a level where we at least start to see the available capacity in the bolts being exhausted (that is in the single worker setup). I still would appreciate any pointers to a good documentation on how the DRPC <--> Trident relationship works in the background. I will try to find the right place in the source code, a pointer to the correct file would be helpful as well. Also I would like to know why multiple workers with increased parallelism actually build up congestion rather than increasing throughput, this seems to be a separate issue. Thanks Jonas On Jul 28, 2017, at 8:18 PM, kalliopi kalantzaki <[email protected]> wrote: > Hi, > > Check whether the whole topology is slow due to processing burden... The > problem with DRPC is that if one thread/call is slow (e.g tuple fails or > passes through a slow bolt etc) then the following calls will get slow as > well--I think because the system waits until the current call finishes... I > am not sure about the last one, check it if you want. > > The most common solution i have seen in such cases, is to put in front of > storm a kafka system where the incoming call writes to a topic, storm then > reads and processes the incoming call and then writes back the result to > another topic via a kafkabolt. From the output topic then you can implement a > producer to take the results.... > This solution definitely works but again maybe this does not fit your use > cases.... > > Hope this helps. > > > > > Στις 5:17 μ.μ. Παρασκευή, 28 Ιουλίου 2017, ο/η Jonas Krauß > <[email protected]> έγραψε: > > > Hi all, > > I am trying to understand how storm's DRPC server and Trident work together. > A particular Trident topology I am developing seems to be limited in its > throughput as batches/tuples are not pushed into it fast enough from the DRPC > server. We are using storm 1.1.0. > > Unfortunately I have not found a good source of information on how the DRPC > server and Trident function together, though I have looked at the various > Trident examples and docs at > https://storm.apache.org/releases/1.1.0/index.html as well as Stackoverflow > and the old storm Google Group. > > What I have tried so far: > - lowering the trident batch emit interval millis from 500 to 200, 100 and > 25, does not seem to have an effect > - using multiple workers in the Trident topology along with different kinds > of partitioning the stream - seems to hurt performance (I suspect network > latency between our storm servers to be the reason for that) > - using a single or multiple spouts, seems to have no effect > > Our drpc is configured as follows: > > drpc.authorizer.acl.filename "drpc-auth-acl.yaml" > drpc.authorizer.acl.strict false > drpc.childopts "-Xmx2048m" > drpc.http.creds.plugin > "org.apache.storm.security.auth.DefaultHttpCredentialsPlugin" > drpc.http.port 3774 > drpc.https.keystore.password "" > drpc.https.keystore.type "JKS" > drpc.https.port -1 > drpc.invocations.port 3773 > drpc.invocations.threads 64 > drpc.max_buffer_size 10485760 > drpc.port 3772 > drpc.queue.size 128 > drpc.request.timeout.secs 600 > drpc.worker.threads 64 > > I appreciate any pointers to a good documentation or concrete suggestions. > Sorry if this question has been asked before. > > Thanks > > Jonas > >
smime.p7s
Description: S/MIME cryptographic signature
