I am trying to download a large amount of files from an smb:// endpoint and
I would like the file consumption to be as fast as possible - meaning I
would like to download and process as many files as possible, for example
polling for 2000 files instead of whatever the default polling value in
Camel is. To achieve this, I was looking for a way to increase the incoming
batch size or add multi-threaded/parallel consumption on a single folder.
So far I have tried the following, but I noticed the download rate stayed
the same as with one thread:
from("smb:\\someFolder?&maxMessagesPerPoll=1000")...
from("smb:\\someFolder?&maxMessagesPerPoll=-1")...
from("smb:\\someFolder").threads(20)...
Even with the threads option, despite seeing that the thread IDs vary, the
download rate was the same. I suppose Camel simply assings a thread pool so
that a different thread will pick up the incoming file each time, but the
polling amount is the same.
I also looked up the concurrentConsumers flag, but from what I gather it is
not supported for files, rather for ActiveMQ.
The only efficiency fix I managed to apply was consuming from multiple
smb:// endpoints. The throughput rate increased rapidly when I did that. I
am wondering if there is a way to achieve the same result as with multiple
folders but using a single folder only?
