I can upload topologies bigger than 90 MB in around 30 seconds in a Storm
1.X cluster.
I'm using a large value for *nimbus.thrift.max_buffer_size* and I have also
increased *nimbus.thrift.threads. *Don't know if this helps the upload
process or how slow your upload is.
On Tue, 7 Feb 2023 at 00:35, Minwoo Kang <its.minwoo.k...@gmail.com> wrote:
> Thank you for your reply.
>
> It is about 60 MB.
> The size is not huge. But the upload speed is too slow.
>
> No, I did not change "nimbus.thrift.max_buffer_size".
> The "getMaxBufferSize" method uses a "nimbus.thrift.max_buffer_size".
> So I thought the "getMaxBufferSize" method only impacts a size check in
> "stringLengthLimit_" and "maxLength" in the thrift frame.
>
> There are two upload processes when submitting a topology.
> first: client -> nimbus
> second: nimbus -> supervisor
>
> The second(nimbus -> supervisor) is also slow. However, the upload speed
> is faster after I increase "storm.blobstore.inputstream.buffer.size.bytes"
> to 4MB.
>
> I changed the code to increase the "THRIFT_CHUNK_SIZE_BYTES" value.
> client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES"
> to 4MB.
>
> So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES"
> like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value).
>
> Thanks.
>
> 2023년 2월 6일 (월) 오후 9:51, Rui Abreu <rui.ab...@gmail.com>님이 작성:
>
>> How big the uber jar that you are submitting?
>> Have you tried tweaking the Nimbus side property?
>> https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67
>>
>> On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <its.minwoo.k...@gmail.com>
>> wrote:
>>
>>> Hello, Users.
>>>
>>> When I submit storm topology, Occasionally it takes a long time.
>>> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
>>> small.
>>>
>>> However, That value can not be modified because it is the final value.
>>> It can be changed (STORM-241[2]) for the first time. But 70174d[3]
>>> commit changed chunk size to private constant.
>>>
>>> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>>>
>>> Thanks.
>>>
>>> [1]:
>>> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
>>> [2]: https://issues.apache.org/jira/browse/STORM-241
>>> [3]:
>>> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>>>
>>
