Thank you for your reply.
I also upload a jar fast in the 1.X cluster.
In the 1.X cluster is not a problem.
However 2.X cluster is too slow to upload a jar.
I do not understand why the 2.X cluster is slower than the 1.X cluster.
2023년 2월 9일 (목) 오전 4:14, Rui Abreu <rui.ab...@gmail.com>님이 작성:
> I can upload topologies bigger than 90 MB in around 30 seconds in a Storm
> 1.X cluster.
> I'm using a large value for *nimbus.thrift.max_buffer_size* and I have
> also increased *nimbus.thrift.threads. *Don't know if this helps the
> upload process or how slow your upload is.
>
> On Tue, 7 Feb 2023 at 00:35, Minwoo Kang <its.minwoo.k...@gmail.com>
> wrote:
>
>> Thank you for your reply.
>>
>> It is about 60 MB.
>> The size is not huge. But the upload speed is too slow.
>>
>> No, I did not change "nimbus.thrift.max_buffer_size".
>> The "getMaxBufferSize" method uses a "nimbus.thrift.max_buffer_size".
>> So I thought the "getMaxBufferSize" method only impacts a size check in
>> "stringLengthLimit_" and "maxLength" in the thrift frame.
>>
>> There are two upload processes when submitting a topology.
>> first: client -> nimbus
>> second: nimbus -> supervisor
>>
>> The second(nimbus -> supervisor) is also slow. However, the upload speed
>> is faster after I increase "storm.blobstore.inputstream.buffer.size.bytes"
>> to 4MB.
>>
>> I changed the code to increase the "THRIFT_CHUNK_SIZE_BYTES" value.
>> client upload speed is faster after I increase "THRIFT_CHUNK_SIZE_BYTES"
>> to 4MB.
>>
>> So I thought maybe the client should change the "THRIFT_CHUNK_SIZE_BYTES"
>> like nimbus ("storm.blobstore.inputstream.buffer.size.bytes" value).
>>
>> Thanks.
>>
>> 2023년 2월 6일 (월) 오후 9:51, Rui Abreu <rui.ab...@gmail.com>님이 작성:
>>
>>> How big the uber jar that you are submitting?
>>> Have you tried tweaking the Nimbus side property?
>>> https://github.com/apache/storm/blob/master/conf/defaults.yaml#L67
>>>
>>> On Mon, 6 Feb 2023 at 04:14, Minwoo Kang <its.minwoo.k...@gmail.com>
>>> wrote:
>>>
>>>> Hello, Users.
>>>>
>>>> When I submit storm topology, Occasionally it takes a long time.
>>>> So I investigated that, and I found THRIFT_CHUNK_SIZE_BYTES[1] is too
>>>> small.
>>>>
>>>> However, That value can not be modified because it is the final value.
>>>> It can be changed (STORM-241[2]) for the first time. But 70174d[3]
>>>> commit changed chunk size to private constant.
>>>>
>>>> I wonder why the user cannot change THRIFT_CHUNK_SIZE_BYTES.
>>>>
>>>> Thanks.
>>>>
>>>> [1]:
>>>> https://github.com/apache/storm/blob/bf29d1cc9914d4fe596b5e65532322e3dfd3e4ff/storm-client/src/jvm/org/apache/storm/StormSubmitter.java#L62
>>>> [2]: https://issues.apache.org/jira/browse/STORM-241
>>>> [3]:
>>>> https://github.com/apache/storm/commit/70174deaa758fc1e3f3a881445e8750ae7218b3d
>>>>
>>>
