Another thing to consider when increasing the buffer size is how to adjust the "Q2" limit. This limits the number of buffers that will be stored in a streaming message at any one time. By increasing the buffer size by 8x, the Q2 limit in bytes is also increased 8x.
This won't have any effect on your small-message test, but it will affect the router's memory consumption with the transfer of large or streaming messages. -Ted On Fri, Feb 12, 2021 at 11:56 AM Ted Ross <tr...@apache.org> wrote: > > > On Fri, Feb 12, 2021 at 2:44 AM Michael Goulish <mgoul...@redhat.com> > wrote: > >> >> >> *Can you explain how you are measuring AMQP throughput? What message >> sizes are you using? Credit windows? How many senders and receivers? Max >> frame* >> * size?* >> >> Oops! Good point. Describe the Test! >> >> 100 senders, 100 receivers, 100 unique addresses -- each sender sends to >> one receiver. >> Each sender is throttled to 100 messages per second (Apparently I Really >> Like the number 100). >> And message size is .... wait for it ... 100. (payload size .. so >> really 139 or something like that.) >> > > Though I'm not sure exactly what's causing the strange things you are > seeing, this is not a good test to evaluate the effect of the larger buffer > size. > > Since the message sizes are so small, they will be using the same number > of buffers in both size cases (512 and 4096). The 100 byte messages fit > into both buffer sizes. The router will not place multiple messages into > the same buffer. So, with 512 byte buffers, this test leaves ~400 bytes > unused per buffer. With 4096 byte buffers, it leaves ~4000 bytes unused > per buffer. You are allocating a lot more buffer space for no benefit. > > A better test would involve much larger messages, maybe 64K, 128K, or more. > > >> >> Credit window is 1000. >> >> I can't find anything in my router config nor in my C client code about >> max frame size. What do I get by default? Or, how can I check that? >> >> The way I measured throughput was that -- first -- I noticed that when I >> made the test go longer, i.e. send 20 million total messages instead of the >> original 1 million -- it was taking much longer than I expected. So I had >> each receiver log a message every time its total received messages was >> divisible by 1000. >> >> What I saw was that the first thousand came after 11 seconds (just about >> as expected because of sender-throttle to 100/sec) but that later thousands >> became slower. By the time I stopped the test -- after more than 50,000 >> messages per receiver -- each thousand was taking ... well ... look at this >> very interesting graph that I made of one receiver's behavior. >> >> This graph is made by just noting the time when you receive each >> thousandth message (time since test started) and graphing that -- so we >> expect to see an upward-sloping straight line whose slope is determined by >> how long it takes to receive each 1000 messages (should be close to 10 >> seconds). >> >> [image: messages_vs_time.jpg] >> >> I'm glad I graphed this! This inflection point was a total shock to me. >> NOTE TO SELF: always graph everything from now on forever. >> >> I guess Something Interesting happened at about 28 seconds! >> >> Maybe what I need ... is a reading from "qdstat -m" just before and after >> that inflection point !?!?? >> >> >> >> On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <tr...@apache.org> wrote: >> >>> On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <mgoul...@redhat.com> >>> wrote: >>> >>> > OK, so in the file Dispatch Router file src/buffer.c I changed this: >>> > size_t BUFFER_SIZE = 512; >>> > to this: >>> > size_t BUFFER_SIZE = 4096; >>> > >>> > Gordon tells me that's like 8 times bigger. >>> > >>> > >>> > It makes a terrific difference in throughput in the TCP adapter, and >>> if you >>> > limit the sender to the throughput that the receiver can accept, it >>> can go >>> > Real Fast with no memory bloat. ( Like 15 Gbit/sec ) >>> > >>> > But. >>> > AMQP throughput is Not Happy with this change. >>> > >>> > Some of the managed fields grow rapidly (although not enough to >>> account for >>> > total memory growth) -- and throughput gradually drops to a crawl. >>> > >>> > Here are the fields that increase dramatically (like 10x or more) -- >>> and >>> > the ones that don't much change. >>> > >>> > qd_bitmask_t >>> > *qd_buffer_t * >>> > qd_composed_field_t >>> > qd_composite_t >>> > qd_connection_t >>> > qd_hash_handle_t >>> > qd_hash_item_t >>> > qd_iterator_t >>> > *qd_link_ref_t* >>> > qd_link_t >>> > qd_listener_t >>> > qd_log_entry_t >>> > qd_management_context_t >>> > *qd_message_content_t* >>> > *qd_message_t* >>> > qd_node_t >>> > qd_parse_node_t >>> > qd_parse_tree_t >>> > qd_parsed_field_t >>> > qd_session_t >>> > qd_timer_t >>> > *qdr_action_t* >>> > qdr_address_config_t >>> > qdr_address_t >>> > qdr_connection_info_t >>> > qdr_connection_t >>> > qdr_connection_work_t >>> > qdr_core_timer_t >>> > qdr_delivery_cleanup_t >>> > *qdr_delivery_ref_t* >>> > *qdr_delivery_t* >>> > qdr_field_t >>> > qdr_general_work_t >>> > qdr_link_ref_t >>> > qdr_link_t >>> > qdr_link_work_t >>> > qdr_query_t >>> > qdr_terminus_t >>> > >>> > >>> > Does anyone have a great idea about any experiment I could do, >>> > instrumentation I could add, whatever -- that might help to further >>> > diagnose what is going on? >>> > >>> >>> Can you explain how you are measuring AMQP throughput? What message >>> sizes >>> are you using? Credit windows? How many senders and receivers? Max >>> frame >>> size? >>> >>
