On Fri, Feb 12, 2021 at 2:44 AM Michael Goulish <[email protected]> wrote:
> > > *Can you explain how you are measuring AMQP throughput? What message > sizes are you using? Credit windows? How many senders and receivers? Max > frame* > * size?* > > Oops! Good point. Describe the Test! > > 100 senders, 100 receivers, 100 unique addresses -- each sender sends to > one receiver. > Each sender is throttled to 100 messages per second (Apparently I Really > Like the number 100). > And message size is .... wait for it ... 100. (payload size .. so > really 139 or something like that.) > Though I'm not sure exactly what's causing the strange things you are seeing, this is not a good test to evaluate the effect of the larger buffer size. Since the message sizes are so small, they will be using the same number of buffers in both size cases (512 and 4096). The 100 byte messages fit into both buffer sizes. The router will not place multiple messages into the same buffer. So, with 512 byte buffers, this test leaves ~400 bytes unused per buffer. With 4096 byte buffers, it leaves ~4000 bytes unused per buffer. You are allocating a lot more buffer space for no benefit. A better test would involve much larger messages, maybe 64K, 128K, or more. > > Credit window is 1000. > > I can't find anything in my router config nor in my C client code about > max frame size. What do I get by default? Or, how can I check that? > > The way I measured throughput was that -- first -- I noticed that when I > made the test go longer, i.e. send 20 million total messages instead of the > original 1 million -- it was taking much longer than I expected. So I had > each receiver log a message every time its total received messages was > divisible by 1000. > > What I saw was that the first thousand came after 11 seconds (just about > as expected because of sender-throttle to 100/sec) but that later thousands > became slower. By the time I stopped the test -- after more than 50,000 > messages per receiver -- each thousand was taking ... well ... look at this > very interesting graph that I made of one receiver's behavior. > > This graph is made by just noting the time when you receive each > thousandth message (time since test started) and graphing that -- so we > expect to see an upward-sloping straight line whose slope is determined by > how long it takes to receive each 1000 messages (should be close to 10 > seconds). > > [image: messages_vs_time.jpg] > > I'm glad I graphed this! This inflection point was a total shock to me. > NOTE TO SELF: always graph everything from now on forever. > > I guess Something Interesting happened at about 28 seconds! > > Maybe what I need ... is a reading from "qdstat -m" just before and after > that inflection point !?!?? > > > > On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <[email protected]> wrote: > >> On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <[email protected]> >> wrote: >> >> > OK, so in the file Dispatch Router file src/buffer.c I changed this: >> > size_t BUFFER_SIZE = 512; >> > to this: >> > size_t BUFFER_SIZE = 4096; >> > >> > Gordon tells me that's like 8 times bigger. >> > >> > >> > It makes a terrific difference in throughput in the TCP adapter, and if >> you >> > limit the sender to the throughput that the receiver can accept, it can >> go >> > Real Fast with no memory bloat. ( Like 15 Gbit/sec ) >> > >> > But. >> > AMQP throughput is Not Happy with this change. >> > >> > Some of the managed fields grow rapidly (although not enough to account >> for >> > total memory growth) -- and throughput gradually drops to a crawl. >> > >> > Here are the fields that increase dramatically (like 10x or more) -- and >> > the ones that don't much change. >> > >> > qd_bitmask_t >> > *qd_buffer_t * >> > qd_composed_field_t >> > qd_composite_t >> > qd_connection_t >> > qd_hash_handle_t >> > qd_hash_item_t >> > qd_iterator_t >> > *qd_link_ref_t* >> > qd_link_t >> > qd_listener_t >> > qd_log_entry_t >> > qd_management_context_t >> > *qd_message_content_t* >> > *qd_message_t* >> > qd_node_t >> > qd_parse_node_t >> > qd_parse_tree_t >> > qd_parsed_field_t >> > qd_session_t >> > qd_timer_t >> > *qdr_action_t* >> > qdr_address_config_t >> > qdr_address_t >> > qdr_connection_info_t >> > qdr_connection_t >> > qdr_connection_work_t >> > qdr_core_timer_t >> > qdr_delivery_cleanup_t >> > *qdr_delivery_ref_t* >> > *qdr_delivery_t* >> > qdr_field_t >> > qdr_general_work_t >> > qdr_link_ref_t >> > qdr_link_t >> > qdr_link_work_t >> > qdr_query_t >> > qdr_terminus_t >> > >> > >> > Does anyone have a great idea about any experiment I could do, >> > instrumentation I could add, whatever -- that might help to further >> > diagnose what is going on? >> > >> >> Can you explain how you are measuring AMQP throughput? What message sizes >> are you using? Credit windows? How many senders and receivers? Max >> frame >> size? >> >
