*Can you explain how you are measuring AMQP throughput? What message sizes are you using? Credit windows? How many senders and receivers? Max frame* * size?*
Oops! Good point. Describe the Test! 100 senders, 100 receivers, 100 unique addresses -- each sender sends to one receiver. Each sender is throttled to 100 messages per second (Apparently I Really Like the number 100). And message size is .... wait for it ... 100. (payload size .. so really 139 or something like that.) Credit window is 1000. I can't find anything in my router config nor in my C client code about max frame size. What do I get by default? Or, how can I check that? The way I measured throughput was that -- first -- I noticed that when I made the test go longer, i.e. send 20 million total messages instead of the original 1 million -- it was taking much longer than I expected. So I had each receiver log a message every time its total received messages was divisible by 1000. What I saw was that the first thousand came after 11 seconds (just about as expected because of sender-throttle to 100/sec) but that later thousands became slower. By the time I stopped the test -- after more than 50,000 messages per receiver -- each thousand was taking ... well ... look at this very interesting graph that I made of one receiver's behavior. This graph is made by just noting the time when you receive each thousandth message (time since test started) and graphing that -- so we expect to see an upward-sloping straight line whose slope is determined by how long it takes to receive each 1000 messages (should be close to 10 seconds). [image: messages_vs_time.jpg] I'm glad I graphed this! This inflection point was a total shock to me. NOTE TO SELF: always graph everything from now on forever. I guess Something Interesting happened at about 28 seconds! Maybe what I need ... is a reading from "qdstat -m" just before and after that inflection point !?!?? On Thu, Feb 11, 2021 at 5:37 PM Ted Ross <[email protected]> wrote: > On Thu, Feb 11, 2021 at 2:08 PM Michael Goulish <[email protected]> > wrote: > > > OK, so in the file Dispatch Router file src/buffer.c I changed this: > > size_t BUFFER_SIZE = 512; > > to this: > > size_t BUFFER_SIZE = 4096; > > > > Gordon tells me that's like 8 times bigger. > > > > > > It makes a terrific difference in throughput in the TCP adapter, and if > you > > limit the sender to the throughput that the receiver can accept, it can > go > > Real Fast with no memory bloat. ( Like 15 Gbit/sec ) > > > > But. > > AMQP throughput is Not Happy with this change. > > > > Some of the managed fields grow rapidly (although not enough to account > for > > total memory growth) -- and throughput gradually drops to a crawl. > > > > Here are the fields that increase dramatically (like 10x or more) -- and > > the ones that don't much change. > > > > qd_bitmask_t > > *qd_buffer_t * > > qd_composed_field_t > > qd_composite_t > > qd_connection_t > > qd_hash_handle_t > > qd_hash_item_t > > qd_iterator_t > > *qd_link_ref_t* > > qd_link_t > > qd_listener_t > > qd_log_entry_t > > qd_management_context_t > > *qd_message_content_t* > > *qd_message_t* > > qd_node_t > > qd_parse_node_t > > qd_parse_tree_t > > qd_parsed_field_t > > qd_session_t > > qd_timer_t > > *qdr_action_t* > > qdr_address_config_t > > qdr_address_t > > qdr_connection_info_t > > qdr_connection_t > > qdr_connection_work_t > > qdr_core_timer_t > > qdr_delivery_cleanup_t > > *qdr_delivery_ref_t* > > *qdr_delivery_t* > > qdr_field_t > > qdr_general_work_t > > qdr_link_ref_t > > qdr_link_t > > qdr_link_work_t > > qdr_query_t > > qdr_terminus_t > > > > > > Does anyone have a great idea about any experiment I could do, > > instrumentation I could add, whatever -- that might help to further > > diagnose what is going on? > > > > Can you explain how you are measuring AMQP throughput? What message sizes > are you using? Credit windows? How many senders and receivers? Max frame > size? >
