Re: [jira] [Created] (PROTON-2189) proactor C client has abnormally long pauses during message send
Ken -- I have a proactor client in Mercury and it handles timeouts differently than your code. I don't remember now exactly why this was necessary, but I do vaguely recall weird behavior before I fixed it, at aconway's suggestion. I think that, in the PN_PROACTOR_TIMEOUT case in the event handler, in my code it was not correct to send at that time and then reset the timer. (As you are doing.) Instead, in that case all I did was to wake the cnx: pn_connection_wake ( context->connection ); You can then send, and reset the timer only when you get an event of type: PN_CONNECTION_WAKE: send_message ( context ); pn_proactor_set_timeout ( context->proactor, context->throttle ); In my case, I am sending entire messages -- but this is how I get it to 'throttle' the send-rate, i.e. send 1 message per N msec. I don't remember exactly what Bad Thing was happening to me before I started doing it this way, but I have a Bad Feeling that it may have been similar to what you describe. On Sat, Apr 4, 2020 at 12:00 PM Ken Giusti (Jira) wrote: > Ken Giusti created PROTON-2189: > -- > > Summary: proactor C client has abnormally long pauses during > message send > Key: PROTON-2189 > URL: https://issues.apache.org/jira/browse/PROTON-2189 > Project: Qpid Proton > Issue Type: Bug > Components: proton-c > Affects Versions: proton-c-0.30.0 > Environment: To compile the clients install qpid-proton-c-devel > and simply compile: > > gcc -O2 -g -Wall -lqpid-proton -lm -o clogger clogger.c > > To reproduce my test, build qdrouterd and run it in the background. > You need to have a consumer attached. There is a test receiver client in > the qdrouterd build in /tests/test-receiver. This receiver is > designed to handle streaming messages (by default sent to 'test-address') > > Run the consumer in the background then run each clogger (default params > are fine). > > You should observe that clogger-reactor runs smoothly (use PN_TRACE_FRM=1 > on qdrouterd as well). > > You'll see clogger-reactor send the message header, then nothing for > awhile, then send the entire message. > > Use "-D" for debug output to see how many bytes have been written to > pn_link_send() > > > Reporter: Ken Giusti > Attachments: clogger-proactor.c, clogger-reactor.c > > I have a proactor-based C test client that has the ability to slowly send > extremely large messages slowly. This is done by sending 'chunks' of body > data with pauses in between. > > This client was designed to test large streaming messages against > qdrouterd. > > The behavior of this client is unexpected - I would expect the message > data to appear "on the wire" in bursts relatively quickly. In reality the > data is buffered - in some cases over 1 GB is buffered - before it is > written (as indicated by the lack @transfer frames dumped by the client AND > the qdrouterd). In some cases it takes up to 30 seconds before the > client's data starts being written to the client. > > I've refactored the client to use reactor instead and the data flows as > expected. There is minimal buffering and no abnormally long pauses. > > The clients are attached. > > It is quite likely the proactor client is incorrectly implemented, but I > used the qdrouterd I/O loop as the model and cannot see what may be wrong. > > > > > > -- > This message was sent by Atlassian Jira > (v8.3.4#803005) > > - > To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org > For additional commands, e-mail: dev-h...@qpid.apache.org > >
[jira] [Created] (PROTON-2189) proactor C client has abnormally long pauses during message send
Ken Giusti created PROTON-2189: -- Summary: proactor C client has abnormally long pauses during message send Key: PROTON-2189 URL: https://issues.apache.org/jira/browse/PROTON-2189 Project: Qpid Proton Issue Type: Bug Components: proton-c Affects Versions: proton-c-0.30.0 Environment: To compile the clients install qpid-proton-c-devel and simply compile: gcc -O2 -g -Wall -lqpid-proton -lm -o clogger clogger.c To reproduce my test, build qdrouterd and run it in the background. You need to have a consumer attached. There is a test receiver client in the qdrouterd build in /tests/test-receiver. This receiver is designed to handle streaming messages (by default sent to 'test-address') Run the consumer in the background then run each clogger (default params are fine). You should observe that clogger-reactor runs smoothly (use PN_TRACE_FRM=1 on qdrouterd as well). You'll see clogger-reactor send the message header, then nothing for awhile, then send the entire message. Use "-D" for debug output to see how many bytes have been written to pn_link_send() Reporter: Ken Giusti Attachments: clogger-proactor.c, clogger-reactor.c I have a proactor-based C test client that has the ability to slowly send extremely large messages slowly. This is done by sending 'chunks' of body data with pauses in between. This client was designed to test large streaming messages against qdrouterd. The behavior of this client is unexpected - I would expect the message data to appear "on the wire" in bursts relatively quickly. In reality the data is buffered - in some cases over 1 GB is buffered - before it is written (as indicated by the lack @transfer frames dumped by the client AND the qdrouterd). In some cases it takes up to 30 seconds before the client's data starts being written to the client. I've refactored the client to use reactor instead and the data flows as expected. There is minimal buffering and no abnormally long pauses. The clients are attached. It is quite likely the proactor client is incorrectly implemented, but I used the qdrouterd I/O loop as the model and cannot see what may be wrong. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[GitHub] [qpid-proton] jiridanek commented on issue #204: PROTON-2133 PROTON-2134 asan and tsan suppressions and travis jobs
jiridanek commented on issue #204: PROTON-2133 PROTON-2134 asan and tsan suppressions and travis jobs URL: https://github.com/apache/qpid-proton/pull/204#issuecomment-609015894 Ok, done running, and it stayed exactly the same :( https://travis-ci.org/github/jdanekrh/qpid-proton/builds/670898974 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (PROTON-2133) c-fdlimit-tests with epoll is not tsan clean
[
https://issues.apache.org/jira/browse/PROTON-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17075119#comment-17075119
]
ASF GitHub Bot commented on PROTON-2133:
jiridanek commented on issue #204: PROTON-2133 PROTON-2134 asan and tsan
suppressions and travis jobs
URL: https://github.com/apache/qpid-proton/pull/204#issuecomment-609015894
Ok, done running, and it stayed exactly the same :(
https://travis-ci.org/github/jdanekrh/qpid-proton/builds/670898974
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> c-fdlimit-tests with epoll is not tsan clean
>
>
> Key: PROTON-2133
> URL: https://issues.apache.org/jira/browse/PROTON-2133
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
>Affects Versions: proton-c-0.29.0
>Reporter: Jiri Daněk
>Priority: Major
>
> {noformat}
> 6: Test timeout computed to be: 1500
> 6: E
> 6: ==
> 6: ERROR: test_fd_limit_broker (__main__.FdLimitTest)
> 6: Check behaviour when running out of file descriptors on accept
> 6: --
> 6: Traceback (most recent call last):
> 6: File "/home/jdanek/repos/qpid/qpid-proton/c/tests/fdlimit.py", line 87,
> in test_fd_limit_broker
> 6: self.assertIn("10 messages received",
> test_subprocess.check_output(["receive", "", b.port],
> universal_newlines=True))
> 6: File "/home/jdanek/repos/qpid/qpid-proton/tests/py/test_subprocess.py",
> line 110, in __exit__
> 6: self.on_exit()
> 6: File "/home/jdanek/repos/qpid/qpid-proton/tests/py/test_subprocess.py",
> line 84, in check_kill
> 6: raise TestProcessError(self, "check_kill found error output")
> 6: TestProcessError: ['prlimit', '-n256', 'broker', '', '0'] pid=7358
> exit=-9: check_kill found error output
> 6: stderr(7358)
>
> 6: ==
> 6: WARNING: ThreadSanitizer: data race (pid=7358)
> 6: Write of size 4 at 0x7b729818 by thread T3:
> 6: #0 stop_polling
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:313:10
> (libqpid-proton-proactor.so.1+0xd22a)
> 6: #1 pconnection_cleanup
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:896:3
> (libqpid-proton-proactor.so.1+0xcacb)
> 6: #2 pconnection_done
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:1044:7
> (libqpid-proton-proactor.so.1+0xa670)
> 6: #3 pn_proactor_done
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:2166:5
> (libqpid-proton-proactor.so.1+0xa670)
> 6: #4 broker_thread
> /home/jdanek/repos/qpid/qpid-proton/c/examples/broker.c:439:5
> (broker+0x4cb4e6)
> 6:
> 6: Previous read of size 4 at 0x7b729818 by thread T2 (mutexes: write
> M2131):
> 6: #0 rearm
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:720:48
> (libqpid-proton-proactor.so.1+0x8703)
> 6: #1 pconnection_rearm
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:1014:5
> (libqpid-proton-proactor.so.1+0xcf78)
> 6: #2 pconnection_done
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:1051:14
> (libqpid-proton-proactor.so.1+0x9e24)
> 6: #3 pn_proactor_done
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:2166:5
> (libqpid-proton-proactor.so.1+0x9e24)
> 6: #4 broker_thread
> /home/jdanek/repos/qpid/qpid-proton/c/examples/broker.c:439:5
> (broker+0x4cb4e6)
> 6:
> 6: Location is heap block of size 1880 at 0x7b729800 allocated by
> thread T2:
> 6: #0 calloc (broker+0x45b124)
> 6: #1 pn_listener_accept2
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:1795:40
> (libqpid-proton-proactor.so.1+0x7d91)
> 6: #2 handle
> /home/jdanek/repos/qpid/qpid-proton/c/examples/broker.c:306:6
> (broker+0x4cb794)
> 6: #3 broker_thread
> /home/jdanek/repos/qpid/qpid-proton/c/examples/broker.c:437:14
> (broker+0x4cb794)
> 6:
> 6: Mutex M2131 (0x7b729ee8) created at:
> 6: #0 pthread_mutex_init (broker+0x42e922)
> 6: #1 pmutex_init
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:105:7
> (libqpid-proton-proactor.so.1+0x560a)
> 6: #2 pconnection_setup
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:853:3
> (libqpid-proton-proactor.so.1+0x560a)
> 6: #3 pn_listener_accept2
> /home/jdanek/repos/qpid/qpid-proton/c/src/proactor/epoll.c:1797:21
> (libqpid-proton-proactor.so.1+0x7dd8)
> 6: #4 handle
>
[jira] [Commented] (PROTON-2133) c-fdlimit-tests with epoll is not tsan clean
[
https://issues.apache.org/jira/browse/PROTON-2133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17075118#comment-17075118
]
ASF GitHub Bot commented on PROTON-2133:
jiridanek commented on issue #204: PROTON-2133 PROTON-2134 asan and tsan
suppressions and travis jobs
URL: https://github.com/apache/qpid-proton/pull/204#issuecomment-609015289
Previously, before last rebase of PR, the list of used suppressions in
python-tests was the following, taken from asan in clang at
https://travis-ci.org/github/apache/qpid-proton/jobs/64073?utm_medium=notification&utm_source=github_status
and asan in gcc at
https://travis-ci.org/github/apache/qpid-proton/jobs/64072?utm_medium=notification&utm_source=github_status.
There isn't a listing of used tsan suppressions in the tsan job.
h3. asan in clang
```
{noformat}
26: Totals: 344 tests, 343 passed, 1 skipped, 0 ignored, 0 failed
26: -
26: Suppressions used:
26: count bytes template
26: 2 2304 ^pn_data$
26: 6 1728 ^pni_data_grow$
26: 28 5344 ^pn_buffer$
26: 12928 ^pn_buffer_ensure$
26: 1 16 ^pn_error$
26: 81 10368 ^pn_list$
26: 42 21504 ^pni_map_allocate$
26: 826 55696 ^pn_object_new$
26: 132 4248 ^pni_record_create$
26: 101 1616 ^pn_stringn$
26: 23 2340 ^pn_string_grow$
26: 2 16384 ^pn_transport$
26: 44 2464 ^pn_ssl_domain$
26: 12260 ^SWIG_Python_addvarlink$
26:1804 312708 libcrypto.so
26:15322262203 bin/python2
26: 37120 _ctypes
26: -
{noformat}
```
h3. asan in gcc
```
{noformat}
26: Totals: 344 tests, 343 passed, 1 skipped, 0 ignored, 0 failed
26: -
26: Suppressions used:
26: count bytes template
26: 2 2304 ^pn_data$
26: 6 1728 ^pni_data_grow$
26: 28 5344 ^pn_buffer$
26: 12928 ^pn_buffer_ensure$
26: 1 16 ^pn_error$
26: 81 10368 ^pn_list$
26: 42 21504 ^pni_map_allocate$
26: 826 55696 ^pn_object_new$
26: 132 4248 ^pni_record_create$
26: 101 1616 ^pn_stringn$
26: 23 2340 ^pn_string_grow$
26: 2 16384 ^pn_transport$
26: 44 2464 ^pn_ssl_domain$
26: 12260 ^SWIG_Python_addvarlink$
26:1804 312708 libcrypto.so
26:15362264363 bin/python2
26: 37120 _ctypes
26: -
{noformat}
```
[~cliffjansen] I'm rebasing the PR, so let's see what it shows now, after I
fix rebase issues (if any) and rerun it on top of current master in Travis.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> c-fdlimit-tests with epoll is not tsan clean
>
>
> Key: PROTON-2133
> URL: https://issues.apache.org/jira/browse/PROTON-2133
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
>Affects Versions: proton-c-0.29.0
>Reporter: Jiri Daněk
>Priority: Major
>
> {noformat}
> 6: Test timeout computed to be: 1500
> 6: E
> 6: ==
> 6: ERROR: test_fd_limit_broker (__main__.FdLimitTest)
> 6: Check behaviour when running out of file descriptors on accept
> 6: --
> 6: Traceback (most recent call last):
> 6: File "/home/jdanek/repos/qpid/qpid-proton/c/tests/fdlimit.py", line 87,
> in test_fd_limit_broker
> 6: self.assertIn("10 messages received",
> test_subprocess.check_output(["receive", "", b.port],
> universal_newlines=True))
> 6: File "/home/jdanek/repos/qpid/qpid-proton/tests/py/test_subprocess.py",
> line 110, in __exit__
> 6: self.on_exit()
> 6: File "/home/jdanek/repos/qpid/qpid-proton/tests/py/test_subprocess.py",
> line 84, in check_kill
> 6: raise TestProcessError(self, "check_kill found error output")
> 6: TestProcessError: ['prlimit', '-n256', 'broker', '', '0'] pid=7358
> exit=-9: check_kill found error output
> 6: stderr(7358)
>
> 6: ==
> 6: WARNING: ThreadSanitizer: data race (pid=7358)
>
[GitHub] [qpid-proton] jiridanek commented on issue #204: PROTON-2133 PROTON-2134 asan and tsan suppressions and travis jobs
jiridanek commented on issue #204: PROTON-2133 PROTON-2134 asan and tsan
suppressions and travis jobs
URL: https://github.com/apache/qpid-proton/pull/204#issuecomment-609015289
Previously, before last rebase of PR, the list of used suppressions in
python-tests was the following, taken from asan in clang at
https://travis-ci.org/github/apache/qpid-proton/jobs/64073?utm_medium=notification&utm_source=github_status
and asan in gcc at
https://travis-ci.org/github/apache/qpid-proton/jobs/64072?utm_medium=notification&utm_source=github_status.
There isn't a listing of used tsan suppressions in the tsan job.
h3. asan in clang
```
{noformat}
26: Totals: 344 tests, 343 passed, 1 skipped, 0 ignored, 0 failed
26: -
26: Suppressions used:
26: count bytes template
26: 2 2304 ^pn_data$
26: 6 1728 ^pni_data_grow$
26: 28 5344 ^pn_buffer$
26: 12928 ^pn_buffer_ensure$
26: 1 16 ^pn_error$
26: 81 10368 ^pn_list$
26: 42 21504 ^pni_map_allocate$
26: 826 55696 ^pn_object_new$
26: 132 4248 ^pni_record_create$
26: 101 1616 ^pn_stringn$
26: 23 2340 ^pn_string_grow$
26: 2 16384 ^pn_transport$
26: 44 2464 ^pn_ssl_domain$
26: 12260 ^SWIG_Python_addvarlink$
26:1804 312708 libcrypto.so
26:15322262203 bin/python2
26: 37120 _ctypes
26: -
{noformat}
```
h3. asan in gcc
```
{noformat}
26: Totals: 344 tests, 343 passed, 1 skipped, 0 ignored, 0 failed
26: -
26: Suppressions used:
26: count bytes template
26: 2 2304 ^pn_data$
26: 6 1728 ^pni_data_grow$
26: 28 5344 ^pn_buffer$
26: 12928 ^pn_buffer_ensure$
26: 1 16 ^pn_error$
26: 81 10368 ^pn_list$
26: 42 21504 ^pni_map_allocate$
26: 826 55696 ^pn_object_new$
26: 132 4248 ^pni_record_create$
26: 101 1616 ^pn_stringn$
26: 23 2340 ^pn_string_grow$
26: 2 16384 ^pn_transport$
26: 44 2464 ^pn_ssl_domain$
26: 12260 ^SWIG_Python_addvarlink$
26:1804 312708 libcrypto.so
26:15362264363 bin/python2
26: 37120 _ctypes
26: -
{noformat}
```
[~cliffjansen] I'm rebasing the PR, so let's see what it shows now, after I
fix rebase issues (if any) and rerun it on top of current master in Travis.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
