Turns out Justin’s original suggestion to “to set SNDHWM=0 on the SUB socket and RCVHWM=0 on the PUB socket” resolves the issue. In particular on the SUB socket. I had mistakenly set the RCVHWM for the SUB socket.
Thanks for the help. Benjy From: zeromq-dev <zeromq-dev-boun...@lists.zeromq.org> On Behalf Of Bekritsky, Benjamin via zeromq-dev Sent: Wednesday, May 15, 2019 12:58 PM To: James Harvey <jamesdillonhar...@gmail.com>; ZeroMQ development list <zeromq-dev@lists.zeromq.org> Cc: Bekritsky, Benjamin <benjamin.bekrit...@zebra.com> Subject: Re: [zeromq-dev] max PUB/SUB topics [External Email] If I have another process that continually cycles through 0-1024 and anything greater than 1000 doesn’t come through… the issue is not a simply missing the original messages, it is that the subscriptions don’t seem to go through… From: James Harvey <jamesdillonhar...@gmail.com<mailto:jamesdillonhar...@gmail.com>> Sent: Wednesday, May 15, 2019 12:54 PM To: ZeroMQ development list <zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org>> Cc: Bekritsky, Benjamin <benjamin.bekrit...@zebra.com<mailto:benjamin.bekrit...@zebra.com>> Subject: Re: [zeromq-dev] max PUB/SUB topics [External Email] And regarding the c example where you see the issue if you start publishing before subscribing. " However, if instead of alternating pub/sub, I do the pubs first and then the subs, I occasionally hit the 1000 limit. " That is never going to be reliable as the publisher will throw away the data if there are no current subscriptions Search up "late joiner" for possible solutions if that is a feature you need to support. On Wed, May 15, 2019 at 10:39 AM James Harvey <jamesdillonhar...@gmail.com<mailto:jamesdillonhar...@gmail.com>> wrote: Hi Benjy, From your original example you are mixing binary for subscription subjects and strings for publish subjects. Use the same for both. The issue is not with the number of subscriptions, its the fact the subjects you send do not match the subjects you subscribed to. You std::string insert() is expecting a signed char, not unsigned incoming_string.insert(0,1,i%256); after 127 its going to turn negative and the std string is going to fail to print a unicode character, spitting out unprintable data James On Wed, May 15, 2019 at 9:53 AM Bekritsky, Benjamin via zeromq-dev <zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org>> wrote: Justin, I rewrote your code in c and get the same results as you showed for Python. I’m on Ubuntu 16.04 and using the “latest_release” branch (4.3.1). https://gist.github.com/bekritsk/c73af9c3080fb976328b053e0c454166<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_bekritsk_c73af9c3080fb976328b053e0c454166&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=EeNomsL8Of6KU6kxFDGILaAiG5URWw2vnonkRJA-39c&s=ZRtffhCaa7lqlH3Tc1MHsRtDiu0ISOVKdmY_wnrxQq4&e=> However, if instead of alternating pub/sub, I do the pubs first and then the subs, I occasionally hit the 1000 limit. I tried doing the same thing in Python, but Python seems to work whether the pub/subs are alternating or all the pubs are first followed by all of the subs. Thanks, Benjy From: zeromq-dev <zeromq-dev-boun...@lists.zeromq.org<mailto:zeromq-dev-boun...@lists.zeromq.org>> On Behalf Of Justin Karneges Sent: Tuesday, May 14, 2019 6:01 PM To: Jamie Stewart via zeromq-dev <zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org>> Subject: Re: [zeromq-dev] max PUB/SUB topics [External Email] Works for me with 4.2.5 on Ubuntu 18.04. https://gist.github.com/jkarneges/0b8f6e6d70c23807de0ad02c8ed72fcb<https://urldefense.proofpoint.com/v2/url?u=https-3A__gist.github.com_jkarneges_0b8f6e6d70c23807de0ad02c8ed72fcb&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=if5ZcgfMqCPKYZ553AP14LU19id8HIg6pMiaE2ez4Yw&s=9owZiXJwjq6awqaVthdwfEAbzrKv2cMkcA2YEHTGbZA&e=> On Tue, May 14, 2019, at 12:59 AM, Bekritsky, Benjamin via zeromq-dev wrote: Justin, Thanks for the guidance, but I set the HWMs to 0 and neither application crashed. I also played with changing the HWMs to 2048 and am still encountering the same issue. I then subscribed to “all” and received the messages from all the topics (0-1024); It seems as though it is limiting the number of subscriptions to 1000. If I subscribed from 20 -1024, I get topics 20 – 1019. I am using lib 4.2.3 and the publisher and subscriber on the same Ubuntu box. Thanks, Benjy From: zeromq-dev <zeromq-dev-boun...@lists.zeromq.org<mailto:zeromq-dev-boun...@lists.zeromq.org>> On Behalf Of Justin Karneges Sent: Tuesday, May 14, 2019 7:10 AM To: Jamie Stewart via zeromq-dev <zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org>> Subject: Re: [zeromq-dev] max PUB/SUB topics [External Email] Hi, Subscribing sends a message to the publisher. I believe you're hitting the default high water mark, and subscription messages are getting dropped. If I'm expecting lots of subscriptions, my rule of thumb is to set SNDHWM=0 on the SUB socket and RCVHWM=0 on the PUB socket. This way the application will sooner die than lose a subscription message. Justin On Mon, May 13, 2019, at 10:06 AM, Bekritsky, Benjamin via zeromq-dev wrote: I am using libzmq and setting up topics using binary data. In other words, my topic is the first 2 bytes of my message. I subscribe by setting a and b to the values I’m interested in: for (uint16_t i = a; i <= b; i++) { d = zmq_setsockopt(sub_socket, ZMQ_SUBSCRIBE, &i, 2); if (d) { printf ("E: subscription failed: %s\n", strerror (errno)); return -1; } } My publisher publishes like this: string incoming_string = " this message"; incoming_string.insert(0,1,i>>8); incoming_string.insert(0,1,i%256); int rc = zmq_send(pub_socket, incoming_string.c_str(), incoming_string.size(), 0); I can’t seem to subscribe to more than 1000 topics (values 0x0000 – 0x03e7). From what I understand, thousands of topics should be available. Any guidance would be helpful. Thanks, Benjy Benjamin Bekritsky c/o Zebra Technologies 2 Negev St. (Motorola Building) 4th Floor Airport City 7019900 Israel ________________________________ - CONFIDENTIAL- This email and any files transmitted with it are confidential, and may also be legally privileged. If you are not the intended recipient, you may not review, use, copy, or distribute this message. If you receive this email in error, please notify the sender immediately by reply email and then delete this email. ________________________________ - CONFIDENTIAL- This email and any files transmitted with it are confidential, and may also be legally privileged. If you are not the intended recipient, you may not review, use, copy, or distribute this message. If you receive this email in error, please notify the sender immediately by reply email and then delete this email. _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org> https://lists.zeromq.org/mailman/listinfo/zeromq-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=if5ZcgfMqCPKYZ553AP14LU19id8HIg6pMiaE2ez4Yw&s=MFBykfTCfPngIL34eSeJ2V9NMDX85SYfQxo28ITv7RM&e=> _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org> https://lists.zeromq.org/mailman/listinfo/zeromq-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=EeNomsL8Of6KU6kxFDGILaAiG5URWw2vnonkRJA-39c&s=ml270hTRrev__Pw8FDFclDZaU-1DVTGAcU4KzR5bA_c&e=> _______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org<mailto:zeromq-dev@lists.zeromq.org> https://lists.zeromq.org/mailman/listinfo/zeromq-dev<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.zeromq.org_mailman_listinfo_zeromq-2Ddev&d=DwMFaQ&c=Qwsh1H-X9ypOoLLEcAIltQ&r=wDgxT19weImS5vjsvnN8A59V8HMU1jmK8Tt64xQOZwA&m=EeNomsL8Of6KU6kxFDGILaAiG5URWw2vnonkRJA-39c&s=ml270hTRrev__Pw8FDFclDZaU-1DVTGAcU4KzR5bA_c&e=>
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
