Hi Yanone: My take is that the main question you need to answer is network architecture. With PUB/SUB over TCP (http://api.zeromq.org/master:zmq-tcp) you may run into resource scalability issues — e.g., number of sockets, etc. with the number of subscribers you expect. (Bear in mind that with PUB/SUB over TCP each published message needs to be sent individually to each subscriber).
An alternative could be multicast, and ZeroMQ does support PGM protocol with PUB/SUB (http://api.zeromq.org/master:zmq-pgm <http://api.zeromq.org/master:zmq-pgm>). However, I’m not sure if the underlying PGM protocol would be supported on the network(s) you intend to use — my understanding is that PGM requires some minimal support from the network routers/switches. Another choice could be using UDP, which is supported by the (draft) RADIO/DISH socket types in ZeroMQ (http://api.zeromq.org/master:zmq-udp <http://api.zeromq.org/master:zmq-udp>). UDP is more a “fire-and-forget” protocol, but it sounds like that would be OK for your use case. Of course, as with any design, “the proof is in the pudding”. At the risk of being Capt. Obvious, you’re prob. going to have to invest significant effort into being able to test your design at scale. Hope this helps… Bill > On Nov 3, 2020, at 7:48 AM, Yanone <p...@yanone.de> wrote: > > > >> Am 03.11.2020 um 13:08 schrieb Edwin van den Oetelaar <ed...@oetelaar.com >> <mailto:ed...@oetelaar.com>>: >> >> Define : *very* high number of subscribers > > It needs to be able to handle let's say <100K subscribers from the beginning > without changes. Later, that number could go up to many millions when the big > corporations jump onboard, which would be cool because then I can generate > funding for a tool that aims to level the power asymmetry between them and > the small and independent publishers. I want them to jump onboard, so I need > to be prepared. > > The design needs take this into account from the beginning. What this means > for me is that I could continue to work on the messaging infrastructure > without having to update the user-installed apps. > Sure, I *could* roll out updates, but I'd rather not break functionality for > those apps that are not getting updated for whatever reason. > > So I can postpone the actual scaling, but I feel like I need to make the > right decisions from the get-go. Any change to the messaging system on the > client side yields a functionality break. > > >> >> how about some RSS feed or NNTP ? > > I'm not sure how NNTP works exactly, but for sure constant polling is out of > question I believe (RSS is not a transport protocol). Because the updates in > my system will be very infrequent but need to trigger a somewhat instant GUI > app event when they do happen, frequent polling is out of question. My whole > system is designed rather distributed, so the subscription update payload > will be polled from the small publisher’s web servers directly (after the > discussed update notification) and I can't burden them or my own central > server with a high and growing polling load. > > Pub/Sub is perfect as long as the messaging system can handle high numbers of > subscribers. > > >> Or does it have to be real time in milli-seconds? > > No, up to a few seconds lag is fine. Like a messaging app – as long as you > can use it, it'll work. Messages can arrive a few seconds late and I can > still have a conversation. > >> >> Why ZeroMQ? > > Honestly, I'm really hoping that you guys can answer that question for me. > Really, that's why I'm here. > > I had already implemented Google’s commercial Pub/Sub, which works like a > charm for my app, but topics and subscribers per topic are both limited to > 10K each, so I can't even launch with that. AWS’s solution handles more, but > also limited. After there, you need to roll your own for specs alone, plus > it'll be cheaper in the long run. > > Thanks for your time. > Jan > > >> >> Best regards, >> Edwin #oetelx >> >> On Tue, 3 Nov 2020 at 12:51, Yanone <p...@yanone.de> wrote: >> Hello friends, >> >> I'm new to the world of messaging systems. I'm developing an open source app >> in the world of graphic design that aims to change the way fonts are >> installed on people’s computers and with that tool help small and >> independent publishers to a better market position against the big evil >> corporations in our industry. >> >> I need the GUI app to react to when either the user data or the font >> subscription data has changed. The central server will inform the GUI app of >> these changes and the app needs to react to them somewhat instantly. A few >> seconds lag are no problem, neither are missing messages as I can record the >> update timestamp for all datasets on the central server and have the GUI app >> ask the central server for the timestamps on startup or network-reconnect >> for all datasets that the GUI app instance is concerned with. >> >> Back to ZeroMQ. >> So I need to design a pub/sub system that is horizontally scalable if >> necessary. At first I'm sure I can do with a few tens of thousands of >> subscribers, but later that number could grow dramatically. The thing >> doesn't need to scale now, but eventually it may need to. So I'd rather make >> the right decisions now. >> >> I read the guide (discarded non-pub/sub topics), and from what I understand, >> I can do with the simple-most pub/sub pattern. I don't need to store >> anything as the central server can keep the update timestamps as described. >> Messages can go lost. Only as long as the client is running and connected, I >> want updates to trigger. >> >> I'm expecting extremely little throughput (could be one update per topic in >> many months), but I'm expecting high number of topics and *very* high number >> of subscribers. Theoretically, two topics are sufficient (user accounts + >> subscriptions) when applying filters. This will mean even higher number of >> subscribers per topic. Otherwise, each user account and each subscription >> gets their own topic. In that case, normal number of subscribers per topic >> will be very low, except in rarer cases of free font subscriptions, when a >> single subscription could still get a very high number of subscribers. >> >> I haven't found *any* information anywhere on ZeroMQ limitations. So I need >> to ask you for clarifications. >> >> Here are a precise set of questions: >> 1. What are the limitations of number of topics and number of subscribers, >> either per topic or in total? >> 2. Which setup scales better? Only two topics with all subscribers in them >> and filters, or subscribers spread out across a high number of topics? In >> any case, the total number of subscribers stays the same. >> 3. In case of hard limitations on the above, how do I scale a cluster >> horizontally? How are the subscribers and topics spread across containers >> and brokered between them? >> 4. What is a good environment in the Google Cloud for that? I was eyeing >> Kubernetes. I have no experience with it as I come from a Google App Engine >> (Python) background, but I can learn it. The real question is: Will it scale? >> >> Thank you so much for your input! 🙏 >> >> >> >> >> >> -- >> >> Yanone >> Dipl.-Designer, >> Master of Design in Type & Media >> https://yanone.de >> Twitter/Instagram: @yanone >> +49 176 24023268 (Germany) >> >> Type.World project: >> https://type.world >> >> Crowd funding on: >> https://www.patreon.com/typeWorld >> >> >> _______________________________________________ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev >> _______________________________________________ >> zeromq-dev mailing list >> zeromq-dev@lists.zeromq.org >> https://lists.zeromq.org/mailman/listinfo/zeromq-dev > > > > -- > > Yanone > Dipl.-Designer, > Master of Design in Type & Media > https://yanone.de <https://yanone.de/> > Twitter/Instagram: @yanone > +49 176 24023268 (Germany) > > Type.World project: > https://type.world <https://type.world/> > > Crowd funding on: > https://www.patreon.com/typeWorld <https://www.patreon.com/typeWorld> > > > _______________________________________________ > zeromq-dev mailing list > zeromq-dev@lists.zeromq.org <mailto:zeromq-dev@lists.zeromq.org> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev > <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
_______________________________________________ zeromq-dev mailing list zeromq-dev@lists.zeromq.org https://lists.zeromq.org/mailman/listinfo/zeromq-dev
