Re: GSoC project: fedmsg for the Debian infrastructure
Hi. Daniel Pocock writes: > > Red Hat promotes a number of messaging solutions, I've used several of > these things commercially - they publish a very interesting roadmap[1]: > - - HornetQ "new ultra high performance enterprise grade messaging" > - - MRG Messaging (based on Qpid) "Exploits Linux-specific optimizations > for performance" > - - Fuse MQ (based on ActiveMQ) > - - JBoss XQ Messaging > > and I'm surprised that they ruled them all out and went for ZeroMQ > > OpenStack is working with RabbitMQ, it is based on a broker paradigm too > > My own feeling is that brokers do scale to some extent: if reliable > delivery is a requirement, then you just have to buy the right > hardware to run the broker at scale. > I'm not at all an expert in Message Queuing middleware, but I noticed that Aapache Allura [0], the infrastructure running SF.net/SourceForge (or at least, part of it), seems to have been built around a similar architecture. Maybe there's interesting feedback to draw from there, in addition to FedMsg in Fedora. Hope this helps. Best regards, [0] http://sourceforge.net/p/allura/wiki/ -- Olivier BERGER http://www-public.telecom-sudparis.eu/~berger_o/ - OpenPGP-Id: 2048R/5819D7E8 Ingenieur Recherche - Dept INF Institut Mines-Telecom, Telecom SudParis, Evry (France) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87fvydk6qa@inf-8657.int-evry.fr
Re: [Soc-coordination] GSoC project: fedmsg for the Debian infrastructure
This one time, at band camp, Simon Chopin said: > Quoting Stephen Gran (2013-04-25 21:17:29) > > This one time, at band camp, Simon Chopin said: > > > The thing itself is based on the ZeroMQ protocol. > > > > One of the principles, up to now, of system design for the debian.org > > infrastructure has been that it can tolerate single nodes being off line > > for periods of time. My understanding of ZeroMQ is that it doesn't do > > very well when the sender and the receiver aren't on line at the same > > time. I have not used ZeroMQ in any serious way, so I'm only repeating > > what I've heard. Please correct me if I'm wrong. > > Well, as I understand it, when sender or receiver are not online, there > is simply no message passing. If your concern is about what happens to > the backlog when the consumer comes back online, then I've already > written about that earlier today :-) OK, that's fairly straight forward, then. I'm not convinced yet that this is going to work out as a debian.org service. But, that being said, I think we need something like this, and no one else appears to be working on it at the moment. Go on and make it easily installable and make people interested in it. If there are architectural problems we can't live with for our needs, we'll still benefit from your work. Thanks for doing this. Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: [Soc-coordination] GSoC project: fedmsg for the Debian infrastructure
On Thu, 25 Apr 2013, Simon Chopin wrote: > > One of the principles, up to now, of system design for the debian.org > > infrastructure has been that it can tolerate single nodes being off line > > for periods of time. My understanding of ZeroMQ is that it doesn't do > > very well when the sender and the receiver aren't on line at the same > > time. I have not used ZeroMQ in any serious way, so I'm only repeating > > what I've heard. Please correct me if I'm wrong. > > Well, as I understand it, when sender or receiver are not online, there > is simply no message passing. If your concern is about what happens to > the backlog when the consumer comes back online, then I've already > written about that earlier today :-) Does that imply you expect us to run services on core infrastructure machines that listen to the world? Cheers, weasel -- | .''`. ** Debian ** Peter Palfrader | : :' : The universal http://www.palfrader.org/ | `. `' Operating System | `-http://www.debian.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130425204936.gs23...@anguilla.noreply.org
Re: [Soc-coordination] GSoC project: fedmsg for the Debian infrastructure
Quoting Peter Palfrader (2013-04-25 22:49:36) > On Thu, 25 Apr 2013, Simon Chopin wrote: > > > > One of the principles, up to now, of system design for the debian.org > > > infrastructure has been that it can tolerate single nodes being off line > > > for periods of time. My understanding of ZeroMQ is that it doesn't do > > > very well when the sender and the receiver aren't on line at the same > > > time. I have not used ZeroMQ in any serious way, so I'm only repeating > > > what I've heard. Please correct me if I'm wrong. > > > > Well, as I understand it, when sender or receiver are not online, there > > is simply no message passing. If your concern is about what happens to > > the backlog when the consumer comes back online, then I've already > > written about that earlier today :-) > > Does that imply you expect us to run services on core infrastructure > machines that listen to the world? I don't know where you have read this implication, but no that is certainly not what I meant. PS: Removing the extra MLs from the Cc list. Please keep Ralph in Cc though. signature.asc Description: signature
Re: GSoC project: fedmsg for the Debian infrastructure
+++ Nicolas Dandrimont [2013-04-25 19:13 +0200]: > * Daniel Pocock [2013-04-25 17:34:03 +0200]: > > > - do we want to use an AMQP broker? In theory, this is an open standard > > like SMTP: the clients and brokers are interchangeable > > As Simon already said, the Fedora people have tested this and rejected it > because it didn't scale. > > we will keep going on with fedmsg. > > There are several reasons for doing so: > - It is currently implemented in Fedora, on an infra that is different to > ours >but has similar components > > - This allows for tool interoperability between distros, and why not for >inter-distro collaboration > > - It's python, which fits very well with most of our stack > > - Upstream has been super, super helpful so far, and I don't see this > changing > > I wanted Simon to start this thread to gather ideas: I know some people, > mostly > in DSA, have thought about such a system for some time, so this was more of a > heads-up than wanting to reconsider the project from scratch. It landed on > -devel because, well, we couldn't find a more appropriate list :) I'm not sure how much this helps, but it seems relevant so I'll add it to the thoughts: Someone has already written a RabbitMQ (APMQ) based distributed debian buildd system in python: https://github.com/nicholasdavidson/pybit This seems to cover at least some of the same ground as the proposed project, but I just thought it should be mentioned in case it helped as example code or reducing the wheel-reinventing. This one supports cross-building. I hope fedmsg can too, but they may not have thought about that as they don't currently do any. But perhaps it's sufficiently generic and low-level that this is not a relevant point. Wookey -- Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM http://wookware.org/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130425203435.gb2...@stoneboat.aleph1.co.uk
Re: [Soc-coordination] GSoC project: fedmsg for the Debian infrastructure
Quoting Stephen Gran (2013-04-25 21:17:29) > Hi, > > This one time, at band camp, Simon Chopin said: > > Hi, > > > > Nicolas Dandrimont and I are currently working on a project proposal for > > the Google Summer of Code to use the messaging system written by Fedora, > > fedmsg[0][1], within the Debian infrastructure (some of you might have seen > > the various ITPs related to that on -devel). > > > > Tollef kindly pointed out to us that Debian service administrators would > > probably have something to say about all this, so here we are. > > > > As a premise, please note that we obviously plan to make fedmsg > > distro-agnostic before anything else (than packaging). The original > > upstream author seems very enthousiastic about the project, which makes > > it probable that we won't have to carry those patches on our own. > > > > The thing itself is based on the ZeroMQ protocol. > > One of the principles, up to now, of system design for the debian.org > infrastructure has been that it can tolerate single nodes being off line > for periods of time. My understanding of ZeroMQ is that it doesn't do > very well when the sender and the receiver aren't on line at the same > time. I have not used ZeroMQ in any serious way, so I'm only repeating > what I've heard. Please correct me if I'm wrong. Well, as I understand it, when sender or receiver are not online, there is simply no message passing. If your concern is about what happens to the backlog when the consumer comes back online, then I've already written about that earlier today :-) > It may be time to revisit our assumptions, of course - our hosting is > dramatically better than it was when I joined the project, and even > since I started doing DSA work. > > Thanks for looking into it - something like this would be very useful > for us. Thanks for your input, Cheers, Simon signature.asc Description: signature
Re: GSoC project: fedmsg for the Debian infrastructure
Hi, This one time, at band camp, Simon Chopin said: > Hi, > > Nicolas Dandrimont and I are currently working on a project proposal for > the Google Summer of Code to use the messaging system written by Fedora, > fedmsg[0][1], within the Debian infrastructure (some of you might have seen > the various ITPs related to that on -devel). > > Tollef kindly pointed out to us that Debian service administrators would > probably have something to say about all this, so here we are. > > As a premise, please note that we obviously plan to make fedmsg > distro-agnostic before anything else (than packaging). The original > upstream author seems very enthousiastic about the project, which makes > it probable that we won't have to carry those patches on our own. > > The thing itself is based on the ZeroMQ protocol. One of the principles, up to now, of system design for the debian.org infrastructure has been that it can tolerate single nodes being off line for periods of time. My understanding of ZeroMQ is that it doesn't do very well when the sender and the receiver aren't on line at the same time. I have not used ZeroMQ in any serious way, so I'm only repeating what I've heard. Please correct me if I'm wrong. It may be time to revisit our assumptions, of course - our hosting is dramatically better than it was when I joined the project, and even since I started doing DSA work. Thanks for looking into it - something like this would be very useful for us. Cheers, -- - | ,''`.Stephen Gran | | : :' :sg...@debian.org | | `. `'Debian user, admin, and developer | |`- http://www.debian.org | - signature.asc Description: Digital signature
Re: GSoC project: fedmsg for the Debian infrastructure
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 25/04/13 18:07, Simon Chopin wrote: > Quoting Daniel Pocock (2013-04-25 17:34:03) >> ZeroMQ is a very lightweight solution - it is brokerless (like >> multicast) so won't necessarily support the requirement for >> durable subscriptions (keeping messages queued up for clients >> that are disconnected) >> http://www.zeromq.org/topics:requirements-for-reliability > As I mentionned in my original email (granted, without expanding > much on it), I plan to implement a mechanism to solve this very > problem by keeping the messages sent on the emitter side and > resending them on demand. That seems like extra work to do something that can already be done in a standard way, and it also seems like it is just transferring the scalability problems from the brokers to the message sources. >> Some things that are worth looking at: >> >> - do we want to use an AMQP broker? In theory, this is an open >> standard like SMTP: the clients and brokers are interchangeable > > This solution has been already examined by the Fedora folks, and > rejected by lack of scalability WRT the broker. This sort of > question is actually why I think it is a good idea to use fedmsg > because its authors have needs that are quite similar to ours, and > virtually the same use cases. Red Hat promotes a number of messaging solutions, I've used several of these things commercially - they publish a very interesting roadmap[1]: - - HornetQ "new ultra high performance enterprise grade messaging" - - MRG Messaging (based on Qpid) "Exploits Linux-specific optimizations for performance" - - Fuse MQ (based on ActiveMQ) - - JBoss XQ Messaging and I'm surprised that they ruled them all out and went for ZeroMQ OpenStack is working with RabbitMQ, it is based on a broker paradigm too My own feeling is that brokers do scale to some extent: if reliable delivery is a requirement, then you just have to buy the right hardware to run the broker at scale. >> - as an alternative, could we use something like SIP or XMPP as >> a messaging platform? Then people can receive messages in their >> chat client. In this case it doesn't appear to be the optimal >> solution, but these protocols are quite good for systems that are >> very widely distributed over public networks. > Correct me if I'm wrong, but isn't SIP specific to VoIP? I don't > know much about XMPP outside of its application in chatting. In practice, that is how they are used, but they are interchangeable for both purposes. (and please see [2], feedback needed!) SIP supports methods such as MESSAGE, SUBSCRIBE and NOTIFY which could provide some very interesting ways to distribute data to intermittently connected developers in all the odd locations where we choose to have our workstations. These protocols may not be at the core of the solution, they could just be an external interface. On 25/04/13 19:13, Nicolas Dandrimont wrote: > * Daniel Pocock [2013-04-25 17:34:03 > +0200]: >> - then again, there are plenty of examples of systems like Apache >> Camel that can take a message from a traditional broker and >> deliver it to just >> about anything: SIP, XMPP, email, IRC channel, syslog, Nagios + >> 200 other possible destinations: >> http://camel.apache.org/components.html and it can use Java or >> XML to describe various filters and transforms > > Those are interesting suggestions, I actually had a look at Camel when it was > suggested to us on another thread, but we will keep going on with fedmsg. I didn't necessarily suggest that Camel would replace fedmsg. I have used it in much larger environments where it has been the core component for mediating messaging workflows, running multiple instances in parallel off a HA message broker. As such, I'm confident it could scale well enough to replace fedmsg, but that is not what I'm arguing - I'm just suggesting that it is a useful tool. fedmsg could well be the core of the solution, and a couple of Camel instances could be set up to implement custom workflows and deliver messages (or derivatives of the messages) over transports that fedmsg doesn't natively support. > I wanted Simon to start this thread to gather ideas: I know some people, mostly > in DSA, have thought about such a system for some time, so this > was more of a > heads-up than wanting to reconsider the project from scratch. It landed on > -devel because, well, we couldn't find a more appropriate list :) > > To get back to your remark, it would definitely be possible to > build a fedmsg-to-xmpp filtering bridge, if there is interest, but > that's not really > the scope of this GSoC project. I agree the GSoC scope needs to be contained, but it would also be good to have a wider architecture in mind and understand how things fit together, so please don't feel I'm hijacking that - I just saw the request for comments and was keen to help. fedmsg may well be the best way forward, but it would be interesting to know if we can
Re: GSoC project: fedmsg for the Debian infrastructure
On Thu, Apr 25, 2013 at 07:20:09PM +0200, Jonas Smedegaard wrote: > Quoting Simon Chopin (2013-04-25 18:07:29) > > Quoting Daniel Pocock (2013-04-25 17:34:03) > > > ZeroMQ is a very lightweight solution - it is brokerless (like > > > multicast) so won't necessarily support the requirement for durable > > > subscriptions (keeping messages queued up for clients that are > > > disconnected) > > > http://www.zeromq.org/topics:requirements-for-reliability > > > > As I mentionned in my original email (granted, without expanding much > > on it), I plan to implement a mechanism to solve this very problem by > > keeping the messages sent on the emitter side and resending them on > > demand. > > Hmm - so essentially you will follow a standard for low-level transport > of messages, but invent something custom for the slightly higher level > QoS handling. Sounds odd to me. > > > > Some things that are worth looking at: > > > > > > - do we want to use an AMQP broker? In theory, this is an open > > > standard like SMTP: the clients and brokers are interchangeable > > > > This solution has been already examined by the Fedora folks, and > > rejected by lack of scalability WRT the broker. This sort of question > > is actually why I think it is a good idea to use fedmsg because its > > authors have needs that are quite similar to ours, and virtually the > > same use cases. > > Fair enough. > > Could you perhaps point to the relevant prior discussions - I am curious > what they found bad about AMQP, and also curious if they had considered > MQTT which seems the perfect fit for this (simpler than AMQP, yet > includes simple QoS and simple authentication not in 0MQ). Indeed. Both MQTT [1] and PZQ [2] seem interesting, which is hard or me to say given my unabashed bias towards WSO2. If we are wed to fedmsg and fedmsg is wed to ZeroMQ, then I'd be interested in leveragingone of MQTT or PZQ for QoS. [1] http://mqtt.org/ [2] https://github.com/mkoppanen/pzq/wiki/An-Introduction-To-PZQ -- Luca Filipozzi http://www.crowdrise.com/SupportDebian -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130425182408.ga24...@emyr.net
Re: GSoC project: fedmsg for the Debian infrastructure
Quoting Simon Chopin (2013-04-25 18:07:29) > Quoting Daniel Pocock (2013-04-25 17:34:03) > > ZeroMQ is a very lightweight solution - it is brokerless (like > > multicast) so won't necessarily support the requirement for durable > > subscriptions (keeping messages queued up for clients that are > > disconnected) > > http://www.zeromq.org/topics:requirements-for-reliability > As I mentionned in my original email (granted, without expanding much > on it), I plan to implement a mechanism to solve this very problem by > keeping the messages sent on the emitter side and resending them on > demand. Hmm - so essentially you will follow a standard for low-level transport of messages, but invent something custom for the slightly higher level QoS handling. Sounds odd to me. > > Some things that are worth looking at: > > > > - do we want to use an AMQP broker? In theory, this is an open > > standard like SMTP: the clients and brokers are interchangeable > > This solution has been already examined by the Fedora folks, and > rejected by lack of scalability WRT the broker. This sort of question > is actually why I think it is a good idea to use fedmsg because its > authors have needs that are quite similar to ours, and virtually the > same use cases. Fair enough. Could you perhaps point to the relevant prior discussions - I am curious what they found bad about AMQP, and also curious if they had considered MQTT which seems the perfect fit for this (simpler than AMQP, yet includes simple QoS and simple authentication not in 0MQ). Good luck with the project - it sounds quite exciting! - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20130425172009.28196.54...@bastian.jones.dk
Re: GSoC project: fedmsg for the Debian infrastructure
* Daniel Pocock [2013-04-25 17:34:03 +0200]: > On 25/04/13 13:50, Simon Chopin wrote: > > [ description of the fedmsg project ] > > Questions, comments? > > ZeroMQ is a very lightweight solution - it is brokerless (like > multicast) so won't necessarily support the requirement for durable > subscriptions (keeping messages queued up for clients that are disconnected) > http://www.zeromq.org/topics:requirements-for-reliability > > Some things that are worth looking at: > > - do we want to use an AMQP broker? In theory, this is an open standard > like SMTP: the clients and brokers are interchangeable As Simon already said, the Fedora people have tested this and rejected it because it didn't scale. > - as an alternative, could we use something like SIP or XMPP as a > messaging platform? Then people can receive messages in their chat > client. In this case it doesn't appear to be the optimal solution, but > these protocols are quite good for systems that are very widely > distributed over public networks. > > - then again, there are plenty of examples of systems like Apache Camel > that can take a message from a traditional broker and deliver it to just > about anything: SIP, XMPP, email, IRC channel, syslog, Nagios + 200 > other possible destinations: > http://camel.apache.org/components.html > and it can use Java or XML to describe various filters and transforms Those are interesting suggestions, I actually had a look at Camel when it was suggested to us on another thread, but we will keep going on with fedmsg. There are several reasons for doing so: - It is currently implemented in Fedora, on an infra that is different to ours but has similar components - This allows for tool interoperability between distros, and why not for inter-distro collaboration - It's python, which fits very well with most of our stack - Upstream has been super, super helpful so far, and I don't see this changing I wanted Simon to start this thread to gather ideas: I know some people, mostly in DSA, have thought about such a system for some time, so this was more of a heads-up than wanting to reconsider the project from scratch. It landed on -devel because, well, we couldn't find a more appropriate list :) To get back to your remark, it would definitely be possible to build a fedmsg-to-xmpp filtering bridge, if there is interest, but that's not really the scope of this GSoC project. Cheers, -- Nicolas Dandrimont BOFH excuse #124: user to computer ration too low. signature.asc Description: Digital signature
Re: GSoC project: fedmsg for the Debian infrastructure
Quoting Paul Tagliamonte (2013-04-25 18:04:26) > OK. It's not Bikesheading, I read the rest of the thread. I'm a bit out > of order. > > I'm still not pleased it's still up for discussion, this puts slot > allocation into a funny place where the GSoC team has to decide if we > can bet on a project with a huge payout but is ill defined. As far as I'm concerned, this would almost be an "all or nothing" deal, I've already done quite a bit of work on fedmsg and I don't really have the courage to throw it all away and begin again with another software stack. /me goes and ask ansgar about the dak test GSoC stuff :-) signature.asc Description: signature
Re: GSoC project: fedmsg for the Debian infrastructure
Quoting Daniel Pocock (2013-04-25 17:34:03) > ZeroMQ is a very lightweight solution - it is brokerless (like > multicast) so won't necessarily support the requirement for durable > subscriptions (keeping messages queued up for clients that are disconnected) > http://www.zeromq.org/topics:requirements-for-reliability As I mentionned in my original email (granted, without expanding much on it), I plan to implement a mechanism to solve this very problem by keeping the messages sent on the emitter side and resending them on demand. > Some things that are worth looking at: > > - do we want to use an AMQP broker? In theory, this is an open standard > like SMTP: the clients and brokers are interchangeable This solution has been already examined by the Fedora folks, and rejected by lack of scalability WRT the broker. This sort of question is actually why I think it is a good idea to use fedmsg because its authors have needs that are quite similar to ours, and virtually the same use cases. > - as an alternative, could we use something like SIP or XMPP as a > messaging platform? Then people can receive messages in their chat > client. In this case it doesn't appear to be the optimal solution, but > these protocols are quite good for systems that are very widely > distributed over public networks. Correct me if I'm wrong, but isn't SIP specific to VoIP? I don't know much about XMPP outside of its application in chatting. > - then again, there are plenty of examples of systems like Apache Camel > that can take a message from a traditional broker and deliver it to just > about anything: SIP, XMPP, email, IRC channel, syslog, Nagios + 200 > other possible destinations: > http://camel.apache.org/components.html > and it can use Java or XML to describe various filters and transforms See above. Cheers, Simon signature.asc Description: signature
Re: GSoC project: fedmsg for the Debian infrastructure
On Thu, Apr 25, 2013 at 11:44:35AM -0400, Paul Tagliamonte wrote: > On Thu, Apr 25, 2013 at 05:34:03PM +0200, Daniel Pocock wrote: > > On 25/04/13 13:50, Simon Chopin wrote: > > > Hi, > > > > > > Nicolas Dandrimont and I are currently working on a project proposal for > > > the Google Summer of Code to use the messaging system written by Fedora, > > > fedmsg[0][1], within the Debian infrastructure (some of you might have > > > seen > > > the various ITPs related to that on -devel). > > > > > > Tollef kindly pointed out to us that Debian service administrators would > > > probably have something to say about all this, so here we are. > > > > > > As a premise, please note that we obviously plan to make fedmsg > > > distro-agnostic before anything else (than packaging). The original > > > upstream author seems very enthousiastic about the project, which makes > > > it probable that we won't have to carry those patches on our own. > > > > > > The thing itself is based on the ZeroMQ protocol. > > > > > > To quote Nicolas: > > >> One of the key outcomes of getting such a system in place, is that > > >> everyone, > > >> everywhere, can start listening to the messages and using them, opening > > >> up lots > > >> of doors for people to make amazing services based on Debian. > > >> > > >> A few ideas: > > >> - getting a signal from the archive on an accepted package (I'm > > >> confusing > > >>binaries and sources for the sake of brevity): > > >>→ Trigger a piuparts run > > >>→ Trigger lintian checks > > >>→ Let any derivative intent a rebuild > > >>→ Signal ports to rebuild > > >>→ Trigger a jenkins job on specific package uploads > > >>→ Post to pump.io/identi.ca/twitter > > >>→ get a notification on your desktop > > >>→ ... > > >> - one of your pet packages gets a git commit > > >>→ try a rebuild > > >>→ run QA checks > > >>→ ... > > >> > > >> (boy, that escalated quickly) > > >> > > >> I think the possibilities are quite nice, and, as the fedmsg webpage > > >> says, that > > >> "gives new meaning to open infrastructure". > > > Two features I'd like to implement during this GSoC that are not AFAICT > > > already present in fedmsg are GPG support and some kind of playback > > > mechanism for the systems where it is important that all messages are > > > sent and received (there are some others where the information would > > > have value only at the time of emitting, I suppose). > > > > > > Questions, comments? > > > > ZeroMQ is a very lightweight solution - it is brokerless (like > > multicast) so won't necessarily support the requirement for durable > > subscriptions (keeping messages queued up for clients that are disconnected) > > http://www.zeromq.org/topics:requirements-for-reliability > > > > Some things that are worth looking at: > > > > - do we want to use an AMQP broker? In theory, this is an open standard > > like SMTP: the clients and brokers are interchangeable > > > > - as an alternative, could we use something like SIP or XMPP as a > > messaging platform? Then people can receive messages in their chat > > client. In this case it doesn't appear to be the optimal solution, but > > these protocols are quite good for systems that are very widely > > distributed over public networks. > > > > - then again, there are plenty of examples of systems like Apache Camel > > that can take a message from a traditional broker and deliver it to just > > about anything: SIP, XMPP, email, IRC channel, syslog, Nagios + 200 > > other possible destinations: > > http://camel.apache.org/components.html > > and it can use Java or XML to describe various filters and transforms > > > > > > > > -- > > To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org > > with a subject of "unsubscribe". Trouble? Contact > > listmas...@lists.debian.org > > Archive: http://lists.debian.org/51794ceb@pocock.com.au > > > > Guys, we're *days* into student applications. We should have had this > nailed down weeks ago. > > Pick something, go with it, or let's stop this early. Really; we should > be fielding students now, now bikesheading over this stuff. OK. It's not Bikesheading, I read the rest of the thread. I'm a bit out of order. I'm still not pleased it's still up for discussion, this puts slot allocation into a funny place where the GSoC team has to decide if we can bet on a project with a huge payout but is ill defined. I'll shut up and get out of this thread, but we should *really* nail this down, like, in the next few days. > > Cheers, > Paul > Sorry! Paul -- .''`. Paul Tagliamonte : :' : Proud Debian Developer `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 `- http://people.debian.org/~paultag signature.asc Description: Digital signature
Re: GSoC project: fedmsg for the Debian infrastructure
On Thu, Apr 25, 2013 at 05:34:03PM +0200, Daniel Pocock wrote: > On 25/04/13 13:50, Simon Chopin wrote: > > Hi, > > > > Nicolas Dandrimont and I are currently working on a project proposal for > > the Google Summer of Code to use the messaging system written by Fedora, > > fedmsg[0][1], within the Debian infrastructure (some of you might have seen > > the various ITPs related to that on -devel). > > > > Tollef kindly pointed out to us that Debian service administrators would > > probably have something to say about all this, so here we are. > > > > As a premise, please note that we obviously plan to make fedmsg > > distro-agnostic before anything else (than packaging). The original > > upstream author seems very enthousiastic about the project, which makes > > it probable that we won't have to carry those patches on our own. > > > > The thing itself is based on the ZeroMQ protocol. > > > > To quote Nicolas: > >> One of the key outcomes of getting such a system in place, is that > >> everyone, > >> everywhere, can start listening to the messages and using them, opening up > >> lots > >> of doors for people to make amazing services based on Debian. > >> > >> A few ideas: > >> - getting a signal from the archive on an accepted package (I'm confusing > >>binaries and sources for the sake of brevity): > >>→ Trigger a piuparts run > >>→ Trigger lintian checks > >>→ Let any derivative intent a rebuild > >>→ Signal ports to rebuild > >>→ Trigger a jenkins job on specific package uploads > >>→ Post to pump.io/identi.ca/twitter > >>→ get a notification on your desktop > >>→ ... > >> - one of your pet packages gets a git commit > >>→ try a rebuild > >>→ run QA checks > >>→ ... > >> > >> (boy, that escalated quickly) > >> > >> I think the possibilities are quite nice, and, as the fedmsg webpage says, > >> that > >> "gives new meaning to open infrastructure". > > Two features I'd like to implement during this GSoC that are not AFAICT > > already present in fedmsg are GPG support and some kind of playback > > mechanism for the systems where it is important that all messages are > > sent and received (there are some others where the information would > > have value only at the time of emitting, I suppose). > > > > Questions, comments? > > ZeroMQ is a very lightweight solution - it is brokerless (like > multicast) so won't necessarily support the requirement for durable > subscriptions (keeping messages queued up for clients that are disconnected) > http://www.zeromq.org/topics:requirements-for-reliability > > Some things that are worth looking at: > > - do we want to use an AMQP broker? In theory, this is an open standard > like SMTP: the clients and brokers are interchangeable > > - as an alternative, could we use something like SIP or XMPP as a > messaging platform? Then people can receive messages in their chat > client. In this case it doesn't appear to be the optimal solution, but > these protocols are quite good for systems that are very widely > distributed over public networks. > > - then again, there are plenty of examples of systems like Apache Camel > that can take a message from a traditional broker and deliver it to just > about anything: SIP, XMPP, email, IRC channel, syslog, Nagios + 200 > other possible destinations: > http://camel.apache.org/components.html > and it can use Java or XML to describe various filters and transforms > > > > -- > To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org > with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org > Archive: http://lists.debian.org/51794ceb@pocock.com.au > Guys, we're *days* into student applications. We should have had this nailed down weeks ago. Pick something, go with it, or let's stop this early. Really; we should be fielding students now, now bikesheading over this stuff. Cheers, Paul -- .''`. Paul Tagliamonte : :' : Proud Debian Developer `. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87 `- http://people.debian.org/~paultag signature.asc Description: Digital signature
Re: GSoC project: fedmsg for the Debian infrastructure
On 25/04/13 13:50, Simon Chopin wrote: > Hi, > > Nicolas Dandrimont and I are currently working on a project proposal for > the Google Summer of Code to use the messaging system written by Fedora, > fedmsg[0][1], within the Debian infrastructure (some of you might have seen > the various ITPs related to that on -devel). > > Tollef kindly pointed out to us that Debian service administrators would > probably have something to say about all this, so here we are. > > As a premise, please note that we obviously plan to make fedmsg > distro-agnostic before anything else (than packaging). The original > upstream author seems very enthousiastic about the project, which makes > it probable that we won't have to carry those patches on our own. > > The thing itself is based on the ZeroMQ protocol. > > To quote Nicolas: >> One of the key outcomes of getting such a system in place, is that everyone, >> everywhere, can start listening to the messages and using them, opening up >> lots >> of doors for people to make amazing services based on Debian. >> >> A few ideas: >> - getting a signal from the archive on an accepted package (I'm confusing >>binaries and sources for the sake of brevity): >>→ Trigger a piuparts run >>→ Trigger lintian checks >>→ Let any derivative intent a rebuild >>→ Signal ports to rebuild >>→ Trigger a jenkins job on specific package uploads >>→ Post to pump.io/identi.ca/twitter >>→ get a notification on your desktop >>→ ... >> - one of your pet packages gets a git commit >>→ try a rebuild >>→ run QA checks >>→ ... >> >> (boy, that escalated quickly) >> >> I think the possibilities are quite nice, and, as the fedmsg webpage says, >> that >> "gives new meaning to open infrastructure". > Two features I'd like to implement during this GSoC that are not AFAICT > already present in fedmsg are GPG support and some kind of playback > mechanism for the systems where it is important that all messages are > sent and received (there are some others where the information would > have value only at the time of emitting, I suppose). > > Questions, comments? ZeroMQ is a very lightweight solution - it is brokerless (like multicast) so won't necessarily support the requirement for durable subscriptions (keeping messages queued up for clients that are disconnected) http://www.zeromq.org/topics:requirements-for-reliability Some things that are worth looking at: - do we want to use an AMQP broker? In theory, this is an open standard like SMTP: the clients and brokers are interchangeable - as an alternative, could we use something like SIP or XMPP as a messaging platform? Then people can receive messages in their chat client. In this case it doesn't appear to be the optimal solution, but these protocols are quite good for systems that are very widely distributed over public networks. - then again, there are plenty of examples of systems like Apache Camel that can take a message from a traditional broker and deliver it to just about anything: SIP, XMPP, email, IRC channel, syslog, Nagios + 200 other possible destinations: http://camel.apache.org/components.html and it can use Java or XML to describe various filters and transforms -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/51794ceb@pocock.com.au
GSoC project: fedmsg for the Debian infrastructure
Hi, Nicolas Dandrimont and I are currently working on a project proposal for the Google Summer of Code to use the messaging system written by Fedora, fedmsg[0][1], within the Debian infrastructure (some of you might have seen the various ITPs related to that on -devel). Tollef kindly pointed out to us that Debian service administrators would probably have something to say about all this, so here we are. As a premise, please note that we obviously plan to make fedmsg distro-agnostic before anything else (than packaging). The original upstream author seems very enthousiastic about the project, which makes it probable that we won't have to carry those patches on our own. The thing itself is based on the ZeroMQ protocol. To quote Nicolas: > One of the key outcomes of getting such a system in place, is that everyone, > everywhere, can start listening to the messages and using them, opening up > lots > of doors for people to make amazing services based on Debian. > > A few ideas: > - getting a signal from the archive on an accepted package (I'm confusing >binaries and sources for the sake of brevity): >→ Trigger a piuparts run >→ Trigger lintian checks >→ Let any derivative intent a rebuild >→ Signal ports to rebuild >→ Trigger a jenkins job on specific package uploads >→ Post to pump.io/identi.ca/twitter >→ get a notification on your desktop >→ ... > - one of your pet packages gets a git commit >→ try a rebuild >→ run QA checks >→ ... > > (boy, that escalated quickly) > > I think the possibilities are quite nice, and, as the fedmsg webpage says, > that > "gives new meaning to open infrastructure". Two features I'd like to implement during this GSoC that are not AFAICT already present in fedmsg are GPG support and some kind of playback mechanism for the systems where it is important that all messages are sent and received (there are some others where the information would have value only at the time of emitting, I suppose). Questions, comments? Cheers, Simon [0] http://wiki.debian.org/SummerOfCode2013/StudentApplications/SimonChopin [1] http://fedmsg.com/ PS: debian-admin, debian-services-admin and soc-coordination Cc-ed for reference, but further discussion should be on -devel. Upstream author Ralph Bean also Cc-ed, it would probably be useful to keep him in the loop. signature.asc Description: signature
