Re: How to consume fedora-messaging?
On Thu, 2019-06-20 at 10:11 +0200, Pavel Raiskup wrote: > Hi Adam, or anyone, > > On Monday, June 10, 2019 6:06:42 PM CEST Adam Williamson wrote: > > If you're writing an AMQP consumer in Python, what you'll ultimately > > get for your consumer to process is a `message` object which is an > > instance of a fedora_messaging.api.Message() (or a subclass of it - a > > message schema class). > > I fail to see an example of this, I mean ... [1] says > > Message bodies are JSON objects, that adhere to a schema. Message schemas > live in their own Python package, so they can be installed on the producer > and on the consumer. > > do we have any such package so I (as a consumer) can install the package > with schema class, and use it to parse the message body? Yeah, Bodhi. Write a consumer that listens to Bodhi messages, then run it without python3-bodhi installed. When it gets a message, it'll just show up as an instance of Message(), and your system log will have a warning like this: Jun 20 14:44:40 openqa-stg01.qa.fedoraproject.org fedora- messaging[11082]: [WARNING fedora_messaging.message] The schema "bodhi.update.request.testing.v1" is not in the schema registry! Either install the package with its schema definition or define a schema. Falling back to the default schema... Now install python3-bodhi, run the consumer again, and this time the message should show up as an instance of the appropriate schema class. (I have not actually tested this as the consumers I'm writing don't need to use any of the schema methods really...but that's how it's supposed to work). > I have seen > the example of producing the message using the schema [2], but not > consuming - only the toy example which is not really using the schema > class. > > > This will have a `body` attribute which should be a dict of the message > > 'body' - the main meat of the message. > > ... I can use the "meat" to instantiate the message object manually, by > Message(body=body) perhaps, but I still have to first check the topic, > etc, I hoped there's something like: > > from MYAPP import MessageConsumer > from fedora_messaging.api import consume > ... > > class Consumer(MessageConsumer): > def consume(self, message): > """ message _is_ instance of the schema class """ > message.do_some_stuff() > if message.some_property: > do_something() > > consume(Consumer) Yeah, so long as the package that provides the schema class is installed, this actually should work. I guess whether you write your consumer to work with or without the schema class, or just have it blow up if the schema class is not installed, is kinda up to you. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: How to consume fedora-messaging?
Hi, On 6/20/19 4:11 AM, Pavel Raiskup wrote: Hi Adam, or anyone, On Monday, June 10, 2019 6:06:42 PM CEST Adam Williamson wrote: If you're writing an AMQP consumer in Python, what you'll ultimately get for your consumer to process is a `message` object which is an instance of a fedora_messaging.api.Message() (or a subclass of it - a message schema class). I fail to see an example of this, I mean ... [1] says Message bodies are JSON objects, that adhere to a schema. Message schemas live in their own Python package, so they can be installed on the producer and on the consumer. do we have any such package so I (as a consumer) can install the package with schema class, and use it to parse the message body? I have seen the example of producing the message using the schema [2], but not consuming - only the toy example which is not really using the schema class. Only if the publisher of the message has created a schema. There are a couple at this point, but not many. The consumer will log (at warning level) if a message arrived that indicates it has a schema and you don't have it installed. It doesn't give you a ton of information how to find that schema, though, so I filed an issue[0] to fix that. You can try it out with Bodhi's[1] messages, for example. This will have a `body` attribute which should be a dict of the message 'body' - the main meat of the message. ... I can use the "meat" to instantiate the message object manually, by Message(body=body) perhaps, but I still have to first check the topic, etc, I hoped there's something like: from MYAPP import MessageConsumer from fedora_messaging.api import consume ... class Consumer(MessageConsumer): def consume(self, message): """ message _is_ instance of the schema class """ message.do_some_stuff() if message.some_property: do_something() consume(Consumer) This example wouldn't work because you're not passing the actual callable, but that issue aside, your consumer gets an instance of the Message class or one of its sub-classes. The Message class is used if the publisher used it or if you don't have the schema available. [0] https://github.com/fedora-infra/fedora-messaging/issues/187 [1] https://pypi.org/project/bodhi-messages/ - Jeremy ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: How to consume fedora-messaging?
Hi Adam, or anyone, On Monday, June 10, 2019 6:06:42 PM CEST Adam Williamson wrote: > If you're writing an AMQP consumer in Python, what you'll ultimately > get for your consumer to process is a `message` object which is an > instance of a fedora_messaging.api.Message() (or a subclass of it - a > message schema class). I fail to see an example of this, I mean ... [1] says Message bodies are JSON objects, that adhere to a schema. Message schemas live in their own Python package, so they can be installed on the producer and on the consumer. do we have any such package so I (as a consumer) can install the package with schema class, and use it to parse the message body? I have seen the example of producing the message using the schema [2], but not consuming - only the toy example which is not really using the schema class. > This will have a `body` attribute which should be a dict of the message > 'body' - the main meat of the message. ... I can use the "meat" to instantiate the message object manually, by Message(body=body) perhaps, but I still have to first check the topic, etc, I hoped there's something like: from MYAPP import MessageConsumer from fedora_messaging.api import consume ... class Consumer(MessageConsumer): def consume(self, message): """ message _is_ instance of the schema class """ message.do_some_stuff() if message.some_property: do_something() consume(Consumer) Well, if not that far -- I'd at least like to see something like `python3-foo-fedora-messaging` package which I could install and give it a try (this sounds like it is common pattern from the docs). I'd like to design the copr.git/fedora-messaging directory according to that. [1] https://fedora-messaging.readthedocs.io/en/latest/tutorial/schemas.html [2] https://github.com/fedora-infra/fedora-messaging/blob/master/docs/tutorial/schemas.rst#using-it [3] https://github.com/fedora-infra/fedora-messaging/blob/master/docs/tutorial/usage.rst#python-script Pavel ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: How to consume fedora-messaging?
On 6/10/19 12:06 PM, Adam Williamson wrote: > On Mon, 2019-06-10 at 09:24 +0900, Tristan Cacqueray wrote: >> On Mon, Jun 10, 2019 at 01:24 Igor Gnatenko wrote: >>> Hello, >>> >>> I have been trying to write some script which would listen on >>> generation of new repository / successful build is tagged in Koji and >>> do some actions locally. Or basically whenever someone pushes commits, >>> I want to fetch repo locally. >>> >>> I was reading >>> https://fedora-messaging.readthedocs.io/en/latest/consuming.html, >>> but it is not very clear to me where I can find list of topics and >>> what data messages will contain... > > As well as the doc, there is a sample consumer: > https://github.com/fedora-infra/fedora-messaging/blob/master/fedora_messaging/example.py > and a couple of sample config files: > https://github.com/fedora-infra/fedora-messaging/blob/master/configs/fedora.toml > https://github.com/fedora-infra/fedora-messaging/blob/master/configs/fedora.stg.toml > that you will probably find useful. > >> >> Hi Igor, >> >> you can find the list of topics and their associated schema here: >> https://fedora-fedmsg.readthedocs.io/en/latest/topics.html > > Treat this with caution, because it is...sometimes a lie. related: https://github.com/fedora-infra/fedora-messaging/issues/180 Dusty ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: How to consume fedora-messaging?
On Mon, 2019-06-10 at 09:24 +0900, Tristan Cacqueray wrote: > On Mon, Jun 10, 2019 at 01:24 Igor Gnatenko wrote: > > Hello, > > > > I have been trying to write some script which would listen on > > generation of new repository / successful build is tagged in Koji and > > do some actions locally. Or basically whenever someone pushes commits, > > I want to fetch repo locally. > > > > I was reading > > https://fedora-messaging.readthedocs.io/en/latest/consuming.html, > > but it is not very clear to me where I can find list of topics and > > what data messages will contain... As well as the doc, there is a sample consumer: https://github.com/fedora-infra/fedora-messaging/blob/master/fedora_messaging/example.py and a couple of sample config files: https://github.com/fedora-infra/fedora-messaging/blob/master/configs/fedora.toml https://github.com/fedora-infra/fedora-messaging/blob/master/configs/fedora.stg.toml that you will probably find useful. > > Hi Igor, > > you can find the list of topics and their associated schema here: > https://fedora-fedmsg.readthedocs.io/en/latest/topics.html Treat this with caution, because it is...sometimes a lie. It is in fact auto-generated from the test cases for the fedmsg_meta_fedora_infrastructure project: https://github.com/fedora-infra/fedmsg_meta_fedora_infrastructure/ which is where the metadata providers that produce things like the human-readable summaries you get from FMN live. If you maintain a thing that publishes fedmsgs, you are *supposed to* also maintain a provider in fedmsg_meta that can interpret all the fedmsgs your thing publishes, and a test suite for that provider which includes tests for every topic your provider publishes on and useful docstrings for each test. That doc page is then actually built more or less just by gluing all the test docstrings together. However, there is no enforcement of any of this. Which means that it quite often isn't really all in sync. I think there may still be some fedmsg publishers which don't have a metadata provider or tests at all. There are definitely some where a provider and tests exist, but the fedmsg publisher's behaviour has changed a lot since they were written and no-one has updated the provider or the tests, so the doc no longer actually lists the topics on which messages are now being published. I don't really use this list any more because of issues like that... > And you can also find samples on this website: > https://apps.fedoraproject.org/datagrepper/raw ...this is much more useful, because it is just a log of all the *actual* fedmsgs that have ever been published. It can be a bit unwieldy to work with, but it's not too bad. You can see all Koji messages from the last two days here: https://apps.fedoraproject.org/datagrepper/raw?category=buildsys=172800 You can tweak the 'delta' arg (given in seconds), or use 'start' and 'end' (given as Unix timestamps) instead, to get messages from different periods. Note, datagrepper consumes and publishes *fedmsgs*, not fedora- messaging messages (at least at present). We are currently in the middle of trying to move things over from fedmsg (the old 0MQ-based system) to fedora-messaging (the new AMQP-based system). To aid this, there are bridges in place in both directions: messages published to fedora-mesaging are re-published as fedmsgs, and messages published as fedmsgs are re-published to fedora-messaging. Even for messages that are actually natively published to fedora-messaging, what you see on datagrepper is the result of the AMQP->0MQ bridge: it's not the actual original message, but the re-published fedmsg. AFAIK there is no publicly available 'datagrepper-for-fedora-messaging' ATM, though, so you can't really look at an archive of the actual messages as they're seen on the new system. There is one very important wrinkle you should bear in mind thanks to this: the message format. If you're writing an AMQP consumer in Python, what you'll ultimately get for your consumer to process is a `message` object which is an instance of a fedora_messaging.api.Message() (or a subclass of it - a message schema class). This will have a `body` attribute which should be a dict of the message 'body' - the main meat of the message. If the message was natively published as a fedora-messaging AMQP message, that is indeed what it will be. However, if it was published as a fedmsg, what you get as `message.body` is actually the *entire* fedmsg dict - the whole dict as you see it when you view a message on datagrepper, with a bunch of headers and a 'msg' dict with the real body in it, e.g. https://apps.fedoraproject.org/datagrepper/id?id=2019-633e9879-8974-4ad4-8fa2-ef07f93600b4_raw=true=extra-large . To get to what is effectively the 'body' of a fedmsg, you need to take message.body['msg']. This wrinkle is currently under discussion at https://github.com/fedora-infra/fedmsg-migration-tools/issues/20 . I came across this while converting fedmsg consumers to
Re: How to consume fedora-messaging?
On Mon, Jun 10, 2019 at 01:24 Igor Gnatenko wrote: > Hello, > > I have been trying to write some script which would listen on > generation of new repository / successful build is tagged in Koji and > do some actions locally. Or basically whenever someone pushes commits, > I want to fetch repo locally. > > I was reading > https://fedora-messaging.readthedocs.io/en/latest/consuming.html, > but it is not very clear to me where I can find list of topics and > what data messages will contain... > Hi Igor, you can find the list of topics and their associated schema here: https://fedora-fedmsg.readthedocs.io/en/latest/topics.html And you can also find samples on this website: https://apps.fedoraproject.org/datagrepper/raw > Bonus point who can tell me how does it know which messages should be > re-queued and how to manipulate that. > It doesn't seems like you have to worry about re-queue when consuming the bus. Regards, -Tristan signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
How to consume fedora-messaging?
Hello, I have been trying to write some script which would listen on generation of new repository / successful build is tagged in Koji and do some actions locally. Or basically whenever someone pushes commits, I want to fetch repo locally. I was reading https://fedora-messaging.readthedocs.io/en/latest/consuming.html, but it is not very clear to me where I can find list of topics and what data messages will contain... Bonus point who can tell me how does it know which messages should be re-queued and how to manipulate that. Thanks! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
