[jira] Updated: (QPID-2997) C++ broker creates duplicate management objects under rapid create/delete/create scenarios.
[ https://issues.apache.org/jira/browse/QPID-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ken Giusti updated QPID-2997: - Attachment: disambigfix.diff proposed patch > C++ broker creates duplicate management objects under rapid > create/delete/create scenarios. > --- > > Key: QPID-2997 > URL: https://issues.apache.org/jira/browse/QPID-2997 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: 0.8 >Reporter: Ken Giusti >Assignee: Ken Giusti > Attachments: disambigfix.diff > > > When a management object (a queue, for example) is rapidly created, deleted > and then recreated, the management agent in the broker may duplicate the > object. This only happens when the recreate cycle is done entirely within > the mgmt periodic interval (default: 10 seconds). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: QIP proposal: Easy Broker Info
On Thu, 13 Jan 2011, Andrew Kennedy wrote: I would suggest a nice pluggable architecture. There would be a broker info provider class, just like you suggested, which would be feeding into a one or many publisheres. These could be, for example: * A '/var/lib/qpid/' filesystem driver on Linux * A 'C:\Documents and Settings\Qpid User\Local Settings\Application Data\' filesystem driver for Windows * A Registry driver on Windows * An Active Directory driver on Windows Server * an SNMP MIB generator for systems with an SNMP agent * A JMX driver on the Java broker * A QMF driver for both Java and C++ brokers You could include a lot of information in the provider's dump and the publisher plugins will decide what information to filter out of this and select and how (and how often) to present it. I think this is a different proposal to Mick's. Yours is about adding indirection for the broker's internal mgmt event system, so you could for instance get a bunch of log output for mgmt events, or you could get them published as qmf events. To my impression, Mick's is much smaller. It's about a simple way to discover some basic info (and I'd argue it's strictly runtime configuration). Justin - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: QIP proposal: Easy Broker Info
On 13 Jan 2011, at 18:00, Justin Ross wrote: On Thu, 13 Jan 2011, Steve Huston wrote: -Original Message- From: Gordon Sim [mailto:g...@redhat.com] ... My concern is that the specific problem of communicating the port used is driving a less clearly defined need for an alternative mechanism for data retrieval from the broker. I'd like to see a more use cases or alternative concrete problems. That's a nice, succinct way to express my reservations as well. I see this problem as well. We have one open-ended broker info service in qmf. We don't need another. I'm still attracted, however, by the "easy". I'd like to consider narrowing the scope of the "info" part to make it more palatable. Currently, it's not easy to recover the configuration of a running broker. You'd need to recreate the broker's internal configuration routine, parsing config files and then command-line arguments, to arrive at a reasonable facsimile, and even then any config that can be altered during runtime wouldn't be represented. I think it would be reasonable to write that runtime config out to a file in the manner that Mick described. It would include: - auth config - module dir - data dir - port, ssl port - max connections It *would not* include: - process id, memory use, etc. (os services do this) - queue and exchange definitions (qmf tools do this) - debug info (logging does this) This is useful in two primary ways that I can think of: (1) processes that need this basic info for integration, and (2) recovering debug info when a broker has a problem. Qmf can publish all of the same information, but it's imo not as easy to integrate with. Qmf doesn't, for instance, have a lua client at present. But any programming environment can read and parse a simple file format. Second, qmf is not something you can access when a broker is unresponsive. I'm in total agreement with you and Gordon about needing good, concrete use cases. Nonetheless, I see potential for utility here. I would suggest a nice pluggable architecture. There would be a broker info provider class, just like you suggested, which would be feeding into a one or many publisheres. These could be, for example: * A '/var/lib/qpid/' filesystem driver on Linux * A 'C:\Documents and Settings\Qpid User\Local Settings\Application Data\' filesystem driver for Windows * A Registry driver on Windows * An Active Directory driver on Windows Server * an SNMP MIB generator for systems with an SNMP agent * A JMX driver on the Java broker * A QMF driver for both Java and C++ brokers You could include a lot of information in the provider's dump and the publisher plugins will decide what information to filter out of this and select and how (and how often) to present it. That gives you a lot of flexibility and extends your use cases into things like automated broker discovery and enterprise auditing and management tools. Easy is good, but single purpose features are dead ends. We should be looking at creating a full-on best-of-breed enterprise messaging system that can *really* compete with all the commercial entities out there, and we need many of these features, like active discovery, for that. Andrew. -- -- andrew d kennedy ? do not fold, bend, spindle, or mutilate ; -- http://grkvlt.blogspot.com/ ? edinburgh : +44 7582 293 255 ; - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Resolved: (QPID-2499) Stale federation routes remain after route deletion.
[ https://issues.apache.org/jira/browse/QPID-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Robie resolved QPID-2499. -- Resolution: Fixed Assignee: Jonathan Robie (was: Ken Giusti) Fixed in revision 1058747. > Stale federation routes remain after route deletion. > > > Key: QPID-2499 > URL: https://issues.apache.org/jira/browse/QPID-2499 > Project: Qpid > Issue Type: Bug > Components: C++ Broker >Affects Versions: 0.7 >Reporter: Ken Giusti >Assignee: Jonathan Robie >Priority: Minor > Fix For: 0.9 > > Attachments: fed-direct-stale-routes.sh > > > When the same routing key is used for multiple bindings, deleting one of the > bindings may not remove routes that are dependent on that binding. This will > result in unnecessarily forwarding some messages. > I will attach a script that exhibits this bug shortly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
[jira] Created: (QPID-2997) C++ broker creates duplicate management objects under rapid create/delete/create scenarios.
C++ broker creates duplicate management objects under rapid create/delete/create scenarios. --- Key: QPID-2997 URL: https://issues.apache.org/jira/browse/QPID-2997 Project: Qpid Issue Type: Bug Components: C++ Broker Affects Versions: 0.8 Reporter: Ken Giusti Assignee: Ken Giusti When a management object (a queue, for example) is rapidly created, deleted and then recreated, the management agent in the broker may duplicate the object. This only happens when the recreate cycle is done entirely within the mgmt periodic interval (default: 10 seconds). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
RE: QIP proposal: Easy Broker Info
On Thu, 13 Jan 2011, Justin Ross wrote: On Thu, 13 Jan 2011, Steve Huston wrote: This is useful in two primary ways that I can think of: (1) processes that need this basic info for integration, and (2) recovering debug info when a broker has a problem. Eeks! I should have said config info, not debug info. I'm off to a good start by contradicting myself, :). - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
RE: QIP proposal: Easy Broker Info
On Thu, 13 Jan 2011, Steve Huston wrote: -Original Message- From: Gordon Sim [mailto:g...@redhat.com] ... My concern is that the specific problem of communicating the port used is driving a less clearly defined need for an alternative mechanism for data retrieval from the broker. I'd like to see a more use cases or alternative concrete problems. That's a nice, succinct way to express my reservations as well. I see this problem as well. We have one open-ended broker info service in qmf. We don't need another. I'm still attracted, however, by the "easy". I'd like to consider narrowing the scope of the "info" part to make it more palatable. Currently, it's not easy to recover the configuration of a running broker. You'd need to recreate the broker's internal configuration routine, parsing config files and then command-line arguments, to arrive at a reasonable facsimile, and even then any config that can be altered during runtime wouldn't be represented. I think it would be reasonable to write that runtime config out to a file in the manner that Mick described. It would include: - auth config - module dir - data dir - port, ssl port - max connections It *would not* include: - process id, memory use, etc. (os services do this) - queue and exchange definitions (qmf tools do this) - debug info (logging does this) This is useful in two primary ways that I can think of: (1) processes that need this basic info for integration, and (2) recovering debug info when a broker has a problem. Qmf can publish all of the same information, but it's imo not as easy to integrate with. Qmf doesn't, for instance, have a lua client at present. But any programming environment can read and parse a simple file format. Second, qmf is not something you can access when a broker is unresponsive. I'm in total agreement with you and Gordon about needing good, concrete use cases. Nonetheless, I see potential for utility here. Justin - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
RE: QIP proposal: Easy Broker Info
> -Original Message- > From: Gordon Sim [mailto:g...@redhat.com] > ... > My concern is that the specific problem of communicating the > port used > is driving a less clearly defined need for an alternative > mechanism for > data retrieval from the broker. I'd like to see a more use cases or > alternative concrete problems. That's a nice, succinct way to express my reservations as well. Without getting into details, can QMF find a broker w/o knowing the port number a-priori, or is QMF being noted as a good way to get all the other info once you can get the port number(s)? -Steve - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: QIP proposal: Easy Broker Info
On 01/12/2011 07:43 PM, mick wrote: Problem I wrote a script in which I needed to know the port that my broker was listening on for both TCP, and for SSL. There was no way to get this info. Right now, we use a mechanism where the broker writes the port number to stdout that it has chosen ( when you use --port 0 ). You can tell it to report on a different transport's port (i.e. SSL ) by using the "--transport" flag. I understand the specific problem, i.e. the need to be able to get both ssl and tcp ports (primarily in testing where you have specified port 0 for both in order to find a free port). However... But what if you want to know several different pieces of info about the broker? What if you are not the script that started it, but are just some other program or script that is coming late to the party? ...other broker information is available via QMF. What sorts of information are you thinking about? Anything that is not (or would not be) in the management schema? Is this essentially another mechanism for simple local access to that data? Do you have any other use cases where the usual management mechanism is not a good solution and a file based solution would be a significant improvement? There should be one central, easy way to discover all running brokers, discover which of them are clustered, etc. What ports they are listening on. Maybe even info that gets updated periodically like health information, throughput, etc. I think discovery is a very interesting problem. However I'm not convinced that a filesystem based approach is a particularly effective solution for that, at least for general use cases. My concern is that the specific problem of communicating the port used is driving a less clearly defined need for an alternative mechanism for data retrieval from the broker. I'd like to see a more use cases or alternative concrete problems. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: QIP proposal: Easy Broker Info
On Wed, 2011-01-12 at 15:31 -0500, Justin Ross wrote:
> On Wed, 12 Jan 2011, mick wrote:
>
> I like this proposal. It combines low ambition
well crap, there goes my Nobel Prize :-(
> and high utility, a good
> combination!
>
> From a QIP standpoint, however, I think the utility part needs more
> explanation. It's not hard for me to imagine uses of this, but it ought
> to be explicit. Testing in particular could gain from a simple protocol
> like this, so I think such an example would be great.
roger that. see below in "Rationale"
>
> > Index
> >
> > Easy Broker Info Service
> >
> > Status
> >
> > Draft
> >
> >
> >
> > Summary
> >
> >Brokers write simple name-value pairs, one to a line, into PID
> >files in /var/run/qpid. The names are tree-structured. Files are
> >updated periodically, and taken down when broker shuts down
> >normally. This allows easy discovery of broker info by any
> >script or program.
>
> I might consider dropping the tree-structured names part. Based on my
> experience with java properties files, I find people tend to be confused
> as to whether ".", for instance, is a word separator or a parent-child
> operator. The result is a big mess.
>
> I see two options: decide up front that data with an implicit tree
> structure is not necessary and remove the language about trees, or switch
> to a format that truly supports nested data (json, yaml, etc.).
I replied to this point after Andrew's reply in a future email.
>
> > Problem
> >
> >I wrote a script in which I needed to know the port that my
> >broker was listening on for both TCP, and for SSL. There was no
> >way to get this info. Right now, we use a mechanism where the
> >broker writes the port number to stdout that it has chosen
> >( when you use --port 0 ). You can tell it to report on a
> >different transport's port (i.e. SSL ) by using the
> >"--transport" flag. But what if you want to know several
> >different pieces of info about the broker? What if you are not
> >the script that started it, but are just some other program or
> >script that is coming late to the party? There should be one
> >central, easy way to discover all running brokers, discover
> >which of them are clustered, etc. What ports they are listening
> >on. Maybe even info that gets updated periodically like health
> >information, throughput, etc.
> >
> >
> > Solution
> >
> >Brokers write information about themselves to files in a
> >well-known directory ( i.e. /var/run/qpid ). This allows any
>
> On linux I think it would be /var/lib/qpid. What else can you say about
> how these files are namespaced?
Ah, I see.
"/var/lib : Variable state information"
OK -- that's me! ( That name with those semantics has never made the
least bit of sense to me. )
I was just imagining a single file for each broker at /var/lib/PID
That file contains all the current status info for the broker, one attr
value pair per line.
>
> >running program or script to easily discover what brokers are
> >running, what ports they are listening to for which transports,
> >and any other information that the brokers want to share. This
> >is strictly broker-based, and works whether or not management is
> >enabled. Brokers only write the info, and non-brokers only read
> >it. The info is in a simple, easily grepped name-value format,
> >whose names are tree-structured. For example:
> >"transports_tcp_port ". There is a single name-value pair
> >per line.
> >
> >
> > Rationale
> >
> >I think this is the simplest possible solution for this problem.
> >I want something that is strictly broker-based, i.e. not
> >dependent on management, language-neutral, OS-neutral, easily
> >greppable. At first I thought of /proc, but that's too hinky.
Here is an example of how this facility might be used in a script,
against brokers that are already running -- not started by this script.
for broker_path in $QPID_INFO_ROOT/*
do
broker_pid=`basename $broker_path`
ssl_port=`cat ${broker_path} | grep transport_ssl | awk '{print $2}'`
if [ $ssl_port ]
then
echo "I will connect to ssl port: $ssl_port on broker $broker_pid "
break
fi
done
> >
> >
> > Implementation Notes
> >
> >I think this is dead easy, in its first impl. make a class
> >called something like ... infoPublisher ... or something. Make a
> >public instance of it in the Broker. Anything in the code that
> >can reach the Broker can call broker->infoPublisher ( name,
> >value ); And that pair will get written out to the appropriate
> >file. You can make it either replace any previous instance of
> >that attr-value pair in the file, or just append this latest
> >
Re: Qpid Improvement Process
I would like to see the existing toolset used properly before we think about bringing another process into use, specifically JIRA. Having recently undertaken the Release Manager role I now more fully appreciate the pain those who have undertaken the process in the past have gone through in this area. JIRA is there to allow tracking issues and the related changes to the code base, however at present JIRAs often aren't created at all, or are simply created and then never updated/referenced. Just taking a quick look back through recent commits, the vast majority don't specify a JIRA in the commit log. If these actually have an associated JIRA and it isn't sitting in the resolved state then how is anyone looking at it meant to know whether it has actually been worked on? If it was worked on then how is anyone interested (user or developer) meant to know what changes were actually made in order to fix/implement it? If there was a deep and meanigful discussion about a change on a JIRA, how is someone meant to find it and vice versa? Alternatively, if no JIRA at all was created how are people meant to find out a problem was even fixed / feature added etc without searching through the commits and knowing what to look for? Not updating a JIRA to say its in progress / resolved / reopened etc is one thing, but making it unnecessarily difficult / impossible for anyone else to do so is another. Thanks to some additional prodding from Justin I did actually get some help with the process eventually for 0.8, but with those JIRAs I had to tidy up myself it took a lot more work than it should have. I'm also sure there will be JIRAs that have been pushed out to the next version during the release cycles but have actually been actioned already, except it wasn't possible for anyone to tell 6 months later. I realise there are changes where it sometimes doesn't seem necessary to make a JIRA (e.g. add a comment, correct a typo, update a readme, etc) and I will admit to doing a commit without one on occasion, but that is the vast minority of the time and should not be the norm for anyone. Ideally we should just have a JIRA for everything, but we are so bad at this as a community I would actually say we should have a commit hook put in place to enforce presence of a JIRA tag in commit logs to encourage the behaviour. Robbie. - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
Re: QIP proposal: Easy Broker Info
On Wed, 2011-01-12 at 16:00 -0500, Andrew Stitcher wrote: > On Wed, 2011-01-12 at 15:31 -0500, Justin Ross wrote: > > ... > > > Summary > > > > > >Brokers write simple name-value pairs, one to a line, into PID > > >files in /var/run/qpid. The names are tree-structured. Files are > > >updated periodically, and taken down when broker shuts down > > >normally. This allows easy discovery of broker info by any > > >script or program. > > > > I might consider dropping the tree-structured names part. Based on my > > experience with java properties files, I find people tend to be confused > > as to whether ".", for instance, is a word separator or a parent-child > > operator. The result is a big mess. > > > > I see two options: decide up front that data with an implicit tree > > structure is not necessary and remove the language about trees, or switch > > to a format that truly supports nested data (json, yaml, etc.). > > I'm not wholly convinced by Justin's argument, but you must specify the > file format including the separation of the hierarchy - Justin has > assumed the separator is "." which is consistent with many other config > files. However Mick seems to have used "_" as his separator. > > I note that you could have flat name space that looks like a hierarchy > merely by allowing a de facto separator character to occur in the names. > My intention was as Andrew says -- to have a flat namespace that can look tree-formed if you wish, by including an underscore as a legal character. It seems really nice to do it this way for grepping purposes, so i can get all of a category of info by grepping for a leading substring. You know -- the problem of using real nesting comes up a lot -- but we do not have any tool that is nearly as ubiquitous as grep that can deal with simple markup. Hey, wait -- should I just write it out as a Python tree ( nested lists ). Would that make it easy to extract nodes of the tree? > > ... > > > Implementation Notes > > > > > >I think this is dead easy, in its first impl. make a class > > >called something like ... infoPublisher ... or something. Make a > > >public instance of it in the Broker. Anything in the code that > > >can reach the Broker can call broker->infoPublisher ( name, > > >value ); And that pair will get written out to the appropriate > > >file. You can make it either replace any previous instance of > > >that attr-value pair in the file, or just append this latest > > >line to the end of the file. This code would be in > > >cpp/src/qpid/broker . /em> > > > > I find the alternative append behavior surprising. I tend to think it > > should be simpler: there's a map in memory, and sometimes it gets written > > to disk. > > > > I find the append only behaviour fairly reasonable - it would act as a > journal. It's easy to implement by the broker - just write out a single > new line whenever something changes. A reading program would have to > read every line and allow subsequent lines to overwrite existing values. i'm happy to do it either way -- i was initially thinking of a /proc-like behavior, where it gets updated whenever something changes. this discussion makes me realize that there is no point in making this a sort of logging-lite facility -- we already have logging -- in fact, this facility would tell you where the log for the(each) broker is being stored. I will have to lock the file when i write to it anyway, since different threads may all want to write info. what i think would be nicest is just an image in memory as Jross said, it gets written out to a (locked) file occasionally -- the trigger is simply when-something-changes. i think this should remain a simple-as-possible facility that gives sort of vital broker info that many processes or scripts would need -- including a pointer to the logfile, where you can find other stuff. (if it's being logged). oh, you know -- current log level would be shown by this facility. current number of connections. ports&transports. etc. > > As Justin notes if you write out the whole map to some trigger than you > have to figure out what that trigger might be. > > Andrew > > > > - > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:dev-subscr...@qpid.apache.org > - Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:dev-subscr...@qpid.apache.org
