Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, 2011-02-09 at 18:58 +0100, Lars Ellenberg wrote: > On Wed, Feb 09, 2011 at 06:22:09PM +0100, Dejan Muhamedagic wrote: > > Hi Holger, > > > > On Wed, Feb 09, 2011 at 06:08:51PM +0100, Holger Teutsch wrote: > > > Guys, > > > coming back from a 3 hour break I'm a bit shocked about the very > > > *active* discussion and it is difficult to me to find an entry point. > > > > No worries, I found it very difficult to follow to. There were > > like a few thousand subthreads. > > ;-) > > > Right now, I think the best option would be to put your agent in > > /usr/lib/ocf/resource.d/testing and ask people to help test it > > and log that as well from the old db2 resource agent (sth like > > "this db2 RA is about to be deprecated, please try ocf:testing:db2"). > > Actually, because it is db2, I expect anyone running db2 to have a test > system where he will excercise any and all changes to the system, > be it config changes, tuning parameters, or package upgrades. > In my previous life I was head of a corporate data center and can fully confirm this. Regards Holger ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] [PATCH] manage PostgreSQL 9.0 streaming replication using Master/Slave
Hello -
I'm not sure that I really like this approach.
1. It doesn't follow standard Master/Slav logic of Pacemaker and that
can create a lot of confusion. On start RA decides on what role shall
be started where based not on Pacemaker configuration but on existence
of recovery.conf file which is out of Pacemaker scope. Moving master
role form one node to another cannot be done simply with Pacemaker
command. I mean master will be moved, but slave won't be created.
Though Pacemaker will be forced to believe that it still exists and
administrator will have to do several manual steps to make it real.
It's too tricky in my opinion.
2, What is the reason of such heavy use of ATTR_UPDATER. What process
is the consumer of that pgsql_status attribute? Calling attr_updater
so often makes RA slow.
I need some time to build PosgreSQL 9.0 test cluster and try some more tests.
On Tue, Feb 8, 2011 at 6:03 AM, Takatoshi MATSUO wrote:
> Hi
>
> Thank you for comments.
>
>> First of all thanks for doing this. It's definitely a neat option to
>> have. Unfortunately I haven't yet worked with ProstgreSQL 9.0. Could
>> you please elaborate little bit more on how it works? What happens if
>> master dies? Who is responsible for creating/deleting recovery.conf
>> file?
>
> This patch does not setup recovery.conf for hot standby automatically,
> so you need setup it manually before starting hot standby.
> If recovery.conf exists, resource agent will become slave.
> Die of master causes fail-over correctly.
>
>> Also please be more careful with quoting variables in your if statements.
>
> I checked and quoted.
>
>
>
>
> I elaborate.
>
> Condition *
>
> 1. Streaming replication works correctly.
> 2. Streaming replication works correctly if you swap
> primary server for hot standby server.
>
> FYA I wrote my configuration at the end of the post.
> See "Example of configuration".
>
> Setup and start ***
>
> 1. You decide which server do you want to be master.
> In this instance I assume that you choice node1.
>
> 2. node1# rm /var/lib/pgsql/9.0/data/recovery.conf
>
> 3. node1# /etc/init.d/heartbeat start
> Wait for "ONLINE node1".
>
> 4. node1# crm configure load update pgsql.crm
>
> 5. Wait for master started as below
> node1# crm_mon -1 -fA
>
>
> Last updated: Tue Feb 8 18:51:00 2011
> Stack: Heartbeat
> Current DC: node1 (0c140f90-7de3-438f-b1b5-3b9722bbde21) -
> partition with quorum
> Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
> 2 Nodes configured, unknown expected votes
> 3 Resources configured.
>
>
> Online: [ node1 ]
>
> vip (ocf::heartbeat:IPaddr2): Started node1
> Master/Slave Set: msPostgresql
> Masters: [ node1 ]
> Stopped: [ postgresql:1 ]
> Clone Set: clnPingd
> Started: [ node1 ]
> Stopped: [ pingd:1 ]
>
> Node Attributes:
> * Node node1:
> + master-postgresql:0 : 1000
> + pgsql-status : PRI
> + ping_set : 100
>
> Migration summary:
> * Node node1:
>
>
> If start operation fails, please check PostgreSQL state using "crm_mon -A".
> "pgsql-status" line shows something.
>
> 6. Backup node1's data and restore to node2.
> In my case I use rsync command as below.
>
> node2# psql -h 192.168.2.107 -U postgres -c "SELECT
> pg_start_backup('label')"
> node2# rsync -avr --delete --exclude=postmaster.pid
> 192.168.2.107:/var/lib/pgsql/9.0/data/ /var/lib/pgsql/9.0/data/
> node2# psql -h 192.168.2.107 -U postgres -c "SELECT pg_stop_backup()"
>
> 7. Put recovery.conf for node2.
> node2# cp recovery.conf /var/lib/pgsql/9.0/data/
> node2# chown postgres:postgres /var/lib/pgsql/9.0/data/recovery.conf
>
> 8. node2# /etc/init.d/heartbeat start
>
> 9. Wait for slave started as below.
> node2# crm_mon -1 -fA
>
>
> Last updated: Tue Feb 8 19:41:00 2011
> Stack: Heartbeat
> Current DC: node1 (0c140f90-7de3-438f-b1b5-3b9722bbde21) -
> partition with quorum
> Version: 1.0.10-da7075976b5ff0bee71074385f8fd02f296ec8a3
> 2 Nodes configured, unknown expected votes
> 3 Resources configured.
>
>
> Online: [ node1 node2 ]
>
> vip (ocf::heartbeat:IPaddr2): Started node1
> Master/Slave Set: msPostgresql
> Masters: [ node1 ]
> Slaves: [ node2 ]
> Clone Set: clnPingd
> Started: [ node1 node2 ]
>
> Node Attributes:
> * Node node1:
> + master-postgresql:0 : 1000
> + pgsql-status : PRI
> + ping_set : 100
> * Node node2:
> + master-postgresql:1 : 100
> + pgsql-status
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, 2011-02-09 at 19:30 +0100, Lars Marowsky-Bree wrote: > We don't have them _yet_, which I think is holding us up here. > > I'm wondering if you could automate these tests in a script (perhaps > using the ocft framework that is in the repo)? So that we can show that > the "new" DB2 one passes them just as well as the legacy one? I will look at it but no promises. > > (Actually, I'd expect the new one to pass more than the old one, since > it covers additional scenarios.) > > It may be a bit too much to ask, but hey, there's lunch ;-) And I won't > be too cheap. To go a step further: I live in the Frankfurt area. Any obstacles here ? > > > Regards, > Lars > Regards Holger ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T16:58:12, Dejan Muhamedagic wrote: > Actually, I was also thinking about having another provider for > db2. I think that we should do this. I'm still unconvinced, because it requires users to change their configuration back and forth. > > If the RA is really backwards compatible, they can do that even w/o > > service outage - this should cause a reprobe, and if all is well, the > > new RA will just attach to the existing instance. > Is setting to unmanaged enough in this case? Won't that anyway > trigger a restart because the definition changed? Note the use of "should", not "does". Clearly, if we went down this route, we would have an incentive to make it easy for users to switch w/o outages. > Nothing is, but any of these schemes is anyway better than > introducing a resource agent with the same functionality and > different name. Besides, this should at least be easy to test. > And document. Uhm. So, from a user perspective, whether we introduce the "new" code as "heartbeat:db2-v2" or "testing:db2" doesn't make a lot of difference; they'll see it twice and wonder what to use. Second, how does this make it easier to test whether the RA is really compatible and sufficiently stable to replace the old one? Noone will try it on their production sites. Noone wants "testing" code on their radar towers or trading floors or tactical weapon computers. Or have to go back on features that don't work as advertised, or where the configuration changes again. The only people who will test this are those with staging systems, and they need to be sufficiently aware of us to actually run the tests (which ones?) and report back. If we could define "Which test results will we consider adequate feedback before shipping the new version as stable" today, perhaps in an automated fashion ;-), we wouldn't need the canary testing. We could just find someone with the lab setup we need, run the tests, and be happy. Experience from the pacemaker project shows that there is very very little feedback on "experimental" code. Check the mailing lists for reports on devel, or the ACL feature, before we pulled it into 1.1. Clearly, it'd be good if we had more such feedback, but we don't. To restate more concisely (I know I'm bad at that): Without defining what we consider adequate feedback, canary testing is useless. If we defined it, we would not need it (or need it much less). We _first_ need to define what adequate testing is. Just saying "we've shipped this as devel/staging/testing code for 3 releases and not gotten any (negative) feedback" is not good enough. > There's no way we will ever have regression tests for stuff such > as db2, oracle, SAP, and similar. This is a devastating attitude and not one I'm willing to subscribe to. > I'm not even in a position to do any kind of testing for some of > these, because they are too complex to setup and commercial. And so > forth. We don't need a regression test suite for everything that you and I can run. (Though working for Novell, we're actually in a position to push that to QA, as is Red Hat.) We need one that contributors and RA maintainers can run: both clearly have a test rig available and thus could run it. > It would be great if the software vendor would ship the RA, but that's > also something we cannot expect to happen. Like I said, make the above a requirement for accepting new RAs, even enforce the coding style if you find reading through the variations annoying (which I can understand all too well), and you'll both have created a boost for quality and a huge incentive for the more sloppy developers to contribute code to the service projects. Sounds like a win to me ;-) You and Florian are the resource-agent maintainer these days, you get to set the policy. (Which doesn't mean I won't argue with it if I disagree. ;-) Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
Hi,
please find enclosed the revised version.
-holger
On Wed, 2011-02-09 at 14:03 +0100, Dejan Muhamedagic wrote:
> Hi,
>
> Please find below some comments.
>
> > #!/bin/sh
> > #
> > # db2
> > #
> > # Resource agent that manages a DB2 LUW database in Standard role
> > db2home=$(sh -c "echo ~$instance")
> > db2sql=$db2home/sqllib
> > db2profile=$db2sql/db2profile
> > db2bin=$db2sql/bin
> >
> > STATE_FILE=${HA_RSCTMP}/Dummy-${OCF_RESOURCE_INSTANCE}.state
>
> Why Dummy? Better use db2.
cut&paste error -> changed
>
> > # Let's make sure a few important things are there...
> > if ! [ -d "$db2sql" -a -d "$db2bin" -a -f "$db2profile" -a \
> >-x "$db2profile" -a -x "$db2bin/db2" ]
> > then
> > ocf_is_probe && exit $OCF_NOT_RUNNING
> > ocf_log err "DB2 required directories and/or files not found"
> > exit $OCF_ERR_INSTALLED
> > fi
> >
> > db2instance=$(runasdb2 'echo $DB2INSTANCE')
> > if [ "$db2instance" != "$instance" ]
> > then
> > ocf_is_probe && exit $OCF_NOT_RUNNING
> > ocf_log err "DB2 parameter instance \"$instance\" != DB2INSTANCE
> > \"$db2instance\""
> > exit $OCF_ERR_INSTALLED
>
> Better ERR_CONFIGURED, this looks like a serious setup problem.
agreed
>
> > fi
> >
> > # enough checking for stop to succeed
> > [ $__OCF_ACTION = stop ] && return $OCF_SUCCESS
>
> return? Above it was exit. Or is there more to be done on stop?
The db2 instance has to be stopped. This refers to
stop)
db2_validate
db2_stop
exit $?
It means: further checking may lead to exit $OCF_ERR_INSTALLED which may
give stop failure -> STONITH.
> > #
> > # maintain the fal (first active log) attribute
> > # db2_fal_attrib DB {set val|get|delete}
> > #
> > db2_fal_attrib() {
> > local db=$1
> > local attr
> >
> > attr=db2hadr_${instance}_${db}_fal
> >
> > case "$2" in
> > set)
> > crm_attribute -t crm_config -n $attr -v "$3"
>
> Hmm, this is an attribute which should be on the master
> right? Using crm_config for that looks wrong. I think
> that it should go into the node status.
No, it is an attribute that communicates the log position from the
Master node to a restarting slave node.
>
> >
> > #
> > # unfortunately a first connect after a crash may need several minutes
> > # for some internal cleanup stuff in DB2.
> > # We run a connect in background so other connects (i.e. monitoring!) may
> > proceed.
> > #
> > db2_run_connect() {
> > local db=$1
> >
> > logasdb2 "db2 connect to $db; db2 terminate"
>
> What does this do? It is run in the background from the
> start action, but it is not waited on (no exit code check)
>
From long year experience with DB2 I know that after a DB crash with
crash recovery completed a *first* connect to the DB may need minutes
(for a TB database) for some internal cleanup stuff. All other connects
issued in parallel go through immediately. So that is for getting a
'blocking' connect fired immediately. Status doesn't matter as monitor
has it's own connect.
> > }
> >
> > #
> > # get some data from the database config
> > # sets HADR_ROLE HADR_TIMEOUT HADR_PEER_WINDOW
> > #
> > db2_get_cfg() {
> > local db=$1
> >
> > local output hadr_vars
> >
> > output=$(runasdb2 db2 get db cfg for $db)
> > [ $? != 0 ] && return $OCF_ERR_GENERIC
> >
> > hadr_vars=$(echo "$output" |\
> > awk '$0 ~/HADR database role/ {printf "HADR_ROLE=%s; ", $NF;}
> > $0 ~/HADR_TIMEOUT/ {printf "HADR_TIMEOUT=%s; ", $NF;}
> > $0 ~/First active log file/ {printf "FIRST_ACTIVE_LOG=%s\n", $NF;}
> > $0 ~/HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW=%s\n", $NF;}')
>
> You can leave out "$0 ~". It is going to work this way too,
> but otherwise it does not look very awk :)
Done
> Also, since it goes to eval below, better add single quotes around
> /HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW='%s'\n" ...
I've put it in but the output is *very* expected (hopefully the DB is
really DB2 8-).
...
> > #
> > # Start instance and DB.
> > # Standard mode is through "db2 activate" in order to start in previous
> > # mode (Standy/Primary).
> > # If the database is a primary AND we can determine that the running master
> > # has a higher "first active log" we conclude that we come up after a crash
> > # an the previous Standby is now Primary.
> > # The db is then started as Standby.
> > #
> > # Other cases: danger of split brain, log error and do nothing.
> > #
> > db2_start() {
> > local output start_cmd is_ms db
> >
> > if output=$(runasdb2 db2start dbpartitionnum $db2node)
> > then
> > ocf_log info "DB2 instance $instance($db2node) started: $output"
> > else
> > case $output in
> > SQL1026N*)
> > ocf_log info "DB2 instance $instance($db2node) already running:
> > $output"
> > ;;
> >
> > *)
>
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T17:09:29, Dejan Muhamedagic wrote: > > I agree with that. (Not the coward part, of course!) IMHO though, the > > answer is to improve our test coverage to the point where we can > > refactor and clean up code without taking an unreasonable risk of > > breakage. > Dream on ;-) Uhm. This is standard practice in other projects. Even in some that make up the rest of the cluster stack. Why should this be impossible here? > In that case, it would need to be a set of incremental changes > so that each of them can be easy to understand and check and so > forth. Not necessarily. Code review of individual changesets is excellent for spotting bugs, but an orthogonal practice to automated regression tests. > > See my other mail for a proposal: the Linux-HA lunch fund for testers > > ;-) > Must have missed that one. It helps to read all mails before responding, or to not read them in thread but date order; otherwise, they may contain forward references ;-) Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T18:08:51, Holger Teutsch wrote: > coming back from a 3 hour break I'm a bit shocked about the very > *active* discussion and it is difficult to me to find an entry point. Heh ;-) Sorry, you've prodded a wasp nest. > I've spent a lot of time to make it plug in compatible to the old agent. > I was the one who contributed the last patches to the old agent so I know it > pretty well. [snip] What you describe here with regard to the manual testing you've done clearly convinces me that the new RA is as good as the old one. (As you can probably tell, I'm not a big fan of the idea of shipping both.) > If you have some formal requirements for documentation please feel free > to forward them to me. We don't have them _yet_, which I think is holding us up here. I'm wondering if you could automate these tests in a script (perhaps using the ocft framework that is in the repo)? So that we can show that the "new" DB2 one passes them just as well as the legacy one? (Actually, I'd expect the new one to pass more than the old one, since it covers additional scenarios.) It may be a bit too much to ask, but hey, there's lunch ;-) And I won't be too cheap. Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T18:22:09, Dejan Muhamedagic wrote: We need to get out of this "oh my god fear is bad I'm afraid" attitude as a project/group. It doesn't make me happy. > Right now, I think the best option would be to put your agent in > /usr/lib/ocf/resource.d/testing and ask people to help test it > and log that as well from the old db2 resource agent (sth like > "this db2 RA is about to be deprecated, please try ocf:testing:db2"). I think this is a horrible idea, in particular logging in the "stable" code base. And having them change their configurations back and forth. Think about the message this sends to users: "OK, so this RA is going to be depreciated; I don't want to use that! But that RA is classified as 'testing', I don't want something unstable either! I'll go use Veritas!" Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T17:05:08, Dejan Muhamedagic wrote: > > That, by itself, doesn't strike me as a reasonable argument for > > duplicating the RA. > Look, if I could tell that changes were safe, then I certainly > wouldn't moan about it. If you can, please go ahead. ... I thought I covered that in the following paragraphs. > And it definitely is a bad idea. But this case is different from > IPaddr*. IPaddr* are two RAs covering the same functionality > but using different tools. Oh, no, IPaddr and IPaddr2 are by no means compatible except for the base functionality. Not just the different tools, they provide different features. Over a couple of years divergence (to which I shamefully admit to have contributed) have brought us no closer to dropping one. I am _very_ opposed to creating another such situation with another agent. The precedent is not good. > > To make up for that, I'll personally invite everyone who contributes a > > reasonably comprehensive, non-trivial test suite for an RA to lunch. > > > > (Just sayin', so that not everyone tries to get a free lunch out of > > testing the Dummy RA ;-) > > OK. You've been a PM for a while, perhaps you could estimate the > effort needed to setup and run a regression test for db2? Or SAP? We don't need to regress SAP or DB2. We need to regress the resource agent. The "happy path" is probably easy tested for DB2 by anyone who has a DB2 setup, or even with the (supposedly freely available download) test version. The alternate or sad paths could be tested in similar setups. I'm not expecting a test suite that _everyone_ can run - clearly, having access to the software under test would be required. But if we had the test suite, we could ask our maintainers (assuming we have a RA maintainer, or definitely whoever wants to contribute a change - I'd sort of expect a contributor to have a test setup available, and I'd also not be surprised if this is something where Novell/RHT/Canonical QA can chime in) to run the test suite before accepting the patch, and send us the test protocol. The same, essentially, holds true for SAP. Hey, as soon as we make "OK, if you want us to ship your code, you provide us with a reasonable regression test suite" and "your code must match our coding style guidelines and formatting so we can review it sanely" requirements to code inclusion we've created a _very strong_ incentive for people to include the RAs with their upstream projects. That is a win-win for everyone! ;-) Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 06:22:09PM +0100, Dejan Muhamedagic wrote: > Hi Holger, > > On Wed, Feb 09, 2011 at 06:08:51PM +0100, Holger Teutsch wrote: > > Guys, > > coming back from a 3 hour break I'm a bit shocked about the very > > *active* discussion and it is difficult to me to find an entry point. > > No worries, I found it very difficult to follow to. There were > like a few thousand subthreads. ;-) > Right now, I think the best option would be to put your agent in > /usr/lib/ocf/resource.d/testing and ask people to help test it > and log that as well from the old db2 resource agent (sth like > "this db2 RA is about to be deprecated, please try ocf:testing:db2"). Actually, because it is db2, I expect anyone running db2 to have a test system where he will excercise any and all changes to the system, be it config changes, tuning parameters, or package upgrades. So I'm all for doing an inplace replace this time. What can possibly go wrong? * test cluster failing during pre-deployment upgrade tests. Great. That's _exactly_ the feedback we'd need. > > > I know it's somewhat heavy to put the burden of tests on to the new > > > contributor (it'd be better if we already had some to verify what we > > > already ship), but we desperately need test coverage for our RAs, > > > since they provide crucial and essential integration code. > > > > > > To make up for that, I'll personally invite everyone who contributes a > > > reasonably comprehensive, non-trivial test suite for an RA to lunch. > > > > This lunch thing sounds very attractive. > > Aha, so here's that lunch offer :-) Who is cooking? -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
Hi Holger, On Wed, Feb 09, 2011 at 06:08:51PM +0100, Holger Teutsch wrote: > Guys, > coming back from a 3 hour break I'm a bit shocked about the very > *active* discussion and it is difficult to me to find an entry point. No worries, I found it very difficult to follow to. There were like a few thousand subthreads. > On Wed, 2011-02-09 at 14:19 +0100, Lars Marowsky-Bree wrote: > > A reasonable approach - in my opinion - would be to add a few test vcases > > (starting with ocf-tester, and perhaps testing one or two failure > > scenarios), that work with the old code, and which would help prove the > > new RA is compatible and doesn't introduce regressions. Of course, tests > > that also exercise the new functionality are welcome. > > > > If they pass, we can be sure that the new RA is, indeed, a drop-in > > replacement like it should be, and we can avoid shipping both. > > > > I've spent a lot of time to make it plug in compatible to the old agent. > I was the one who contributed the last patches to the old > agent so I know it pretty well. > > I perfectly understand concerns of changing production code. > Nevertheless I would prefer to keep *one* version of the agent. > > So that is what I can offer to provide test cases: > > The code was developed using 2 test clusters with DB2. > > Custer 1) shared disk cluster with multiple partitions > Custer 2) HADR cluster > > Of course I've run ocf-tester and verified with Cluster1) that it is > indeed a plugin replacement. > > I tested tons of crashes as well on both clusters. I have never had any doubts that you did as much testing as humanly possible with the resources you have available. Your contributions did earn you quite a bit of credit here. Even if it were me to have implemented this new db2 agent, I'm sure I wouldn't be able to just replace the old one. It's really not a matter of trust. My experience also tells me that even when one is absolutely sure that there are no problems and that the code was tested in every possible way, there's no way telling what kind of installation people run and what will they do to your code. That's simply so with any new code. And trust me, you wouldn't want to take responsibility for your code without having it out in the field for a while. BTW, only recently I had an issue with the iscsi RA for which I was sure that it was fine and which I tested very thoroughly and it was already out for about two years, just because the user's setup was very unusual. Right now, I think the best option would be to put your agent in /usr/lib/ocf/resource.d/testing and ask people to help test it and log that as well from the old db2 resource agent (sth like "this db2 RA is about to be deprecated, please try ocf:testing:db2"). > If you have some formal requirements for documentation please feel free > to forward them to me. I'm not sure what do you mean here. > > I know it's somewhat heavy to put the burden of tests on to the new > > contributor (it'd be better if we already had some to verify what we > > already ship), but we desperately need test coverage for our RAs, > since > > they provide crucial and essential integration code. > > > > To make up for that, I'll personally invite everyone who contributes a > > reasonably comprehensive, non-trivial test suite for an RA to lunch. > > This lunch thing sounds very attractive. Aha, so here's that lunch offer :-) Cheers, Dejan > Regards > Holger > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 05:54:38PM +0100, Lars Ellenberg wrote:
> On Wed, Feb 09, 2011 at 05:00:31PM +0100, Dejan Muhamedagic wrote:
> > On Wed, Feb 09, 2011 at 03:41:26PM +0100, Lars Ellenberg wrote:
> > > On Wed, Feb 09, 2011 at 02:03:04PM +0100, Dejan Muhamedagic wrote:
> > > > > hadr_vars=$(echo "$output" |\
> > > > > awk '$0 ~/HADR database role/ {printf "HADR_ROLE=%s; ", $NF;}
> > > > > $0 ~/HADR_TIMEOUT/ {printf "HADR_TIMEOUT=%s; ", $NF;}
> > > > > $0 ~/First active log file/ {printf "FIRST_ACTIVE_LOG=%s\n",
> > > > > $NF;}
> > > > > $0 ~/HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW=%s\n",
> > > > > $NF;}')
> > > >
> > > > You can leave out "$0 ~". It is going to work this way too,
> > > > but otherwise it does not look very awk :)
> > > > Also, since it goes to eval below, better add single quotes around
> > > > /HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW='%s'\n" ...
> > > >
> > > > > # sets HADR_ROLE HADR_TIMEOUT HADR_PEER_WINDOW
> > > > > eval $hadr_vars
> > >
> > > Hmm. Since both shell and awk by default do word split on white space,
> > > how about this, without doing eval:
> > > set -- $(echo "$output" |
> > > awk '/HADR database role/ { role=$NF; }
> > >/HADR_TIMEOUT/ { timeout=$NF; }
> > >/First active log file/{ log=$NF; }
> > >/HADR_PEER_WINDOW/ { window=$NF; }
> > >END { printf ".%s\t.%s\t.%s\t.%s\n",
> > > role, timeout, log, window; }')
> > > HADR_ROLE=${1#.}
> > > HADR_TIMEOUT=${2#.}
> > > FIRST_ACTIVE_LOG=${3#.}
> > > HADR_PEER_WINDOW=${4#.}
> >
> > Isn't this a bashism?
>
> No.
> Which part of it seems suspicious to you?
${1#.}
> > Which is absolutely understandable. But note that this input
> > comes from the db2 program, so it shouldn't be totally
> > unexpected.
>
> As long as you can be sure there never will be single quotes part of the
> value, you can try with your suggestiong of '%s'. Otherwise, you'd need
> to first quote any and all singlequotes.
>
> I like mine better, still ;-)
If it can be done without eval and without bash, I'm sold.
Cheers,
Dejan
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> ___
> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
___
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
Guys, coming back from a 3 hour break I'm a bit shocked about the very *active* discussion and it is difficult to me to find an entry point. On Wed, 2011-02-09 at 14:19 +0100, Lars Marowsky-Bree wrote: > A reasonable approach - in my opinion - would be to add a few test vcases > (starting with ocf-tester, and perhaps testing one or two failure > scenarios), that work with the old code, and which would help prove the > new RA is compatible and doesn't introduce regressions. Of course, tests > that also exercise the new functionality are welcome. > > If they pass, we can be sure that the new RA is, indeed, a drop-in > replacement like it should be, and we can avoid shipping both. > I've spent a lot of time to make it plug in compatible to the old agent. I was the one who contributed the last patches to the old agent so I know it pretty well. I perfectly understand concerns of changing production code. Nevertheless I would prefer to keep *one* version of the agent. So that is what I can offer to provide test cases: The code was developed using 2 test clusters with DB2. Custer 1) shared disk cluster with multiple partitions Custer 2) HADR cluster Of course I've run ocf-tester and verified with Cluster1) that it is indeed a plugin replacement. I tested tons of crashes as well on both clusters. If you have some formal requirements for documentation please feel free to forward them to me. I know it's somewhat heavy to put the burden of tests on to the new > contributor (it'd be better if we already had some to verify what we > already ship), but we desperately need test coverage for our RAs, since > they provide crucial and essential integration code. > > To make up for that, I'll personally invite everyone who contributes a > reasonably comprehensive, non-trivial test suite for an RA to lunch. This lunch thing sounds very attractive. Regards Holger ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 05:00:31PM +0100, Dejan Muhamedagic wrote:
> On Wed, Feb 09, 2011 at 03:41:26PM +0100, Lars Ellenberg wrote:
> > On Wed, Feb 09, 2011 at 02:03:04PM +0100, Dejan Muhamedagic wrote:
> > > > hadr_vars=$(echo "$output" |\
> > > > awk '$0 ~/HADR database role/ {printf "HADR_ROLE=%s; ", $NF;}
> > > > $0 ~/HADR_TIMEOUT/ {printf "HADR_TIMEOUT=%s; ", $NF;}
> > > > $0 ~/First active log file/ {printf "FIRST_ACTIVE_LOG=%s\n",
> > > > $NF;}
> > > > $0 ~/HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW=%s\n", $NF;}')
> > >
> > > You can leave out "$0 ~". It is going to work this way too,
> > > but otherwise it does not look very awk :)
> > > Also, since it goes to eval below, better add single quotes around
> > > /HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW='%s'\n" ...
> > >
> > > > # sets HADR_ROLE HADR_TIMEOUT HADR_PEER_WINDOW
> > > > eval $hadr_vars
> >
> > Hmm. Since both shell and awk by default do word split on white space,
> > how about this, without doing eval:
> > set -- $(echo "$output" |
> > awk '/HADR database role/ { role=$NF; }
> > /HADR_TIMEOUT/ { timeout=$NF; }
> > /First active log file/{ log=$NF; }
> > /HADR_PEER_WINDOW/ { window=$NF; }
> > END { printf ".%s\t.%s\t.%s\t.%s\n",
> > role, timeout, log, window; }')
> > HADR_ROLE=${1#.}
> > HADR_TIMEOUT=${2#.}
> > FIRST_ACTIVE_LOG=${3#.}
> > HADR_PEER_WINDOW=${4#.}
>
> Isn't this a bashism?
No.
Which part of it seems suspicious to you?
> Which is absolutely understandable. But note that this input
> comes from the db2 program, so it shouldn't be totally
> unexpected.
As long as you can be sure there never will be single quotes part of the
value, you can try with your suggestiong of '%s'. Otherwise, you'd need
to first quote any and all singlequotes.
I like mine better, still ;-)
--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
___
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 02:47:03PM +0100, Lars Marowsky-Bree wrote: > On 2011-02-09T14:43:17, Andrew Beekhof wrote: > > > It happens often enough - its just normally by a core developer. > > And realistically, almost every RA is going to get similar treatment > > (over time) as they're merged with the Red Hat ones. > > The pending big refactoring & merge highlights just how badly we need > better test coverage. > > The poor db2 contributor just got sucked into this discussion. No > offense ;-) Oh, I think that Holger perfectly understood my concerns. Cheers, Dejan > > Regards, > Lars > > -- > Architect Storage/HA, OPS Engineering, Novell, Inc. > SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) > "Experience is the name everyone gives to their mistakes." -- Oscar Wilde > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 02:21:15PM +0100, Lars Marowsky-Bree wrote: > On 2011-02-09T14:17:58, Dejan Muhamedagic wrote: > > > At any rate, I wouldn't want to take responsibility for replacing > > the existing (and working RA) with a completely new and not yet > > tested code. Call me coward :) > > I agree with that. (Not the coward part, of course!) IMHO though, the > answer is to improve our test coverage to the point where we can > refactor and clean up code without taking an unreasonable risk of > breakage. Dream on ;-) > > Finally, I expected that the new functionality is going to be > > added without much changes to the existing code. But it turned > > out to be a rewrite. > > Yes, but that's an important part of evolving code; probably it could > have also be tacked on somehow, but that doesn't bode well for long-term > maintenance. In that case, it would need to be a set of incremental changes so that each of them can be easy to understand and check and so forth. Hmm, sometimes I feel like preaching to the choire. I wonder why do we discuss all this. > See my other mail for a proposal: the Linux-HA lunch fund for testers > ;-) Must have missed that one. Cheers, Dejan > Regards, > Lars > > -- > Architect Storage/HA, OPS Engineering, Novell, Inc. > SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) > "Experience is the name everyone gives to their mistakes." -- Oscar Wilde > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 02:19:02PM +0100, Lars Marowsky-Bree wrote: > On 2011-02-09T11:56:53, Dejan Muhamedagic wrote: > > > Great! Unfortunately, we can't replace the old db2 now, the > > number of changes is very large: > > That, by itself, doesn't strike me as a reasonable argument for > duplicating the RA. Look, if I could tell that changes were safe, then I certainly wouldn't moan about it. If you can, please go ahead. > It may seem a reasonable idea to protect existing > setups, but consider that it leads to user confusion (which should they > use?), requirements to change setups if they drift apart, and not to > mention code duplication, which is a maintenance nightmare. > > We've done it once (with IPaddr2 and IPaddr), and personally, that has > convinced me that it is a bad idea ;-) And it definitely is a bad idea. But this case is different from IPaddr*. IPaddr* are two RAs covering the same functionality but using different tools. db2* use the same tools. > So, working from the assumption that the new 'db2' RA is supposed to be > backwards-compatible with the existing one, except that it adds new > features: > > A reasonable approach - in my opinion - would be to add a few test cases > (starting with ocf-tester, and perhaps testing one or two failure > scenarios), that work with the old code, and which would help prove the > new RA is compatible and doesn't introduce regressions. Of course, tests > that also exercise the new functionality are welcome. > > If they pass, we can be sure that the new RA is, indeed, a drop-in > replacement like it should be, and we can avoid shipping both. > > I know it's somewhat heavy to put the burden of tests on to the new > contributor (it'd be better if we already had some to verify what we > already ship), but we desperately need test coverage for our RAs, since > they provide crucial and essential integration code. > > To make up for that, I'll personally invite everyone who contributes a > reasonably comprehensive, non-trivial test suite for an RA to lunch. > > (Just sayin', so that not everyone tries to get a free lunch out of > testing the Dummy RA ;-) OK. You've been a PM for a while, perhaps you could estimate the effort needed to setup and run a regression test for db2? Or SAP? Cheers, Dejan > Regards, > Lars > > -- > Architect Storage/HA, OPS Engineering, Novell, Inc. > SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) > "Experience is the name everyone gives to their mistakes." -- Oscar Wilde > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 03:41:26PM +0100, Lars Ellenberg wrote:
> On Wed, Feb 09, 2011 at 02:03:04PM +0100, Dejan Muhamedagic wrote:
> > > hadr_vars=$(echo "$output" |\
> > > awk '$0 ~/HADR database role/ {printf "HADR_ROLE=%s; ", $NF;}
> > > $0 ~/HADR_TIMEOUT/ {printf "HADR_TIMEOUT=%s; ", $NF;}
> > > $0 ~/First active log file/ {printf "FIRST_ACTIVE_LOG=%s\n", $NF;}
> > > $0 ~/HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW=%s\n", $NF;}')
> >
> > You can leave out "$0 ~". It is going to work this way too,
> > but otherwise it does not look very awk :)
> > Also, since it goes to eval below, better add single quotes around
> > /HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW='%s'\n" ...
> >
> > > # sets HADR_ROLE HADR_TIMEOUT HADR_PEER_WINDOW
> > > eval $hadr_vars
>
> Hmm. Since both shell and awk by default do word split on white space,
> how about this, without doing eval:
> set -- $(echo "$output" |
> awk '/HADR database role/ { role=$NF; }
>/HADR_TIMEOUT/ { timeout=$NF; }
>/First active log file/{ log=$NF; }
>/HADR_PEER_WINDOW/ { window=$NF; }
>END { printf ".%s\t.%s\t.%s\t.%s\n",
> role, timeout, log, window; }')
> HADR_ROLE=${1#.}
> HADR_TIMEOUT=${2#.}
> FIRST_ACTIVE_LOG=${3#.}
> HADR_PEER_WINDOW=${4#.}
Isn't this a bashism?
> /me tries to avoid eval wherever possible ;-)
Which is absolutely understandable. But note that this input
comes from the db2 program, so it shouldn't be totally
unexpected.
Cheers,
Dejan
> --
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
>
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> ___
> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
> Home Page: http://linux-ha.org/
___
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 04:01:46PM +0100, Lars Marowsky-Bree wrote: > On 2011-02-09T15:35:01, Lars Ellenberg wrote: > > > > So don't do that :-) > > > Put up a wiki page with instructions for how to download+use the new > > > agent and give feedback. > > > > How about a staging area? > > /usr/lib/ocf/resource.d/staging/ Perhaps "testing" would be better than "staging". The latter sounds to me as something still needing work like "under construction". > > we can also add a > > /usr/lib/ocf/resource.d/deprecated/ Hmm, I'm not sure we need this one, but anyway that's not so important. > I like the idea of putting infrastructure in place to ease canary > testing. Actually, I was also thinking about having another provider for db2. I think that we should do this. > But we probably don't even need the symlink; if they've got to change > something anyway, they can as well change the provider from "stable" to > "staging" (and back, if they want to). > > If the RA is really backwards compatible, they can do that even w/o > service outage - this should cause a reprobe, and if all is well, the > new RA will just attach to the existing instance. Is setting to unmanaged enough in this case? Won't that anyway trigger a restart because the definition changed? > However, see below. > > > Once settled, we copy over the staging one to the "real" directory, > > replacing the "original" one, and add a "please fix your config" to the > > thing that remains in staging/, so we will be able to start a further > > rewrite with the next "merge window". > > This bit I don't like so much: yes, they should explicitly need to > accept that they're participating in the canary testing of a new > version. > > What I'd like to avoid is for them to have to change anything when we > promote the canary-tested version to "stable". > > Two ideas: > > - If we do this via symlinks - e.g., if they switch to the staging > branch for a RA, we adjust the symlink -, they don't need to change > anything. We can adjust the symlink back automatically once that > version has become the stable one. > > Downside - it's a per-node change, and possibly another point of > divergence in the cluster. Positive is that it is a per-node change > and that rolling upgrades are more readily possible. > > - Alternatively, we can add a "version" meta-attribute that is parsed by > the lrmd/OCF plugin to chose which one to run. > > In practice though, both approaches and the whole approach are not > without complexity. Nothing is, but any of these schemes is anyway better than introducing a resource agent with the same functionality and different name. Besides, this should at least be easy to test. And document. > I'd really feel more comfortable if we could instead get reasonable test > coverage so that we don't have to be afraid. The "canary testing" is the > icing on the top, but not a replacement for automatic regression > testing. There's no way we will ever have regression tests for stuff such as db2, oracle, SAP, and similar. I'm not even in a position to do any kind of testing for some of these, because they are too complex to setup and commercial. And so forth. It would be great if the software vendor would ship the RA, but that's also something we cannot expect to happen. Cheers, Dejan > Regards, > Lars > > -- > Architect Storage/HA, OPS Engineering, Novell, Inc. > SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) > "Experience is the name everyone gives to their mistakes." -- Oscar Wilde > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 04:01:46PM +0100, Lars Marowsky-Bree wrote: > In practice though, both approaches and the whole approach are not > without complexity. > > I'd really feel more comfortable if we could instead get reasonable test > coverage so that we don't have to be afraid. The "canary testing" is the > icing on the top, but not a replacement for automatic regression > testing. Absolutely. Please go ahead. -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T15:35:01, Lars Ellenberg wrote: > > So don't do that :-) > > Put up a wiki page with instructions for how to download+use the new > > agent and give feedback. > > How about a staging area? > /usr/lib/ocf/resource.d/staging/ > > we can also add a > /usr/lib/ocf/resource.d/deprecated/ I like the idea of putting infrastructure in place to ease canary testing. But we probably don't even need the symlink; if they've got to change something anyway, they can as well change the provider from "stable" to "staging" (and back, if they want to). If the RA is really backwards compatible, they can do that even w/o service outage - this should cause a reprobe, and if all is well, the new RA will just attach to the existing instance. However, see below. > Once settled, we copy over the staging one to the "real" directory, > replacing the "original" one, and add a "please fix your config" to the > thing that remains in staging/, so we will be able to start a further > rewrite with the next "merge window". This bit I don't like so much: yes, they should explicitly need to accept that they're participating in the canary testing of a new version. What I'd like to avoid is for them to have to change anything when we promote the canary-tested version to "stable". Two ideas: - If we do this via symlinks - e.g., if they switch to the staging branch for a RA, we adjust the symlink -, they don't need to change anything. We can adjust the symlink back automatically once that version has become the stable one. Downside - it's a per-node change, and possibly another point of divergence in the cluster. Positive is that it is a per-node change and that rolling upgrades are more readily possible. - Alternatively, we can add a "version" meta-attribute that is parsed by the lrmd/OCF plugin to chose which one to run. In practice though, both approaches and the whole approach are not without complexity. I'd really feel more comfortable if we could instead get reasonable test coverage so that we don't have to be afraid. The "canary testing" is the icing on the top, but not a replacement for automatic regression testing. Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 9, 2011 at 3:35 PM, Lars Ellenberg wrote: > On Wed, Feb 09, 2011 at 02:43:17PM +0100, Andrew Beekhof wrote: >> >> Are you going to change the name of every agent that gets a rewrite? >> >> >> >> IPaddr2-ng-ng-again-and-one-more-plus-one >> > >> > I don't think it is going to happen that often. >> >> It happens often enough - its just normally by a core developer. >> And realistically, almost every RA is going to get similar treatment >> (over time) as they're merged with the Red Hat ones. >> >> > >> >> Solicit feedback, like was done for kliend's new agent, and replace >> >> the existing one it if/when people respond positively. > > >> >> Its not like the old one disappears from the face of the earth after >> >> you merge the new one. >> >> >> >> wget -o /usr/lib/ocf/resource.d/heartbeat/db2 >> >> http://hg.linux-ha.org/agents/file/agents-1.0.3/heartbeat/db2 >> > >> > What do you suggest? That we add to the release announcement: >> > >> > The db2 RA has been rewritten and didn't get yet a lot of >> > field testing. Please help test it. >> >> So don't do that :-) >> Put up a wiki page with instructions for how to download+use the new >> agent and give feedback. > > How about a staging area? > /usr/lib/ocf/resource.d/staging/ I was thinking along the same lines when I said "testing". Either name works for me :-) > > we can also add a > /usr/lib/ocf/resource.d/deprecated/ > > The thing in .../heartbeat/ can become a symlink, > and be given "config file status" by the package manager? > Something like that. > > So we have it bundled with the release, > it is readily available without much "go to that web page and download > and save to there and make executable and then blah". > > It would simply pop up in crm shell and DRBD-MC and so on. > > We can add "please give feedback" to the description, > and "this will replace the current RA with release + 2 > unless we get veto-ing feedback" to the release notes. > > Once settled, we copy over the staging one to the "real" directory, > replacing the "original" one, and add a "please fix your config" to the > thing that remains in staging/, so we will be able to start a further > rewrite with the next "merge window". > > * does not break existing setups > * new RAs and rewrites are readily available > > > -- > : Lars Ellenberg > : LINBIT | Your Way to High Availability > : DRBD/HA support and consulting http://www.linbit.com > > DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ > ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 02:03:04PM +0100, Dejan Muhamedagic wrote:
> > hadr_vars=$(echo "$output" |\
> > awk '$0 ~/HADR database role/ {printf "HADR_ROLE=%s; ", $NF;}
> > $0 ~/HADR_TIMEOUT/ {printf "HADR_TIMEOUT=%s; ", $NF;}
> > $0 ~/First active log file/ {printf "FIRST_ACTIVE_LOG=%s\n", $NF;}
> > $0 ~/HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW=%s\n", $NF;}')
>
> You can leave out "$0 ~". It is going to work this way too,
> but otherwise it does not look very awk :)
> Also, since it goes to eval below, better add single quotes around
> /HADR_PEER_WINDOW/ {printf "HADR_PEER_WINDOW='%s'\n" ...
>
> > # sets HADR_ROLE HADR_TIMEOUT HADR_PEER_WINDOW
> > eval $hadr_vars
Hmm. Since both shell and awk by default do word split on white space,
how about this, without doing eval:
set -- $(echo "$output" |
awk '/HADR database role/ { role=$NF; }
/HADR_TIMEOUT/ { timeout=$NF; }
/First active log file/{ log=$NF; }
/HADR_PEER_WINDOW/ { window=$NF; }
END { printf ".%s\t.%s\t.%s\t.%s\n",
role, timeout, log, window; }')
HADR_ROLE=${1#.}
HADR_TIMEOUT=${2#.}
FIRST_ACTIVE_LOG=${3#.}
HADR_PEER_WINDOW=${4#.}
/me tries to avoid eval wherever possible ;-)
--
: Lars Ellenberg
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting http://www.linbit.com
DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
___
Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org
http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev
Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 02:43:17PM +0100, Andrew Beekhof wrote: > >> Are you going to change the name of every agent that gets a rewrite? > >> > >> IPaddr2-ng-ng-again-and-one-more-plus-one > > > > I don't think it is going to happen that often. > > It happens often enough - its just normally by a core developer. > And realistically, almost every RA is going to get similar treatment > (over time) as they're merged with the Red Hat ones. > > > > >> Solicit feedback, like was done for kliend's new agent, and replace > >> the existing one it if/when people respond positively. > >> Its not like the old one disappears from the face of the earth after > >> you merge the new one. > >> > >> wget -o /usr/lib/ocf/resource.d/heartbeat/db2 > >> http://hg.linux-ha.org/agents/file/agents-1.0.3/heartbeat/db2 > > > > What do you suggest? That we add to the release announcement: > > > > The db2 RA has been rewritten and didn't get yet a lot of > > field testing. Please help test it. > > So don't do that :-) > Put up a wiki page with instructions for how to download+use the new > agent and give feedback. How about a staging area? /usr/lib/ocf/resource.d/staging/ we can also add a /usr/lib/ocf/resource.d/deprecated/ The thing in .../heartbeat/ can become a symlink, and be given "config file status" by the package manager? Something like that. So we have it bundled with the release, it is readily available without much "go to that web page and download and save to there and make executable and then blah". It would simply pop up in crm shell and DRBD-MC and so on. We can add "please give feedback" to the description, and "this will replace the current RA with release + 2 unless we get veto-ing feedback" to the release notes. Once settled, we copy over the staging one to the "real" directory, replacing the "original" one, and add a "please fix your config" to the thing that remains in staging/, so we will be able to start a further rewrite with the next "merge window". * does not break existing setups * new RAs and rewrites are readily available -- : Lars Ellenberg : LINBIT | Your Way to High Availability : DRBD/HA support and consulting http://www.linbit.com DRBD® and LINBIT® are registered trademarks of LINBIT, Austria. ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T14:43:17, Andrew Beekhof wrote: > It happens often enough - its just normally by a core developer. > And realistically, almost every RA is going to get similar treatment > (over time) as they're merged with the Red Hat ones. The pending big refactoring & merge highlights just how badly we need better test coverage. The poor db2 contributor just got sucked into this discussion. No offense ;-) Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 9, 2011 at 2:17 PM, Dejan Muhamedagic wrote: > Hi Andrew, > > On Wed, Feb 09, 2011 at 01:33:03PM +0100, Andrew Beekhof wrote: >> On Wed, Feb 9, 2011 at 12:15 PM, Dejan Muhamedagic >> wrote: >> > On Wed, Feb 09, 2011 at 12:06:04PM +0100, Florian Haas wrote: >> >> On 2011-02-09 11:56, Dejan Muhamedagic wrote: >> >> >> It is plugin compatible to the old version of the agent. >> >> > >> >> > Great! Unfortunately, we can't replace the old db2 now, the >> >> > number of changes is very large: >> >> > >> >> > db2 | 1076 >> >> > +++- >> >> > 1 file changed, 687 insertions(+), 389 deletions(-) >> >> > >> >> > And the code is completely new (though I have no doubt that it is >> >> > of excellent quality). So, I'd suggest to add this as another db2 >> >> > RA. Once it gets some field testing we can mark the old one as >> >> > deprecated. What name would you suggest? db2db2? >> >> >> >> Just making sure: Is that a joke? >> > >> > A bit of a joke, yes. But the alternatives such as db22 or db2new >> > looked a bit boring. >> >> I think boring is the least of our problems with those names. >> Are you going to change the name of every agent that gets a rewrite? >> >> IPaddr2-ng-ng-again-and-one-more-plus-one > > I don't think it is going to happen that often. It happens often enough - its just normally by a core developer. And realistically, almost every RA is going to get similar treatment (over time) as they're merged with the Red Hat ones. > >> Solicit feedback, like was done for kliend's new agent, and replace >> the existing one it if/when people respond positively. > > That would be for the best, but it takes time. We may opt for it, > but I wanted to add the this agent to the new release. Understood - but I think the long-term pain that is created outweighs any perceived benefit in the short-term. > Also, it > is very seldom that people test anything which is not contained > in the release. Unless there's no alternative as was the case > with conntrac. > >> Its not like the old one disappears from the face of the earth after >> you merge the new one. >> >> wget -o /usr/lib/ocf/resource.d/heartbeat/db2 >> http://hg.linux-ha.org/agents/file/agents-1.0.3/heartbeat/db2 > > What do you suggest? That we add to the release announcement: > > The db2 RA has been rewritten and didn't get yet a lot of > field testing. Please help test it. So don't do that :-) Put up a wiki page with instructions for how to download+use the new agent and give feedback. If the new version is significantly better, you're going to hear people pleading for its inclusion pretty soon. > But, if you want to keep > the old agent, download the old one from the repository and > use it instead of the new one. And don't forget to do the > same when installing the next resource-agents release. > > At any rate, I wouldn't want to take responsibility for replacing > the existing (and working RA) with a completely new and not yet > tested code. Call me coward :) I wouldn't either - which is why I keep saying "test then replace" :-) Another alternative, create a "testing" provider... not sure if its a good idea or not, just putting it out there. > Finally, I expected that the new functionality is going to be > added without much changes to the existing code. But it turned > out to be a rewrite. > > Cheers, > > Dejan > >> >> > HADR is a very different beast from non-HADR db, right? Why not >> >> > then add the "hadr" boolean parameter and use that instead of >> >> > checking if the resource has been configured as multi-state? >> >> >> >> I'll take responsibility for suggesting the use of ocf_is_ms(), and I'd >> >> be curious to find out what you think is wrong with that approach. >> > >> > There's nothing wrong in the sense whether it is going to work. >> > But someday, db2 may sport say HADR2 or VHA or whatever else >> > which may run as a ms resource. I just think that it's better to >> > make it obvious in the configuration that the user runs HADR. >> > Does that make sense? >> > >> >> Because if anything is, then the mysql RA needs fixing too. >> > >> > No idea what's up with mysql. >> > >> > Cheers, >> > >> > Dejan >> > >> >> Florian >> >> >> > >> > >> > >> >> ___ >> >> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org >> >> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev >> >> Home Page: http://linux-ha.org/ >> > >> > ___ >> > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org >> > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev >> > Home Page: http://linux-ha.org/ >> > >> ___ >> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org >> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev >> Home Page: http://linux-ha.org/ > ___
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T14:17:58, Dejan Muhamedagic wrote: > At any rate, I wouldn't want to take responsibility for replacing > the existing (and working RA) with a completely new and not yet > tested code. Call me coward :) I agree with that. (Not the coward part, of course!) IMHO though, the answer is to improve our test coverage to the point where we can refactor and clean up code without taking an unreasonable risk of breakage. > Finally, I expected that the new functionality is going to be > added without much changes to the existing code. But it turned > out to be a rewrite. Yes, but that's an important part of evolving code; probably it could have also be tacked on somehow, but that doesn't bode well for long-term maintenance. See my other mail for a proposal: the Linux-HA lunch fund for testers ;-) Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09T11:56:53, Dejan Muhamedagic wrote: > Great! Unfortunately, we can't replace the old db2 now, the > number of changes is very large: That, by itself, doesn't strike me as a reasonable argument for duplicating the RA. It may seem a reasonable idea to protect existing setups, but consider that it leads to user confusion (which should they use?), requirements to change setups if they drift apart, and not to mention code duplication, which is a maintenance nightmare. We've done it once (with IPaddr2 and IPaddr), and personally, that has convinced me that it is a bad idea ;-) So, working from the assumption that the new 'db2' RA is supposed to be backwards-compatible with the existing one, except that it adds new features: A reasonable approach - in my opinion - would be to add a few test cases (starting with ocf-tester, and perhaps testing one or two failure scenarios), that work with the old code, and which would help prove the new RA is compatible and doesn't introduce regressions. Of course, tests that also exercise the new functionality are welcome. If they pass, we can be sure that the new RA is, indeed, a drop-in replacement like it should be, and we can avoid shipping both. I know it's somewhat heavy to put the burden of tests on to the new contributor (it'd be better if we already had some to verify what we already ship), but we desperately need test coverage for our RAs, since they provide crucial and essential integration code. To make up for that, I'll personally invite everyone who contributes a reasonably comprehensive, non-trivial test suite for an RA to lunch. (Just sayin', so that not everyone tries to get a free lunch out of testing the Dummy RA ;-) Regards, Lars -- Architect Storage/HA, OPS Engineering, Novell, Inc. SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) "Experience is the name everyone gives to their mistakes." -- Oscar Wilde ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
Hi Andrew, On Wed, Feb 09, 2011 at 01:33:03PM +0100, Andrew Beekhof wrote: > On Wed, Feb 9, 2011 at 12:15 PM, Dejan Muhamedagic > wrote: > > On Wed, Feb 09, 2011 at 12:06:04PM +0100, Florian Haas wrote: > >> On 2011-02-09 11:56, Dejan Muhamedagic wrote: > >> >> It is plugin compatible to the old version of the agent. > >> > > >> > Great! Unfortunately, we can't replace the old db2 now, the > >> > number of changes is very large: > >> > > >> > db2 | 1076 > >> > +++- > >> > 1 file changed, 687 insertions(+), 389 deletions(-) > >> > > >> > And the code is completely new (though I have no doubt that it is > >> > of excellent quality). So, I'd suggest to add this as another db2 > >> > RA. Once it gets some field testing we can mark the old one as > >> > deprecated. What name would you suggest? db2db2? > >> > >> Just making sure: Is that a joke? > > > > A bit of a joke, yes. But the alternatives such as db22 or db2new > > looked a bit boring. > > I think boring is the least of our problems with those names. > Are you going to change the name of every agent that gets a rewrite? > >IPaddr2-ng-ng-again-and-one-more-plus-one I don't think it is going to happen that often. > Solicit feedback, like was done for kliend's new agent, and replace > the existing one it if/when people respond positively. That would be for the best, but it takes time. We may opt for it, but I wanted to add the this agent to the new release. Also, it is very seldom that people test anything which is not contained in the release. Unless there's no alternative as was the case with conntrac. > Its not like the old one disappears from the face of the earth after > you merge the new one. > >wget -o /usr/lib/ocf/resource.d/heartbeat/db2 > http://hg.linux-ha.org/agents/file/agents-1.0.3/heartbeat/db2 What do you suggest? That we add to the release announcement: The db2 RA has been rewritten and didn't get yet a lot of field testing. Please help test it. But, if you want to keep the old agent, download the old one from the repository and use it instead of the new one. And don't forget to do the same when installing the next resource-agents release. At any rate, I wouldn't want to take responsibility for replacing the existing (and working RA) with a completely new and not yet tested code. Call me coward :) Finally, I expected that the new functionality is going to be added without much changes to the existing code. But it turned out to be a rewrite. Cheers, Dejan > >> > HADR is a very different beast from non-HADR db, right? Why not > >> > then add the "hadr" boolean parameter and use that instead of > >> > checking if the resource has been configured as multi-state? > >> > >> I'll take responsibility for suggesting the use of ocf_is_ms(), and I'd > >> be curious to find out what you think is wrong with that approach. > > > > There's nothing wrong in the sense whether it is going to work. > > But someday, db2 may sport say HADR2 or VHA or whatever else > > which may run as a ms resource. I just think that it's better to > > make it obvious in the configuration that the user runs HADR. > > Does that make sense? > > > >> Because if anything is, then the mysql RA needs fixing too. > > > > No idea what's up with mysql. > > > > Cheers, > > > > Dejan > > > >> Florian > >> > > > > > > > >> ___ > >> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > >> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > >> Home Page: http://linux-ha.org/ > > > > ___ > > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > > Home Page: http://linux-ha.org/ > > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 12:33:19PM +0100, Holger Teutsch wrote: > On Wed, 2011-02-09 at 11:56 +0100, Dejan Muhamedagic wrote: > > Hi Holger, > > > > On Wed, Feb 09, 2011 at 10:23:06AM +0100, Holger Teutsch wrote: > > > Dejan + Florian, > > > please find attached the RA agent for db2 that supports HADR as well. > > > > > > It is plugin compatible to the old version of the agent. > > > > Great! Unfortunately, we can't replace the old db2 now, the > > number of changes is very large: > > > > db2 | 1076 > > +++- > > 1 file changed, 687 insertions(+), 389 deletions(-) > > > > And the code is completely new (though I have no doubt that it is > > of excellent quality). So, I'd suggest to add this as another db2 > > RA. Once it gets some field testing we can mark the old one as > > deprecated. What name would you suggest? db2db2? > > db2-ng ? Yes, db2-ng sounds fine. > > > Here is how it > > > works. > > > > > > The old agent has instance parameters: > > > > > > instance (r) > > > admin (o) Boils down to use value of admin instead of value > > > of instance > > > > Oops. Strange. > > > > > dbpartitionnum(o) defaults to 0 > > > > > > The old agent starts all defined databases in the instance, they fail > > > together as well. > > > > > > The new agent has instance parameters: > > > > > > instance (r) > > > admin (o) retained same behavior for compatibility but > > > flagged and logged as deprecated > > > > How often people actually use different instance and admin name? > > IIRC, all db2 installations I saw had them equal. Furthermore, > > instance is also user input, so I can't see any sense in > > overwriting that. I'm quite confused. At any rate, if we're > > adding this as a separate agent, let's try to do the most > > sensible thing. > > With standard setup (since I know DB2 V8 in 2003) this is not possible > at all. If it will be a new agent I suggest to remove this altogether. Agreed. > > > dbpartitionnum(o) defaults to 0 > > > dblist(o) defaults to all defined databases in the instance > > > > > > When configured as primitive without dblist: exactly the same behavior as > > > the old agent. > > > When configured as primitive with dblist: only starts the listed > > > databases which fail together as well. > > > When configured as m/s and (the resulting dblist) containing excatly one > > > value: implement HADR pair. > > > > HADR is a very different beast from non-HADR db, right? Why not > > then add the "hadr" boolean parameter and use that instead of > > checking if the resource has been configured as multi-state? Then > > the RA can complain if the resource is not ms. And in the > > configuration it is going to be obvious that this is a HADR > > instance. > > > > I would like stick with the ocf_is_ms logic in order to avoid > parameters. I don't think that would be too much of an overhead. And it would make the user's intention clear. As I replied earlier to Florian, there's also a possibility that once in future db2 gets another mode which may be supported in an ms setup. Makes sense? Cheers, Dejan > > I'll write again after reviewing the code. > > > > Many thanks for the contribution and effort! > > > > Cheers, > > > > Dejan > > > > > ___ > > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > > Home Page: http://linux-ha.org/ > > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
Hi,
On Wed, Feb 09, 2011 at 10:23:06AM +0100, Holger Teutsch wrote:
> On Tue, 2011-02-08 at 14:13 +0100, Holger Teutsch wrote:
> > On Tue, 2011-02-08 at 12:12 +0100, Dejan Muhamedagic wrote:
> > >
> > > We're about to finally tag the new release by the end of this
> > > week and it'd of course be great to have this one on board.
> > > Cheers,
> > >
> > > Dejan
> > >
> > I'm in the final compatibility tests right now. I will submit it
> > tomorrow.
> >
> > - holger
> >
> Dejan + Florian,
> please find attached the RA agent for db2 that supports HADR as well.
>
Please find below some comments.
> #!/bin/sh
> #
> # db2
> #
> # Resource agent that manages a DB2 LUW database in Standard role
> # or HADR configuration in master/slave configuration.
> # Multi partition is supported as well.
> #
> # Copyright (c) 2011 Holger Teutsch
> #
> # This agent incoporates code of a previous release created by
> # Alan Robertson and the community.
> #
> # This program is free software; you can redistribute it and/or modify
> # it under the terms of version 2 of the GNU General Public License as
> # published by the Free Software Foundation.
> #
> # This program is distributed in the hope that it would be useful, but
> # WITHOUT ANY WARRANTY; without even the implied warranty of
> # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> #
> # Further, this software is distributed without any warranty that it is
> # free of the rightful claim of any third person regarding infringement
> # or the like. Any license provided herein, whether implied or
> # otherwise, applies only to this software file. Patent licenses, if
> # any, provided herein do not apply to combinations of this program with
> # other software, or any other product whatsoever.
> #
> # You should have received a copy of the GNU General Public License
> # along with this program; if not, write the Free Software Foundation, Inc.,
> # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
> #
>
> ###
> # Initialization:
>
> : ${OCF_FUNCTIONS_DIR=${OCF_ROOT}/resource.d/heartbeat}
> . ${OCF_FUNCTIONS_DIR}/.ocf-shellfuncs
>
> ###
>
>
> db2_usage() {
> echo "db2 start|stop|monitor|promote|demote|validate-all|meta-data"
> }
>
> db2_meta_data() {
> cat <
>
>
> 1.0
>
> Resource Agent that manages an IBM DB2 LUW database in Standard role as
> primitive or HADR roles in master/slave configuration.
>
> Multiple partitions are supported as well. Configure each partition as a
> separate resource.
>
> In case of HADR be very deliberate in specifying intervals/timeouts. The
> detection of a failure including promote must complete within
> HADR_PEER_WINDOW.
>
> In addition to honoring requirements for crash recovery etc. for your
> specific database use the following relations as guidance:
>
> "monitor interval" < HADR_PEER_WINDOW - (appr 30 sec)
> "promote timeout" < HADR_PEER_WINDOW + (appr 20 sec)
>
> For further information and examples consult
> http://www.linux-ha.org/wiki/db2_(resource_agent)
>
> Manages an IBM DB2 LUW database in Standard or HADR
> mode
>
>
>
>
> The instance of the database(s).
>
> instance
>
>
>
>
> List of databases to be managed, e.g "db1 db2".
> Defaults to all databases in the instance. Specify one db for HADR mode.
>
> List of databases to be managed
>
>
>
>
> DEPRECATED: The admin user of the instance.
>
> DEPRECATED: admin
>
>
>
>
> The number of the partion (DBPARTITIONNUM) to be managed.
>
> number of partion
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> END
> }
>
> #
> # validate
> # .. and set global variables
> #
> # exit on error
> #
> db2_validate() {
> local db2home db2sql db2instance
>
> # check required instance vars
> if [ -z "$OCF_RESKEY_instance" ]
> then
> ocf_log err "DB2 required parameter instance is not set!"
> return $OCF_ERR_CONFIGURED
> fi
>
> instance=$OCF_RESKEY_instance
> if [ -n "$OCF_RESKEY_admin" ]
> then
> ocf_log warn "DB2 deprecated parameter admin is set, using
> $OCF_RESKEY_admin as instance."
> instance=$OCF_RESKEY_admin
> fi
>
> db2node=${OCF_RESKEY_dbpartitionnum:-0}
>
> db2home=$(sh -c "echo ~$instance")
> db2sql=$db2home/sqllib
> db2profile=$db2sql/db2profile
> db2bin=$db2sql/bin
>
> STATE_FILE=${HA_RSCTMP}/Dummy-${OCF_RESOURCE_INSTANCE}.state
Why Dummy? Better use db2.
> # Let's make sure a few important things are there...
> if ! [ -d "$db2sql" -a -d "$db2bin" -a -f "$db2profile" -a \
>-x "$db2profile" -a -x "$db2bin/db2" ]
> then
> ocf_is_probe && exit $OCF_NOT_RUNNING
> ocf_log err "DB2 required directories and/or files not found"
> exit $OCF_ERR_INSTALLED
> fi
>
> db2instance=$(runasdb2 'echo $DB2INSTANCE')
> if [ "$db2instance" != "$instance" ]
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 9, 2011 at 12:15 PM, Dejan Muhamedagic wrote: > On Wed, Feb 09, 2011 at 12:06:04PM +0100, Florian Haas wrote: >> On 2011-02-09 11:56, Dejan Muhamedagic wrote: >> >> It is plugin compatible to the old version of the agent. >> > >> > Great! Unfortunately, we can't replace the old db2 now, the >> > number of changes is very large: >> > >> > db2 | 1076 >> > +++- >> > 1 file changed, 687 insertions(+), 389 deletions(-) >> > >> > And the code is completely new (though I have no doubt that it is >> > of excellent quality). So, I'd suggest to add this as another db2 >> > RA. Once it gets some field testing we can mark the old one as >> > deprecated. What name would you suggest? db2db2? >> >> Just making sure: Is that a joke? > > A bit of a joke, yes. But the alternatives such as db22 or db2new > looked a bit boring. I think boring is the least of our problems with those names. Are you going to change the name of every agent that gets a rewrite? IPaddr2-ng-ng-again-and-one-more-plus-one Solicit feedback, like was done for kliend's new agent, and replace the existing one it if/when people respond positively. Its not like the old one disappears from the face of the earth after you merge the new one. wget -o /usr/lib/ocf/resource.d/heartbeat/db2 http://hg.linux-ha.org/agents/file/agents-1.0.3/heartbeat/db2 > >> > HADR is a very different beast from non-HADR db, right? Why not >> > then add the "hadr" boolean parameter and use that instead of >> > checking if the resource has been configured as multi-state? >> >> I'll take responsibility for suggesting the use of ocf_is_ms(), and I'd >> be curious to find out what you think is wrong with that approach. > > There's nothing wrong in the sense whether it is going to work. > But someday, db2 may sport say HADR2 or VHA or whatever else > which may run as a ms resource. I just think that it's better to > make it obvious in the configuration that the user runs HADR. > Does that make sense? > >> Because if anything is, then the mysql RA needs fixing too. > > No idea what's up with mysql. > > Cheers, > > Dejan > >> Florian >> > > > >> ___ >> Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org >> http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev >> Home Page: http://linux-ha.org/ > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ > ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, 2011-02-09 at 11:56 +0100, Dejan Muhamedagic wrote: > Hi Holger, > > On Wed, Feb 09, 2011 at 10:23:06AM +0100, Holger Teutsch wrote: > > Dejan + Florian, > > please find attached the RA agent for db2 that supports HADR as well. > > > > It is plugin compatible to the old version of the agent. > > Great! Unfortunately, we can't replace the old db2 now, the > number of changes is very large: > > db2 | 1076 > +++- > 1 file changed, 687 insertions(+), 389 deletions(-) > > And the code is completely new (though I have no doubt that it is > of excellent quality). So, I'd suggest to add this as another db2 > RA. Once it gets some field testing we can mark the old one as > deprecated. What name would you suggest? db2db2? db2-ng ? > > > Here is how it > > works. > > > > The old agent has instance parameters: > > > > instance(r) > > admin (o) Boils down to use value of admin instead of value > > of instance > > Oops. Strange. > > > dbpartitionnum (o) defaults to 0 > > > > The old agent starts all defined databases in the instance, they fail > > together as well. > > > > The new agent has instance parameters: > > > > instance(r) > > admin (o) retained same behavior for compatibility but > > flagged and logged as deprecated > > How often people actually use different instance and admin name? > IIRC, all db2 installations I saw had them equal. Furthermore, > instance is also user input, so I can't see any sense in > overwriting that. I'm quite confused. At any rate, if we're > adding this as a separate agent, let's try to do the most > sensible thing. With standard setup (since I know DB2 V8 in 2003) this is not possible at all. If it will be a new agent I suggest to remove this altogether. > > > dbpartitionnum (o) defaults to 0 > > dblist (o) defaults to all defined databases in the instance > > > > When configured as primitive without dblist: exactly the same behavior as > > the old agent. > > When configured as primitive with dblist: only starts the listed databases > > which fail together as well. > > When configured as m/s and (the resulting dblist) containing excatly one > > value: implement HADR pair. > > HADR is a very different beast from non-HADR db, right? Why not > then add the "hadr" boolean parameter and use that instead of > checking if the resource has been configured as multi-state? Then > the RA can complain if the resource is not ms. And in the > configuration it is going to be obvious that this is a HADR > instance. > I would like stick with the ocf_is_ms logic in order to avoid parameters. > I'll write again after reviewing the code. > > Many thanks for the contribution and effort! > > Cheers, > > Dejan > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Wed, Feb 09, 2011 at 12:06:04PM +0100, Florian Haas wrote: > On 2011-02-09 11:56, Dejan Muhamedagic wrote: > >> It is plugin compatible to the old version of the agent. > > > > Great! Unfortunately, we can't replace the old db2 now, the > > number of changes is very large: > > > > db2 | 1076 > > +++- > > 1 file changed, 687 insertions(+), 389 deletions(-) > > > > And the code is completely new (though I have no doubt that it is > > of excellent quality). So, I'd suggest to add this as another db2 > > RA. Once it gets some field testing we can mark the old one as > > deprecated. What name would you suggest? db2db2? > > Just making sure: Is that a joke? A bit of a joke, yes. But the alternatives such as db22 or db2new looked a bit boring. > > HADR is a very different beast from non-HADR db, right? Why not > > then add the "hadr" boolean parameter and use that instead of > > checking if the resource has been configured as multi-state? > > I'll take responsibility for suggesting the use of ocf_is_ms(), and I'd > be curious to find out what you think is wrong with that approach. There's nothing wrong in the sense whether it is going to work. But someday, db2 may sport say HADR2 or VHA or whatever else which may run as a ms resource. I just think that it's better to make it obvious in the configuration that the user runs HADR. Does that make sense? > Because if anything is, then the mysql RA needs fixing too. No idea what's up with mysql. Cheers, Dejan > Florian > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On 2011-02-09 11:56, Dejan Muhamedagic wrote: >> It is plugin compatible to the old version of the agent. > > Great! Unfortunately, we can't replace the old db2 now, the > number of changes is very large: > > db2 | 1076 > +++- > 1 file changed, 687 insertions(+), 389 deletions(-) > > And the code is completely new (though I have no doubt that it is > of excellent quality). So, I'd suggest to add this as another db2 > RA. Once it gets some field testing we can mark the old one as > deprecated. What name would you suggest? db2db2? Just making sure: Is that a joke? > HADR is a very different beast from non-HADR db, right? Why not > then add the "hadr" boolean parameter and use that instead of > checking if the resource has been configured as multi-state? I'll take responsibility for suggesting the use of ocf_is_ms(), and I'd be curious to find out what you think is wrong with that approach. Because if anything is, then the mysql RA needs fixing too. Florian signature.asc Description: OpenPGP digital signature ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
Hi Holger, On Wed, Feb 09, 2011 at 10:23:06AM +0100, Holger Teutsch wrote: > On Tue, 2011-02-08 at 14:13 +0100, Holger Teutsch wrote: > > On Tue, 2011-02-08 at 12:12 +0100, Dejan Muhamedagic wrote: > > > > > > We're about to finally tag the new release by the end of this > > > week and it'd of course be great to have this one on board. > > > Cheers, > > > > > > Dejan > > > > > I'm in the final compatibility tests right now. I will submit it > > tomorrow. > > > > - holger > > > Dejan + Florian, > please find attached the RA agent for db2 that supports HADR as well. > > It is plugin compatible to the old version of the agent. Great! Unfortunately, we can't replace the old db2 now, the number of changes is very large: db2 | 1076 +++- 1 file changed, 687 insertions(+), 389 deletions(-) And the code is completely new (though I have no doubt that it is of excellent quality). So, I'd suggest to add this as another db2 RA. Once it gets some field testing we can mark the old one as deprecated. What name would you suggest? db2db2? > Here is how it > works. > > The old agent has instance parameters: > > instance (r) > admin (o) Boils down to use value of admin instead of value of > instance Oops. Strange. > dbpartitionnum(o) defaults to 0 > > The old agent starts all defined databases in the instance, they fail > together as well. > > The new agent has instance parameters: > > instance (r) > admin (o) retained same behavior for compatibility but flagged and > logged as deprecated How often people actually use different instance and admin name? IIRC, all db2 installations I saw had them equal. Furthermore, instance is also user input, so I can't see any sense in overwriting that. I'm quite confused. At any rate, if we're adding this as a separate agent, let's try to do the most sensible thing. > dbpartitionnum(o) defaults to 0 > dblist(o) defaults to all defined databases in the instance > > When configured as primitive without dblist: exactly the same behavior as the > old agent. > When configured as primitive with dblist: only starts the listed databases > which fail together as well. > When configured as m/s and (the resulting dblist) containing excatly one > value: implement HADR pair. HADR is a very different beast from non-HADR db, right? Why not then add the "hadr" boolean parameter and use that instead of checking if the resource has been configured as multi-state? Then the RA can complain if the resource is not ms. And in the configuration it is going to be obvious that this is a HADR instance. I'll write again after reviewing the code. Many thanks for the contribution and effort! Cheers, Dejan > Regards > Holger > > ___ > Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org > http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev > Home Page: http://linux-ha.org/ ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
Re: [Linux-ha-dev] New master/slave resource agent for DB2 databases in HADR (High Availability Disaster Recovery) mode
On Tue, 2011-02-08 at 14:13 +0100, Holger Teutsch wrote: > On Tue, 2011-02-08 at 12:12 +0100, Dejan Muhamedagic wrote: > > > > We're about to finally tag the new release by the end of this > > week and it'd of course be great to have this one on board. > > Cheers, > > > > Dejan > > > I'm in the final compatibility tests right now. I will submit it > tomorrow. > > - holger > Dejan + Florian, please find attached the RA agent for db2 that supports HADR as well. It is plugin compatible to the old version of the agent. Here is how it works. The old agent has instance parameters: instance(r) admin (o) Boils down to use value of admin instead of value of instance dbpartitionnum (o) defaults to 0 The old agent starts all defined databases in the instance, they fail together as well. The new agent has instance parameters: instance(r) admin (o) retained same behavior for compatibility but flagged and logged as deprecated dbpartitionnum (o) defaults to 0 dblist (o) defaults to all defined databases in the instance When configured as primitive without dblist: exactly the same behavior as the old agent. When configured as primitive with dblist: only starts the listed databases which fail together as well. When configured as m/s and (the resulting dblist) containing excatly one value: implement HADR pair. Regards Holger db2 Description: application/shellscript ___ Linux-HA-Dev: Linux-HA-Dev@lists.linux-ha.org http://lists.linux-ha.org/mailman/listinfo/linux-ha-dev Home Page: http://linux-ha.org/
