Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
With respect to using a seed - the facility to supply one to the
rebalance operation has recently been added to puppet-swift master
branch (commit b8b4434), however the seed parameter is not available to
any of the usual calling methods (this looks to be deliberate from the
commit message), so is not immediately useful without surgery :-)
Regards
Mark
On 13/06/15 18:05, Mark Kirkwood wrote:
From what I can see, the ring gets created and rebalanced in
puppet-swift/manifest/ringbuilder.pp i.e calling:
class { '::swift::ringbuilder':
# the part power should be determined by assuming 100 partitions
per drive
part_power = '18',
replicas = '3',
min_part_hours = 1,
require= Class['swift'],
}
*not* when each device is added.
Yeah, using a seed is probably a good solution too. For the moment I'm
using the idea of one proxy being a 'ring server/master' which achieves
the same thing (identical rings everywhere). However I'll have a look at
using a seed, as this may simplify the code and also the operational
procedure needed to replace said 'master' if it fails (i.e to avoid
accidentally creating a new ring when you really don't need to...)
Regards,
Mark
On 12/06/15 23:10, McCabe, Donagh wrote:
I skimmed the code, but since I'm not familiar with the environment, I
could not find where swift-ring-builder rebalance is invoked. I'm
guessing that each time you add a device to a ring, a rebalance is
also done. Leaving aside how inefficient that is, the key thing is
that the rebalance command has an optional seed parameter. Unless
you explicitly set the seed (to same value on all node obviously), you
won't get the same ring on all nodes. You also need to make sure you
add the same set of drives and in same order.
Regards,
Donagh
-Original Message-
From: Mark Kirkwood [mailto:mark.kirkw...@catalyst.net.nz]
Sent: 12 June 2015 06:28
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will
create out of sync rings
I've looking at using puppet-swift to deploy a swift cluster.
Firstly - without
http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp
I would have struggled a great deal more to get up and running, so a
big thank you for a nice worked example of how to do multiple nodes!
However I have stumbled upon a problem - with respect to creating
multiple proxy nodes. There are some recipes around that follow on
from the site.pp above and explicitly build 1 proxy (e.g
https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp)
Now the problem is - each proxy node does a ring builder create, so
ends up with *different* builder (and therefore) ring files. This is
not good, as the end result is a cluster with all storage nodes and
*one* proxy with the same set of ring files, and *all* other proxies with
*different* ring (and builder) files.
I have used logic similar to the attached to work around this, i.e
only create rings if we are the 'ring server', otherwise get 'em via
rsync.
Thoughts?
Regards
Mark
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
From what I can see, the ring gets created and rebalanced in
puppet-swift/manifest/ringbuilder.pp i.e calling:
class { '::swift::ringbuilder':
# the part power should be determined by assuming 100 partitions
per drive
part_power = '18',
replicas = '3',
min_part_hours = 1,
require= Class['swift'],
}
*not* when each device is added.
Yeah, using a seed is probably a good solution too. For the moment I'm
using the idea of one proxy being a 'ring server/master' which achieves
the same thing (identical rings everywhere). However I'll have a look at
using a seed, as this may simplify the code and also the operational
procedure needed to replace said 'master' if it fails (i.e to avoid
accidentally creating a new ring when you really don't need to...)
Regards,
Mark
On 12/06/15 23:10, McCabe, Donagh wrote:
I skimmed the code, but since I'm not familiar with the environment, I could not find where
swift-ring-builder rebalance is invoked. I'm guessing that each time you add a device
to a ring, a rebalance is also done. Leaving aside how inefficient that is, the key thing is that
the rebalance command has an optional seed parameter. Unless you explicitly set the
seed (to same value on all node obviously), you won't get the same ring on all nodes. You also need
to make sure you add the same set of drives and in same order.
Regards,
Donagh
-Original Message-
From: Mark Kirkwood [mailto:mark.kirkw...@catalyst.net.nz]
Sent: 12 June 2015 06:28
To: OpenStack Development Mailing List (not for usage questions)
Subject: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create
out of sync rings
I've looking at using puppet-swift to deploy a swift cluster.
Firstly - without
http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp
I would have struggled a great deal more to get up and running, so a big thank
you for a nice worked example of how to do multiple nodes!
However I have stumbled upon a problem - with respect to creating multiple proxy
nodes. There are some recipes around that follow on from the site.pp above and
explicitly build 1 proxy (e.g
https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp)
Now the problem is - each proxy node does a ring builder create, so ends up
with *different* builder (and therefore) ring files. This is not good, as the
end result is a cluster with all storage nodes and *one* proxy with the same
set of ring files, and *all* other proxies with
*different* ring (and builder) files.
I have used logic similar to the attached to work around this, i.e only create
rings if we are the 'ring server', otherwise get 'em via rsync.
Thoughts?
Regards
Mark
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
On 12/06/15 17:27, Mark Kirkwood wrote:
I've looking at using puppet-swift to deploy a swift cluster.
Firstly - without
http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp
I would have struggled a great deal more to get up and running, so a big
thank you for a nice worked example of how to do multiple nodes!
However I have stumbled upon a problem - with respect to creating
multiple proxy nodes. There are some recipes around that follow on from
the site.pp above and explicitly build 1 proxy (e.g
https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp)
Now the problem is - each proxy node does a ring builder create, so ends
up with *different* builder (and therefore) ring files. This is not
good, as the end result is a cluster with all storage nodes and *one*
proxy with the same set of ring files, and *all* other proxies with
*different* ring (and builder) files.
I have used logic similar to the attached to work around this, i.e only
create rings if we are the 'ring server', otherwise get 'em via rsync.
Thoughts?
I should have noted that the previously mentioned site.pp introduced the
idea of a ringmaster or ringserver host, and I made use of that i.e only
one proxy actually builds the rings.
Also I see in my effort to provide a simple fragment I left off a $ in
front of some variables, and I should probably have defined the
ringserver ip to make it clear what was happening, e.g:
$ringserver_local_net_ip = '192.168.5.200'
if $ipaddress_eth0 == $ringserver_local_net_ip {
Cheers
Mark
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
I skimmed the code, but since I'm not familiar with the environment, I could not find where swift-ring-builder rebalance is invoked. I'm guessing that each time you add a device to a ring, a rebalance is also done. Leaving aside how inefficient that is, the key thing is that the rebalance command has an optional seed parameter. Unless you explicitly set the seed (to same value on all node obviously), you won't get the same ring on all nodes. You also need to make sure you add the same set of drives and in same order. Regards, Donagh -Original Message- From: Mark Kirkwood [mailto:mark.kirkw...@catalyst.net.nz] Sent: 12 June 2015 06:28 To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings I've looking at using puppet-swift to deploy a swift cluster. Firstly - without http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp I would have struggled a great deal more to get up and running, so a big thank you for a nice worked example of how to do multiple nodes! However I have stumbled upon a problem - with respect to creating multiple proxy nodes. There are some recipes around that follow on from the site.pp above and explicitly build 1 proxy (e.g https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp) Now the problem is - each proxy node does a ring builder create, so ends up with *different* builder (and therefore) ring files. This is not good, as the end result is a cluster with all storage nodes and *one* proxy with the same set of ring files, and *all* other proxies with *different* ring (and builder) files. I have used logic similar to the attached to work around this, i.e only create rings if we are the 'ring server', otherwise get 'em via rsync. Thoughts? Regards Mark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [puppet] [Swift] Multiple proxy recipes will create out of sync rings
I've looking at using puppet-swift to deploy a swift cluster.
Firstly - without
http://git.openstack.org/cgit/stackforge/puppet-swift/tree/tests/site.pp
I would have struggled a great deal more to get up and running, so a big
thank you for a nice worked example of how to do multiple nodes!
However I have stumbled upon a problem - with respect to creating
multiple proxy nodes. There are some recipes around that follow on from
the site.pp above and explicitly build 1 proxy (e.g
https://github.com/CiscoSystems/puppet-openstack-ha/blob/folsom_ha/examples/swift-nodes.pp)
Now the problem is - each proxy node does a ring builder create, so ends
up with *different* builder (and therefore) ring files. This is not
good, as the end result is a cluster with all storage nodes and *one*
proxy with the same set of ring files, and *all* other proxies with
*different* ring (and builder) files.
I have used logic similar to the attached to work around this, i.e only
create rings if we are the 'ring server', otherwise get 'em via rsync.
Thoughts?
Regards
Mark
# create the ring if we are the ring server
if $ipaddress_eth0 == ringserver_local_net_ip {
class { 'swift::ringbuilder':
# the part power should be determined by assuming 100 partitions per drive
part_power = '18'
replicas = '2'
min_part_hours = 1
require= Class['swift'],
}
# sets up an rsync db that can be used to sync the ring DB
class { 'swift::ringserver':
local_net_ip = $ipaddress_eth0,
}
# exports rsync gets that can be used to sync the ring files
@@swift::ringsync { ['account', 'object', 'container']:
ring_server = $ipaddress_eth0,
}
} else {
# collect resources for synchronizing the ring databases
Swift::Ringsync||
}
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
