Re: [ceph-users] S3 Radosgw : how to grant a user within a tenant
On 02/17/2017 06:25 PM, Vincent Godin wrote: > I created 2 users : jack & bob inside a tenant_A > jack created a bucket named BUCKET_A and want to give read access to the > user bob > > with s3cmd, i can grant a user without tenant easylly: s3cmd setacl > --acl-grant=read:user s3://BUCKET_A > > but with an explicit tenant, i tried : > --acl-grant=read:bob > --acl-grant=read:tenant_A$bob > --acl-grant=read:tenant_A\$bob > --acl-grant=read:"tenant_A:bob" > > each time, i got a s3 error : 400 (invalidArgument) > > Does someone know the solution ? Have you tried using email-address instead of tenant:UID? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] async-ms with RDMA or DPDK?
Hi, according to kraken release-notes and documentation, AsyncMessenger now also supports RDMA and DPDK. Is anyone already using async-ms with RDMA or DPDK and might be able to tell us something about real-world performance gains and stability? Best, Bastian ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSPF to the host
Hi, regarding clustered VyOS on KVM: In theory this sounds like a safe plan, but will come with a great performance penalty because of all the context-switches. And even with PCI-passthrough you will also feel increased latency. Docker/LXC/LXD on the other hand does not share the context-switch dilemma. Not sure if VyOS likes to run in a docker container though. I didn't have a chance to play with VPP[1] yet, but it sounds like this could be quite useful for high performance routing/switch inside a container. [1]: https://wiki.fd.io/view/VPP Cheers, Bastian Am 2016-06-08 09:04, schrieb Josef Johansson: Hi, Regarding single points of failure on the daemon on the host I was thinking about doing a cluster setup with i.e. VyOS on kvm-machines on the host, and they handle all the ospf stuff as well. I have not done any performance benchmarks but it should be possible to do at least. Maybe even possible to do in docker or straight in lxc since it's mostly route management in the kernel. Regards, Josef On Mon, 6 Jun 2016, 18:54 Jeremy Hanmer, wrote: We do the same thing. OSPF between ToR switches, BGP to all of the hosts with each one advertising its own /32 (each has 2 NICs). On Mon, Jun 6, 2016 at 6:29 AM, Luis Periquito wrote: Nick, TL;DR: works brilliantly :) Where I work we have all of the ceph nodes (and a lot of other stuff) using OSPF and BGP server attachment. With that we're able to implement solutions like Anycast addresses, removing the need to add load balancers, for the radosgw solution. The biggest issues we've had were around the per-flow vs per-packets traffic load balancing, but as long as you keep it simple you shouldn't have any issues. Currently we have a P2P network between the servers and the ToR switches on a /31 subnet, and then create a virtual loopback address, which is the interface we use for all communications. Running tests like iperf we're able to reach 19Gbps (on a 2x10Gbps network). OTOH we no longer have the ability to separate traffic between public and osd network, but never really felt the need for it. Also spend a bit of time planning how the network will look like and it's topology. If done properly (think details like route summarization) then it's really worth the extra effort. On Mon, Jun 6, 2016 at 11:57 AM, Nick Fisk wrote: Hi All, Has anybody had any experience with running the network routed down all the way to the host? I know the standard way most people configured their OSD nodes is to bond the two nics which will then talk via a VRRP gateway and then probably from then on the networking is all Layer3. The main disadvantage I see here is that you need a beefy inter switch link to cope with the amount of traffic flowing between switches to the VRRP address. I’ve been trying to design around this by splitting hosts into groups with different VRRP gateways on either switch, but this relies on using active/passive bonding on the OSD hosts to make sure traffic goes from the correct Nic to the directly connected switch. What I was thinking, instead of terminating the Layer3 part of the network at the access switches, terminate it at the hosts. If each Nic of the OSD host had a different subnet and the actual “OSD Server” address bound to a loopback adapter, OSPF should advertise this loopback adapter address as reachable via the two L3 links on the physically attached Nic’s. This should give you a redundant topology which also will respect your physically layout and potentially give you higher performance due to ECMP. Any thoughts, any pitfalls? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] OSPF to the host
Hi, for IPoIB this is probably the only way to efficiently use dual-port HCAs. Since IPoIB can - AFAIK - only do bonding in active-passive mode, it won't distribute traffic across both ports like ethernet link-aggregation would do. OTOH, running ceph on dynamically routed networks will put your routing daemon (e.g. bird) in a SPOF position... Cheers, Bastian Am 2016-06-06 15:29, schrieb Luis Periquito: Nick, TL;DR: works brilliantly :) Where I work we have all of the ceph nodes (and a lot of other stuff) using OSPF and BGP server attachment. With that we're able to implement solutions like Anycast addresses, removing the need to add load balancers, for the radosgw solution. The biggest issues we've had were around the per-flow vs per-packets traffic load balancing, but as long as you keep it simple you shouldn't have any issues. Currently we have a P2P network between the servers and the ToR switches on a /31 subnet, and then create a virtual loopback address, which is the interface we use for all communications. Running tests like iperf we're able to reach 19Gbps (on a 2x10Gbps network). OTOH we no longer have the ability to separate traffic between public and osd network, but never really felt the need for it. Also spend a bit of time planning how the network will look like and it's topology. If done properly (think details like route summarization) then it's really worth the extra effort. On Mon, Jun 6, 2016 at 11:57 AM, Nick Fisk wrote: Hi All, Has anybody had any experience with running the network routed down all the way to the host? I know the standard way most people configured their OSD nodes is to bond the two nics which will then talk via a VRRP gateway and then probably from then on the networking is all Layer3. The main disadvantage I see here is that you need a beefy inter switch link to cope with the amount of traffic flowing between switches to the VRRP address. I’ve been trying to design around this by splitting hosts into groups with different VRRP gateways on either switch, but this relies on using active/passive bonding on the OSD hosts to make sure traffic goes from the correct Nic to the directly connected switch. What I was thinking, instead of terminating the Layer3 part of the network at the access switches, terminate it at the hosts. If each Nic of the OSD host had a different subnet and the actual “OSD Server” address bound to a loopback adapter, OSPF should advertise this loopback adapter address as reachable via the two L3 links on the physically attached Nic’s. This should give you a redundant topology which also will respect your physically layout and potentially give you higher performance due to ECMP. Any thoughts, any pitfalls? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] DSS 7000 for large scale object storage
Yes, rebuild in case of a whole chassis failure is indeed an issue. That depends on how the failure domain looks like. I'm currently thinking of initially not running fully equipped nodes. Let's say four of these machines with 60x 6TB drives each, so only loaded 2/3. That's raw 1440TB distributed over eight OSD nodes. Each individual OSD-node would therefore host "only" 30 OSDs but still allow for fast expansion. Usually delivery and installation of a bunch of HDDs is much faster than servers. I really wonder how easy it is to add additional disks and whether chance for node- or even chassis-failure increases. Cheers, Bastian Am 2016-03-21 10:33, schrieb David: Sounds like you’ll have a field day waiting for rebuild in case of a node failure or an upgrade of the crush map ;) David 21 mars 2016 kl. 09:55 skrev Bastian Rosner : Hi, any chance that somebody here already got hands on Dell DSS 7000 machines? 4U chassis containing 90x 3.5" drives and 2x dual-socket server sleds (DSS7500). Sounds ideal for high capacity and density clusters, since each of the server-sleds would run 45 drives, which I believe is a suitable number of OSDs per node. When searching for this model there's not much detailed information out there. Sadly I could not find a review from somebody who actually owns a bunch of them and runs a decent PB-size cluster with it. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] DSS 7000 for large scale object storage
Hi, any chance that somebody here already got hands on Dell DSS 7000 machines? 4U chassis containing 90x 3.5" drives and 2x dual-socket server sleds (DSS7500). Sounds ideal for high capacity and density clusters, since each of the server-sleds would run 45 drives, which I believe is a suitable number of OSDs per node. When searching for this model there's not much detailed information out there. Sadly I could not find a review from somebody who actually owns a bunch of them and runs a decent PB-size cluster with it. Cheers, Bastian ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com