[Openstack-operators] [Large Deployments Team] Summit Meeting Notes & Actions

[Mike Dorman]

Thanks to everyone who was able to attend the LDT session in Tokyo!  I wanted 
to just briefly summarize some of the key points and action items.

* Neutron segmented/routed networks
  * Encourage you to follow & provide feedback on the Neutron spec [1]

* Cells v1 Patches
  * Many of the patches we gathered in [2] are no longer relevant or have been 
addressed some other way
  * Several will require a blueprint to get them into Nova
  * ACTION: mdorman to build a spec for vif plug patch [3].  If this goes well 
we can model future specs for other patches after this.
  * ACTION: sorrison to resubmit trivial cell name display patch [4]

* Public Clouds
  * Gathered good list of requirements/nice-to-haves in the etherpad [5]
  * Will go over these more in future IRC meeting to figure out what next steps 
to take on these.

* Glance Asks
  * List gathered in etherpad [6]
  * Also will discuss these in more detail in later meetings to determine 
appropriate RFE process.

I’ve also updated our couple wiki pages [7,8] with more current info based on 
results from the summit.

Our next meeting will be on Friday, November 20th at 3:00 UTC in 
#openstack-operators [8]  Please join us!

Thanks all!
Mike


[1] https://review.openstack.org/#/c/225384/
[2] https://etherpad.openstack.org/p/PAO-LDT-cells-patches
[3] https://review.openstack.org/#/c/235485/
[4] https://review.openstack.org/#/c/184158/
[5] https://etherpad.openstack.org/p/TYO-ops-large-deployments-team
[6] https://etherpad.openstack.org/p/LDT-glance-asks
[7] https://wiki.openstack.org/wiki/Large_Deployment_Team
[8] https://wiki.openstack.org/wiki/Meetings/LDT
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Large Deployments Team] [nova] Cells v1 attach/detach blueprint

[Mike Dorman]

I reached out to nova today to see what is the best way forward on this patch 
[1].  mriedem had created a blueprint skeleton for this already [2].

It sounds like it’s unlikely that nova would accept it, because it will require 
a new CI job [3] and it goes against the philosophy of not adding to the v1 
code base.

They suggested we add it to the nova meeting agenda [4], by which I hope to get 
to a definite “yea” or “nay” on if this can go forward.

It’s tomorrow (Thursday 11/5) 2100 UTC, #openstack-meeting.  Please attend as 
you’re able.

Thanks,
Mike


[1] https://review.openstack.org/#/c/215459/
[2] https://blueprints.launchpad.net/nova/+spec/cells-v1-interface-events
[3] https://review.openstack.org/#/c/235485/
[4] https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Tuning Guide

[David Medberry]

Great Kevin. Thanks for honchoing and sharing.

On Wed, Nov 4, 2015 at 3:56 PM, Kevin Bringard (kevinbri) <
kevin...@cisco.com> wrote:

> Hey all!
>
> Something that jumped out at me in Tokyo was how much it seemed that
> "basic" tuning stuff wasn't common knowledge. This was especially prevalent
> in the couple of rabbit talks I went to. So, in order to pool our
> resources, I started an Etherpad titled the "OpenStack Tuning Guide" (
> https://etherpad.openstack.org/p/OpenStack_Tuning_Guide). Eventually I
> expect this should go into the documentation project, and much of it may
> already exist in the operators manual (or elsewhere), but I thought that
> getting us all together to drop in our hints, tweaks, and best practices
> for tuning our systems to run OpenStack well, in real production, would be
> time well spent.
>
> It's a work in progress at the moment, and we've only just started, but
> please feel free to check it out. Feedback and community involvement is
> super welcome, so please don't hesitate to modify it as you see fit.
>
> Finally, I hate diverging resources, so if something like this already
> exists please speak up so we can focus our efforts on making sure that's up
> to date and well publicized.
>
> Thanks everyone!
>
> -- Kevin
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] [Large Deployments Team] Summit Meeting Notes & Actions

[Mike Dorman]

Sorry, wrong link for [3] below.



On 11/4/15, 3:05 PM, "Mike Dorman"  wrote:

>Thanks to everyone who was able to attend the LDT session in Tokyo!  I wanted 
>to just briefly summarize some of the key points and action items.
>
>* Neutron segmented/routed networks
>  * Encourage you to follow & provide feedback on the Neutron spec [1]
>
>* Cells v1 Patches
>  * Many of the patches we gathered in [2] are no longer relevant or have been 
> addressed some other way
>  * Several will require a blueprint to get them into Nova
>  * ACTION: mdorman to build a spec for vif plug patch [3].  If this goes well 
> we can model future specs for other patches after this.
>  * ACTION: sorrison to resubmit trivial cell name display patch [4]
>
>* Public Clouds
>  * Gathered good list of requirements/nice-to-haves in the etherpad [5]
>  * Will go over these more in future IRC meeting to figure out what next 
> steps to take on these.
>
>* Glance Asks
>  * List gathered in etherpad [6]
>  * Also will discuss these in more detail in later meetings to determine 
> appropriate RFE process.
>
>I’ve also updated our couple wiki pages [7,8] with more current info based on 
>results from the summit.
>
>Our next meeting will be on Friday, November 20th at 3:00 UTC in 
>#openstack-operators [8]  Please join us!
>
>Thanks all!
>Mike
>
>
>[1] https://review.openstack.org/#/c/225384/
>[2] https://etherpad.openstack.org/p/PAO-LDT-cells-patches
>[3] https://review.openstack.org/#/c/215459/ (FIXED)
>[4] https://review.openstack.org/#/c/184158/
>[5] https://etherpad.openstack.org/p/TYO-ops-large-deployments-team
>[6] https://etherpad.openstack.org/p/LDT-glance-asks
>[7] https://wiki.openstack.org/wiki/Large_Deployment_Team
>[8] https://wiki.openstack.org/wiki/Meetings/LDT
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Tuning Guide

[Donald Talton]

Awesome start. Rabbit fd tweaks are the bane of every install...including some 
of my own...

-Original Message-
From: Kevin Bringard (kevinbri) [mailto:kevin...@cisco.com] 
Sent: Wednesday, November 04, 2015 3:56 PM
To: OpenStack Operators
Subject: [Openstack-operators] OpenStack Tuning Guide

Hey all!

Something that jumped out at me in Tokyo was how much it seemed that "basic" 
tuning stuff wasn't common knowledge. This was especially prevalent in the 
couple of rabbit talks I went to. So, in order to pool our resources, I started 
an Etherpad titled the "OpenStack Tuning Guide" 
(https://etherpad.openstack.org/p/OpenStack_Tuning_Guide). Eventually I expect 
this should go into the documentation project, and much of it may already exist 
in the operators manual (or elsewhere), but I thought that getting us all 
together to drop in our hints, tweaks, and best practices for tuning our 
systems to run OpenStack well, in real production, would be time well spent.

It's a work in progress at the moment, and we've only just started, but please 
feel free to check it out. Feedback and community involvement is super welcome, 
so please don't hesitate to modify it as you see fit.

Finally, I hate diverging resources, so if something like this already exists 
please speak up so we can focus our efforts on making sure that's up to date 
and well publicized.

Thanks everyone!

-- Kevin
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Tuning Guide

[Joe Topjian]

Hi Kevin,

Oops, noticed I didn't reply to all the first time.

I think it's great to see more people who want to collect and distil
knowledge like this. :)

Finally, I hate diverging resources, so if something like this already
> exists please speak up so we can focus our efforts on making sure that's up
> to date and well publicized.
>

These may be complementary / sibling, but thought I'd mention them:

* Hypervisor Tuning Guide[1]
* Performance Team[2]

1:
http://lists.openstack.org/pipermail/openstack-operators/2015-October/008557.html
2:
http://lists.openstack.org/pipermail/openstack-dev/2015-October/078028.html

Thanks,
Joe
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Tuning Guide

[Leslie-Alexandre DENIS]

Hello there,

Very interesting initiative ! I'm currently working on the performance 
side too on our (french astrophysics cloud) OpenStack deployment but to 
say so currently I'm just following what CERN did regarding to Nova/CPU 
configuration.
It's essentially around KSM, NUMA, CPU pinning, EPT and performance 
measures are compared through Spec 06 benchmarks, which is standard for 
HPC/HTC computing.


You can take a look at the very interesting blog from CERN here 
http://openstack-in-production.blogspot.fr/ and if you want I have few 
slides from various meetings like HEPIX on these subjects.


Following the etherpad !

Regards,

Le 05/11/2015 00:11, Donald Talton a écrit :

Awesome start. Rabbit fd tweaks are the bane of every install...including some 
of my own...

-Original Message-
From: Kevin Bringard (kevinbri) [mailto:kevin...@cisco.com]
Sent: Wednesday, November 04, 2015 3:56 PM
To: OpenStack Operators
Subject: [Openstack-operators] OpenStack Tuning Guide

Hey all!

Something that jumped out at me in Tokyo was how much it seemed that "basic" tuning stuff 
wasn't common knowledge. This was especially prevalent in the couple of rabbit talks I went to. So, 
in order to pool our resources, I started an Etherpad titled the "OpenStack Tuning Guide" 
(https://etherpad.openstack.org/p/OpenStack_Tuning_Guide). Eventually I expect this should go into 
the documentation project, and much of it may already exist in the operators manual (or elsewhere), 
but I thought that getting us all together to drop in our hints, tweaks, and best practices for 
tuning our systems to run OpenStack well, in real production, would be time well spent.

It's a work in progress at the moment, and we've only just started, but please 
feel free to check it out. Feedback and community involvement is super welcome, 
so please don't hesitate to modify it as you see fit.

Finally, I hate diverging resources, so if something like this already exists 
please speak up so we can focus our efforts on making sure that's up to date 
and well publicized.

Thanks everyone!

-- Kevin
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

This email and any files transmitted with it are confidential, proprietary and 
intended solely for the individual or entity to whom they are addressed. If you 
have received this email in error please delete it immediately.


___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] OpenStack Tuning Guide

[Matt Kassawara]

The official documentation would greatly benefit from more practical
information such as tuning... most likely in the administration or
operations guide, at least for now. Ideally, I would like to see a
operators contribute documentation for complete production deployments that
others can use as a reference for new deployments and avoid reinventing the
wheel.

On Wed, Nov 4, 2015 at 4:27 PM, Leslie-Alexandre DENIS 
wrote:

> Hello there,
>
> Very interesting initiative ! I'm currently working on the performance
> side too on our (french astrophysics cloud) OpenStack deployment but to say
> so currently I'm just following what CERN did regarding to Nova/CPU
> configuration.
> It's essentially around KSM, NUMA, CPU pinning, EPT and performance
> measures are compared through Spec 06 benchmarks, which is standard for
> HPC/HTC computing.
>
> You can take a look at the very interesting blog from CERN here
> http://openstack-in-production.blogspot.fr/ and if you want I have few
> slides from various meetings like HEPIX on these subjects.
>
> Following the etherpad !
>
> Regards,
>
>
> Le 05/11/2015 00:11, Donald Talton a écrit :
>
>> Awesome start. Rabbit fd tweaks are the bane of every install...including
>> some of my own...
>>
>> -Original Message-
>> From: Kevin Bringard (kevinbri) [mailto:kevin...@cisco.com]
>> Sent: Wednesday, November 04, 2015 3:56 PM
>> To: OpenStack Operators
>> Subject: [Openstack-operators] OpenStack Tuning Guide
>>
>> Hey all!
>>
>> Something that jumped out at me in Tokyo was how much it seemed that
>> "basic" tuning stuff wasn't common knowledge. This was especially prevalent
>> in the couple of rabbit talks I went to. So, in order to pool our
>> resources, I started an Etherpad titled the "OpenStack Tuning Guide" (
>> https://etherpad.openstack.org/p/OpenStack_Tuning_Guide). Eventually I
>> expect this should go into the documentation project, and much of it may
>> already exist in the operators manual (or elsewhere), but I thought that
>> getting us all together to drop in our hints, tweaks, and best practices
>> for tuning our systems to run OpenStack well, in real production, would be
>> time well spent.
>>
>> It's a work in progress at the moment, and we've only just started, but
>> please feel free to check it out. Feedback and community involvement is
>> super welcome, so please don't hesitate to modify it as you see fit.
>>
>> Finally, I hate diverging resources, so if something like this already
>> exists please speak up so we can focus our efforts on making sure that's up
>> to date and well publicized.
>>
>> Thanks everyone!
>>
>> -- Kevin
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>> This email and any files transmitted with it are confidential,
>> proprietary and intended solely for the individual or entity to whom they
>> are addressed. If you have received this email in error please delete it
>> immediately.
>>
>>
>> ___
>> OpenStack-operators mailing list
>> OpenStack-operators@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Cinder mid-cycle planning survey

[Duncan Thomas]

Hi Folks

The Cinder team is trying to plan our mid-cycle meetup again.

Can anybody interested in attending please fill out this quick survey to
help with planning, please?

https://www.surveymonkey.com/r/Q5FZX68

Closing date is 11th November.

Thanks
-- 
-- 
Duncan Thomas
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] cinder volume_clear=zero makes sense with rbd ?

[Saverio Proto]

Hello there,

I am using cinder with rbd, and most volumes are created from glance
images on rbd as well.
Because of ceph features, these volumes are CoW and only blocks
different from the original parent image are really written.

Today I am debugging why in my production system deleting cinder
volumes gets very slow. Looks like the problem happens only at scale,
I can't reproduce it on my small test cluster.

I read all the cinder.conf reference, and I found this default value
=>   volume_clear=0.

Is this parameter evaluated when cinder works with rbd ?

This means that everytime we delete a Volume we first write all blocks
to 0 with a "dd" like operation and then we really delete it. This
default is designed with LVM backend in mind. In fact we dont want
that the next user gets a raw block device that is dirty, and can
potentially can read data out of it.

But what happens when we are using Ceph rbd as cinder backend ? and
our volumes are CoW from Glance Images most of the time, so we only
write in Ceph the blocks that are different from the original image. I
hope this is not writing all the rbd objects with zeros before
actually deleting the ceph volumes.

Does anybody has any advice on volume_clear setting to be used with rbd ?
Or even better, how can I make sure that the setting volume_clear is
not evaluated at all when using the rbd backend ?

thank you

Saverio

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] OVS flow dump fix

[Clayton O'Neill]

As many of you know, Liberty includes a bug fix for OVS L2 agent dropping
all flows when it starts up.  Since this is a bug, I’d like to see this
back ported to Kilo if possible.

We’re planning to do an upgrade to the latest stable/kilo because of the
Nova NUMA issue anyway and we’re still using packages for Nova & Neutron.
If this can be back ported then our upgrade to Liberty would go a lot
smoother since this would be once less network related issue to deal with
during the upgrade.

I’d appreciate if anyone that feels the same could leave a comment to that
effect on the bug:

https://bugs.launchpad.net/neutron/+bug/1383674
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] cinder volume_clear=zero makes sense with rbd ?

[Chris Friesen]

On 11/04/2015 08:46 AM, Saverio Proto wrote:

Hello there,

I am using cinder with rbd, and most volumes are created from glance
images on rbd as well.
Because of ceph features, these volumes are CoW and only blocks
different from the original parent image are really written.

Today I am debugging why in my production system deleting cinder
volumes gets very slow. Looks like the problem happens only at scale,
I can't reproduce it on my small test cluster.

I read all the cinder.conf reference, and I found this default value
=>   volume_clear=0.

Is this parameter evaluated when cinder works with rbd ?


I don't think that's actually used with rbd, since as you say Ceph uses CoW 
internally.


I believe it's also ignored if you use LVM with thin provisioning.

Chris

___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Permission Denied in swift

[Heiko Krämer]

Hi,

you are using an shared storage such as NFS ?

total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

It looks like your file system is broken or your export of your shared
storage isn't working well.

Cheers
Heiko

Am 04.11.2015 um 11:10 schrieb Davíð Örn Jóhannsson:
>
> I have a newly set up swift cluster that is generating a lot of error
> like this where it seems that swift user does not have access to read
> permissions or something
>
>
> Nov  4 10:01:27 swift-04 object-auditor: ERROR Trying to audit
> /srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8:
> #012Traceback (most recent call last):#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 173, in
> failsafe_object_audit#012self.object_audit(location)#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 191, in
> object_audit#012with df.open():#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1292,
> in open#012data_file, meta_file, ts_file =
> self._get_ondisk_file()#012  File
> "/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1381,
> in _get_ondisk_file#012"Error listing directory %s: %s" %
> (self._datadir, err))#012DiskFileError: Error listing directory
> /srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8:
> [Errno 13] Permission denied:
> '/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8'
>
>
> As Root 
>
>
> *root*@swift-04:~# ls -la
> /srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
> total 44
> drwxr--r-- 2 swift swift42 Nov  4 09:52 .
> drw-r--r-- 3 swift swift53 Nov  4 09:52 ..
> -rw-r--r-- 1 swift swift 40957 Nov  4 09:52 1446566608.56781.data
>
> As swift
>
> *swift*@swift-04:/$ *ls -la
> /srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8*
> ls: cannot access
> /srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8:
> Permission denied
>
> *swift*@swift-04:/$ *ls -la /srv/node/Z302XLHP/objects/11808/7c8*
> ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/.: Permission
> denied
> ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/..: Permission
> denied
> ls: cannot access
> /srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8:
> Permission denied
> total 0
> d? ? ? ? ?? .
> d? ? ? ? ?? ..
> ?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8
>
> *swift*@swift-04:/$*ls -la /srv/node/Z302XLHP/objects/11808/*
> total 8
> drwxrwxr-x   3 swift swift   61 Nov  4 10:03 .
> drwxr-xr-x 151 swift swift 4096 Nov  4 09:56 ..
> drwxr--r--   3 swift swift   53 Nov  4 09:52 7c8
> -rw---   1 swift swift   50 Nov  4 10:03 hashes.pkl
> -rwxr-xr-x   1 swift swift0 Nov  4 09:52 .lock
>
> Any ideas to what could be wrong with the permissions?
>
>
>
>
>
>
>
> ___
> OpenStack-operators mailing list
> OpenStack-operators@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-- 
anynines.com


signature.asc
Description: OpenPGP digital signature
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] Permission Denied in swift

[Davíð Örn Jóhannsson]

I have a newly set up swift cluster that is generating a lot of error like this 
where it seems that swift user does not have access to read permissions or 
something


Nov  4 10:01:27 swift-04 object-auditor: ERROR Trying to audit 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 173, in 
failsafe_object_audit#012self.object_audit(location)#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 191, in 
object_audit#012with df.open():#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1292, in 
open#012data_file, meta_file, ts_file = self._get_ondisk_file()#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1381, in 
_get_ondisk_file#012"Error listing directory %s: %s" % (self._datadir, 
err))#012DiskFileError: Error listing directory 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: [Errno 
13] Permission denied: 
'/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8'


As Root


root@swift-04:~# ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
total 44
drwxr--r-- 2 swift swift42 Nov  4 09:52 .
drw-r--r-- 3 swift swift53 Nov  4 09:52 ..
-rw-r--r-- 1 swift swift 40957 Nov  4 09:52 1446566608.56781.data

As swift

swift@swift-04:/$ ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/7c8
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/.: Permission denied
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/..: Permission denied
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied
total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/
total 8
drwxrwxr-x   3 swift swift   61 Nov  4 10:03 .
drwxr-xr-x 151 swift swift 4096 Nov  4 09:56 ..
drwxr--r--   3 swift swift   53 Nov  4 09:52 7c8
-rw---   1 swift swift   50 Nov  4 10:03 hashes.pkl
-rwxr-xr-x   1 swift swift0 Nov  4 09:52 .lock

Any ideas to what could be wrong with the permissions?





___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Permission Denied in swift

[Davíð Örn Jóhannsson]

No, Local disks.


it seems that the /srv/node/Z302XK5V/objects/84990/fbc/ folder is being created 
without execute permissions


Frá: Heiko Krämer 
Sent: 04. nóvember 2015 10:18
Til: openstack-operators@lists.openstack.org
Efni: Re: [Openstack-operators] Permission Denied in swift

Hi,

you are using an shared storage such as NFS ?

total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

It looks like your file system is broken or your export of your shared storage 
isn't working well.

Cheers
Heiko

Am 04.11.2015 um 11:10 schrieb Davíð Örn Jóhannsson:

I have a newly set up swift cluster that is generating a lot of error like this 
where it seems that swift user does not have access to read permissions or 
something


Nov  4 10:01:27 swift-04 object-auditor: ERROR Trying to audit 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 173, in 
failsafe_object_audit#012self.object_audit(location)#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 191, in 
object_audit#012with df.open():#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1292, in 
open#012data_file, meta_file, ts_file = self._get_ondisk_file()#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1381, in 
_get_ondisk_file#012"Error listing directory %s: %s" % (self._datadir, 
err))#012DiskFileError: Error listing directory 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: [Errno 
13] Permission denied: 
'/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8'


As Root


root@swift-04:~# ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
total 44
drwxr--r-- 2 swift swift42 Nov  4 09:52 .
drw-r--r-- 3 swift swift53 Nov  4 09:52 ..
-rw-r--r-- 1 swift swift 40957 Nov  4 09:52 1446566608.56781.data

As swift

swift@swift-04:/$ ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/7c8
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/.: Permission denied
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/..: Permission denied
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied
total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/
total 8
drwxrwxr-x   3 swift swift   61 Nov  4 10:03 .
drwxr-xr-x 151 swift swift 4096 Nov  4 09:56 ..
drwxr--r--   3 swift swift   53 Nov  4 09:52 7c8
-rw---   1 swift swift   50 Nov  4 10:03 hashes.pkl
-rwxr-xr-x   1 swift swift0 Nov  4 09:52 .lock

Any ideas to what could be wrong with the permissions?








___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



--
anynines.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Permission Denied in swift

[Davíð Örn Jóhannsson]

Swift creates the folder with these permissions

drw-r--r--   3 swift swift   53 Nov  3 15:53 fbc


but all other folders ar creteted with these permissions

drwxr-xr-x   3 swift swift   61 Nov  4 10:47 84990


Might the process responsible for creating these directories be misconfigured  
to use 644 ?


Frá: Davíð Örn Jóhannsson
Sent: 04. nóvember 2015 10:38
Til: Heiko Krämer; openstack-operators@lists.openstack.org
Efni: Re: [Openstack-operators] Permission Denied in swift


No, Local disks.


it seems that the /srv/node/Z302XK5V/objects/84990/fbc/ folder is being created 
without execute permissions


Frá: Heiko Krämer 
Sent: 04. nóvember 2015 10:18
Til: openstack-operators@lists.openstack.org
Efni: Re: [Openstack-operators] Permission Denied in swift

Hi,

you are using an shared storage such as NFS ?

total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

It looks like your file system is broken or your export of your shared storage 
isn't working well.

Cheers
Heiko

Am 04.11.2015 um 11:10 schrieb Davíð Örn Jóhannsson:

I have a newly set up swift cluster that is generating a lot of error like this 
where it seems that swift user does not have access to read permissions or 
something


Nov  4 10:01:27 swift-04 object-auditor: ERROR Trying to audit 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 173, in 
failsafe_object_audit#012self.object_audit(location)#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 191, in 
object_audit#012with df.open():#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1292, in 
open#012data_file, meta_file, ts_file = self._get_ondisk_file()#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1381, in 
_get_ondisk_file#012"Error listing directory %s: %s" % (self._datadir, 
err))#012DiskFileError: Error listing directory 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: [Errno 
13] Permission denied: 
'/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8'


As Root


root@swift-04:~# ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
total 44
drwxr--r-- 2 swift swift42 Nov  4 09:52 .
drw-r--r-- 3 swift swift53 Nov  4 09:52 ..
-rw-r--r-- 1 swift swift 40957 Nov  4 09:52 1446566608.56781.data

As swift

swift@swift-04:/$ ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/7c8
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/.: Permission denied
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/..: Permission denied
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied
total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/
total 8
drwxrwxr-x   3 swift swift   61 Nov  4 10:03 .
drwxr-xr-x 151 swift swift 4096 Nov  4 09:56 ..
drwxr--r--   3 swift swift   53 Nov  4 09:52 7c8
-rw---   1 swift swift   50 Nov  4 10:03 hashes.pkl
-rwxr-xr-x   1 swift swift0 Nov  4 09:52 .lock

Any ideas to what could be wrong with the permissions?








___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



--
anynines.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: [Openstack-operators] Permission Denied in swift

[Davíð Örn Jóhannsson]

​I seem to have spotted the error, I'm using Puppet OpenStack Swift Module 
https://github.com/openstack/puppet-swift/blob/master/manifests/storage/server.pp​
 and I seem to have missed the comments regarding permissions in rsyncd.conf


Frá: Davíð Örn Jóhannsson
Sent: 04. nóvember 2015 10:55
Til: Heiko Krämer; openstack-operators@lists.openstack.org
Efni: Re: [Openstack-operators] Permission Denied in swift


Swift creates the folder with these permissions

drw-r--r--   3 swift swift   53 Nov  3 15:53 fbc


but all other folders ar creteted with these permissions

drwxr-xr-x   3 swift swift   61 Nov  4 10:47 84990


Might the process responsible for creating these directories be misconfigured  
to use 644 ?


Frá: Davíð Örn Jóhannsson
Sent: 04. nóvember 2015 10:38
Til: Heiko Krämer; openstack-operators@lists.openstack.org
Efni: Re: [Openstack-operators] Permission Denied in swift


No, Local disks.


it seems that the /srv/node/Z302XK5V/objects/84990/fbc/ folder is being created 
without execute permissions


Frá: Heiko Krämer 
Sent: 04. nóvember 2015 10:18
Til: openstack-operators@lists.openstack.org
Efni: Re: [Openstack-operators] Permission Denied in swift

Hi,

you are using an shared storage such as NFS ?

total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

It looks like your file system is broken or your export of your shared storage 
isn't working well.

Cheers
Heiko

Am 04.11.2015 um 11:10 schrieb Davíð Örn Jóhannsson:

I have a newly set up swift cluster that is generating a lot of error like this 
where it seems that swift user does not have access to read permissions or 
something


Nov  4 10:01:27 swift-04 object-auditor: ERROR Trying to audit 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
#012Traceback (most recent call last):#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 173, in 
failsafe_object_audit#012self.object_audit(location)#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/auditor.py", line 191, in 
object_audit#012with df.open():#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1292, in 
open#012data_file, meta_file, ts_file = self._get_ondisk_file()#012  File 
"/usr/lib/python2.7/dist-packages/swift/obj/diskfile.py", line 1381, in 
_get_ondisk_file#012"Error listing directory %s: %s" % (self._datadir, 
err))#012DiskFileError: Error listing directory 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: [Errno 
13] Permission denied: 
'/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8'


As Root


root@swift-04:~# ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
total 44
drwxr--r-- 2 swift swift42 Nov  4 09:52 .
drw-r--r-- 3 swift swift53 Nov  4 09:52 ..
-rw-r--r-- 1 swift swift 40957 Nov  4 09:52 1446566608.56781.data

As swift

swift@swift-04:/$ ls -la 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/7c8
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/.: Permission denied
ls: cannot access /srv/node/Z302XLHP/objects/11808/7c8/..: Permission denied
ls: cannot access 
/srv/node/Z302XLHP/objects/11808/7c8/1710674aaa385fc75885b2ac7f8967c8: 
Permission denied
total 0
d? ? ? ? ?? .
d? ? ? ? ?? ..
?? ? ? ? ?? 1710674aaa385fc75885b2ac7f8967c8

swift@swift-04:/$ ls -la /srv/node/Z302XLHP/objects/11808/
total 8
drwxrwxr-x   3 swift swift   61 Nov  4 10:03 .
drwxr-xr-x 151 swift swift 4096 Nov  4 09:56 ..
drwxr--r--   3 swift swift   53 Nov  4 09:52 7c8
-rw---   1 swift swift   50 Nov  4 10:03 hashes.pkl
-rwxr-xr-x   1 swift swift0 Nov  4 09:52 .lock

Any ideas to what could be wrong with the permissions?








___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators



--
anynines.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators