[openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Hi,

I'm working on getting our CI working again, and i left it running over the
weekend.
Of the 50 jobs it ran. 6 of them failed because of different tests failed,
but all with the same cause
"tempest.exceptions.VolumeBuildErrorException"

Looking in c-vol log i see "Volume group "stack-volumes-lvmdriver-1" has
insufficient free space (1023 extents): 1024 required.\n'" (for that test)

I have VOLUME_BACKING_FILE_SIZE=32G
and i checked vgdisplay and it shows:

sudo vgdisplay

--- Volume group ---

VG Name   stack-volumes-lvmdriver-1

...

VG Size   32.00 GiB

Total PE  8191


How can i fix this?


Thanks,


-- 

*Eduard Biceri Matei, Senior Software Developer*
www.cloudfounders.com
 | eduard.ma...@cloudfounders.com
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Abhishek Shrivastava]

Hi Eduard,

Can you please send the snippet of failed tests along with their error
cause from your Build Logs.

On Mon, Aug 17, 2015 at 12:02 PM, Eduard Matei <
eduard.ma...@cloudfounders.com> wrote:

> Hi,
>
> I'm working on getting our CI working again, and i left it running over
> the weekend.
> Of the 50 jobs it ran. 6 of them failed because of different tests failed,
> but all with the same cause
> "tempest.exceptions.VolumeBuildErrorException"
>
> Looking in c-vol log i see "Volume group "stack-volumes-lvmdriver-1" has
> insufficient free space (1023 extents): 1024 required.\n'" (for that test)
>
> I have VOLUME_BACKING_FILE_SIZE=32G
> and i checked vgdisplay and it shows:
>
> sudo vgdisplay
>
> --- Volume group ---
>
> VG Name   stack-volumes-lvmdriver-1
>
> ...
>
> VG Size   32.00 GiB
>
> Total PE  8191
>
>
> How can i fix this?
>
>
> Thanks,
>
>
> --
>
> *Eduard Biceri Matei, Senior Software Developer*
> www.cloudfounders.com
>  | eduard.ma...@cloudfounders.com
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 


*Thanks & Regards,*
*Abhishek*
*Cloudbyte Inc. *
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Hi,

Not sure what snippet you mean but here are all the logs for the failed
test run:
http://packages.cloudfounders.com/ci_logs/01/213501/3/check/dsvm-tempest-full/37aa6fe/
(console.html and c-vol.txt show more details).

Thanks,
Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Abhishek Shrivastava]

Hi Eduard,

This is what I gathered the cause of the test failure:

2015-08-17 03:13:44.239 ERROR oslo_messaging.rpc.dispatcher
[req-95c1bc0f-e333-493b-9730-cfba9c3dfd9a
tempest-VolumesV1ActionsTest-418947994] Exception during message
handling: 500 Internal Server Error: Failed to upload image
32be6815-3571-48e8-b664-7902613ffd04 (HTTP 500)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
Traceback (most recent call last):
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 142, in _dispatch_and_reply
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
executor_callback))
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 186, in _dispatch
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
executor_callback)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 129, in _do_dispatch
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
result = func(ctxt, **new_args)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py",
line 105, in wrapper
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
return f(*args, **kwargs)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/volume/manager.py", line 976, in
copy_volume_to_image
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
payload['message'] = six.text_type(error)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py",
line 119, in __exit__
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/volume/manager.py", line 963, in
copy_volume_to_image
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
image_meta)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py",
line 105, in wrapper
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
return f(*args, **kwargs)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/volume/drivers/lvm.py", line 449,
in copy_volume_to_image
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
self.local_path(volume))
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/image/image_utils.py", line 351, in
upload_volume
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
image_service.update(context, image_id, {}, image_file)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/image/glance.py", line 334, in
update
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
_reraise_translated_image_exception(image_id)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/image/glance.py", line 516, in
_reraise_translated_image_exception
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
six.reraise(type(new_exc), new_exc, exc_trace)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/image/glance.py", line 332, in
update
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
**image_meta)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/opt/stack/new/cinder/cinder/image/glance.py", line 177, in call
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
return getattr(controller, method)(*args, **kwargs)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/glanceclient/v1/images.py",
line 360, in update
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
resp, body = self.client.put(url, headers=hdrs, data=image_data)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py",
line 268, in put
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
return self._request('PUT', url, **kwargs)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py",
line 230, in _request
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
raise exc.from_response(resp, resp.text)
2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
HTTPInternalServerError: 500 Internal Server Error: Failed to upload
image 32be68

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Andrey Pavlov]

glance log -

2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] REQ: curl -i
http://127.0.0.1:8080/v1/AUTH_93bdc098999d400b9838abb51a7a8126/glance/32be6815-3571-48e8-b664-7902613ffd04
-X PUT -H "X-Auth-Token: 5103ad1cf0684071aa47e36683004ead"
2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] RESP STATUS: 503 Service Unavailable
2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] RESP HEADERS: [('date', 'Mon, 17
Aug 2015 03:13:17 GMT'), ('content-length', '118'), ('content-type',
'text/html; charset=UTF-8'), ('x-trans-id',
'tx78adec8088444befb8faf-0055d15136')]
2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] RESP BODY: Service
navailableThe server is currently unavailable. Please try again at
a later time.
2015-08-17 03:13:43.939 2312 ERROR glance_store._drivers.swift.store
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] Failed to add object to Swift.
Got error from Swift: put_object('glance',
'32be6815-3571-48e8-b664-7902613ffd04', ...) failure and no ability to
reset contents for reupload..
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] Failed to upload image
32be6815-3571-48e8-b664-7902613ffd04
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils Traceback
(most recent call last):
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/opt/stack/new/glance/glance/api/v1/upload_utils.py", line 113, in
upload_data_to_store
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
context=req.context)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 340,
in store_add_to_backend
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
context=context)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line
226, in op_checker
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils return
store_op_fun(store, *args, **kwargs)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py",
line 620, in add
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils raise
glance_store.BackendException(msg)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils Got error
from Swift: put_object('glance', '32be6815-3571-48e8-b664-7902613ffd04',
...) failure and no ability to reset contents for reupload.

swift-object log -

object-server: 127.0.0.1 - - [17/Aug/2015:03:13:12 +] "PUT
/sdb1/419/AUTH_93bdc098999d400b9838abb51a7a8126/glance/68b9d0f8-f20f-42e8-a430-8930608e9ed4"
201 - "PUT
http://127.0.0.1:8080/v1/AUTH_93bdc098999d400b9838abb51a7a8126/glance/68b9d0f8-f20f-42e8-a430-8930608e9ed4";
"txe6ba789f3616452aa2a12-0055d15148" "proxy-server 2111" 0.0039 "-" 2159 0
object-server: ERROR __call__ error with PUT
/sdb1/134/AUTH_93bdc098999d400b9838abb51a7a8126/glance/32be6815-3571-48e8-b664-7902613ffd04
: [Errno 28] No space left on device (txn:
tx78adec8088444befb8faf-0055d15136)

maybe test tries to upload very huge image?


On Mon, Aug 17, 2015 at 10:02 AM, Abhishek Shrivastava <
abhis...@cloudbyte.com> wrote:

> Hi Eduard,
>
> This is what I gathered the cause of the test failure:
>
> 2015-08-17 03:13:44.239 ERROR oslo_messaging.rpc.dispatcher 
> [req-95c1bc0f-e333-493b-9730-cfba9c3dfd9a 
> tempest-VolumesV1ActionsTest-418947994] Exception during message handling: 
> 500 Internal Server Error: Failed to upload image 
> 32be6815-3571-48e8-b664-7902613ffd04 (HTTP 500)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher Traceback 
> (most recent call last):
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", 
> line 142, in _dispatch_and_reply
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher 
> executor_callback))
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File 
> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", 
> line 186, in _dispatch
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher 
> executor_callback)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_mes

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Thanks,
I didn't see that errror, seems to be caused by swift:
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.

I'll investigate further.
But what about the lvmdriver-1 error? Isn't that related.

Thanks,
Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Thanks,
I will try to increase the size of the dsvm disk, maybe that will help.

No idea about the image size.
The problem is that not all test runs fail, so it's not easy to trace.

Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Hi,

I can't find any way to change the image size of the vm used for testing
(it's been created using nodepool-dib and managed by nodepoold).

Anyway, tests are still (randomly) failing with lvm error (Volume group
"stack-volumes-lvmdriver-1" has insufficient free space (1023 extents):
1024 required.\n')

Any idea how to fix this?

Thanks,

Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Hi,

Further investigation shows there might be a relation between a test
failing because it can't upload an image (most likely because there's not
enough space left) and another test failing because lvm doesn't have enough
extents - at the same time (due to tests running on two workers). So if
these tests run at the same time, they will both fail.

I increased VOLUME_BACKING_FILE_SIZE to 32G but still same error.

Any idea how to fix this?
Can i increase the number of "extents"? (how?) Or can i increase the disk
size of the image? (again how?)

Thanks,

Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Hi,

Might be because volume backing size of 32G results in 8191 PE instead of
8192, so last volume create attempt sees only 1023 free extents.

Will retry with 32768M

Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [third-party-ci]Tests randomly failing because of lvm

[Eduard Matei]

Hi,

In case someone is still interested, 32768 is the fix. The number of PEs
must be multiple of 1024 (not sure why).

Eduard
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev