This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu2~cloud0
---
python-glance-store (2.0.0-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
python-glance-store (2.0.0-0ubuntu2) focal; urgency=medium
.
[ Corey
This bug was fixed in the package python-glance-store - 2.3.0-0ubuntu1~cloud1
---
python-glance-store (2.3.0-0ubuntu1~cloud1) focal-victoria; urgency=medium
.
[ Chris MacNaughton ]
* d/control: Update VCS paths for move to lp:~ubuntu-openstack-dev.
.
[ Corey Bryant ]
*
This bug was fixed in the package python-glance-store - 2.5.0-0ubuntu2~cloud0
---
python-glance-store (2.5.0-0ubuntu2~cloud0) focal-wallaby; urgency=medium
.
* New update for the Ubuntu Cloud Archive.
.
python-glance-store (2.5.0-0ubuntu2) hirsute; urgency=medium
.
*
This bug was fixed in the package python-glance-store - 2.0.0-0ubuntu2
---
python-glance-store (2.0.0-0ubuntu2) focal; urgency=medium
[ Corey Bryant ]
* d/gbp.conf: Create stable/ussuri branch.
[ Chris MacNaughton ]
* d/control: Update VCS paths for move to
This bug was fixed in the package python-glance-store - 2.5.0-0ubuntu2
---
python-glance-store (2.5.0-0ubuntu2) hirsute; urgency=medium
* d/gbp.conf: Create stable/wallaby branch.
* d/p/s3-optimize-write-chunksize.patch: Cherry-picked from upstream
to fix S3 write performance
[focal-victoria]
All of the uploads succeeded. And -proposed shortened time for the
larger sizes.
$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api
$ apt-cache policy python3-glance-store
python3-glance-store:
Installed: 2.3.0-0ubuntu1~cloud1
[bionic-ussuri]
All of the uploads succeeded. And -proposed shortened time for the
larger sizes.
$ sudo apt-get install python3-glance-store/bionic-proposed
$ sudo systemctl restart glance-api
$ apt-cache policy python3-glance-store
python3-glance-store:
Installed: 2.0.0-0ubuntu2~cloud0
Just for the record, this is the current status with focal-victoria. No
diff between -updates and -proposed.
$ apt-cache policy python3-glance-store
python3-glance-store:
Installed: 2.3.0-0ubuntu1~cloud0
Candidate: 2.3.0-0ubuntu1~cloud0
Version table:
*** 2.3.0-0ubuntu1~cloud0 500
@Corey,
Somehow the binary package for cloud-archive:victoria-proposed is not
published yet. Can you please double-check the build status of the
package? I just don't know where to look.
cloud1 in the source vs cloud0 in the binary.
$ curl -s
[focal-wallaby]
All of the uploads succeeded. And -proposed shortened time for the
larger sizes.
$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api
$ apt-cache policy python3-glance-store
python3-glance-store:
Installed: 2.5.0-0ubuntu2~cloud0
[focal]
All of the uploads succeeded. And -proposed shortened time for the
larger sizes.
$ sudo apt-get install python3-glance-store/focal-proposed
$ sudo systemctl restart glance-api
$ apt-cache policy python3-glance-store
python3-glance-store:
Installed: 2.0.0-0ubuntu2
Candidate:
[hirsute]
All of the uploads succeeded. And -proposed shortened time for the
larger sizes.
$ sudo apt-get install python3-glance-store/hirsute-proposed
$ sudo systemctl restart glance-api
$ apt-cache policy python3-glance-store
python3-glance-store:
Installed: 2.5.0-0ubuntu2
Candidate:
Hello Nobuto, or anyone else affected,
Accepted python-glance-store into hirsute-proposed. The package will
build now and be available at
https://launchpad.net/ubuntu/+source/python-glance-store/2.5.0-0ubuntu2
in a few hours, and then in the -proposed repository.
Please help us by testing this
My update in the bug description was somehow rolled back (by me in the
record), trying again.
** Description changed:
[Impact]
- [Test Case]
- I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with Glance, an
** Description changed:
[Impact]
-
- Glance with S3 backend cannot accept image uploads in a realistic time
- frame. For example, an 1GB image upload takes ~60 minutes although other
- backends such as swift can complete it with 10 seconds.
-
- [Test Plan]
-
- 1. Deploy a partial OpenStack
** Description changed:
[Impact]
- [Test Case]
- I have a test Ceph cluster as an object storage with both Swift and S3
protocols enabled for Glance (Ussuri). When I use Swift backend with Glance, an
image upload completes quickly enough. But with S3 backend Glance, it takes
much more time
Uploaded to hirsute unapproved queue, focal unapproved queue, and
victoria-staging PPA:
https://launchpad.net/ubuntu/hirsute/+queue?queue_state=1_text=python-glance-store
https://launchpad.net/ubuntu/focal/+queue?queue_state=1_text=python-glance-store
** Description changed:
- I have a test Ceph cluster as an object storage with both Swift and S3
- protocols enabled for Glance (Ussuri). When I use Swift backend with
- Glance, an image upload completes quickly enough. But with S3 backend
- Glance, it takes much more time to upload an image and
Subscribing Canonical's ~field-high to initiate the Ubuntu package's SRU
process in a timely manner.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
Reviewed: https://review.opendev.org/c/openstack/glance_store/+/799870
Committed:
https://opendev.org/openstack/glance_store/commit/32f9a1509bc94baa6acaf508c922f3b7edf5b65f
Submitter: "Zuul (22348)"
Branch:master
commit 32f9a1509bc94baa6acaf508c922f3b7edf5b65f
Author: Nobuto Murata
Date:
I'm facing the same issue and can confirm the suggested fix improves the
upload speed dramatically (without it, I had ~3.5min for 200mb image,
now the same time is consumed for 10GB image upload).
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed
Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance_store/+/799870
** Changed in: glance-store
Status: New => In Progress
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
> I *think* hash calculation and verifier have to be outside of the loop
to avoid the overhead. I will confirm it with a manual testing.
This hypothesis wasn't true, it was really about the chunk size.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is
I *think* hash calculation and verifier have to be outside of the loop
to avoid the overhead. I will confirm it with a manual testing.
for chunk in utils.chunkreadable(image_file, self.WRITE_CHUNKSIZE):
image_data += chunk
image_size += len(chunk)
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: python-glance-store (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Yeah, I put the same config on purpose for both s3 and swift. But
tweaking large_object_size didn't make any difference.
[swift]
large_object_size = 5120
large_object_chunk_size = 200
[s3]
s3_store_large_object_size = 5120
s3_store_large_object_chunk_size = 200
After digging into the actual
Actually that's only used for images over 5GB
On Wed, Jul 7, 2021 at 1:59 PM James Page
wrote:
> I think the swift driver uses 200M:
>
> DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M
>
> On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <1934...@bugs.launchpad.net>
> wrote:
>
>> And by using "4 *
I think the swift driver uses 200M:
DEFAULT_LARGE_OBJECT_CHUNK_SIZE = 200 # 200M
On Wed, Jul 7, 2021 at 1:50 PM Nobuto Murata <1934...@bugs.launchpad.net>
wrote:
> And by using "4 * units.Mi" it can be 20s.
>
> --
> You received this bug notification because you are a member of Ubuntu
>
And by using "4 * units.Mi" it can be 20s.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
To manage notifications about this bug go to:
Okay, as the utils.chunkreadable loop is taking time I've tried a larger
WRITE_CHUNKSIZE by hand. It can decrease the amount of time of uploading
a 512MB image from 14 minutes to 60 seconds.
$ git diff
diff --git a/glance_store/_drivers/s3.py b/glance_store/_drivers/s3.py
index 1c18531..576c573
The code part in question is this for loop:
https://opendev.org/openstack/glance_store/src/branch/stable/ussuri/glance_store/_drivers/s3.py#L638-L644
2021-07-07 11:50:06.735 - def _add_singlepart
2021-07-07 11:50:06.736 - getting into utils.chunkreadable loop
2021-07-07 11:50:06.736 - loop
S3 performance itself is not bad. Uploading 512MB object can complete
within a few seconds. So I suppose it's on how Glance S3 driver is using
boto3.
$ time python3 upload.py
real0m3.644s
user0m3.124s
sys 0m1.835s
$ cat upload.py
import boto3
s3 = boto3.client(
"s3",
Debug log of when uploading a 512MB image with S3 backend.
** Attachment added: "glance-api.log"
https://bugs.launchpad.net/ubuntu/+source/python-glance-store/+bug/1934849/+attachment/5509534/+files/glance-api.log
** Also affects: glance-store
Importance: Undecided
Status: New
--
python3-boto3 1.9.253-1
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1934849
Title:
s3 backend takes time exponentially
To manage notifications about this bug go to:
34 matches
Mail list logo
