Public bug reported: The typical behavior of the read function which has length in the parameter is to return data <= that length. It's similar to read system call that has count parameter and it's not expected behavior that the function returns more. There are some places that assume that, e.g. class ChunkReader(object) [1]. it doesn't expect that the len(self.do_read(i)) can be bigger than i.
But read function in class _UWSGIChunkFile can return more data than requested. Due to announced deprecation of eventlets in glance we switched to uwsgi. We use swift backend for glance in our setup. When we tried making snapshot of the VM, created from 2.01 GB QCOW2 image with Create New Volume = No, we started getting really high consumption of memory which caused OOM. During the investigation we found that _UWSGIChunkFile.read returns sometimes more data than it was requested (by adding debug outputs into ChunkReader [2]. Because of this behavior ChunkReader gets negative value of i (see the last 2 lines) and in its turn the last call causes _UWSGIChunkFile to read the whole remanings of the file into memory (~5GB). IF we increase OOM limit, glance tries to send this 5+Gb to swift and gets error, because swift typically has limit and under normal condition it sends data using 200Mb (swift_store_large_object_chunk_size can change that limit). Before switching to uwsgi we didn't see that issue and the data was transferred by 200Mb. [1] https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/swift/store.py#L1672 [2] def read(self, i): left = self.total - self.bytes_read if i > left: i = left ... if i == 0: result = b'' else: result = self.do_read(i) LOG.info(("read i: %d, res: %d" % (i, len(result)))) self.bytes_read += len(result) self.checksum.update(result) self.os_hash_value.update(result) if self.verifier: self.verifier.update(result) we see: 2025-10-10 01:34:28.291 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 65536, res: 73720 ... 2025-10-10 01:34:31.712 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 34726, res: 40960 2025-10-10 01:35:00.645 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: -6234, res: 5053605798 ** Affects: glance Importance: Undecided Status: New ** Also affects: glance Importance: Undecided Status: New ** No longer affects: python-openstackclient -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/2127521 Title: read function in class _UWSGIChunkFile can return more data than requested Status in Glance: New Bug description: The typical behavior of the read function which has length in the parameter is to return data <= that length. It's similar to read system call that has count parameter and it's not expected behavior that the function returns more. There are some places that assume that, e.g. class ChunkReader(object) [1]. it doesn't expect that the len(self.do_read(i)) can be bigger than i. But read function in class _UWSGIChunkFile can return more data than requested. Due to announced deprecation of eventlets in glance we switched to uwsgi. We use swift backend for glance in our setup. When we tried making snapshot of the VM, created from 2.01 GB QCOW2 image with Create New Volume = No, we started getting really high consumption of memory which caused OOM. During the investigation we found that _UWSGIChunkFile.read returns sometimes more data than it was requested (by adding debug outputs into ChunkReader [2]. Because of this behavior ChunkReader gets negative value of i (see the last 2 lines) and in its turn the last call causes _UWSGIChunkFile to read the whole remanings of the file into memory (~5GB). IF we increase OOM limit, glance tries to send this 5+Gb to swift and gets error, because swift typically has limit and under normal condition it sends data using 200Mb (swift_store_large_object_chunk_size can change that limit). Before switching to uwsgi we didn't see that issue and the data was transferred by 200Mb. [1] https://opendev.org/openstack/glance_store/src/branch/master/glance_store/_drivers/swift/store.py#L1672 [2] def read(self, i): left = self.total - self.bytes_read if i > left: i = left ... if i == 0: result = b'' else: result = self.do_read(i) LOG.info(("read i: %d, res: %d" % (i, len(result)))) self.bytes_read += len(result) self.checksum.update(result) self.os_hash_value.update(result) if self.verifier: self.verifier.update(result) we see: 2025-10-10 01:34:28.291 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 65536, res: 73720 ... 2025-10-10 01:34:31.712 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: 34726, res: 40960 2025-10-10 01:35:00.645 7 INFO glance_store._drivers.swift.store [None req-b7061fc2-d029-441f-aef8-2a7d5efe088b 1662740891134e449807a022d29fd011 313fda1bd54c4a7c81d29aead98833e4 - - default default] read i: -6234, res: 5053605798 To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/2127521/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
