Reviewed: https://review.openstack.org/290148 Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=0f7f1de233919a0b046349a3e31ae7fc8675a1c5 Submitter: Jenkins Branch: feature/hummingbird
commit d6b4587a554b51ba733b151e0d924735b63d07e0 Author: Olga Saprycheva <osap...@us.ibm.com> Date: Tue Mar 8 10:57:56 2016 -0600 Removed redundant file for flake8 check Change-Id: I4322978aa20ee731391f7709bbd79dee140fc703 commit 643dbce134140530eef2ae62c42fef1107f905ed Author: OpenStack Proposal Bot <openstack-in...@lists.openstack.org> Date: Tue Mar 8 06:35:49 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: I96b8ff1287bf219c5f8d56a3a4868c1063a953f9 commit 83713d37f0331c5ce9d377f4b4e8724551ae30ca Author: Daisuke Morita <morita.dais...@ntti3.com> Date: Mon Mar 7 18:30:47 2016 -0800 Missing comments for storage policy parameter There are missing comments about storege_policy_index so appropriate comments are added. Change-Id: I3de3f0e6864e65918ca1a13cce70f19c23d295f5 commit 2cff2dec3d1c4588f5103e39679c43b3dded6dcb Author: Olga Saprycheva <osap...@us.ibm.com> Date: Fri Mar 4 15:19:39 2016 -0600 Fixed pep8 and flake8 errors in doc/source/conf.py and updated flake8 commands in tox.ini to test it. Change-Id: I2add370e4cfb55d1388e3a8b41f688a7f3f2c621 commit 043fbca6d08648baa314ea2236f1ccdca8785f16 Author: Christian Schwede <cschw...@redhat.com> Date: Fri Mar 4 09:33:17 2016 +0000 Remove Erasure Coding beta status from docs This removes notes stating support for Erasure coding as beta. Questions regarding the stability of EC are coming up regularly, and are often referring to the docs that state EC as still in beta. Besides this, a note marking statsd support as beta has been removed as well. Change-Id: If4fb6a5c4cb741d42953db3cee8cb17a1d774e15 commit 09c73b86e9255f28fbd4cf571a52c17d549a8f9a Author: Pete Zaitcev <zait...@kotori.zaitcev.us> Date: Thu Mar 3 10:24:28 2016 -0700 Fix a crash in exception printout Says the number of arguments does not match the number of '%'. Change-Id: I8b5e395a07328fb9d4ac7a19f8ed2ae1637bee3b commit fad5fabe0a22e8a86635a66523dd3d3d3b1fa705 Author: Tim Burke <tim.bu...@gmail.com> Date: Thu Mar 3 15:07:08 2016 +0000 During functional tests, 404 response to a DELETE is successful Previously, we would only consider 204 responses successful, which would cause some spurious gate failures, such as http://logs.openstack.org/66/287666/3/check/gate-swift-dsvm-functional/c6d2673/console.html#_2016-03-03_13_41_07_846 Change-Id: Ic8c300647924352a297a2781b50064f7657038b4 commit e91de49d6864b3794f8dc5acd9c1bf0c2f7409d1 Author: Alistair Coles <alistair.co...@hp.com> Date: Mon Aug 10 10:30:10 2015 -0500 Update container on fast-POST This patch makes a number of changes to enable content-type metadata to be updated when using the fast-POST mode of operation, as proposed in the associated spec [1]. * the object server and diskfile are modified to allow content-type to be updated by a POST and the updated value to be stored in .meta files. * the object server accepts PUTs and DELETEs with older timestamps than existing .meta files. This is to be consistent with replication that will leave a later .meta file in place when replicating a .data file. * the diskfile interface is modified to provide accessor methods for the content-type and its timestamp. * the naming of .meta files is modified to encode two timestamps when the .meta file contains a content-type value that was set prior to the latest metadata update; this enables consistency to be achieved when rsync is used for replication. * ssync is modified to sync meta files when content-type differs between local and remote copies of objects. * the object server issues container updates when handling POST requests, notifying the container server of the current immutable metadata (etag, size, hash, swift_bytes), content-type with their respective timestamps, and the mutable metadata timestamp. * the container server maintains the most recently reported values for immutable metadata, content-type and mutable metadata, each with their respective timestamps, in a single db row. * new probe tests verify that replication achieves eventual consistency of containers and objects after discrete updates to content-type and mutable metadata, and that container-sync sync's objects after fast-post updates. [1] spec change-id: I60688efc3df692d3a39557114dca8c5490f7837e Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01 commit 3c61ab4678a7aa9ff256ace4bc97ab449607fd49 Author: asettle <alexandra.set...@rackspace.com> Date: Wed Feb 10 17:58:05 2016 +1000 Operational procedures guide This is the operational procedures guide that HPE used to operate and monitor their public Swift systems. It has been made publicly available. Change-Id: Iefb484893056d28beb69265d99ba30c3c84add2b commit 8e4347afd59aa30708ef03a78202652aaadda2e9 Author: Takashi Kajinami <kajina...@nttdata.co.jp> Date: Tue Nov 17 16:15:59 2015 +0900 Fix proxy-server's support for chunked transferring in GET object Proxy-server now requires Content-Length in the response header when getting object and does not support chunked transferring with "Transfer-Encoding: chunked" This doesn't matter in normal swift, but prohibits us from putting any middelwares to execute something like streaming processing of objects, which can't calculate the length of their response body before they start to send their response. Change-Id: I60fc6c86338d734e39b7e5f1e48a2647995045ef commit 9dda1c9526c5b3e0ac905985921605d833bef3cc Author: Takashi Kajinami <kajina...@nttdata.co.jp> Date: Fri Dec 25 10:30:01 2015 +0900 Update parameters about authtoken middleware in proxy-server.conf.sample This patch replaces some configration parameters about authtoken middleware in proxy-server.conf, which will be deprecated. The following parameters are to be deprecated. auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, admin_tenant_name Change-Id: I373734933189c87c4094203b0752dd3762689034 commit 087fa4fa01fc36343f2ce0974d3b36ec0b98afc8 Author: Andreas Jaeger <a...@suse.com> Date: Mon Feb 29 20:00:38 2016 +0100 Remove unused pngmath Sphinx extension There's no RST file that uses ".. math" and thus the pngmath Sphinx extension is not used and can get removed. Change-Id: I3ea4b529025c8ac3c4092f3720124647861e6668 commit 9c5b01e7c7699fbfd221e98ccde1afa9361c8119 Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Wed Dec 9 08:08:22 2015 +0000 Add container quota functional test This patch provides one container quota functional test. Change-Id: I16354c15fc434d70f9bb6b9b1e18b09357195c83 commit efdc6ee8dbc434e9189a6b85b924c322c3ff6ae3 Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Thu Dec 10 01:15:30 2015 +0000 Add X-Delete-After and X-Delete-At functional test This patch provides functional tests for X-Delete-After and X-Delete-At. Change-Id: I55b0d0584428413aa7a31f66defec55f9b1bbcd9 commit bd93d44bb46350f716f1dfb66fb43c2ab4ad8c50 Author: Takashi Kajinami <kajina...@nttdata.co.jp> Date: Sun Feb 28 13:41:19 2016 +0900 Make sure all temp files get deleted in test_ringbuilder This patch makes test_ringbuilder create a temporal directory, run ring builder commands under it, and delete it after testing for each test cases, to fix temp file leaking. Change-Id: I6f59fe095ea6485af0e60b5a8e8fc3892e0a0f90 commit 10b5765c60790f8a3ae2a94d6a51736ec2c99ce0 Author: benjkeller <bjkel...@us.ibm.com> Date: Thu Feb 25 18:54:00 2016 -0600 Rename test_object_delete_at_aysnc This renames test_object_delete_at_aysnc to test_object_delete_at_async to match with the function async_update which it tests. Change-Id: I726afcbf08a3449c7af2834e573b97be378a86cd Closes-Bug: 1550067 commit 9e53bb47efa28983109da8ceca5d316925bc6804 Author: John Dickinson <m...@not.mn> Date: Wed Feb 24 09:36:40 2016 -0800 updated some weird whitespace character to a normal one Change-Id: I187fb8fc8d10f950bbebc586999eced0bc117432 commit 54e53ae87b6e87229dc77cb7c814ab9733ea7013 Author: Alistair Coles <alistair.co...@hpe.com> Date: Fri Feb 19 09:30:10 2016 +0000 Silence SkipTest noise in in-process test teardown setup_package and teardown_package get called twice when running functional tests using nose. With in-process mode the first call to teardown_package clears global config which causes the test swiftclient to raise a SkipTest error during the second call to teardown_package. To reproduce: SWIFT_TEST_IN_PROCESS=1 nosetests ./test/functional/test_account.py This patch simply tests for config existence before attempting to create a test swiftclient in the teardown_package function. Also fix a related issue whereby in_process flag would be reset to False during second call to setup_package, thus causing some of the in process cleanup in teardown_package to never be executed. Change-Id: I074dcd3d39aa46b262632024b047556ca471e8b8 commit 076134784b4eb1c2e55b28218dc0d64a57795d63 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Tue Feb 23 17:28:08 2016 -0800 Add warn method back to LogAdapter The change in 684c4c04 to improve py3 support breaks older middleware that consumes the SwiftLogger. Good for Swift code to not to use warn because it's deprecated. Bad for LogAdapter to not faithfully provide the current python logging module API (deprecated or otherwise). Change-Id: Ib0baf66b7f33f91562867076ff0526a287d8e5a6 commit eb7ca115e6501edd9225dce8784b8b3d3e094bb1 Author: Samuel Merritt <s...@swiftstack.com> Date: Tue Feb 23 14:00:34 2016 -0800 Fix StatsD tests to not use real DNS In common/test_utils.py, TestStatsdLogging had the majority of its test cases calling the real socket.getaddrinfo(), which uses real DNS. This is very slightly slower than using a mock getaddrinfo() when the machine running the tests has functioning DNS, but on a machine with no network connection at all, the tests are excruciatingly slow due to timeouts. This commit mocks things out as appropriate. There's still one user of the real getaddrinfo(), but it's for ::1, so that's just local resolution based on /etc/hosts. Timing numbers for "./.unittests test.unit.common.test_utils:TestStatsdLogging": * network, without this patch: 1.8s * no network, without this patch: 221.2s (ouch) * network, with this patch: 1.1s * no network, with this patch: 1.1s Change-Id: I1a2d6f24fc9bb928894fb1fd8383516250e29e0c commit b97adf70ade2982fea5354c55f1eea9f98442fa7 Author: Thiago da Silva <thi...@redhat.com> Date: Tue Feb 23 16:23:51 2016 -0500 fix formatting of bulk operations doc Change-Id: I247d2867c55d6bbf2288dfb74271f57c6f88b243 commit cfd50212358372694ba0713ff8c3bc313c6f1a92 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Thu Feb 18 21:06:40 2016 -0800 dry out container unittests Change Ifcc4392ffa4df485ce3c62a35d0fcc8c93de74aa introduced TestContainerController4Replicas as a subclass of TestContainerController. And there is duplicated code block to override setUp method. This patch removed the duplicated code block. Change-Id: Iad0b64cba52554b89dbd959475a2ce7c0d9a20eb commit 406e8996fb288e9851e02c3f3faa536a9919b28f Author: Takashi Kajinami <kajina...@nttdata.co.jp> Date: Mon Feb 22 16:10:09 2016 +0900 Use 201 insted of 200 in PUT test case There are some unit test cases which assumes 200 as the response of PUT request, but 200 is never returned in PUT. This patch replaces 200 in PUT by 201, and adapt unit test cases to real case. Change-Id: I35fc7a4fe7fcc5558a9e6c6371105793c2710f3e commit 5b18455cef57d052df3cda3e9179c923d3aadcad Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Fri Jan 22 07:06:45 2016 +0000 Add unit tests for checking responses in ContainerController This patch provides unit tests to check responses for various situation in three/four replicas env. Change-Id: Ifcc4392ffa4df485ce3c62a35d0fcc8c93de74aa commit 002513d21786df0d6c71e9772ec4688e2c14d8de Author: Tim Burke <tim.bu...@gmail.com> Date: Thu Jan 7 11:56:33 2016 -0800 Make write-only container access consistent Previously, if a user could write to (but not read from) a container, the behavior for object POST would vary depending on whether object_post_as_copy was enabled (403 response) or disabled (202 response). Now, POSTs will consistently be allowed, regardless of whether fast-POST is enabled. Change-Id: I1d6dcbc4f5034a322a1073850fc3b059ebb1c0fa commit 9a6c6b55713f896609b7127560ae99fabbf759e7 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Thu Feb 18 17:38:42 2016 +0100 Fixed log format in object auditor Change-Id: I218e08446c5ffda2db7c774ec8bb9d28ce9f63e7 commit 2d55960a221c9934680053873bf1355c4690bb19 Author: Alistair Coles <alistair.co...@hpe.com> Date: Thu Jan 14 18:31:21 2016 +0000 Fix inconsistent suffix hashes after ssync of tombstone Consider two replicas of the same object whose ondisk files have diverged due to failures: A has t2.ts B has t1.data, t4.meta (The DELETE at t2 did not make it to B. The POST at t4 was rejected by A.) After ssync replication the two ondisk file sets will not be consistent: A has t2.ts (ssync cannot POST t4.meta to this node) B has t2.ts, t4.meta (ssync should not delete t4.meta, there may be a t3.data somewhere) Consequenty the two nodes will report different hashes for the object's suffix, and replication will repeat, always with the inconsistent outcome. This scenario is reproduced by the probe test added in this patch. (Note that rsync replication does result in (t2.ts, t4.meta) on both nodes.) The solution is to change the way that suffix hashes are calculated. Currently the names of *all* files found in each object dir are added to the hash. With this patch the timestamps of only those files that could be used to construct a valid diskfile are added to the hash. File extensions are appended to the timestamp so that in most 'normal' situations the result of the hashing is the same as before this patch. That avoids a storm of hash mismatches when this patch is deployed in an existing cluster. In the problem case described above, t4.meta is no longer added to the hash, since it is not useful for constructing a diskfile. (Note that t4.meta is not deleted because it may become useful should a t3.data be replicated in future). Closes-Bug: 1534276 Change-Id: I99e88b8d5f5d9bc22b42112a99634ba942415e05 commit ddeb0cde9fc828fa79dce90df4975f66737ba054 Author: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Date: Wed Feb 17 21:17:34 2016 -0800 Fix object-server to handle newer ts file Swift object-server will return 409 Conflict when the incomming request X-Timestamp is older than the local disk file to prevent wasted data transfer and disk space consumption. However, IFF, the local disk file is a tombstone, current object-server will write the data in the device and cleanup it according to timestamp comparison result in hash_cleanup_list_dir, and then return 201 Created. That's wasted and far from the semantics in the normal case. This patch fixes object-server to catch the DiskFileDeleted exception at PUT request, and then, set the timestamp from tombstone as original_timestamp to be able to compare with request timestamp. Co-Authored-By: Kazuhiro Miyahara <miyahara.kazuh...@lab.ntt.co.jp> Change-Id: I078c9cb90707a3c320708e76ea42cbfa73e1ea4b Closes-Bug: #1546865 commit 878688cae00576ec0bdb0a66eca7ada95e661c0a Author: Béla Vancsics <vancs...@inf.u-szeged.hu> Date: Mon Dec 7 17:02:40 2015 +0100 Reduce code duplication Reduced source code by extracting duplicated code (swift/common/direct_client.py) http://openqa.sed.hu/dashboard/index/5?did=1 Change-Id: I47fa6ae12c646e8e42fb0a46a017971f05d883fc Closes-Bug: 1536067 commit c3201f256cf415988a3cb1c42e9c6b7835195fa3 Author: Kazuhiro MIYAHARA <miyahara.kazuh...@lab.ntt.co.jp> Date: Fri Jan 8 18:03:33 2016 +0900 Remove execute permissions from doc files and swift.conf-sample Some doc files and swift.conf-sample were added execute permissions in past changes. This patch remove execute permissions from them. Change-Id: Id8844989a8321578e9207566ebd6660f5b9523f0 commit e46d6b17251ab538595b1deb5581360a286b8fd1 Author: Christian Schwede <cschw...@redhat.com> Date: Wed Feb 17 20:47:26 2016 +0100 Add note on using printable chars for swift_hash_path_suffix/prefix Using unprintable characters for swift_hash_path_prefix/suffix might lead to hard problems when parsing of these values changes, for example due to newer Python versions or changes in the parsers itself. Let's avoid this and add a note that deployers should use printable strings for these values. Change-Id: I976982b753b6af831ab91d7190f50f8f15bf73bf commit b8fa48080812ab0f6bac7ae19c0290da08f6a6e4 Author: Donagh McCabe <donagh.mcc...@hp.com> Date: Thu Feb 4 16:19:13 2016 +0000 Added links to API reference and usage guides Added a link to the API reference (where headers and query strings are documented; click the "detail" button to see). Also added a reference to Swift section of the OpenStack end user guide. This contains some additional details about the API. No attempt was made to reconcile duplicate information. Instead this patch links documents that might otherwise be overlooked. However, I fixed text, originally in a table, that had become garbled in a prior patch. Change-Id: I0910cbeb0c8bffc00e510f35585603e7b7a67790 commit aa7204d106ae33eba9219514f31f008510c9db53 Author: Kazuhiro MIYAHARA <miyahara.kazuh...@lab.ntt.co.jp> Date: Fri Feb 12 20:48:34 2016 +0900 Remove '#! /usr/bin/env python' from unexecutable files 'cli/recon.py' and 'cli/ring_builder_analyzer.py' have '#! /usr/bin/env python' in spite of they don't have execute permissions. This patch removes '#! /usr/bin/env python' from them. Change-Id: I1917ccc84b1673af3d862be1796f54595f94c5ca commit db87a5a9889d436f2e2aacdc8923d55ca0d3d1ca Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Tue Feb 16 10:52:01 2016 -0800 Mock sleep in direct_client retry tests ... and fixup some assertions, and other mocks Change-Id: Iadf839d1d3d735109fed24049fc3d829776b7b65 commit 8eb30afd7c39f6e6ba031f72c6a61f45b28d37f6 Author: Tim Burke <tim.bu...@gmail.com> Date: Tue Feb 16 10:25:39 2016 -0800 Static methods should be @staticmethods Change-Id: Ifee5d68e00bbb3571aaac885cdd7490c79732985 commit 4a44e27e00795c5f12176ef346cd2249fca54fd8 Author: Chaozhe.Chen <chaozhe.c...@easystack.cn> Date: Tue Feb 16 23:44:27 2016 +0800 Replace assertEqual(None, *) with assertIsNone in tests As swift no longer supports Python 2.6, replace assertEqual(None, *) with assertIsNone in tests to have more clear messages in case of failure. Change-Id: I94af3e8156ef40465d4f7a2cb79fb99fc7bbda56 Closes-Bug: #1280522 commit 42f4b3fc1e69a046c1e70e73afe847e4423fa2da Author: Christian Schwede <cschw...@redhat.com> Date: Tue Feb 16 10:08:55 2016 +0000 Add SwiftHLM to associated projects Change-Id: I5ea3de34e61f22abec803b35fec8adde18a793e9 commit 025ebf2901d5a972796b73538bb306847d7d06b6 Author: Ankur Gupta <ankur.gu...@intel.com> Date: Wed Feb 10 14:36:16 2016 -0600 Docstring omission in class BaseDiskFileManager. Added documentation for missing Docstring variables. Change-Id: I29a53b8141c5607815f234a6123e2289200bca34 commit 4906b4c431edc436f165b163a228b0a221950c79 Author: Janonymous <janonymous.codevult...@gmail.com> Date: Tue Jan 12 12:50:43 2016 +0530 Fix missing txn_id logs in GreenAsyncPile's spawned functions This commit ensures that the logger thread_locals value is passed to and set in _get_conn_response methods executed in a green thread. Added partial bug tag because in bug description a more relevant fix is suggested which would fix the bug completely but for now this makes sense to add this commit for logging. Co-Authored-By: Clay Gerrard <clay.gerr...@gmail.com> Change-Id: I13bbf174fdca89318d69bb0674ed23dc9ec25b9a Partial-Bug: #1409302 commit 511a1b8698889c430bf673ed916ae41a96b99c5c Author: Kazuhiro MIYAHARA <miyahara.kazuh...@lab.ntt.co.jp> Date: Fri Feb 12 19:01:59 2016 +0900 Remove execute permission from swift/cli/ringbuilder.py In Change-Id: Ia63f59a8faca1fad990784f27532ca07a2125454, 'bin/swift-ring-builder' is renamed to 'swift/cli/ringbuilder.py'. However, execute permission and '#! /usr/bin/env python' are left on the file. This patch removes execute permission and '#! /usr/bin/env python' from the file. Change-Id: I66370f825ce9ed9a40ec21f2745aa6aff22c8709 commit 973d9358d2e6e029409d1902831b4680e02952c9 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Thu Feb 11 10:20:10 2016 +0100 Bump PyECLib requirement to >= 1.2.0 This was discussed at last Swift meeting [1] and conclusion was to require newest PyECLib version. [1] http://eavesdrop.openstack.org/meetings/swift/2016/swift.2016-02-10-21.00.log.html#l-268 Depends-On: I0fd58f26bc0a3e7715356ff74344981f22786152 Change-Id: I54e89d99b48e00e614d2d457814c00354763c83b commit d3dd137cad896c7f741592e88f076acab2636a3c Author: Tomas Matlocha <tomas.matlo...@firma.seznam.cz> Date: Tue Jan 5 12:32:50 2016 +0100 Additional info log message for drive-audit An additional info log message was added for case of running drive-audit without failed device unmounting. Change-Id: I11abee40a712b6c6de65e63626b6f7f0a9c9f4c7 commit c4158bd674f34c8a0648c98974ba474fcd1c2b7a Author: Sarafraj Singh <sarafraj.si...@intel.com> Date: Thu Feb 11 20:52:47 2016 +0000 Fixed params on autocreate_account docstring ... in proxy.controllers.base Change-Id: Ic5c27cbf0f19b65fcaa3e6587583f03e56d646a0 commit f56d18e143d0a38584e4c383ea9afe46e67a0af6 Author: Samuel Merritt <s...@swiftstack.com> Date: Thu Feb 11 10:59:43 2016 -0800 Fix typos in comments Change-Id: I4f98d447bd2ddd8eeb2f4da66d069bd7d6f00dc6 commit cc8dc88b55b44ca2a1fd65c049f3cfec22eb5525 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Thu Feb 4 16:02:42 2016 +0100 Script for checking sanity of manpages Change-Id: Ia27b0ecf6a3267eb51f8a6562721dca72f128915 commit bcada66b9065c94cd39498a5330c727cc575a6fe Author: root <lr6...@att.com> Date: Wed Feb 10 09:21:53 2016 -0600 Removed unused parameter in server.py The variable max_large_object_get_time is no longer used and was removed to reflect the change. Change-Id: I43051181dcb38245de6d13fab63876e83f46fc39 Closes-Bug: #1538834 commit b5311f63dbf5787bda3495a58c8ef7a6a557daa4 Author: gh159m <gh1...@att.com> Date: Wed Feb 10 10:36:59 2016 -0600 Removed default value for log_statsd_host Multiple files and documents showed that log_statsd_host had a default value, usually localhost. This was incorrect, instead setting a value for log_statsd_host enables statsd logging. Removed any reference of log_statsd_host having a default value. Also changed descriptions to show setting a value enables logging. Change-Id: I3ca5c0e8b8e4981de3aa6db0c476072b5a59723d Closes-Bug: #1542227 commit b98c1c60f9c581c76677c19baae60ae720e3c7f3 Author: Christian Berendt <bere...@b1-systems.de> Date: Tue Feb 9 07:32:48 2016 +0100 Remove executable flag from sample config file swift.conf-sample should not be executbale Change-Id: I960058e2af10b43efc6324a7534d9a23c810e39a commit 81a4355c2d0907035805c300a9fa5aaa6d0e70c4 Author: Christopher Bartz <ba...@dkrz.de> Date: Fri Feb 5 09:51:11 2016 +0100 Adds storage policy option to recon With this patch, recon gets policy-aware in regard to the hosts to retrieve. If no policy is passed and the server_type is object, all hosts of all policies are retrieved. Previously, recon did only retrieve the hosts of the storage-policy 0. Change-Id: If5735cd6721eac504aed8aaf3884cb91b6a0fcac Closes-Bug: 1541491 commit 87fc21c7cfcce2a3e23a84fddcfd1309cd884716 Author: Alistair Coles <alistair.co...@hpe.com> Date: Mon Feb 8 15:31:42 2016 +0000 Speed up functional testing test/functional/tests.py:TestObjectVersioningUTF8 does not clean up the versions files it creates because the class's multiple inheritance is such that it does not call the tearDown method in TestObjectVersioning. As a result, any attempt to clean up account containers wastes time retrying container delete requests. This occurs either in the setUp for TestSloEnv, if the TestSlo class is included in a test run, or in the tests.py package tearDown method. On the author's dev machine this patch reduces the execution time of functional tests in tests.py by approx 30% or 1 minute. Change-Id: I8194672bf2ca82435df5868720b6a55a79b94413 commit 3b94bd45406f9d457d37caa10c856a19c4cec22a Author: Mingyu Li <li.min...@99cloud.net> Date: Sun Feb 7 21:41:08 2016 +0800 Keep the Usage of exit()/sys.exit() Consistent It's better to keep the usage of exit()/sys.exit() consistent in one file. Furthermore, sys.exit() is considered good to be used in production code, while exit is for interactive shell. Change-Id: Ia3092853a648922588e2bc11db37d6decdec1b48 commit f05da4c8fa37a285bb2ca9b45a355dbe57f7add4 Author: janonymous <janonymous.codevult...@gmail.com> Date: Sat Feb 6 16:59:57 2016 +0530 Changing the base version to python2.7 for pep8 checks tox was taking python3 as it's base version which lead to tox pep8 check failing on local env: http://paste.openstack.org/show/486183/ Change-Id: Ia822ec5b5e12a1fc30b676f1e6d95e287b9b3368 commit 13edc9a865cd203ddf28a3f782426a1b3b94f7a1 Author: Sarafraj Singh <sarafraj.si...@intel.com> Date: Fri Feb 5 17:28:26 2016 +0000 Improved test coverage of form_signature Added four new test cases to execute path of invalid arguments. This improved test coverage of file specified from 80% to 91%. Change-Id: I63c2e7bab3f01121301d78b687687208a58401c0 commit b173995666b026def3f6558d9a8c972b75449323 Author: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Date: Thu Feb 4 05:18:42 2016 -0800 Fix missing Accept-Ranges Since commit 4f2ed8bcd0468f3b69d5fded274d8d6b02ac3d10, the response header for GET EC object doesn't include the Accept-Ranges header. This patch fixes it and also adds a few unittests to prevent regression. Closes-Bug: #1542168 Change-Id: Ibafe56ac87b14bc0028953e620a653cd68dd3f84 commit d78e861d850db96989f4fb77ad31e3170034f15d Author: Richard Hawkins <richard.hawk...@rackspace.com> Date: Wed Jan 13 13:15:07 2016 -0600 Add option for label to static web listings By default listings are rendered with a label of: "Listing of /v1/account/container/path" This change adds a new custom metadata key/value pair "X-Container-Meta-Web-Listings-Label: <label>" that when set, will cause the following: "Listing of <label>/path" to be rendered instead. Change-Id: I47c5d21834f47e566b380920196926882ff83abf commit ae632abbd802228ef98a23c98ee49988e7e5b942 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Tue Feb 2 21:58:52 2016 +0100 Fixed manpages errors. account-server.conf.5 105: warning: numeric expression expected (got `)') container-server.conf.5 111: warning: numeric expression expected (got `)') object-expirer.conf.5 79: warning: numeric expression expected (got `)') object-server.conf.5 114: warning: numeric expression expected (got `)') proxy-server.conf.5 121: warning: numeric expression expected (got `)') 331: warning: numeric expression expected (got `[') 1005: warning: macro `*' not defined Change-Id: I203dcfde83035e3b1dcb91109b72b5d08bb7840e commit f2fca9aafa17e45b668c6f76b256927681cc463a Author: Alistair Coles <alistair.co...@hpe.com> Date: Tue Feb 2 11:43:44 2016 +0000 Fix time-shifting of objects PUT with container-sync When container-sync PUTs an object to a destination container it uses the timestamp from the container row rather than the actual timestamp of the object being copied. The actual timestamp of the object can be newer, so the sync'd object may end up with the right content but at the wrong, older, timestamp. This patch changes the timestamp sent with the sync'd object to be that of the actual source object being sent. Drive-by fix to make code more readable by removing a variable rename mid-function, fix a typo and remove a redundant function call. Change-Id: I800e6de4cdeea289864414980a96f5929281da04 Closes-Bug: #1540884 commit 050e3b35dd88f7b31b6810d9ac88548ef371e3ce Author: venkatamahesh <venkatamaheshko...@gmail.com> Date: Thu Feb 4 16:47:15 2016 +0530 Use uppercase 'S' in word "OpenStack" Change-Id: I4692aa58b5173b1b662d5eee19fac770ae5aaed0 commit e47aaaacf15e8f0c7226400541a97238447a7f56 Author: Tim Burke <tim.bu...@gmail.com> Date: Wed Feb 3 12:52:29 2016 -0800 Stop nesting functions unnecessarily Change-Id: Iff120d0bac8a075c37bbddcd2bb0fe85145f1749 commit 26327e1e8b1d37faa764ec586f5bee0e1560eea2 Author: Darrell Bishop <darr...@swiftstack.com> Date: Thu Jan 21 11:18:18 2016 -0800 Allow IPv6 addresses/hostnames in StatsD target The log_statsd_host value can now be an IPv6 address or a hostname which only resolves to an IPv6 address. In both cases, the new behavior is to use an AF_INET6 socket on which .sendto() is called with the originally-configured hostname (or IP). This means the Swift process is not caching a DNS resolution for the lifetime of the process (a good thing). If a hostname resolves to both an IPv6 or IPv4 address, an AF_INET socket is used (i.e. only the IPv4 address will receive the UDP packet). The old behavior is preserved: any invalid IP address literals and failures in DNS resolution or actual StatsD packet sending do not halt the process or bubble up; they are caught, logged, and otherwise ignored. Change-Id: Ibddddcf140e2e69b08edf3feed3e9a5fa17307cf commit 2f8e6040470948cf17c5fc5fddd0991a415d2a4c Author: Eran Rom <er...@il.ibm.com> Date: Wed Feb 3 08:59:53 2016 +0200 Fixing typo in unit test The typo does not influence the correctness of the test yet its ugly. bhhha. Change-Id: I62e511bae23523892b87309dd3c50aea61742f2d commit f27ad34e1d4e5dc306eed3ff620a294a4c301f31 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Tue Feb 2 11:44:39 2016 -0800 Document use-case for slow option Change-Id: Iec4087a896a2277179e3720d802cca101fa7ad54 commit 16976a0f14ee2b5f7abc76086f9abfb9f337a655 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Sun Jan 31 10:30:06 2016 +0100 Changed EC backend from jerasure to liberasurecode in examples and docs liberasurecode_rs_vand is build-in liberasurecode, so you don't need another depedency libjerasure2. liberasurecode_rs_vand is supported by pyeclib from 1.0.8 version, so bumping version up. Closes-Bug: #1534325 Change-Id: If2d96875694df8fd48c5278395859aaa165cb566 commit 30d74af6534b754e6ad9bfdcbff4ec494277ca83 Author: Alistair Coles <alistair.co...@hpe.com> Date: Wed Jan 27 13:50:57 2016 +0000 Insert versioned_writes in correct pipeline position If not explicitly configured the versioned_writes middleware should be auto-inserted in the pipeline after slo and dlo, which is where the versioned_writes filter section's comments say it should be in proxy-server.conf-sample. At the moment it can end up being placed ahead of slo and dlo if they have been explicitly configured, which results in the linked bug manifesting. Closes-Bug: #1537042 Change-Id: I6ac95a331f4ef0d4887311940acc6f8bc00fb4eb commit ccdf4a9f309af5fe1401a6e4b776b08c3f6a655c Author: Christian Schwede <cschw...@redhat.com> Date: Tue Feb 2 09:38:55 2016 +0100 Document slow option in etc/object-server.conf Change-Id: Ic9940b0b830a468887878f7b0d7ca42c2cbbebd5 commit b19dc1ddecdc68760ca2f5a2b7b4bbdce2199b35 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Mon Feb 1 15:30:59 2016 -0800 Always fix devices with multiple part-replica assignments I've found that given a sufficiently bad replica2part2dev table we can accidently not entirely fix palcement when more than two replicas of a part are assigned to the duplicate devices. It shows up most on > 3 replica rings when you have two *different* devices both holding two replicas. But you can see it on a three replica ring when all three replicas are assigned to the same device. Change-Id: Ieb213c1a259815a2ed657291242919cda568c7b5 commit c722fa1ab318b233fc11f8e1bd8758dbe33e87bb Author: dangming <dangm...@unitedstack.com> Date: Mon Feb 1 16:13:54 2016 +0800 Fix spelling error Change-Id: I4a1400c4d383acd59727ada3661a8dffc67825fb commit c7e5e125003de1a0f288f79783e9a41650e6f686 Author: Alistair Coles <alistair.co...@hpe.com> Date: Fri Jan 29 14:22:19 2016 +0000 Enable in-process func tests to optionally use fast-post Running functional tests in the in-process mode uses the default value for proxy-server object_post_as_copy, which is True. This patch adds support for an environment variable to change this value to False so that the fast-post mode is tested when running in-process functional tests. The patch also adds a new tox environment func-in-process-fast-post which forces in-process functional testing with object_post_as_copy=False. The motivation for this change, apart from enabling configurable local testing, is to put support in place for an upstream CI job that will actually functionally test the fast-post mode, which is otherwise only covered by unit tests. There are currently two gate jobs that run the functional tests: - gate-swift-dsvm-functional runs the tests *twice* against a devstack swift service, once using tempauth and once using keystoneauth. The devstack swift service uses the default object_post_as_copy=True. - gate-swift-tox-func runs the func tests in in-process mode which also uses tempauth and object_post_as_copy=True. This duplicates one of the config scenarios above. With this change either the gate-swift-tox-func job or a new job could run the functional tests using object_post_as_copy=False. Change-Id: Ia37f6df1dc38e44ef7404fbf0a52f6fc22fae0c2 commit 4ba290bac19b1b6e8f709848fd35cda33b8bd1c4 Author: Qiaowei Ren <qiaowei....@intel.com> Date: Mon Feb 1 16:44:52 2016 +0800 Replace exit() by sys.exit() sys.exit() is considered good to use here. Change-Id: I365b0dbb50c8f1953b41665ba49145ac6db1b24f commit 9aff76bd18e609eb657db14f2d60e605ee28d472 Author: Qiaowei Ren <qiaowei....@intel.com> Date: Mon Feb 1 17:15:32 2016 +0800 fix typo informations -> information Change-Id: I6bcb9bb9fd24b873b49de4a686b223550dd703ac commit a8e0dc550bf35010595ce791433cd68c4b8d5287 Author: OpenStack Proposal Bot <openstack-in...@lists.openstack.org> Date: Sun Jan 31 06:15:17 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: If1cd20e187720f5ee02c6a1574f612c8a8e26e30 commit 02c06585e6e0403d1a983fccaba5093e3d3fa7c5 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Fri Jan 29 22:46:50 2016 +0100 Renamed variable for better code readability Change-Id: I22d8db0dd9edc39672fc9997895a24f669975e15 commit 595ec464e3fdc5a766eee9baa06edd074576c73e Author: Alistair Coles <alistair.co...@hpe.com> Date: Fri Jan 29 15:28:56 2016 +0000 Trivial fix for sphinx warnings Including a * in the param name causes a sphinx warning, messes up the html formatting, and strictly the * isn't part of the param name. Change-Id: I6be6a7aa2a371eaae9f8e501435838c1e15f4a1c commit cc90834a8ed7bd84440bf132a94978a47ee2f3da Author: Mingyu Li <li.min...@99cloud.net> Date: Fri Jan 29 03:48:47 2016 +0900 A minor modification of the Deployment Guide In the section of General Service Tuning, there is a sentence "Our Storage servers all run together on the same servers". The word "servers" is used twice with different meanings, which would be a little confusing. This patch makes it more understandable. Change-Id: I6d806456aeb7d0ecf297c04dacbd271b94a0425c Closes-Bug: #1526697 commit a1776b9c1f23fbd604114b377434e4269d2f1a2e Author: Alistair Coles <alistair.co...@hpe.com> Date: Thu Jan 28 16:47:37 2016 +0000 Let equal Timestamps not be unequal Make the result of Timestamp(x) != Timestamp(x) be False. In python 2.7 this requires the __ne__ method to be defined [1]. "The truth of x==y does not imply that x!=y is false." The functools.total_ordering decorator does not autocreate a __ne__ method. In python 3 the __ne__ method is not required [2]. "By default, __ne__() delegates to __eq__() and inverts the result". This patch puts back the __ne__ method removed in [3]. Whilst no tests fail on master with python2.7, they do on this patch [4] and it seems dangerous to have this absurd behaviour lurking. [1] https://docs.python.org/2/reference/datamodel.html#object.__ne__ [2] https://docs.python.org/3.4/reference/datamodel.html#object.__ne__ [3] Change-Id: Id26777ac2c780316ff10ef7d954c48cc1fd480b5 [4] Change-Id: Ia597cd460bb5fd40aa92e886e3e18a7542603d01 Change-Id: I01fbfa310df3c74390f8e8c2e9ffff81bbf05e47 commit c3886eea15f11104764fb201e85202dd871c3027 Author: Thomas Goirand <tho...@goirand.fr> Date: Thu Jan 28 09:15:28 2016 +0000 Fix a few English mistakes in man These errors are producing lintian warnings, so fixing them helps having less errors when checking for Debian packages. Change-Id: Iff99a8d5f2276515f42d758d110a43cae757db28 commit 400cc707273bab0b70a3cdf5dec1c55c2edc4e22 Author: Alistair Coles <alistair.co...@hpe.com> Date: Wed Jan 27 15:35:40 2016 +0000 Remove a print statement from tests Change-Id: I213b7e4bc4f9e6de76ebab572c377af61072d9c5 commit c871e423ef0bed6c774693af3d54750235d6ce1f Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Fri Jan 22 05:19:06 2016 +0000 Add unit tests for checking responses in AccountController This patch provides unit tests to check responses for various situation in three/four replicas env. Change-Id: I1af48728b28da9780a2a78f3110e1c7487047ff8 commit e45d30d6b7b596b80c63e75018ad07582d705ee1 Author: Alistair Coles <alistair.co...@hpe.com> Date: Wed Jan 27 10:18:37 2016 +0000 Remove unnecessary code from test_account.py Change-Id: Ia197ba08a606421244d5b94a900bdc39bc5da647 commit f39cffbb5f7f1853f7f67261c4652b9875d302be Author: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Date: Tue Jan 26 20:16:08 2016 -0800 Follow up mem_server diskfile fixes This is a follow-up patch for change https://review.openstack.org/#/c/270422 That patch fixes POST behavior on object mem server and adds some tests for last-modified. However, that one doesn't check the POST updates the last-modified on object server. This patch adds assertions to make sure the actual POST semantics. *bonus* Add docs and cleanup variable name to fit actual instance at mem diskfile. Change-Id: I78185cdfb0ea604b2122821870ec34322236f48d commit 9ef15453fadb847db0f0fb92e935785b544da3fc Author: Kazuhiro MIYAHARA <miyahara.kazuh...@lab.ntt.co.jp> Date: Thu Jan 21 14:18:31 2016 +0900 Fix handling of "Permission Denied" error from NamedTemporaryFile function If "Permission Denied" has happen in NamedTemporaryFile function in dump_recon_cache method, swift will log a message of reference to a variable without assignment and not log a message of "Permission Denied". This patch fixes the handling and add an unit test. Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Change-Id: Iafdd94905e9e9c81f5966a923324b50c18fcf592 commit 01410129dac6903ce7f486997a48e36072fa0401 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Fri Aug 21 18:15:25 2015 -0700 Make handoffs_first a more useful "mode" Based on experience using handoffs_first and feedback from other operators it has become clear that handoffs_first is only used during periods of problematic cluster behavior (e.g. full disks) when replication attempts are failing to quickly drain off the partitions from the nodes which they have been rebalanced from. In order to focus on the most important work (getting handoff partitions off the node) handoffs_first mode will abort the current replication sweep before attempting any primary suffix syncing if any of the handoff partitions were not removed for any reason - and start over with replication of handoffs jobs as the highest priority. Note that handoffs_first being enabled will emit a warning on start up, even if no handoff jobs fail, because of the negative impact it can have during normal operations by dog piling on a node that was temporarily unavailable. Change-Id: Ia324728d42c606e2f9e7d29b4ab5fcbff6e47aea commit 78cfb137d5d3d6f4770923a1fb086adaa9c0676f Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Wed Nov 11 19:05:09 2015 -0800 Ring Overview updates Change-Id: Ic5a8ec5be9fec7fa812fdb7b91890d5fc2fbf967 commit 24227e8bb0d4d369eeab61ab5c20730b4f84b36a Author: Richard Hawkins <richard.hawk...@rackspace.com> Date: Wed Jan 13 10:16:29 2016 -0600 _get_info_cache should encode unicode metadata Closes-Bug: 1533768 Change-Id: I67bbd0e8802d2a34cb3a0b0fafe26bbc62221032 commit fc9a3852e44b2bf1258fcb24a6c12ecfb2795f44 Author: Tim Burke <tim.bu...@gmail.com> Date: Mon Jan 25 09:25:20 2016 -0800 Don't .lower() a literal It feels silly, and we don't do it for any of the other headers in headers_to_container_info. While we're at it, clean up a stray ' Change-Id: I0745038cc3832a77d064e515c37cacbdcb97c4d9 Related-Change: Iea3d06de80210e9e504e296d4572583d7ffabeac commit 5c6576d1eb3c4c042bb46dd48f23d2fd474eba07 Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Mon Jan 25 18:20:45 2016 +0100 Fixed docs: Title underline too short Change-Id: If96b0e799a4671e7b7f3f5bea2628db75afa9578 commit 9c403d5e1bb1f404445f1731e3ff1b5c034336a6 Author: Victor Stinner <vstin...@redhat.com> Date: Mon Oct 19 16:31:12 2015 +0200 Port swift.common.utils.Timestamp to Python 3 * Fix Timestamp.isoformat(): on Python 3, round manually to use the same rounding method than Python 2. * Use @functoos.total_ordering on Timestamp instead of the __cmp__() special method. On Python 3, the __cmp__() method is no more used, and other special methods must be defined (__eq__, __lt__, etc.). Only define __eq__() and __lt__() methods but use @functools.total_ordering to generate other methods. * Fix also a Python 3 issues in swift.common.swob._datetime_property: replace (float, int, long) with (float,) + six.integer_types. The long type was renamed to int in Python 3. Change-Id: Id26777ac2c780316ff10ef7d954c48cc1fd480b5 commit c790db89369221473bcbfc23f301424ab8d30d60 Author: Andrew Welleck <awel...@us.ibm.com> Date: Mon Jan 25 10:03:18 2016 -0600 Additional step for Debugging Issues in SAIO Adding an additional debugging step for the SAIO "Debugging Issues" section. Adding step to check if memcache is running and to start it if its not. Change-Id: I0bfa563f595dbefcf907494451fad45e6310db88 commit d47155af262d52c284f262bf6c303d0587000bba Author: Victor Stinner <vstin...@redhat.com> Date: Mon Oct 19 17:22:24 2015 +0200 Add __next__() methods to utils iterators for py3 On Python 3, next(obj) calls obj.__next__(), not obj.next(). Add an alias from __next__() to next() to be compatible with Python 2 and Python 3. Change-Id: Ida104d3bd7cdba557e523f18df43d56847060054 commit 0f8319dceb2d21a9d1ec74f78203b033f25c0969 Author: Tim Burke <tim.bu...@gmail.com> Date: Fri Jan 22 13:31:34 2016 -0800 Use calendar.timegm not time.mktime time.mktime produces a POSIX timestamp that takes into account the local timezone, but we're parsing a Last-Modified header with a known (GMT) timezone. As a result, if one proxy-server is configured with a timezone other than GMT/UTC, we may have previous versions in the wrong order. Change-Id: I320e2368b243f4245725e73bfabc7ad19bc5bacb commit 72d972166ff66d3c5313356c1e1e3984fee9f012 Author: paul luse <paul.e.l...@intel.com> Date: Thu Jan 21 13:30:15 2016 -0700 Update AUTHORS +Luse core emeritus Change-Id: Ie175f410ddc24a2d5e64bc191364418a3c450f3f commit 2f8105e5fcf4ecee2202797ba1ab48fa8fb1a1f6 Author: Alistair Coles <alistair.co...@hpe.com> Date: Wed Jan 20 18:14:16 2016 +0000 Fix bug in mem_diskfile write_metadata method The mem_diskfile DiskFile.write_metadata method was apparently never called in any existing test, as if it were it would blow up as reported in the bug. This patch fixes the method and adds a test that exercises it. The test addition itself should be useful since it verifies the behaviour of Last-Modified after POSTs to an object. Drive-by fixes for bad docstring and undefined references in the _quarantine method. Change-Id: I17fd62e5f02be5b48bfd9ba7fa25315e30a0a4bf Closes-Bug: #1536037 commit e13a03c379273ee10e678818078b9c40a96a7dc9 Author: Tim Burke <tim.bu...@gmail.com> Date: Wed Jan 20 16:06:26 2016 -0800 Stop overriding builtin range Change-Id: I315f8b554bb9e96659b455f4158f074961bd6498 commit 0a404def7d54d1ef1c85c11a378052260c4fda4c Author: John Dickinson <m...@not.mn> Date: Wed Jan 20 15:19:35 2016 -0800 remove unneeded duplicate dict keys Change-Id: I926d7aaa9df093418aaae54fe26e8f7bc8210645 commit 221f94fdd39fd2dcd9a2e5565adceab615d55913 Author: John Dickinson <m...@not.mn> Date: Tue Jan 19 14:50:24 2016 -0800 authors and changelog updates for 2.6.0 Change-Id: Idd0ff9e70abc0773be183c37cd6125fe852da7c0 commit 58359269b0e971e52f0eb7f97221566ca2148014 Author: Samuel Merritt <s...@swiftstack.com> Date: Tue Dec 8 16:36:05 2015 -0800 Fix memory/socket leak in proxy on truncated SLO/DLO GET When a client disconnected while consuming an SLO or DLO GET response, the proxy would leak a socket. This could be observed via strace as a socket that had shutdown() called on it, but was never closed. It could also be observed by counting entries in /proc/<pid>/fd, where <pid> is the pid of a proxy server worker process. This is due to a memory leak in SegmentedIterable. A SegmentedIterable has an 'app_iter' attribute, which is a generator. That generator references 'self' (the SegmentedIterable object). This creates a cyclic reference: the generator refers to the SegmentedIterable, and the SegmentedIterable refers to the generator. Python can normally handle cyclic garbage; reference counting won't reclaim it, but the garbage collector will. However, objects with finalizers will stop the garbage collector from collecting them* and the cycle of which they are part. For most objects, "has finalizer" is synonymous with "has a __del__ method". However, a generator has a finalizer once it's started running and before it finishes: basically, while it has stack frames associated with it**. When a client disconnects mid-stream, we get a memory leak. We have our SegmentedIterable object (call it "si"), and its associated generator. si.app_iter is the generator, and the generator closes over si, so we have a cycle; and the generator has started but not yet finished, so the generator needs finalization; hence, the garbage collector won't ever clean it up. The socket leak comes in because the generator *also* refers to the request's WSGI environment, which contains wsgi.input, which ultimately refers to a _socket object from the standard library. Python's _socket objects only close their underlying file descriptor when their reference counts fall to 0***. This commit makes SegmentedIterable.close() call self.app_iter.close(), thereby unwinding its generator's stack and making it eligible for garbage collection. * in Python < 3.4, at least. See PEP 442. ** see PyGen_NeedsFinalizing() in Objects/genobject.c and also has_finalizer() in Modules/gcmodule.c in Python. *** see sock_dealloc() in Modules/socketmodule.c in Python. See sock_close() in the same file for the other half of the sad story. This closes CVE-2016-0738. Closes-Bug: 1493303 Co-Authored-By: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Change-Id: Ib86c4c45641485ce1034212bf6f53bb84f02f612 commit d9b22ac51cca579e97ec7bd5a6df3ad7bf56c3bf Author: Victor Stinner <vstin...@redhat.com> Date: Mon Oct 19 16:45:33 2015 +0200 Port parse_mime_headers() to Python 3 Port swift.common.utils.parse_mime_headers() to Python 3: * On Python 3, tries to decode headers from UTF-8. If an header was was not encoded to UTF-8, decode the header from Latin1. * Update the parse_mime_headers() tests: on Python 3, HTTP header values are Unicode strings. This change is a follow-up of the change Ia5ee2ead67e36e8c6416183667f64ae255887736. Change-Id: I042dd13e9eb0e9844ccd832d538cdac84359ed42 commit cec2aa0b4dbf0e1bf2de62945f5abdf494925a8e Author: Victor Stinner <vstin...@redhat.com> Date: Fri Oct 9 17:56:39 2015 +0200 py3: Use the six module in the xprofile middleware * Replace bytes_() with a isinstance(content, six.text_type) and content.encode('utf-8') * Get parse_qs() from six.moves.urllib.parse Change-Id: I4a506975814bff098ade25c950c154e50dc9626e commit bc4b298b6e208d3188641712c9d66ae82d172c14 Author: Samuel Merritt <s...@swiftstack.com> Date: Tue Jan 19 15:33:13 2016 -0800 Fix a comment's indentation Change-Id: I34514525b606cf82767ddce7769bc42fa5457717 commit 6c32da14f48b435aa4e5efea48904bd288a532a2 Author: Victor Stinner <vstin...@redhat.com> Date: Mon Oct 19 16:38:24 2015 +0200 Port FileLikeIter to Python 3 Port FileLikeIter and _MultipartMimeFileLikeObject and swift.common.utils to Python 3: * Add a __next__() alias to the next() method. On Python 3, the next() method is no more used, __next__() is required. * Use literal byte strings: FileLikeIter _MultipartMimeFileLikeObject are written to handle binary files. * test_close(): replace .FileLikeIter('abcdef') with FileLikeIter([b'a', b'b', b'c']). On Python 3, list(b'abc') returns [97, 98, 99], whereas ['a', 'b', 'c'] is returned on Python 2. * Update unit FileLikeIter tests to use byte strings. Change-Id: Ibacddb70b22f624ecd83e374749578feddf8bca8 commit caa127d59a63a25cc6d08e985e536f90300a6156 Author: Samuel Merritt <s...@swiftstack.com> Date: Thu Nov 12 11:34:44 2015 -0800 Monkeypatch everything the same There's three spots in the code where eventlet monkeypatches things. In one place, it was monkeypatching socket and thread; in the other two, only socket. This commit monkeypatches socket and thread everywhere. Hopefully, this lets us avoid code that, for example, works in the object server but fails in the object updater. Change-Id: Ia30d069c5ac89f4bfbe52233d03d96738144b614 commit 3a0486e532f22af0d3c8a5c5d78613c22e786ff6 Author: Sivasathurappan Radhakrishnan <siva.radhakrish...@intel.com> Date: Fri Dec 4 17:43:00 2015 +0000 Deleted comment about part power in FakeRing Deleted comment about parameter part power in Class FakeRing as its behavior got dropped in I8bfc388a04eff6491038991cdfd7686c9d961545. Change-Id: Iec7d2565a77e48493b0056021066d8d8eab65d0b Closes-Bug: #1488704 commit 999479f9b17b42ccc5da54ce01651960cf7cf970 Author: John Dickinson <m...@not.mn> Date: Tue Jan 19 10:30:30 2016 -0800 Bump eventlet min version to 0.17.4 IPv6 support in Swift is dependent on IPv6 support in eventlet. eventlet itself only claims support for IPv6 post v0.17 (https://github.com/eventlet/eventlet/issues/8). This update matches the OpenStack global requirements version. Change-Id: I9d8433cdd3bf7d7a93b8f50b991cc21721a80d22 commit 133a3ea601a3fea84af36a42845f27b8182fd901 Author: Christopher Bartz <ba...@dkrz.de> Date: Mon Dec 21 14:17:00 2015 +0100 Use the correct split_path in handle_request Change-Id: I86d423309f0b2091ee2e82b2245caf925b6a75ef Closes-Bug: #1528189 commit 09b188f03c4a40838ea6b3e54e0714d93077b066 Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Thu Jul 16 09:35:37 2015 +0900 Add functional test for access control (RBAC) with Keystone This patch adds test cases for PUT, DELETE, GET, HEAD, POST and OPTIONS requests to accounts, containers and objects using various combinations of users/projects, roles and/or service tokens. Change-Id: Iea8141ac74ad949a3ae7fa47fda3135d0f2612f6 commit bf10974cdefffdaaebc58d21e8a9912638a0405a Author: Tim Burke <tim.bu...@gmail.com> Date: Wed Dec 16 15:46:13 2015 -0800 Expose token expiration time in tempauth auth response Previously, we gave no indication of when a token would expire. Users would have to just use it until it stopped working, then re-auth. Now, a successful auth response will include a new header, X-Auth-Token-Expires, with the number of seconds remaining until the token is invalid. This allows the client to attempt to re-auth before sending a request that will definitely fail. For comparison, swauth already uses the X-Auth-Token-Expires header with identical semantics. Additionally, Keystone (v2 and v3) already exposes expiration times in its JSON responses. The security impact should be minimal. Change-Id: I5a4a74276bc0df6dda94e4bc150065c0d77de0eb commit f56f29ef7a692b6c3204e25d41eace3d68dceece Author: Peter Lisák <peter.li...@firma.seznam.cz> Date: Thu Dec 3 16:48:18 2015 +0100 Add info about state of ring file to default command. Try to find ring file, load and compare it with builder file, then show result state. Examples: Ring file object.ring.gz not found, probably it hasn't been written yet Ring file object.ring.gz is up-to-date Ring file object.ring.gz is obsolete Ring file object.ring.gz is invalid: ValueError('string length not a multiple of item size',) Change-Id: I4d769aa5fe1c2b1167ec088aa372874f7d13ae48 commit 47e226418bad35ccad2a1525f392ba69f6165027 Author: OpenStack Proposal Bot <openstack-in...@lists.openstack.org> Date: Mon Jan 18 06:20:14 2016 +0000 Imported Translations from Zanata For more information about this automatic import see: https://wiki.openstack.org/wiki/Translations/Infrastructure Change-Id: Ic416c9afc8a1c76f552803a7c70fc905eda7b3cb commit 165fa1fd40b5e4b13cf5188712593e2275bef821 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Fri Jan 15 11:44:21 2016 -0800 add test for zero weight region get_more_nodes Change-Id: If537981e8deadd9c3528dcb30a15011c7781e334 commit 5d449471b12c67b31ebb5a383d9bb35bace36213 Author: Samuel Merritt <s...@swiftstack.com> Date: Thu Jan 14 17:26:01 2016 -0800 Remove some Python 2.6 leftovers Change-Id: I798d08722c90327c66759aa0bb4526851ba38d41 commit 3c0cf549f1e822cce8f905b069b317e676cf306b Author: Samuel Merritt <s...@swiftstack.com> Date: Wed Jan 13 18:08:45 2016 -0800 Speed up get_more_nodes() when there is an empty zone The ring has some optimizations in get_more_nodes() so that it can find handoffs that span all the regions/zones/et cetera and then stop looking. The stopping is the important part. Previously, it would quickly find a handoff in each unused region, then spend way too long looking for more unused regions; the same was true for zones, IPs, and so on. Thus, in commit 9cd7c6c, we started counting regions and zones, then stopping when we found them all. This count included all regions and zones in the ring, regardless of whether or not there were actually any parts assigned or not. In rings with an empty region, i.e. a region for which there are only zero-weight devices, get_more_nodes() would be very slow. This commit ignores devices with no assigned partitions when counting regions, zones, and so forth, thus greatly speeding things up. The output of get_more_nodes() is unchanged. This is purely an optimization. Closes-Bug: 1534303 Change-Id: I4a5c57205e87e1205d40fd5d9458d4114e524332 commit 70047709fc9885df7019f791e17a3240682cc6cb Author: keliang <ke.li...@easystack.cn> Date: Fri Jan 15 00:31:51 2016 +0800 Drop python 2.6 support Change-Id: Id6329c863dacb189fccfc304453ed7b6f9607c14 commit a4c2fe95ab2fbe59379a69914ed0fac49c28efbb Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Tue Jan 12 21:26:33 2016 +0100 Allow to change auditor sleep interval in config Change-Id: Ic451c5e0b686509f8982ed1bf65a223a2d77b9a0 commit edc823e8030640184071fee4920d34f9a1cc6b3e Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Sun Nov 29 18:46:47 2015 +0100 Show UTC time in swift-recon. It's not consistent now for example local time in replication part and UTC time at begging of line. Use _ptime in swift-recon for all time printing and this function returns UTC now. Change-Id: I732d9851db157130a08e825e8093b7e244b63e9c commit fa5b32d27964478dfcccf71155d2aaa946c561f0 Author: Tim Burke <tim.bu...@gmail.com> Date: Tue Jan 12 14:18:30 2016 -0800 Make object-auditor storage-policy-aware Previously, the object-auditor would always use a (replication) DiskFileManager when walking through AuditLocations, which would cause it to skip EC fragment archives with a warning like: Unexpected file <hash_path>/1452557187.03610#3.data: Invalid Timestamp value in filename '1452557187.03610#3.data' Now, the AuditLocation's policy will be used to find an appropriate manager to get the diskfile. Additionally, several .commit()s were added to the auditor tests so the .durable will be written, which is required when auditing EC fragment archives. Change-Id: I960e7d696fd9ad704ca1872b4ac821f9078838c7 Closes-Bug: 1533002 commit 725a166ebd3fde2c34f2c9578ff6f9911cf206d7 Author: Timur Alperovich <timur...@swiftstack.com> Date: Fri Jan 8 18:42:33 2016 -0800 Make _get_addr() method a function in utils. The patch moves the MemcacheConnPool._get_addr() method a function in swift.common.utils. The function is renamed to parse_socket_string() and the documentation is updated accordingly. The test for it has also been moved. Change-Id: Ida65b2fded28d0a059e668646f5b89714298f348 commit 0647aea9c5fc95f0860758cc54bc23f965da460d Author: Timur Alperovich <timur...@swiftstack.com> Date: Fri Jan 8 14:54:56 2016 -0800 Fail early if the memcache address is invalid. In cases when the memcache address is invalid, we should fail early. This patch addresses the cases when the IPv6 addresses are not enclosed in "[]". It does not, however, fix the case of an invalid hostname. These improvements could also be added to the _get_addr() method. Change-Id: I4743dcda45a1fc1640989325c4a2e1fea591fc69 commit 33476460239c9cdb08dd8065d22d84a4717da7be Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Fri Jan 8 16:15:54 2016 -0800 fixups for ipv6 memcache_servers docs Change-Id: I20d91c1e276014eaf210fa9eb43788bc17f4e8df commit d5ff5447be30b44bf4acc8b912b6241a44f710be Author: Samuel Merritt <s...@swiftstack.com> Date: Tue Jan 12 16:42:06 2016 -0800 Install liberasurecode packages in SAIO. Change-Id: If673afa2b61a3e388612debf4860d561960963a3 commit 4ffc4ba411f67c8407ba38d082a3a51a96ad7e04 Author: Jonathan Hinson <jlhin...@us.ibm.com> Date: Tue Jan 12 11:46:21 2016 -0600 Functional tests for if-match with multiple etags Multiple etags can be provided on an if-match or if-none-match request. This is currently being tested in the unit tests, but not in the functional tests. Since these etags can be modified by middleware, we need functional tests to assert multiple-etag requests are handled correctly. Change-Id: Idc409c85e8aa82b59dc2bc28af6ca2617de82699 commit 1292361176aec385137dd5873c5cc9714de1fa17 Author: Clay Gerrard <clay.gerr...@gmail.com> Date: Mon Jan 11 16:25:50 2016 -0800 functest for x-timestamp validation Change-Id: I05af7d837af00cd1059a8912df05c131a52ffb90 commit e6194113a3c81563590eabf8f761ccb988bb917c Author: Tim Burke <tim.bu...@gmail.com> Date: Fri Jan 8 16:38:31 2016 -0800 Validate X-Timestamps Previously, attempting to PUT a new object with an X-Timestamp header less than or equal to zero (ie, for a timestamp on or before 1970-01-01 00:00:00) would cause the object-server to 500. While we're at it, cap X-Timestamp at 9999999999 (2286-11-20 17:46:40) so we don't get an eleventh digit before the decimal point. Closes-Bug: 1532471 Change-Id: I23666ec8a067d829eaf9bfe54bd086c320b3429e commit 1f3304c5153e01988b8f4493875b6489e93f76d0 Author: Ben Martin <blmar...@us.ibm.com> Date: Mon Dec 14 15:28:17 2015 -0600 Print min_part_hours lockout time remaining swift-ring-builder currently only displays min_part_hours and not the amount of time remaining before a rebalance can occur. This information is readily available and has been displayed as a quality of life improvement. Additionally, a bug where the time since the last rebalance was always updated when rebalance was called regardless of if any partitions were reassigned. This can lead to partitions being unable to be reassigned as they never age according to the time since last rebalance. Change-Id: Ie0e2b5e25140cbac7465f31a26a4998beb3892e9 Closes-Bug: #1526017 commit 167bb5eeb82886d67c1b382417fb22b8ea85f0d3 Author: Timur Alperovich <timur...@swiftstack.com> Date: Wed Dec 16 12:07:27 2015 -0800 Fix IPv6 handling in MemcacheConnPool. The patch removes the assumption of IPv4-only addresses in the MemcacheConnPool. The changes are around address handling. Namely, if a server is specified with an address [<address>]:port (port is optional), it is assumed to be an IPv6 address [1]. If an IPv6 address is specified without "[]", an exception is raised as it is impossible to parse such addresses correctly. For testing, memcache can be configured to listen on the link-local, unique-local, or ::1 (equivalent to 127.0.0.1) addresses. Link-local addresses are assigned by default to each interface and are of the form "fe80::dead:beef". These addresses require a scope ID, which would look like "fe80::dead:beef%eth0" (replacing eth0 with the correct interface). Unique-local addresses are any addresses in the fc00::/7 subnet. To add a ULA to an interface use the "ip" utility. For example: "ip -6 address add fc01::dead:beef dev eth0". Lastly, and probably simplest, memcache can be configured to listen on "::1". The same address would be used in the swift configuration, e.g. "[::1]:11211". Note: only memcached version 1.4.25 or greater supports binding to an IPv6 address. Fixes #1526570 [1] IPv6 host literals: https://tools.ietf.org/html/rfc3986#section-3.2.2 Change-Id: I8408143c1d47d24e70df56a08167c529825276a2 commit fb6751d8ba133c57e1ebb76be71a96f2f120b8ca Author: Paul Dardeau <paul.dard...@intel.com> Date: Fri Jan 8 22:49:05 2016 +0000 Look for device holes that can be reused when adding new device. Change-Id: I1980ebdd9dc89848173d8ca2fe2afb74029dcfa2 Closes-Bug: 1532276 commit b35f3c90bde8a7ccb50440bda5800cbb8274a5a1 Author: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Date: Fri Jan 8 01:29:11 2016 -0800 Add note COPY with conditional headers Swift now supports Range header for COPY (or PUT with X-Copy-From) to make a partial copied object. This patch adds the note to show it obviously supported in Swift community. Change-Id: I6bf28f0932c90e7b305cd61aabce4ed028ae691e Partial-Bug: #1532126 commit 23c7a58f8f1412c28b3a16b79be09c224c9f7d55 Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Fri Dec 11 18:26:34 2015 +0900 Fix ClientException handling in Container Sync swift/container/sync.py uses swift.common.internal_client.delete_object and put_object and expected these methods raise ClientException. But delete_object and put_object never raise the exception so this patch raises ClientException when urllib2 library raises HTTPError. Co-Authored-By: Eran Rom <er...@il.ibm.com> Closes-Bug: #1419901 Change-Id: I58cbf77988979a07998a46d9d81be84d29b0d9bf commit 6786cdf036b4faabe3928c3d0dd9615d94834801 Author: Harshada Mangesh Kakad <harsha...@metsi.co.uk> Date: Thu Dec 31 01:44:00 2015 -0800 Fixing the deprecated library function. os.popen() is deprecated since version 2.6. Resolved with use of subprocess module. Change-Id: I4409cdd9edbc1a26d6f99c125c9100fadda5d758 Partial-Bug: #1529836 commit 85a0a6a28e166bc076cf8786de2b46248d8786a2 Author: Eran Rom <er...@il.ibm.com> Date: Sun Jul 26 13:31:17 2015 +0300 Container-Sync to iterate only over synced containers This change introduces a sync_store which holds only containers that are enabled for sync. The store is implemented using a directory structure that resembles that of the containers directory, but has entries only for containers enabled for sync. The store is maintained in two ways: 1. Preemptively by the container server when processing PUT/POST/DELETE operations targeted at containers with x-container-sync-key / x-container-sync-to 2. In the background using the containers replicator whenever it processes a container set up for sync The change updates [1] [1] http://docs.openstack.org/developer/swift/overview_container_sync.html Change-Id: I9ae4d4c7ff6336611df4122b7c753cc4fa46c0ff Closes-Bug: #1476623 commit e75888b281d59df0889f28d0b32241dac3a34aa2 Author: HugoKuo <tonyt...@gmail.com> Date: Wed Jan 6 14:33:23 2016 +0800 Add more description for write_affinity_node_count parameter in the doc. Change-Id: Iad410a2be4f9a2cd5c53e860b9f91993aa7f2369 Closes-Bug: #1531173 commit 3b1591f235f4b85796917507be5e7fd80365ff9e Author: Ondřej Nový <ondrej.n...@firma.seznam.cz> Date: Wed Sep 30 19:08:09 2015 +0200 swift-init: New option kill-after-timeout This option send SIGKILL to daemon after kill_wait period. When daemon hangs and doesn't respond to SIGTERM/SIGHUP there is no way to stop it using swift-init now. Classic init scripts in Linux kills hanged process after grace period and this patch add same behaviour. This is most usefull when using "restart" on hanged daemon. Change-Id: I8c932b673a0f51e52132df87ea2f4396f4bba9d8 commit 4691d0592ff3deeea2713195312bf1effd5a7a7b Author: Hisashi Osanai <osanai.hisa...@jp.fujitsu.com> Date: Fri Dec 25 14:00:16 2015 +0900 Add object replicator's succeeded and failed counts in log account-replicator and container-replicator outputted the following info in log. info account-replicator: 17 successes, 4 failures info container-replicator: 178 successes, 29 failures But there is no output info for object-replicator. This info is important to check replicator's status from log. So this patch adds object-replicator's succeeded and failed counts in log like account-replicator and container-replicator. Change-Id: I7fbaf161540bdb06c0a65a1032aae83764bd970c commit 9fe0e25604dff35db7eab1bca312821a81db6c1d Author: Kota Tsuyuzaki <tsuyuzaki.k...@lab.ntt.co.jp> Date: Tue Dec 8 22:27:44 2015 -0800 Sleep enough for trampoline When running unite test suite in local poor resource environment, sometimes test/unit/proxy/test_server.py fails due to a lack of waiting time to trampoline of eventlet thread. This patch enables to sleep 1 more second when it doesn't seem to have enough time to tranpoline. Change-Id: I0bbc8fc245919d3c0a071ff87ff6e20b8d58f9b8 commit 7f636a557296ecc6ae4727700cfcf9f82573bd16 Author: Samuel Merritt <s...@swiftstack.com> Date: Mon Nov 30 18:06:09 2015 -0800 Allow smaller segments in static large objects The addition of range support for SLO segments (commit 25d5e68) required the range size to be at least the SLO minimum segment size (default 1 MiB). However, if you're doing something like assembling a video of short clips out of a larger one, then you might not need a full 1 MiB. The reason for the 1 MiB restriction was to protect Swift from resource overconsumption. It takes CPU, RAM, and internal bandwidth to connect to an object server, so it's much cheaper to serve a 10 GiB SLO if it has 10 MiB segments than if it has 10 B segments. Instead of a strict limit, now we apply ratelimiting to small segments. The threshold for "small" is configurable and defaults to 1 MiB. SLO segments may now be as small as 1 byte. If a client makes SLOs as before, it'll still be able to download the objects as fast as Swift can serve them. However, a SLO with a lot of small ranges or segments will be slowed down to avoid resource overconsumption. This is similar to how DLOs work, except that DLOs ratelimit *every* segment, not just small ones. UpgradeImpact For operators: if your cluster has enabled ratelimiting for SLO, you will want to set rate_limit_under_size to a large number prior to upgrade. This will preserve your existing behavior of ratelimiting all SLO segments. 5368709123 is a good value, as that's 1 greater than the default max object size. Alternately, hold down the 9 key until you get bored. If your cluster has not enabled ratelimiting for SLO (the default), no action is needed. Change-Id: Id1ff7742308ed816038a5c44ec548afa26612b95 ** Tags added: in-feature-hummingbird -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1493303 Title: [OSSA 2016-004] Swift proxy memory leak on unfinished read (CVE-2016-0738) To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/1493303/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs