Re: [sheepdog] [PATCH v8 00/19] object reclaim based on generational reference counting

On Thu, May 22, 2014 at 11:29:59PM +0900, Hitoshi Mitake wrote: > At Thu, 22 May 2014 16:54:32 +0800, > Liu Yuan wrote: > > > > On Fri, May 16, 2014 at 12:22:27AM +0900, Hitoshi Mitake wrote: > > > The object reclaim doesn't support hypervolume yet. But hypervolume > > > cannot be > > > used as a

Re: [sheepdog] [PATCH] add dog vdi object exist command to check which objects are lost.

On 2014年05月22日 21:30, Liu Yuan wrote: On Thu, May 22, 2014 at 07:56:06PM +0800, Ruoyu wrote: Sometimes we want to quickly check whether some of the vdi objects or data objects are lost due to unexpected issue. Although dog vdi check will do, it spends a lot of time because of too many client-s

[sheepdog] [PATCH v9 14/19] tests: show ledger objects in list

From: Hitoshi Mitake Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- v7: - refine output -- previous way of counting data object was incorrect v4: - refine output tests/functional/016.out | 14 ++--- tests/functional/042 |

[sheepdog] [PATCH v9 11/19] sheep, dog: migrate deletion process from sheep to dog

From: Hitoshi Mitake Deletion process also requires modification of generational reference status so it must not be executed with snapshot creation or COW request at the same time. This patch migrates deletion process from sheep to dog. The new deletion process in dog is simply implemented as CO

[sheepdog] [PATCH v9 19/19] tests/functional: new test of object reclaim for a case of continuously snapshotted VDI

From: Hitoshi Mitake Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- tests/functional/093 | 32 tests/functional/093.out | 26 ++ tests/functional/group | 1 + 3 files chang

[sheepdog] [PATCH v9 09/19] sheep, dog: prevent COW during snapshot creation

Current generational reference counting scheme for garbage collection of snapshot object requires mutual exclusion between COW operation and snapshot creation. Because both of COW operation and snapshot creation requires modification of generation reference status. This patch adds operations for pr

[sheepdog] [PATCH v9 17/19] update CHANGELOG.md for new object reclaim scheme

From: Hitoshi Mitake Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- CHANGELOG.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 865fa4c..57e01a5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -

[sheepdog] [PATCH v9 18/19] tests/functional: add a new test of the object reclaim for cloned VDIs

From: Hitoshi Mitake This patch adds a new test of the object reclaim algorithm. The test creates branch of VDI tree and checks the GC works well for the case. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- tests/functional/092 | 4

[sheepdog] [PATCH v9 15/19] doc: add documentation of object reclaim algorithm

From: Hitoshi Mitake Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- doc/object-reclaim.txt | 73 ++ 1 file changed, 73 insertions(+) create mode 100644 doc/object-reclaim.txt diff --git

[sheepdog] [PATCH v9 13/19] sheep: update store format version to v5

From: Hitoshi Mitake This patchset introduces two important change for sheep store: 1. generational reference counting: adding ledger object and expand inode object for generation reference 2. changing a place of btree_counter in inode object Incrementing SD_FORMAT_VERSION in either of them w

[sheepdog] [PATCH v9 04/19] sheep: decrement generational reference count on copy-on-write

This decrements a reference count of the old data object when allocating a new data object on CoW. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- sheep/gateway.c | 107 +++- 1 file chan

[sheepdog] [PATCH v9 12/19] dog: print status of generational reference of inodes in dump-inode

From: Hitoshi Mitake This patch lets "dog vdi object dump-inode" print status of generational reference. Example of output: name: test tag: create_time: 53704f51349aa108 snap_ctime: 0 vm_clock_nsec: 0 copy_policy: 0 store_policy: 0 nr_copies: 3 block_size_shift: 22 snap_id: 2 vdi_id: 7c2b26 pare

[sheepdog] [PATCH v9 03/19] sheep: decrement generational reference count on vdi deletion

This removes old vdi deletion code, which reclaims objects only when all relevant snapshots are deleted, and uses a generational reference counting algorithm. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- sheep/vdi.c | 272 ++---

[sheepdog] [PATCH v9 08/19] sheep: get_request() for deferred response

With the new function, response message to clients can be deferred safely without blocking the main thread. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- sheep/request.c| 5 + sheep/sheep_priv.h | 1 + 2 files changed, 6 inserti

[sheepdog] [PATCH v9 10/19] sheep: reduce needless remove request during object reclamation

From: Hitoshi Mitake The previous reference decrement operation issued too many (and not required) remove requests. It harms performance and produces meaningless error logs. This patch reduces the needless requests. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-o

[sheepdog] [PATCH v9 07/19] sheepdog proto: recycle old child_vdi_id for storing btree_counter

From: Hitoshi Mitake Now the area isn't used and clients (QEMU and tgt) don't care about it, we can recycle it for storing the btree_counter. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- include/sheepdog_proto.h | 20

[sheepdog] [PATCH v9 06/19] sheep: introduce sparse objects

We don't need to care about a fragmentation when - the object is unlikely to be accessed sequentially, and - the object is read-only. In that sense, we can make the objects sparse if they are not data objects and writable ones. This fixes the problem that sheepdog consumes many disk spaces for

[sheepdog] [PATCH v9 05/19] sheep: remove max children limit

We no longer need it. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Tested-by: Valerio Pachera Signed-off-by: Hitoshi Mitake --- v3: - Keep SD_INODE_HEADER_SIZE as offsetof(struct sd_inode, data_vdi_id), not offsetof(struct sd_inode, __unused). Because hypervolume code depends on the old va

[sheepdog] [PATCH v9 01/19] sheep: introduce generational reference counting for object reclaim

Generational reference counting is an algorithm to reclaim data efficiently without race conditions on distributed system. This extends vdi objects structure to store generational reference counts, and increments the counts when creating snapshots. Cc: MORITA Kazutaka Cc: Alessandro Bolgia Test

[sheepdog] [PATCH v9 02/19] sheep: introduce ledger objects

This introduces ledger objects, which keeps track of the number of outstanding references of each generation. Sheep decrements a generational reference count with a gateway request SD_OP_DECREF_OBJ, and reclaims objects when there is no generational reference. Cc: MORITA Kazutaka Cc: Alessandro

[sheepdog] [PATCH v9 00/19] object reclaim based on generational reference counting

The object reclaim doesn't support hypervolume yet. But hypervolume cannot be used as a virtual disk (both of qemu and tgt don't support it) currently. And the removal of old vdi deletion is acceptable for hypervolume because it doesn't support snapshot, etc. So I think this patchset can be applied

Re: [sheepdog] [PATCH v8 00/19] object reclaim based on generational reference counting

At Thu, 22 May 2014 16:54:32 +0800, Liu Yuan wrote: > > On Fri, May 16, 2014 at 12:22:27AM +0900, Hitoshi Mitake wrote: > > The object reclaim doesn't support hypervolume yet. But hypervolume cannot > > be > > used as a virtual disk (both of qemu and tgt don't support it) currently. > > And > >

Re: [sheepdog] [PATCH] add dog vdi object exist command to check which objects are lost.

On Thu, May 22, 2014 at 09:30:01PM +0800, Liu Yuan wrote: > On Thu, May 22, 2014 at 07:56:06PM +0800, Ruoyu wrote: > > Sometimes we want to quickly check whether some of the vdi objects > > or data objects are lost due to unexpected issue. > > > > Although dog vdi check will do, it spends a lot of

Re: [sheepdog] [PATCH] add dog vdi object exist command to check which objects are lost.

On Thu, May 22, 2014 at 07:56:06PM +0800, Ruoyu wrote: > Sometimes we want to quickly check whether some of the vdi objects > or data objects are lost due to unexpected issue. > > Although dog vdi check will do, it spends a lot of time because of > too many client-server communication. And the pro

[sheepdog] [PATCH] add dog vdi object exist command to check which objects are lost.

Sometimes we want to quickly check whether some of the vdi objects or data objects are lost due to unexpected issue. Although dog vdi check will do, it spends a lot of time because of too many client-server communication. And the probability of triggering data auto fixing is quite low since the wr

Re: [sheepdog] [PATCH v8 19/19] tests/functional: new test of object reclaim for a case of continuously snapshotted VDI

On Thu, May 22, 2014 at 09:59:22AM +0200, Valerio Pachera wrote: > Tested-by: Valerio Pachera > > Result of my testing is that snapshots get correctly removed freeing space > in the cluster. > I've been testing full replication (-c 2) and erasure code (-c 2:1). > > Removing all snapshots and vdi

Re: [sheepdog] [PATCH v8 00/19] object reclaim based on generational reference counting

On Fri, May 16, 2014 at 12:22:27AM +0900, Hitoshi Mitake wrote: > The object reclaim doesn't support hypervolume yet. But hypervolume cannot be > used as a virtual disk (both of qemu and tgt don't support it) currently. And > the removal of old vdi deletion is acceptable for hypervolume because it

Re: [sheepdog] [PATCH v8 19/19] tests/functional: new test of object reclaim for a case of continuously snapshotted VDI

Tested-by: Valerio Pachera Result of my testing is that snapshots get correctly removed freeing space in the cluster. I've been testing full replication (-c 2) and erasure code (-c 2:1). Removing all snapshots and vdi (so no vdi left in the cluster), most of the time leave almost no objects (e.g.