[sheepdog] [PATCH v4 RESEND 0/8] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-23 Thread levin li
From: levin li xingke@taobao.com v3 -- v4: 1. Refactor the code in proc.c and device.c 2. Split the big patch into some small patches 3. Fixed coding style to pass checkpatch.pl check v2 -- v3: 1. Make it Ok for QEMU to run VDIs from the driver 2. Code refactor 3. Use the format 'del

[sheepdog] [PATCH v4 RESEND 1/8] sheep: modify sheepdog_proto.h to make it usable in linux kernel module

2013-01-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- include/sheepdog_proto.h | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h index 865f451..fb35f3a 100644 --- a/include

[sheepdog] [PATCH v4 RESEND 2/8] sheepdev: use proc entry to control adding/removing devices

2013-01-23 Thread levin li
From: levin li xingke@taobao.com Add a proc entry /proc/sheep for sheep device controlling, by writing to this entry to add or remove device For example: Add a sheep VDI as a block device: echo add 127.0.0.1 linux:tag /proc/sheep Remove a sheep device: echo del linux:tag /proc/sheep

[sheepdog] [PATCH v4 RESEND 4/8] sheepdev: add sheep.c to process the sheepdog protocol

2013-01-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheepdev/Kbuild | 3 +- sheepdev/sheep.c| 186 sheepdev/sheepdev.h | 8 +++ 3 files changed, 196 insertions(+), 1 deletion(-) create mode

[sheepdog] [PATCH v4 RESEND 5/8] sheepdev: make it possible to add or remove a sheep device

2013-01-23 Thread levin li
From: levin li xingke@taobao.com This patch provides functions to create a sheep device by writing to the proc entry, but it hasn't processed the IO requests, it just ends the requests with -EIO. It creates sheep devices named /dev/sheepX, but until now, it can not handle IO request, any

[sheepdog] [PATCH v4 RESEND 3/8] sheepdev: add connect.c for network operations

2013-01-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheepdev/Kbuild | 3 +- sheepdev/connect.c | 178 sheepdev/sheepdev.h | 7 +++ 3 files changed, 187 insertions(+), 1 deletion(-) create mode

[sheepdog] [PATCH v4 RESEND 6/8] sheepdev: create kernel threads to process the IO request in the driver

2013-01-23 Thread levin li
From: levin li xingke@taobao.com This patch creates two kernel threads 'sheep_req' and 'sheep_fin' to process the IO requests and sheep response respectively, sheep_req just fetches IO request from the pending list, and forward the request to sheep, sheep_fin waits for sheep response

[sheepdog] [PATCH v4 RESEND 7/8] sheepdev: make it safe to open a snapshot VDI

2013-01-23 Thread levin li
From: levin li xingke@taobao.com In case we open a snapshot VDI to make it as a block device, we immediately create a new writable VDI for this snapshot just as what QEMU does, then we can safely write to this VDI Signed-off-by: levin li xingke@taobao.com --- sheepdev/sheep.c | 60

[sheepdog] [PATCH v4 RESEND 8/8] sheepdev: add Makefile.in and modify configure.ac to adopt sheepdev

2013-01-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- Makefile.am | 4 configure.ac | 9 + sheepdev/Makefile.in | 8 3 files changed, 21 insertions(+) create mode 100644 sheepdev/Makefile.in diff --git a/Makefile.am b

Re: [sheepdog] [PATCH v3 2/4] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-22 Thread levin li
On 01/22/2013 01:20 PM, Liu Yuan wrote: On 01/22/2013 11:16 AM, levin li wrote: sheepdev/connect.c | 178 ++ sheepdev/device.c | 985 As a general rule, if the proper code of one functions exceed 60 lines, try hard to rethink

[sheepdog] [PATCH v4 0/8] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-22 Thread levin li
From: levin li xingke@taobao.com v3 -- v4: 1. Refactor the code in proc.c and device.c 2. Split the big patch into some small patches 3. Fixed coding style to pass checkpatch.pl check v2 -- v3: 1. Make it Ok for QEMU to run VDIs from the driver 2. Code refactor 3. Use the format 'del

[sheepdog] [PATCH v4 1/8] sheep: modify sheepdog_proto.h to make it usable in linux kernel module

2013-01-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- include/sheepdog_proto.h | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h index 865f451..fb35f3a 100644 --- a/include

[sheepdog] [PATCH v4 2/8] sheepdev: use proc entry to control adding/removing devices

2013-01-22 Thread levin li
From: levin li xingke@taobao.com Add a proc entry /proc/sheep for sheep device controlling, by writing to this entry to add or remove device For example: Add a sheep VDI as a block device: echo add 127.0.0.1 linux:tag /proc/sheep Remove a sheep device: echo del linux:tag /proc/sheep

[sheepdog] [PATCH v4 4/8] sheepdev: add sheep.c to process the sheepdog protocol

2013-01-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheepdev/Kbuild | 3 +- sheepdev/sheep.c| 186 sheepdev/sheepdev.h | 8 +++ 3 files changed, 196 insertions(+), 1 deletion(-) create mode

[sheepdog] [PATCH v4 5/8] sheepdev: make it possible to add or remove a sheep device

2013-01-22 Thread levin li
From: levin li xingke@taobao.com This patch provides functions to create a sheep device by writing to the proc entry, but it hasn't processed the IO requests, it just ends the requests with -EIO. It creates sheep devices named /dev/sheepX, but until now, it can not handle IO request, any

[sheepdog] [PATCH v4 6/8] sheepdev: create kernel threads to process the IO request in the driver

2013-01-22 Thread levin li
From: levin li xingke@taobao.com This patch creates two kernel threads 'sheep_req' and 'sheep_fin' to process the IO requests and sheep response respectively, sheep_req just fetches IO request from the pending list, and forward the request to sheep, sheep_fin waits for sheep response

[sheepdog] [PATCH v4 7/8] sheepdev: make it safe to open a snapshot VDI

2013-01-22 Thread levin li
From: levin li xingke@taobao.com In case we open a snapshot VDI to make it as a block device, we immediately create a new writable VDI for this snapshot just as what QEMU does, then we can safely write to this VDI Signed-off-by: levin li xingke@taobao.com --- sheepdev/sheep.c | 60

[sheepdog] [PATCH v4 8/8] sheepdev: add Makefile.in and modify configure.ac to adopt sheepdev

2013-01-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- Makefile.am | 4 configure.ac | 9 + 2 files changed, 13 insertions(+) diff --git a/Makefile.am b/Makefile.am index 53d18b9..4db40bf 100644 --- a/Makefile.am +++ b/Makefile.am @@ -23,6 +23,10

[sheepdog] [PATCH v3 0/4] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-21 Thread levin li
From: levin li xingke@taobao.com v2 -- v3: 1. Make it Ok for QEMU to run VDIs from the driver 2. Code refactor 3. Use the format 'del vdiname:snap_tag[snap_id]' to delete a device v1 -- v2: 1. Rename module.c to device.c 2. Split proc operations from device.c to proc.c 3. Remove

[sheepdog] [PATCH v3 1/4] sheep: modify sheepdog_proto.h to make it usable in linux kernel module

2013-01-21 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- include/sheepdog_proto.h | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h index 865f451..fb35f3a 100644 --- a/include

[sheepdog] [PATCH v3 2/4] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-21 Thread levin li
From: levin li xingke@taobao.com This module provides function for users to take sheepdog VDIs as block devices in linux, users can register a VDI to its kernel space, it just like that a new hard disk is added to the computer, users can create partitions for the disk, format the disk

[sheepdog] [PATCH v3 4/4] sheepdev: add kbuild and Makefile.in

2013-01-21 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- Makefile.am | 4 configure.ac | 9 + sheepdev/Kbuild | 13 + sheepdev/Makefile.in | 8 4 files changed, 34 insertions(+) create mode 100644 sheepdev

[sheepdog] [PATCH v2 0/4] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-06 Thread levin li
From: levin li xingke@taobao.com v1 -- v2: 1. Rename module.c to device.c 2. Split proc operations from device.c to proc.c 3. Remove sheepdog_proto.h from sheepdev, use the one in include directory 4. Supprt write snapshot VDI 5. Kill the compile error in linux kernel 3.7 6. Modify some

[sheepdog] [PATCH v2 1/4] sheep: modify sheepdog_proto.h to make it usable in linux kernel module

2013-01-06 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- include/sheepdog_proto.h | 11 --- 1 files changed, 8 insertions(+), 3 deletions(-) diff --git a/include/sheepdog_proto.h b/include/sheepdog_proto.h index 865f451..fb35f3a 100644 --- a/include

[sheepdog] [PATCH v2 3/4] sheepdev: make it safe to open a snapshot VDI

2013-01-06 Thread levin li
From: levin li xingke@taobao.com In case we open a snapshot VDI to make it as a block device, we immediately create a new writable VDI for this snapshot just as what QEMU does, then we can safely write to this VDI Signed-off-by: levin li xingke@taobao.com --- sheepdev/sheep.c | 59

[sheepdog] [PATCH v2 2/4] sheepdev: linux kernel module of block device driver for sheepdog

2013-01-06 Thread levin li
From: levin li xingke@taobao.com This module provides function for users to take sheepdog VDIs as block devices in linux, users can register a VDI to its kernel space, it just like that a new hard disk is added to the computer, users can create partitions for the disk, format the disk

[sheepdog] [PATCH RFC 0/2] sheepdev: linux kernel module of block device driver for sheepdog

2012-12-27 Thread levin li
From: levin li xingke@taobao.com This module provides function for users to take sheepdog VDIs as block devices in linux, users can register a VDI to its kernel space, it just like that a new hard disk is added to the computer, users can create partitions for the disk, format the disk

[sheepdog] [PATCH RFC 2/2] sheepdev: add kbuild and Makefile.in

2012-12-27 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- Makefile.am | 4 configure.ac | 9 + sheepdev/Kbuild | 12 sheepdev/Makefile.in | 8 4 files changed, 33 insertions(+) create mode 100644 sheepdev

Re: [sheepdog] [PATCH RFC 1/2] sheepdev: linux kernel module of block device driver for sheepdog

2012-12-27 Thread levin li
On 12/27/2012 11:57 PM, Hitoshi Mitake wrote: At Thu, 27 Dec 2012 22:00:41 +0800, levin li wrote: From: levin li xingke@taobao.com This module provides function for users to take sheepdog VDIs as block devices in linux, users can register a VDI to its kernel space, it just like

[sheepdog] [PATCH 1/2] collie: add 'collie vdi cache flush' to flush dirty cache

2012-12-15 Thread levin li
From: levin li xingke@taobao.com This command can flush cache of an VDI in specified node, which is useful when QEMU terminates accidentally, left the dirty cache data unflushed, this command helps us to flush and delete this cache into the cluster. Signed-off-by: levin li xingke

Re: [sheepdog] [PATCH] sheep: no need to wait for nr_objs of an object

2012-11-29 Thread levin li
On 2012年11月29日 17:24, Liu Yuan wrote: On 11/29/2012 05:17 PM, levin li wrote: + dprintf(ERROR: can not find copy number for object % + PRIx64 \n, oids[i]); maybe use eprintf is better which will show us message even without '-d' specified

[sheepdog] [PATCH] sheep: notify vdi copies number before we create the vdi

2012-11-27 Thread levin li
From: levin li xingke@taobao.com We can not notify the vdi copies number to the cluster after creating the vdi, because if a recovery happens in the time window after cluster_vdi_add() finishes, and just before post_cluster_vdi_add() notifies the copies number of the new vdi to all the nodes

[sheepdog] [PATCH] sheep: fix a dead-lock bug in screen_object_list()

2012-11-25 Thread levin li
From: levin li xingke@taobao.com The bug is introduced in the patch 97ccd87ea15e606b6ec9fecb54f5de453f9c5c1f which causes the cluster hangs in recovery, this patch is like a revert patch except that I add comment to explain why we should not ignore the objects whose nr_objs are zero

Re: [sheepdog] [PATCH] sheep: fix a dead-lock bug in screen_object_list()

2012-11-25 Thread levin li
On 2012年11月26日 13:55, Liu Yuan wrote: On 11/26/2012 01:48 PM, levin li wrote: screen_object_list() is called in recovery, if a VDI creation is in progress, then the VDI creation operation is blocked by gateway to wait for completion of recovery, and if we make screen_object_list() to retry

[sheepdog] [PATCH 0/2] object cache: try to delete cache entris in every node when deleting a VDI

2012-09-05 Thread levin li
From: levin li xingke@taobao.com levin li (2): sheep: read inode directly from backend in VDI deletion work object cache: try to delete cache entris in every node when deleting a VDI sheep/object_cache.c |2 +- sheep/ops.c | 45

[sheepdog] [PATCH 1/2] sheep: read inode directly from backend in VDI deletion work

2012-09-05 Thread levin li
From: levin li xingke@taobao.com This patch is preparation for next patch that deletes object cache when deleting VDI, if object cache deletion work is processed in notify, then it may causes race between cache deletion and next VDI deletion work, because VDI deletion work may needs to read

[sheepdog] [PATCH 2/2] object cache: try to delete cache entris in every node when deleting a VDI

2012-09-05 Thread levin li
From: levin li xingke@taobao.com Every node that starts VM in writeback mode can create cache entries, so we can not only remove cache entries in the node that executes the VDI deletion work, but should notify to try to delete cache entries on every node Signed-off-by: levin li xingke

[sheepdog] [PATCH UPDATE] sheep: read inode directly from backend in VDI deletion work

2012-09-05 Thread levin li
From: levin li xingke@taobao.com This patch is preparation for next patch that deletes object cache when deleting VDI, if object cache deletion work is processed in notify, then it may causes race between cache deletion and next VDI deletion work, because VDI deletion work may needs to read

[sheepdog] [PATCH] object cache: fix crash when deleting object cache

2012-09-04 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/object_cache.c |2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/sheep/object_cache.c b/sheep/object_cache.c index 431fe26..5bd7060 100644 --- a/sheep/object_cache.c +++ b/sheep

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-31 Thread levin li
On 08/31/2012 01:55 PM, Liu Yuan wrote: On 08/31/2012 01:53 PM, levin li wrote: Maybe I misunderstood you, but this patch can do more than you expected, it's no harm for users to rollback a VDI to any snapshot if they choose to. So this implementation is wider solution that include

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-31 Thread levin li
On 08/31/2012 02:10 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 13:53:29 +0800, levin li wrote: On 08/31/2012 12:17 PM, MORITA Kazutaka wrote: At Thu, 30 Aug 2012 13:59:54 +0800, levin li wrote: From: levin li xingke@taobao.com With this command we can rollback a cloned VDI to any

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-31 Thread levin li
On 08/31/2012 02:38 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 14:25:04 +0800, levin li wrote: On 08/31/2012 02:10 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 13:53:29 +0800, levin li wrote: On 08/31/2012 12:17 PM, MORITA Kazutaka wrote: At Thu, 30 Aug 2012 13:59:54 +0800, levin

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-31 Thread levin li
On 08/31/2012 03:03 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 14:59:43 +0800, levin li wrote: On 08/31/2012 02:38 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 14:25:04 +0800, levin li wrote: On 08/31/2012 02:10 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 13:53:29 +0800, levin

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-31 Thread levin li
On 08/31/2012 04:25 PM, MORITA Kazutaka wrote: Here is a draft patch which implements my example but doesn't support resetting clone vdis. I hope it would help you. == commit 138511431d4f87642572b0e2a51d869240b5ac83 Author: MORITA Kazutaka morita.kazut...@lab.ntt.co.jp Date: Fri Aug 31

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-31 Thread levin li
On 08/31/2012 05:37 PM, MORITA Kazutaka wrote: At Fri, 31 Aug 2012 17:09:22 +0800, levin li wrote: I still don't understand what's the use of your rollback of a snapshot, it seems A rollback operation is supported by many systems. See their use cases. it just move the current VDI

Re: [sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-30 Thread levin li
On 08/31/2012 12:17 PM, MORITA Kazutaka wrote: At Thu, 30 Aug 2012 13:59:54 +0800, levin li wrote: From: levin li xingke@taobao.com With this command we can rollback a cloned VDI to any specified snapshot, or easily reset to its parent VDI, the usage is easy. To rollback

[sheepdog] [PATCH] sheep: use short thread for recovery

2012-08-29 Thread levin li
From: levin li xingke@taobao.com Most time sheep isn't in recovery state, so the two recovery work queue recovery_wqueue/recovery_notify_wqueue stay there useless, since we have short thread, we can save resource by creating the thread when recovery needs Signed-off-by: levin li xingke

[sheepdog] [PATCH] sheep: initialize recovery work queue only when cache is enabled

2012-08-29 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/sheep.c |8 ++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/sheep/sheep.c b/sheep/sheep.c index ea25a05..b23a312 100644 --- a/sheep/sheep.c +++ b/sheep/sheep.c @@ -373,9

Re: [sheepdog] [PATCH] sheep: initialize recovery work queue only when cache is enabled

2012-08-29 Thread levin li
On 2012年08月29日 15:10, MORITA Kazutaka wrote: At Wed, 29 Aug 2012 14:47:57 +0800, levin li wrote: From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/sheep.c |8 ++-- 1 files changed, 6 insertions(+), 2 deletions(-) The change looks

[sheepdog] [PATCH UPDATE] sheep: initialize reclaim work queue only when cache is enabled

2012-08-29 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/sheep.c |8 ++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/sheep/sheep.c b/sheep/sheep.c index ea25a05..36e059a 100644 --- a/sheep/sheep.c +++ b/sheep/sheep.c @@ -373,9

Re: [sheepdog] [PATCH] collie: add command 'collie vdi reset' to reset a cloned VDI

2012-08-29 Thread levin li
On 2012年08月29日 15:27, MORITA Kazutaka wrote: At Wed, 29 Aug 2012 15:18:12 +0800, levin li wrote: On 2012年08月29日 15:08, MORITA Kazutaka wrote: At Wed, 29 Aug 2012 13:57:57 +0800, levin li wrote: From: levin li xingke@taobao.com With this command, we can reset a cloned VDI to its

[sheepdog] [PATCH] collie: don't fetch node list if not use

2012-08-29 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/cluster.c |5 + collie/common.c |3 --- collie/node.c|4 ++-- collie/vdi.c | 27 +-- 4 files changed, 16 insertions(+), 23 deletions(-) diff --git

[sheepdog] [PATCH 1/2] collie: add command 'collie vdi rollback'

2012-08-29 Thread levin li
From: levin li xingke@taobao.com With this command we can rollback a cloned VDI to any specified snapshot, or easily reset to its parent VDI, the usage is easy. To rollback to a specified snapshot: $ collie vdi rollback -s 1 base_vdi clone_vdi To reset to its parent: $ collie vdi rollback

[sheepdog] [PATCH 2/2] sheep: add VDI rollback function support

2012-08-29 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/ops.c| 13 sheep/sheep_priv.h |1 + sheep/vdi.c| 83 3 files changed, 97 insertions(+), 0 deletions(-) diff --git

[sheepdog] [PATCH] sheep: no need to set SD_FLAG_CMD_WRITE for local request

2012-08-28 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/recovery.c |1 - 1 file changed, 1 deletion(-) diff --git a/sheep/recovery.c b/sheep/recovery.c index 59ac9d6..0033c6b 100644 --- a/sheep/recovery.c +++ b/sheep/recovery.c @@ -346,7 +346,6

Re: [sheepdog] [PATCH] sheep: no need to set SD_FLAG_CMD_WRITE for local request

2012-08-28 Thread levin li
On 08/28/2012 04:38 PM, Liu Yuan wrote: On 08/28/2012 04:14 PM, levin li wrote: -hdr.flags = SD_FLAG_CMD_WRITE; hdr.data_length = sizeof(sys-this_node); Why not? We do write sys-this_node as data to transfer. Thanks, Yuan For cluster requests we send data through cluster

Re: [sheepdog] [PATCH] sheep: no need to set SD_FLAG_CMD_WRITE for local request

2012-08-28 Thread levin li
On 08/28/2012 05:41 PM, MORITA Kazutaka wrote: At Tue, 28 Aug 2012 16:43:50 +0800, levin li wrote: On 08/28/2012 04:38 PM, Liu Yuan wrote: On 08/28/2012 04:14 PM, levin li wrote: - hdr.flags = SD_FLAG_CMD_WRITE; hdr.data_length = sizeof(sys-this_node); Why not? We do write sys

[sheepdog] [PATCH] collie: add command 'collie vdi reset' to reset a cloned VDI

2012-08-28 Thread levin li
From: levin li xingke@taobao.com With this command, we can reset a cloned VDI to its initial state as it's just cloned from a snapshot, the usage is simple: $ collie vdi reset cloned_vdi_name Signed-off-by: levin li xingke@taobao.com --- collie/collie.h |1 + collie/common.c | 35

[sheepdog] [PATCH] collie: fix error output of 'collie node info'

2012-08-27 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/node.c |1 + sheep/ops.c |5 - 2 files changed, 5 insertions(+), 1 deletions(-) diff --git a/collie/node.c b/collie/node.c index a5a57a3..63bb27a 100644 --- a/collie/node.c +++ b/collie

Re: [sheepdog] [PATCH v2 1/2] sheep: don't send vdi_inuse in get_vdis

2012-08-24 Thread levin li
On 2012年08月24日 14:09, MORITA Kazutaka wrote: At Fri, 24 Aug 2012 13:36:46 +0800, levin li wrote: From: levin li xingke@taobao.com Since we already send vdi copy list in get_vdis(), there's no need to send the vdi_inuse bitmap any more, without it we can send less data as the length

Re: [sheepdog] [PATCH v2 2/2] collie: make `collie vdi list` not read vdi_inuse bitmap

2012-08-24 Thread levin li
On 2012年08月24日 14:11, MORITA Kazutaka wrote: At Fri, 24 Aug 2012 13:36:47 +0800, levin li wrote: From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/common.c| 32 include/sheep.h|5 + sheep

[sheepdog] [PATCH v3 2/2] collie: make `collie vdi list` not read vdi_inuse bitmap

2012-08-24 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/common.c| 32 include/sheep.h|5 + sheep/sheep_priv.h |5 - 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/collie/common.c b

[sheepdog] [PATCH 1/3] sheep: return the copy number of the request object in peer_read_obj

2012-08-23 Thread levin li
From: levin li xingke@taobao.com collie uses this return value in the output of 'collie vdi object', without this, we can not pass test 028 Signed-off-by: levin li xingke@taobao.com --- sheep/ops.c |4 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/sheep/ops.c b

[sheepdog] [PATCH 2/3] sheep: fix cloning VDI fail when cloned VDI has greater nr_copies than the base

2012-08-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/vdi.c | 10 -- 1 files changed, 4 insertions(+), 6 deletions(-) diff --git a/sheep/vdi.c b/sheep/vdi.c index 5c9bf68..aa5134b 100644 --- a/sheep/vdi.c +++ b/sheep/vdi.c @@ -225,7 +225,7

[sheepdog] [PATCH 3/3] test: add a test case for different VDI redundancy level

2012-08-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- tests/029 | 52 tests/029.out | 33 + tests/group |1 + 3 files changed, 86 insertions(+), 0 deletions

[sheepdog] [PATCH 1/2] sheep: don't send vdi_inuse in get_vdis

2012-08-23 Thread levin li
From: levin li xingke@taobao.com Since we already send vdi copy list in get_vdis(), there's no need to send the vdi_inuse bitmap any more, without it we can send less data as the length of vdi copy list is variable and it contains the vid just as vdi_inuse does. Signed-off-by: levin li

[sheepdog] [PATCH 2/2] collie: make `collie vdi list` not read vdi_inuse bitmap

2012-08-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/common.c| 30 +++--- include/sheep.h|5 + sheep/sheep_priv.h |5 - 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/collie/common.c b

Re: [sheepdog] [PATCH 1/2] sheep: don't send vdi_inuse in get_vdis

2012-08-23 Thread levin li
On 2012年08月24日 12:38, MORITA Kazutaka wrote: At Fri, 24 Aug 2012 10:56:35 +0800, levin li wrote: From: levin li xingke@taobao.com Since we already send vdi copy list in get_vdis(), there's no need to send the vdi_inuse bitmap any more, without it we can send less data as the length

[sheepdog] [PATCH v2 2/2] collie: make `collie vdi list` not read vdi_inuse bitmap

2012-08-23 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/common.c| 32 include/sheep.h|5 + sheep/sheep_priv.h |5 - 3 files changed, 21 insertions(+), 21 deletions(-) diff --git a/collie/common.c b

[sheepdog] [PATCH] test: sort the output in test 027

2012-08-22 Thread levin li
From: levin li xingke@taobao.com The order of the output of 'find' may be random, so we should sort the output first before we compare the output with 027.out Signed-off-by: levin li xingke@taobao.com --- tests/027 |2 +- tests/027.out |2 +- 2 files changed, 2 insertions

[sheepdog] [PATCH] test: wait to kill other nodes until the recovery finishs in 026

2012-08-22 Thread levin li
From: levin li xingke@taobao.com We kill two nodes in test 026 and then kill another two nodes, if the recovery doesn't finish, four nodes leave may cause object lost, as our default redundancy level is 3. Signed-off-by: levin li xingke@taobao.com --- tests/026 |6 +++--- 1 files

[sheepdog] [PATCH v5 0/8] different redundancy level for different VDIs

2012-08-22 Thread levin li
From: levin li xingke@taobao.com v4 --- v5 1. do not store vdi copy list locally any more, read it from local inode data 2. collect copy list data from other nodes in get_vdi_bitmap 3. make recovery wait until get_vdi_bitmap finished 4. add back the default redundancy level in 'collie cluster

[sheepdog] [PATCH v5 1/8] sheep: use struct vdi_iocb to simplify the vdi_create api

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/ops.c| 12 -- sheep/sheep_priv.h | 12 +- sheep/vdi.c| 55 +-- 3 files changed, 46 insertions(+), 33 deletions

[sheepdog] [PATCH v5 2/8] collie: add -c for 'collie vdi create' to specify redundancy level for per-vdi

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c | 25 +++-- sheep/ops.c |9 ++--- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/collie/vdi.c b/collie/vdi.c index 5781e75..d8fe029 100644

[sheepdog] [PATCH v5 4/8] sheep: add vdi_copy tree to store copies number for every VDI

2012-08-22 Thread levin li
From: levin li xingke@taobao.com The normal requests from QEMU include the copies number in its header, QEMU stores the nr_copies the first time it reads the inode data, but many other local requests such as read_copy_from_replica and recover_object_from_replica don't know the copies number

[sheepdog] [PATCH v5 5/8] sheep: fetch vdi copy list after sheep joins the cluster

2012-08-22 Thread levin li
From: levin li xingke@taobao.com The new joined node doesn't have the vdi copy list, or have incomplete vdi copy list, so we need to fetch the copy list data from other nodes Signed-off-by: levin li xingke@taobao.com --- collie/common.c| 15 +-- sheep/farm/farm.c | 31

[sheepdog] [PATCH v5 6/8] sheep: use the specified copies number for IO requests

2012-08-22 Thread levin li
From: levin li xingke@taobao.com In gateway_read{write,create_and_write}_obj, and read{write}_object, we should use different copies number for different requests and objects, instead of using the global copies number sys-nr_copies or calculated from vnodes Signed-off-by: levin li xingke

[sheepdog] [PATCH v5 8/8] collie: show copy number in the output of 'collie vdi list'

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c | 10 +- tests/016.out |2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/collie/vdi.c b/collie/vdi.c index 52015ed..b354619 100644 --- a/collie/vdi.c +++ b

[sheepdog] [PATCH v6 1/8] sheep: use struct vdi_iocb to simplify the vdi_create api

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/ops.c| 12 -- sheep/sheep_priv.h | 12 +- sheep/vdi.c| 55 +-- 3 files changed, 46 insertions(+), 33 deletions

[sheepdog] [PATCH v6 2/8] collie: add -c for 'collie vdi create' to specify redundancy level for per-vdi

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c | 25 +++-- sheep/ops.c |9 ++--- 2 files changed, 25 insertions(+), 9 deletions(-) diff --git a/collie/vdi.c b/collie/vdi.c index 5781e75..d8fe029 100644

[sheepdog] [PATCH v6 3/8] sheep: rename ctime to create_time in vdi.c

2012-08-22 Thread levin li
From: levin li xingke@taobao.com 'ctime' shadows a variable in pthread.h, in order to use pthread in vdi.c, so we need to rename ctime to get rid of the compile warning Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c |6 +++--- include/sheepdog_proto.h |2

[sheepdog] [PATCH v6 4/8] sheep: add vdi_copy tree to store copies number for every VDI

2012-08-22 Thread levin li
From: levin li xingke@taobao.com The normal requests from QEMU include the copies number in its header, QEMU stores the nr_copies the first time it reads the inode data, but many other local requests such as read_copy_from_replica and recover_object_from_replica don't know the copies number

[sheepdog] [PATCH v6 5/8] sheep: fetch vdi copy list after sheep joins the cluster

2012-08-22 Thread levin li
From: levin li xingke@taobao.com The new joined node doesn't have the vdi copy list, or have incomplete vdi copy list, so we need to fetch the copy list data from other nodes Signed-off-by: levin li xingke@taobao.com --- collie/common.c| 15 +++-- sheep/farm/farm.c | 31

[sheepdog] [PATCH v6 6/8] sheep: use the specified copies number for IO requests

2012-08-22 Thread levin li
From: levin li xingke@taobao.com In gateway_read{write,create_and_write}_obj, and read{write}_object, we should use different copies number for different requests and objects, instead of using the global copies number sys-nr_copies or calculated from vnodes Signed-off-by: levin li xingke

[sheepdog] [PATCH v6 7/8] sheep: remove some unused copies related functions

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/group.c | 36 +++- sheep/ops.c |3 +-- 2 files changed, 12 insertions(+), 27 deletions(-) diff --git a/sheep/group.c b/sheep/group.c index 11ea5ff..4510ab7

[sheepdog] [PATCH v6 8/8] collie: show copy number in the output of 'collie vdi list'

2012-08-22 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c | 10 +- tests/016.out |2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/collie/vdi.c b/collie/vdi.c index 52015ed..b354619 100644 --- a/collie/vdi.c +++ b

[sheepdog] [PATCH] tests: kill sheep with signal KILL

2012-08-21 Thread levin li
From: levin li xingke@taobao.com In some case, we use pkill -f to kill a sheep node, and the log process isn't killed immediately and becomes a defunt process which take problems for the next start of the same node. Signed-off-by: levin li xingke@taobao.com --- tests/common.rc |2

Re: [sheepdog] [PATCH v4 01/10] sheep: use struct vdi_iocb to simplify the vdi_create api

2012-08-20 Thread levin li
On 2012年08月20日 12:53, MORITA Kazutaka wrote: At Thu, 9 Aug 2012 13:27:36 +0800, levin li wrote: +struct vdi_iocb { +char *data; Should be char *name? Yes +uint32_t data_len; +uint64_t size; +uint32_t base_vid; +int is_snapshot; Should be bool is_snapshot

Re: [sheepdog] [PATCH v4 06/10] sheep: fetch vdi copy list after sheep joins the cluster

2012-08-20 Thread levin li
On 2012年08月20日 16:03, MORITA Kazutaka wrote: At Mon, 20 Aug 2012 15:41:03 +0800, levin li wrote: On 2012年08月20日 13:15, MORITA Kazutaka wrote: At Thu, 9 Aug 2012 13:27:41 +0800, levin li wrote: From: levin li xingke@taobao.com The new joined node doesn't have the vdi copy list

Re: [sheepdog] [PATCH 0/4] variable vnodes specified by free space

2012-08-08 Thread levin li
On 2012年08月08日 14:47, Liu Yuan wrote: On 08/07/2012 05:37 PM, MORITA Kazutaka wrote: BTW, it seems that the latest master doesn't pass cluster/002.sh, which was passed when I released 0.4.0. Current master passes it on my box. Thanks, Yuan I tested it yesterday, and find that it

[sheepdog] [PATCH v4 00/10] different redundancy level for different VDIs

2012-08-08 Thread levin li
From: levin li xingke@taobao.com v3 v4: 1. fix a bug that the cluster hangs when using local driver 2. update the existing VDI copy entry when inserting fails v2 v3 : 1. change get_min_copy_number to get_max_copy_number 2. store vdi copy data in disk in case of sheep restarts 3

[sheepdog] [PATCH v4 01/10] sheep: use struct vdi_iocb to simplify the vdi_create api

2012-08-08 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- sheep/ops.c| 12 -- sheep/sheep_priv.h | 12 +- sheep/vdi.c| 57 +-- 3 files changed, 47 insertions(+), 34 deletions

[sheepdog] [PATCH v4 02/10] collie: add -c for 'collie vdi create' to specify redundancy level for per-vdi

2012-08-08 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c | 27 --- sheep/ops.c |9 ++--- 2 files changed, 26 insertions(+), 10 deletions(-) diff --git a/collie/vdi.c b/collie/vdi.c index e4f4f65..4c793c9 100644

[sheepdog] [PATCH v4 03/10] sheep: rename ctime to create_time in vdi.c

2012-08-08 Thread levin li
From: levin li xingke@taobao.com 'ctime' shadows a variable in pthread.h, in order to use pthread in vdi.c, so we need to rename ctime to get rid of the compile warning Signed-off-by: levin li xingke@taobao.com --- collie/vdi.c |6 +++--- include/sheepdog_proto.h |2

[sheepdog] [PATCH v4 04/10] sheep: add vdi_copy tree to store copies number for every VDI

2012-08-08 Thread levin li
From: levin li xingke@taobao.com The normal requests from QEMU include the copies number in its header, QEMU stores the nr_copies the first time it reads the inode data, but many other local requests such as read_copy_from_replica and recover_object_from_replica don't know the copies number

[sheepdog] [PATCH v4 06/10] sheep: fetch vdi copy list after sheep joins the cluster

2012-08-08 Thread levin li
From: levin li xingke@taobao.com The new joined node doesn't have the vdi copy list, or have incomplete vdi copy list, so we need to fetch the copy list data from other nodes Signed-off-by: levin li xingke@taobao.com --- include/internal_proto.h |1 + sheep/group.c| 11

[sheepdog] [PATCH v4 05/10] sheep: store the VDI copies data in disk in case of sheep restarts

2012-08-08 Thread levin li
From: levin li xingke@taobao.com If we only store the copies data in memory, then after restart the copies data is lost, and we can only try to read the inode data to get the copies data, which is expensive Signed-off-by: levin li xingke@taobao.com --- sheep/sheep_priv.h |2 + sheep

[sheepdog] [PATCH v4 07/10] sheep: use the specified copies number for IO requests

2012-08-08 Thread levin li
From: levin li xingke@taobao.com In gateway_read{write,create_and_write}_obj, and read{write}_object, we should use different copies number for different requests and objects, instead of using the global copies number sys-nr_copies or calculated from vnodes Signed-off-by: levin li xingke

[sheepdog] [PATCH v4 08/10] sheep: cleanup the global copies number

2012-08-08 Thread levin li
From: levin li xingke@taobao.com Since every VDI has its own copies number, there's no need to keep the global copies number Signed-off-by: levin li xingke@taobao.com --- sheep/group.c | 50 -- sheep/ops.c| 18

[sheepdog] [PATCH v4 09/10] collie: remove copies from 'collie cluster format'

2012-08-08 Thread levin li
From: levin li xingke@taobao.com Signed-off-by: levin li xingke@taobao.com --- collie/cluster.c | 17 + 1 files changed, 1 insertions(+), 16 deletions(-) diff --git a/collie/cluster.c b/collie/cluster.c index 9302b78..f690324 100644 --- a/collie/cluster.c +++ b/collie

  1   2   3   4   >