Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Vasiliy Tolstov
2015-02-11 15:43 GMT+03:00 Liu Yuan namei.u...@gmail.com: This scheme can build on the sheepdog's current features: 0 use qemu-img (recommenced because better performance) or dog to read the base vdi. 1 use dog to backup the delta data for different snapshots takben by qemu-img

[sheepdog] Build failed in Jenkins: sheepdog-build #631

2015-02-11 Thread sheepdog-jenkins
See http://jenkins.sheepdog-project.org:8080/job/sheepdog-build/631/changes Changes: [liuyuan] dog: type cast miss at vdi_show_progress -- [...truncated 57 lines...] checking for grep that handles long lines and -e... /bin/grep checking for egrep...

Re: [sheepdog] [PATCH] dog: fix to calculate a resizable max VDI size appropriately

2015-02-11 Thread Liu Yuan
On Tue, Feb 10, 2015 at 05:53:44PM +0900, Teruaki Ishizaki wrote: A resizable max VDI size was fixed value, 4TB. So, when block_size_shift was specified more than 22, resizing VDI size over 4TB caused error. This patch enables to calculate a resizable max VDI properly. Signed-off-by:

Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Liu Yuan
On Wed, Feb 11, 2015 at 05:57:25PM +0400, Vasiliy Tolstov wrote: 2015-02-11 15:43 GMT+03:00 Liu Yuan namei.u...@gmail.com: This scheme can build on the sheepdog's current features: 0 use qemu-img (recommenced because better performance) or dog to read the base vdi. 1 use dog to

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Liu Yuan
On Thu, Feb 12, 2015 at 10:51:25AM +0900, Teruaki Ishizaki wrote: (2015/02/10 20:12), Liu Yuan wrote: On Tue, Jan 27, 2015 at 05:35:27PM +0900, Teruaki Ishizaki wrote: Previously, qemu block driver of sheepdog used hard-coded VDI object size. This patch enables users to handle block_size_shift

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Liu Yuan
On Thu, Feb 12, 2015 at 11:33:16AM +0900, Teruaki Ishizaki wrote: (2015/02/12 11:19), Liu Yuan wrote: On Thu, Feb 12, 2015 at 10:51:25AM +0900, Teruaki Ishizaki wrote: (2015/02/10 20:12), Liu Yuan wrote: On Tue, Jan 27, 2015 at 05:35:27PM +0900, Teruaki Ishizaki wrote: Previously, qemu block

[sheepdog] who is in charge of Jenkins-sheepdog

2015-02-11 Thread Liu Yuan
Hi all, who has the right permission to install yasm on the sheepdog Jenkins server? Our list is annoyed everyday, please save us from it. Thanks Yuan -- sheepdog mailing list sheepdog@lists.wpkg.org https://lists.wpkg.org/mailman/listinfo/sheepdog

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Teruaki Ishizaki
(2015/02/10 20:12), Liu Yuan wrote: On Tue, Jan 27, 2015 at 05:35:27PM +0900, Teruaki Ishizaki wrote: Previously, qemu block driver of sheepdog used hard-coded VDI object size. This patch enables users to handle block_size_shift value for calculating VDI object size. When you start qemu, you

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Teruaki Ishizaki
(2015/02/12 11:19), Liu Yuan wrote: On Thu, Feb 12, 2015 at 10:51:25AM +0900, Teruaki Ishizaki wrote: (2015/02/10 20:12), Liu Yuan wrote: On Tue, Jan 27, 2015 at 05:35:27PM +0900, Teruaki Ishizaki wrote: Previously, qemu block driver of sheepdog used hard-coded VDI object size. This patch

Re: [sheepdog] [PATCH 1/2] dog: add a new option for reducing identical snapshots

2015-02-11 Thread Liu Yuan
On Thu, Feb 12, 2015 at 04:40:56PM +0900, Hitoshi Mitake wrote: At Thu, 12 Feb 2015 15:31:15 +0800, Liu Yuan wrote: On Thu, Feb 12, 2015 at 03:59:51PM +0900, Hitoshi Mitake wrote: At Thu, 12 Feb 2015 14:38:37 +0800, Liu Yuan wrote: On Mon, Feb 09, 2015 at 05:25:48PM +0900,

[sheepdog] [PATCH 4/5] tests: fix content of 052.out

2015-02-11 Thread Wang dongxu
Since code is printf(%s\n, sd_strerror(rsp-result));, 052.out should add a new line. Signed-off-by: Wang dongxu wangdon...@cmss.chinamobile.com --- tests/functional/052.out | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/functional/052.out b/tests/functional/052.out index

[sheepdog] [PATCH 0/5] tests: fix some test cases to suitable for new sheepdog and QEMU

2015-02-11 Thread Wang dongxu
QEMU and sheepdog changes some output formats while upgrading to new version, so tests/functional test cases need some changes. Wang dongxu (5): tests: avoid qemu-io warning tests: avoid qemu-img snapshot warning tests: correct vdi list tests: fix content of 052.out tests:fix vnode

[sheepdog] [PATCH 1/5] tests: avoid qemu-io warning

2015-02-11 Thread Wang dongxu
qemu-io command add a warning message because probing a raw img is dangerous. So add -f option to avoid this. Signed-off-by: Wang dongxu wangdon...@cmss.chinamobile.com --- tests/functional/013 | 6 +++--- tests/functional/017 | 2 +- tests/functional/024 | 6 +++--- tests/functional/025 | 4

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Hitoshi Mitake
At Thu, 12 Feb 2015 15:00:49 +0800, Liu Yuan wrote: On Thu, Feb 12, 2015 at 03:19:21PM +0900, Hitoshi Mitake wrote: At Tue, 10 Feb 2015 18:35:58 +0800, Liu Yuan wrote: On Tue, Feb 10, 2015 at 06:56:33PM +0900, Teruaki Ishizaki wrote: (2015/02/10 17:58), Liu Yuan wrote: On

Re: [sheepdog] [PATCH 1/2] dog: add a new option for reducing identical snapshots

2015-02-11 Thread Liu Yuan
On Mon, Feb 09, 2015 at 05:25:48PM +0900, Hitoshi Mitake wrote: Current dog vdi snapshot command creates a new snapshot unconditionally, even if a working VDI doesn't have its own objects. In such a case, the created snapshot is redundant because same VDI is already existing. What kind of use

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Liu Yuan
On Thu, Feb 12, 2015 at 04:28:01PM +0900, Hitoshi Mitake wrote: At Thu, 12 Feb 2015 15:00:49 +0800, Liu Yuan wrote: On Thu, Feb 12, 2015 at 03:19:21PM +0900, Hitoshi Mitake wrote: At Tue, 10 Feb 2015 18:35:58 +0800, Liu Yuan wrote: On Tue, Feb 10, 2015 at 06:56:33PM +0900,

Re: [sheepdog] [PATCH 1/2] dog: add a new option for reducing identical snapshots

2015-02-11 Thread Hitoshi Mitake
At Thu, 12 Feb 2015 15:31:15 +0800, Liu Yuan wrote: On Thu, Feb 12, 2015 at 03:59:51PM +0900, Hitoshi Mitake wrote: At Thu, 12 Feb 2015 14:38:37 +0800, Liu Yuan wrote: On Mon, Feb 09, 2015 at 05:25:48PM +0900, Hitoshi Mitake wrote: Current dog vdi snapshot command creates a new

[sheepdog] [PATCH] sheepdog:show more detail of the crash source

2015-02-11 Thread Wang Zhengyong
In the current sheepdog, when sheepdog crashed, there is too little information about the signal source. This patch use (*handler)(int, siginfo_t *, void *) instead of (*handler)(int). In this way, can show more detail of the crash problem, especially the pid of singal sender Cc: Hitoshi Mitake

[sheepdog] [PATCH 2/5] tests: avoid qemu-img snapshot warning

2015-02-11 Thread Wang dongxu
qemu-img snapshot option will print warining message while probing a raw image, so filter them using sed. Signed-off-by: Wang dongxu wangdon...@cmss.chinamobile.com --- tests/functional/013 | 2 +- tests/functional/017 | 12 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff

[sheepdog] [PATCH 3/5] tests: correct vdi list

2015-02-11 Thread Wang dongxu
dog vdi list add column Block Size Shift, add them to test cases. Signed-off-by: Wang dongxu wangdon...@cmss.chinamobile.com --- tests/functional/073.out | 2 +- tests/functional/081.out | 6 +++--- tests/functional/082.out | 6 +++--- tests/functional/087.out | 10 +-

Re: [sheepdog] [PATCH v2] sheepdog:show more detail of the crash source

2015-02-11 Thread Hitoshi Mitake
At Wed, 11 Feb 2015 23:44:53 -0800, Wang Zhengyong wrote: In the current sheepdog, when sheepdog crashed, there is too little information about the signal source. This patch use (*handler)(int, siginfo_t *, void *) instead of (*handler)(int). In this way, can show more detail of the crash

[sheepdog] [PATCH v2] sheepdog:show more detail of the crash source

2015-02-11 Thread Wang Zhengyong
In the current sheepdog, when sheepdog crashed, there is too little information about the signal source. This patch use (*handler)(int, siginfo_t *, void *) instead of (*handler)(int). In this way, can show more detail of the crash problem, especially the pid of singal sender Cc: Hitoshi Mitake

[sheepdog] [PATCH 5/5] tests:fix vnode strategy output

2015-02-11 Thread Wang dongxu
since commit 5fed9d6, Cluster vnodes strategy information is produced, so fix them in test cases. Signed-off-by: Wang dongxu wangdon...@cmss.chinamobile.com --- tests/functional/030.out | 1 + tests/functional/096.out | 2 ++ 2 files changed, 3 insertions(+) diff --git

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Liu Yuan
On Thu, Feb 12, 2015 at 03:19:21PM +0900, Hitoshi Mitake wrote: At Tue, 10 Feb 2015 18:35:58 +0800, Liu Yuan wrote: On Tue, Feb 10, 2015 at 06:56:33PM +0900, Teruaki Ishizaki wrote: (2015/02/10 17:58), Liu Yuan wrote: On Tue, Feb 10, 2015 at 05:22:02PM +0900, Teruaki Ishizaki wrote:

Re: [sheepdog] [PATCH 1/2] dog: add a new option for reducing identical snapshots

2015-02-11 Thread Hitoshi Mitake
At Thu, 12 Feb 2015 14:38:37 +0800, Liu Yuan wrote: On Mon, Feb 09, 2015 at 05:25:48PM +0900, Hitoshi Mitake wrote: Current dog vdi snapshot command creates a new snapshot unconditionally, even if a working VDI doesn't have its own objects. In such a case, the created snapshot is

Re: [sheepdog] [PATCH 1/2] dog: add a new option for reducing identical snapshots

2015-02-11 Thread Liu Yuan
On Thu, Feb 12, 2015 at 03:59:51PM +0900, Hitoshi Mitake wrote: At Thu, 12 Feb 2015 14:38:37 +0800, Liu Yuan wrote: On Mon, Feb 09, 2015 at 05:25:48PM +0900, Hitoshi Mitake wrote: Current dog vdi snapshot command creates a new snapshot unconditionally, even if a working VDI doesn't

Re: [sheepdog] [PATCH] sheepdog:show more detail of the crash source

2015-02-11 Thread Hitoshi Mitake
At Wed, 11 Feb 2015 21:47:41 -0800, Wang Zhengyong wrote: In the current sheepdog, when sheepdog crashed, there is too little information about the signal source. This patch use (*handler)(int, siginfo_t *, void *) instead of (*handler)(int). In this way, can show more detail of the crash

[sheepdog] [PATCH] zookeeper: add more detailed description on how zk_watcher report states

2015-02-11 Thread Liu Yuan
From: Liu Yuan liuy...@cmss.chinamobile.com Signed-off-by: Liu Yuan liuy...@cmss.chinamobile.com --- sheep/cluster/zookeeper.c | 25 +++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/sheep/cluster/zookeeper.c b/sheep/cluster/zookeeper.c index

Re: [sheepdog] obbject placement

2015-02-11 Thread Liu Yuan
On Sun, Feb 08, 2015 at 08:46:01PM +0100, Corin Langosch wrote: Hi guys, afaik sheepdog uses consistent hashing to map objects to nodes. But how do you choose where the individual ec-chunks of an object should go? Consistent hashing cannot be used here because different ec-chunks of the

Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Liu Yuan
On Wed, Feb 11, 2015 at 03:57:32PM +0400, Vasiliy Tolstov wrote: Hi! I need to store user backups and allows to download it. I see in google that sheepdog supports deduplication, but can't find info in sheepdog docs about it. Does sheepdog support deduplication? This deduplication is for SD's

[sheepdog] effective storing backups and deduplication

2015-02-11 Thread Vasiliy Tolstov
Hi! I need to store user backups and allows to download it. I see in google that sheepdog supports deduplication, but can't find info in sheepdog docs about it. Does sheepdog support deduplication? Also i think not use cluster wide snapshots because i want to dedicate backup server from other

Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Vasiliy Tolstov
2015-02-11 15:08 GMT+03:00 Liu Yuan namei.u...@gmail.com: On Wed, Feb 11, 2015 at 03:57:32PM +0400, Vasiliy Tolstov wrote: Hi! I need to store user backups and allows to download it. I see in google that sheepdog supports deduplication, but can't find info in sheepdog docs about it. Does

Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Vasiliy Tolstov
2015-02-11 15:28 GMT+03:00 Liu Yuan namei.u...@gmail.com: We need to what is user's backups. Is it the whole vdi or dalta data for different vdis? Best scheme as i think is: 1) If backup not exists for vdi - create full backup (this is simple copy all data) 2) If backup already created -

Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Liu Yuan
On Wed, Feb 11, 2015 at 04:32:34PM +0400, Vasiliy Tolstov wrote: 2015-02-11 15:28 GMT+03:00 Liu Yuan namei.u...@gmail.com: We need to what is user's backups. Is it the whole vdi or dalta data for different vdis? Best scheme as i think is: 1) If backup not exists for vdi - create full

Re: [sheepdog] effective storing backups and deduplication

2015-02-11 Thread Liu Yuan
On Wed, Feb 11, 2015 at 04:14:35PM +0400, Vasiliy Tolstov wrote: 2015-02-11 15:08 GMT+03:00 Liu Yuan namei.u...@gmail.com: On Wed, Feb 11, 2015 at 03:57:32PM +0400, Vasiliy Tolstov wrote: Hi! I need to store user backups and allows to download it. I see in google that sheepdog supports

Re: [sheepdog] [PATCH v4] sheepdog: selectable object size support

2015-02-11 Thread Hitoshi Mitake
At Tue, 10 Feb 2015 18:35:58 +0800, Liu Yuan wrote: On Tue, Feb 10, 2015 at 06:56:33PM +0900, Teruaki Ishizaki wrote: (2015/02/10 17:58), Liu Yuan wrote: On Tue, Feb 10, 2015 at 05:22:02PM +0900, Teruaki Ishizaki wrote: (2015/02/10 12:10), Liu Yuan wrote: On Tue, Jan 27, 2015 at