Re: [PATCH] Setup GitLab-CI for btrfs-progs
On Mon, Oct 07, 2019 at 07:52:35PM +0200, David Sterba wrote: > > Is it possible to move the files to ci/gitlab? .gitlab-ci.yml must be > probably in the top-level dir but that's acceptable. > Ok added these changes to Patch V2. > > The scripts look good to me but I have limited knowledge of the CI > environment so I don't have objections against merging the patch. I'll > spend some time experimenting but overall this seems in a good shape and > we'll get further coverage (due to the new kernel) than what travis > provides. Thanks. Yes, please experiment with it. There are some failures in tests https://gitlab.com/giis/btrfs-progs/pipelines/88716001 and I'm not sure whether its due to environment or missing packages or something else. Let me know, if something needs to be updated or fixed. thanks! Cheers. Lakshmipathi.G
[PATCH v2] Setup GitLab-CI for btrfs-progs
Make use of GitLab-CI nested virutal environment to start QEMU instance inside containers and perform btrfs-progs build, execute unit test cases and save the logs. More details can be found at https://github.com/kdave/btrfs-progs/issues/171 Signed-off-by: Lakshmipathi.G --- .gitlab-ci.yml| 158 ++ ci/gitlab/Dockerfile | 3 + ci/gitlab/btrfs-progs-tests.service | 13 +++ ci/gitlab/build_or_run_btrfs-progs.sh | 37 ci/gitlab/kernel_build.sh | 30 +++ ci/gitlab/run_tests.sh| 9 ++ ci/gitlab/setup_image.sh | 42 + 7 files changed, 292 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 ci/gitlab/Dockerfile create mode 100644 ci/gitlab/btrfs-progs-tests.service create mode 100755 ci/gitlab/build_or_run_btrfs-progs.sh create mode 100755 ci/gitlab/kernel_build.sh create mode 100755 ci/gitlab/run_tests.sh create mode 100755 ci/gitlab/setup_image.sh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 000..8ecfc5a --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,158 @@ +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public +# License v2 as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public +# License along with this program; if not, write to the +# Free Software Foundation, Inc., 59 Temple Place - Suite 330, +# Boston, MA 021110-1307, USA. +# + +image: docker:18.09.7 + +services: +- docker:18.09.7-dind + +# To enable or disable Kernel Build set BUILD_KERNEL: "1" or BUILD_KERNEL: "0" +# If you disable Kernel Build, make sure PREBUILT_KERNEL_ID points to previously built the kernel job id. +# To enable or disable image build update BUILD_IMAGE value to "1" or "0". +# If you disable Image Build, make sure PREBUILT_IMAGE_ID points to previously built rootfs job id. + +variables: + DOCKER_DRIVER: overlay2 + IMAGE_TAG: registry.gitlab.com/$CI_PROJECT_NAMESPACE/btrfs-progs:gitlab-ci + BUILD_KERNEL: "1" + PREBUILT_KERNEL_ID: "320566224" + BUILD_IMAGE: "1" + PREBUILT_IMAGE_ID: "288506168" + +stages: + - build + - btrfs-progs build + - test + +before_script: + - docker login --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD $CI_REGISTRY + +docker build: + stage: build + script: +- cd ci/gitlab +- docker pull $IMAGE_TAG > /dev/null && echo "Downloaded image" || ( docker build -t $IMAGE_TAG . && docker push $IMAGE_TAG ) + + +kernel build: + before_script: +- apk add curl unzip + stage: build + script: + - if [ "$BUILD_KERNEL" == "1" ]; then + docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/ci/gitlab/kernel_build.sh; + else + curl -o bzImage.zip --location --header "JOB-TOKEN:$CI_JOB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$PREBUILT_KERNEL_ID/artifacts"; && unzip bzImage.zip; + fi; + artifacts: +when: always +paths: + - bzImage + + +image build: + before_script: +- apk add curl unzip + stage: build + script: + - if [ "$BUILD_IMAGE" == "1" ]; then + docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/ci/gitlab/setup_image.sh; + else + curl -o qemu-image.img.zip --location --header "JOB-TOKEN:$CI_JOB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$PREBUILT_IMAGE_ID/artifacts"; && unzip qemu-image.img.zip; + fi; + artifacts: +when: always +paths: + - qemu-image.img + +btrfs-progs build: + stage: btrfs-progs build + script: + - docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/ci/gitlab/run_tests.sh + artifacts: +expire_in: 1 week +when: always +paths: + - qemu-image.img + +cli tests: + stage: test + script: + - echo "./cli-tests.sh" > $PWD/cmd + - docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/ci/gitlab/run_tests.sh + - test -e "result" || exit 1 # If result doesn't exists, job failed. + artifacts: +when: always +paths: + - "*tests-results.txt" + +convert tests: + only: +- devel + stage: test + script: + - echo "./conv
Re: [PATCH] Setup GitLab-CI for btrfs-progs
On Tue, Oct 01, 2019 at 08:02:29AM +0200, Philipp Hahn wrote: > > You already have a "variables" section above - merge them? > > > + services: > > +- docker:18.09.7-dind > > You already have "services" defined globally - no need to repeat that > here again. > Hi Philipp, Thanks for the comments. I ran into some issue while using global section so started using job section. Let me try placing them in global and update the results. thanks. Cheers. Lakshmipathi.G
[PATCH] Setup GitLab-CI for btrfs-progs
Make use of GitLab-CI nested virutal environment to start QEMU instance inside containers and perform btrfs-progs build, execute unit test cases and save the logs. More details can be found at https://github.com/kdave/btrfs-progs/issues/171 Signed-off-by: Lakshmipathi.G --- .gitlab-ci.yml| 181 ++ gitlab-ci/Dockerfile | 3 + gitlab-ci/btrfs-progs-tests.service | 13 +++ gitlab-ci/build_or_run_btrfs-progs.sh | 37 +++ gitlab-ci/kernel_build.sh | 30 ++ gitlab-ci/run_tests.sh| 9 ++ gitlab-ci/setup_image.sh | 42 7 files changed, 315 insertions(+) create mode 100644 .gitlab-ci.yml create mode 100644 gitlab-ci/Dockerfile create mode 100644 gitlab-ci/btrfs-progs-tests.service create mode 100755 gitlab-ci/build_or_run_btrfs-progs.sh create mode 100755 gitlab-ci/kernel_build.sh create mode 100755 gitlab-ci/run_tests.sh create mode 100755 gitlab-ci/setup_image.sh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml new file mode 100644 index 000..2afde50 --- /dev/null +++ b/.gitlab-ci.yml @@ -0,0 +1,181 @@ +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU General Public +# License v2 as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public +# License along with this program; if not, write to the +# Free Software Foundation, Inc., 59 Temple Place - Suite 330, +# Boston, MA 021110-1307, USA. +# + +image: docker:18.09.7 + +services: +- docker:18.09.7-dind + +variables: + DOCKER_DRIVER: overlay2 + +stages: + - build + - btrfs-progs build + - test + +variables: + DOCKER_DRIVER: overlay2 + IMAGE_TAG: registry.gitlab.com/$CI_PROJECT_NAMESPACE/btrfs-progs:gitlab-ci + +before_script: + - docker login --username $CI_REGISTRY_USER --password $CI_REGISTRY_PASSWORD $CI_REGISTRY + +docker build: + stage: build + script: +- cd gitlab-ci +- docker pull $IMAGE_TAG > /dev/null && echo "Downloaded image" || ( docker build -t $IMAGE_TAG . && docker push $IMAGE_TAG ) +- cd .. + +## To enable or disable Kernel Build set BUILD_KERNEL: "1" or BUILD_KERNEL: "0" +## If you disable Kernel Build, make sure PREBUILT_KERNEL_ID points to previously built the kernel job id. + +kernel build: + variables: +BUILD_KERNEL: "1" +PREBUILT_KERNEL_ID: "288159334" + before_script: +- apk add curl unzip + stage: build + services: +- docker:18.09.7-dind + script: + - if [ "$BUILD_KERNEL" == "1" ]; then + docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/gitlab-ci/kernel_build.sh; + else + curl -o bzImage.zip --location --header "JOB-TOKEN:$CI_JOB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$PREBUILT_KERNEL_ID/artifacts"; && unzip bzImage.zip; + fi; + artifacts: +when: always +paths: + - bzImage + +# To enable or disable image build update BUILD_IMAGE value to "1" or "0". +# If you disable Image Build, make sure PREBUILT_IMAGE_ID points to previously built rootfs job id. + +image build: + variables: +BUILD_IMAGE: "1" +PREBUILT_IMAGE_ID: "288506168" + before_script: +- apk add curl unzip + stage: build + services: +- docker:18.09.7-dind + script: + - if [ "$BUILD_IMAGE" == "1" ]; then + docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/gitlab-ci/setup_image.sh; + else + curl -o qemu-image.img.zip --location --header "JOB-TOKEN:$CI_JOB_TOKEN" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$PREBUILT_IMAGE_ID/artifacts"; && unzip qemu-image.img.zip; + fi; + artifacts: +when: always +paths: + - qemu-image.img + +btrfs-progs build: + stage: btrfs-progs build + services: +- docker:18.09.7-dind + script: + - docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/gitlab-ci/run_tests.sh + artifacts: +expire_in: 1 week +when: always +paths: + - qemu-image.img + +cli tests: + stage: test + services: +- docker:18.09.7-dind + script: + - echo "./cli-tests.sh" > $PWD/cmd + - docker run --cap-add SYS_PTRACE --cap-add sys_admin --privileged --device=/dev/kvm -v $PWD:/repo $IMAGE_TAG /repo/gitlab-ci/run_tests.sh + - test
Re: dduper - Offline btrfs deduplication tool
Apologies for pretty delay in response. Thanks for the suggestions and comments. >On the other hand, full bit-to-bit comparison is faster and 100% safe >With fideduperange the risk is more than minimized--it's completely >eliminated. Okay got it, will use fideduperange() as default option for the tool and keep ficlonerange() as secondary option. Will make the code changes and send a new patch soon. thanks! Cheers. Lakshmipathi.G
Re: dduper - Offline btrfs deduplication tool
> > One question: > Why not ioctl_fideduperange? > i.e. you kill most of benefits from that ioctl - atomicity. > I plan to add fideduperange as an option too. User can choose between fideduperange and ficlonerange call. If I'm not wrong, with fideduperange, kernel performs comparsion check before dedupe. And it will increase time to dedupe files. I believe the risk involved with ficlonerange is minimized by having a backup file(reflinked). We can revert to older original file, if we encounter some problems. > > -- > Have a nice day, > Timofey. Cheers. Lakshmipathi.G
dduper - Offline btrfs deduplication tool
Hi - dduper is an offline dedupe tool. Instead of reading whole file blocks and computing checksum, It works by fetching checksum from BTRFS csum tree. This hugely improves the performance. dduper works like: - Read csum for given two files. - Find matching location. - Pass the location to ioctl_ficlonerange directly instead of ioctl_fideduperange By default, dduper adds safty check to above steps by creating a backup reflink file and compares the md5sum after dedupe. If the backup file matches new deduped file, then backup file is removed. You can skip this check by passing --skip option. Here is sample cli usage [1] and quick demo [2] Some performance numbers: (with -skip option) Dedupe two 1GB files with same content - 1.2 seconds Dedupe two 5GB files with same content - 8.2 seconds Dedupe two 10GB files with same content - 13.8 seconds dduper requires `btrfs inspect-internal dump-csum` command, you can use this branch [3] or apply patch by yourself [4] [1] https://gitlab.collabora.com/laks/btrfs-progs/blob/dump_csum/Documentation/dduper_usage.md [2] http://giis.co.in/btrfs_dedupe.gif [3] git clone https://gitlab.collabora.com/laks/btrfs-progs.git -b dump_csum [4] https://patchwork.kernel.org/patch/10540229/ Please remember its version-0.1, so test it out, if you plan to use dduper real data. Let me know, if you have suggestions or feedback or bugs :) Cheers. Lakshmipathi.G
[PATCH] v3: btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option
Print csum for a given file on stdout.
Sample usage:
btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4
Signed-off-by: Lakshmipathi.G
---
Makefile | 2 +-
cmds-inspect-dump-csum.c | 239 +++
cmds-inspect-dump-csum.h | 22 +
cmds-inspect.c | 3 +
4 files changed, 265 insertions(+), 1 deletion(-)
create mode 100644 cmds-inspect-dump-csum.c
create mode 100644 cmds-inspect-dump-csum.h
diff --git a/Makefile b/Makefile
index 544410e6..8ad28012 100644
--- a/Makefile
+++ b/Makefile
@@ -122,7 +122,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o
cmds-device.o cmds-scrub.o \
cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
- cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o
\
+ cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o
cmds-inspect-dump-csum.o \
mkfs/common.o check/mode-common.o check/mode-lowmem.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o
\
kernel-lib/crc32c.o messages.o \
diff --git a/cmds-inspect-dump-csum.c b/cmds-inspect-dump-csum.c
new file mode 100644
index ..06dba368
--- /dev/null
+++ b/cmds-inspect-dump-csum.c
@@ -0,0 +1,239 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include "kerncompat.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "ctree.h"
+#include "disk-io.h"
+#include "print-tree.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "commands.h"
+#include "crc32c.h"
+#include "cmds-inspect-dump-csum.h"
+#include "help.h"
+#include "volumes.h"
+
+
+const char * const cmd_inspect_dump_csum_usage[] = {
+ "btrfs inspect-internal dump-csum ",
+ "Get csums for the given file.",
+ NULL
+};
+
+int btrfs_lookup_csums(struct btrfs_trans_handle *trans, struct btrfs_root
*root,
+ struct btrfs_path *path, u64 bytenr, int cow, int total_csums)
+{
+ int ret;
+ int i;
+ int start_pos = 0;
+ struct btrfs_key file_key;
+ struct btrfs_key found_key;
+ struct btrfs_csum_item *item;
+ struct extent_buffer *leaf;
+ u64 csum_offset = 0;
+ u16 csum_size =
+ btrfs_super_csum_size(root->fs_info->super_copy);
+ int csums_in_item = 0;
+ unsigned int tree_csum = 0;
+ int pending_csums = total_csums;
+ static int cnt=1;
+
+ file_key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
+ file_key.offset = bytenr;
+ file_key.type = BTRFS_EXTENT_CSUM_KEY;
+ ret = btrfs_search_slot(trans, root, &file_key, path, 0, cow);
+ if (ret < 0)
+ goto fail;
+ while(1){
+ leaf = path->nodes[0];
+ if (ret > 0) {
+ ret = 1;
+ if (path->slots[0] == 0)
+ goto fail;
+ path->slots[0]--;
+ btrfs_item_key_to_cpu(leaf, &found_key, path->slots[0]);
+ if (found_key.type != BTRFS_EXTENT_CSUM_KEY){
+ fprintf(stderr, "\nInvalid key found.");
+ goto fail;
+ }
+
+ csum_offset = ((bytenr - found_key.offset) /
root->fs_info->sectorsize) * csum_size;
+ csums_in_item = btrfs_item_size_nr(leaf,
path->slots[0]);
+ csums_in_item /= csum_size;
+ csums_in_item -= ( bytenr - found_key.offset ) /
root->fs_info->sectorsize;
+ start_pos=csum_offset;
+ }
+ if (path->slots[0] >= btrfs_header_nritems(leaf)) {
+ if (pending_csums > 0){
+ ret = btrfs_next_leaf(root, path);
+ if (ret == 0)
+ con
Re: [PATCH] v2: btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option
> >Sample usage: > >btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4 > I'm not like the way to pass a filename as a argument. > Just print content to stdout, let users decides where to redirect. > And it will decrease amount of codes(part of checking and opening files). > Ok, will remove default output file creation part (filename.csumdump) and print the content to stdout. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] v2: btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option
Dump csum for a given file.
Sample usage:
btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4
csum for /btrfs/50gbfile dumped to /btrfs/50gbfile.csumdump
Signed-off-by: Lakshmipathi.G
---
Makefile | 2 +-
cmds-inspect-dump-csum.c | 256 +++
cmds-inspect-dump-csum.h | 22
cmds-inspect.c | 3 +
4 files changed, 282 insertions(+), 1 deletion(-)
create mode 100644 cmds-inspect-dump-csum.c
create mode 100644 cmds-inspect-dump-csum.h
diff --git a/Makefile b/Makefile
index 544410e..8ad2801 100644
--- a/Makefile
+++ b/Makefile
@@ -122,7 +122,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o
cmds-device.o cmds-scrub.o \
cmds-quota.o cmds-qgroup.o cmds-replace.o check/main.o \
cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
- cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o
\
+ cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o
cmds-inspect-dump-csum.o \
mkfs/common.o check/mode-common.o check/mode-lowmem.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o
\
kernel-lib/crc32c.o messages.o \
diff --git a/cmds-inspect-dump-csum.c b/cmds-inspect-dump-csum.c
new file mode 100644
index 000..38afcc3
--- /dev/null
+++ b/cmds-inspect-dump-csum.c
@@ -0,0 +1,256 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include "kerncompat.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "ctree.h"
+#include "disk-io.h"
+#include "print-tree.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "commands.h"
+#include "crc32c.h"
+#include "cmds-inspect-dump-csum.h"
+#include "help.h"
+#include "volumes.h"
+
+
+const char * const cmd_inspect_dump_csum_usage[] = {
+ "btrfs inspect-internal dump-csum ",
+ "Get csums for the given file.",
+ NULL
+};
+
+int btrfs_lookup_csums(struct btrfs_trans_handle *trans, struct btrfs_root
*root,
+ struct btrfs_path *path, u64 bytenr, int cow, int total_csums, FILE *fp)
+{
+ int ret;
+ int i;
+ int start_pos = 0;
+ struct btrfs_key file_key;
+ struct btrfs_key found_key;
+ struct btrfs_csum_item *item;
+ struct extent_buffer *leaf;
+ u64 csum_offset = 0;
+ u16 csum_size =
+ btrfs_super_csum_size(root->fs_info->super_copy);
+ int csums_in_item = 0;
+ unsigned int tree_csum = 0;
+ int pending_csums = total_csums;
+ static int cnt=1;
+
+ file_key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
+ file_key.offset = bytenr;
+ file_key.type = BTRFS_EXTENT_CSUM_KEY;
+ ret = btrfs_search_slot(trans, root, &file_key, path, 0, cow);
+ if (ret < 0)
+ goto fail;
+ while(1){
+ leaf = path->nodes[0];
+ if (ret > 0) {
+ ret = 1;
+ if (path->slots[0] == 0)
+ goto fail;
+ path->slots[0]--;
+ btrfs_item_key_to_cpu(leaf, &found_key, path->slots[0]);
+ if (found_key.type != BTRFS_EXTENT_CSUM_KEY){
+ fprintf(stderr, "\nInvalid key found.");
+ goto fail;
+ }
+
+ csum_offset = ((bytenr - found_key.offset) /
root->fs_info->sectorsize) * csum_size;
+ csums_in_item = btrfs_item_size_nr(leaf,
path->slots[0]);
+ csums_in_item /= csum_size;
+ csums_in_item -= ( bytenr - found_key.offset ) /
root->fs_info->sectorsize;
+ start_pos=csum_offset;
+ }
+ if (path->slots[0] >= btrfs_header_nritems(leaf)) {
+ if (pending_csums > 0){
+ ret = btrfs_next_leaf(root, path);
+
Re: [PATCH v2.2 04/10] btrfs: qgroup: Use separate meta reservation type for delalloc
> And for anyone who wants to test the latest version of the patchset, it > can be fetched from the following branch: I started a build with qgroup_meta_rework from [0] and its currently running [1], will update the results once it completes in few hours time. [0]: https://github.com/adam900710/linux : [1]: http://btrfs.in:3000/r/Lakshmipathi Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: xfstests/btrfs/100 and 151 failure
> Thanks for the config file, will check with this and update the status. script/100 passed on today's run [1] with my kernel config too. May be a false alarm :-) [1]: https://github.com/Lakshmipathi/btrfsqa/blob/master/results/results_2017-12-22_07:12/ thanks! Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org http://www.btrfs.in -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: xfstests/btrfs/100 and 151 failure
> > I've attached my master config before compiling a kernel with it I just > do make olddefconfig to sync it with latest kbuild options. > >> Thanks for the config file, will check with this and update the status. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: xfstests/btrfs/100 and 151 failure
> Actually 151 has been failing for me as well but not 100 > Okay, can you share the kernel .config file? I'll give it a try with those config and check 100, sometime tomorrow. ---- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
xfstests/btrfs/100 and 151 failure
Hi. While checking with latest misc-next branch from btrfs-devel. btrfs/100 and 151 failed. Details available on below bugzilla links. thanks https://bugzilla.kernel.org/show_bug.cgi?id=198085 https://bugzilla.kernel.org/show_bug.cgi?id=198087 Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 4/4] btrfs: factor __btrfs_open_devices() to create btrfs_open_one_device()
Hi Anand,
Applied your fix, now the issue resolved. thanks!
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
On Mon, Nov 27, 2017 at 5:17 PM, Anand Jain wrote:
>
> Hi Lakshmipathi,
>
> Oops I can see the same. I am trying to narrow down.
>
> Thanks, Anand
>
>
> On 11/27/2017 02:47 PM, Lakshmipathi.G wrote:
>>
>> Hi Anand,
>>
>> With this patch applied, btrfs-progs/misc-test/021 error out. Is this
>> same for you?
>>
>> Without this patch: https://asciinema.org/a/RJmE5469mHlL3S1BIOCifWVn6
>> With this patch:https://asciinema.org/a/1h5UX6DIFNsvvMXgLo4GiEgdE
>>
>> thanks!
>>
>> Cheers,
>> Lakshmipathi.G
>> http://www.giis.co.in http://www.webminal.org
>>
>>
>> On Thu, Nov 9, 2017 at 9:15 PM, Anand Jain wrote:
>>>
>>> No functional changes, create btrfs_open_one_device() from
>>> __btrfs_open_devices(). This is a preparatory work to add dynamic
>>> device scan.
>>>
>>> Signed-off-by: Anand Jain
>>> ---
>>> fs/btrfs/volumes.c | 126
>>> +
>>> 1 file changed, 69 insertions(+), 57 deletions(-)
>>>
>>> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
>>> index 0857b580014d..d24e966ee29f 100644
>>> --- a/fs/btrfs/volumes.c
>>> +++ b/fs/btrfs/volumes.c
>>> @@ -601,6 +601,73 @@ void btrfs_free_stale_device(struct btrfs_device
>>> *cur_dev)
>>> }
>>> }
>>>
>>> +static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
>>> + struct btrfs_device *device, fmode_t flags,
>>> + void *holder)
>>> +{
>>> + struct request_queue *q;
>>> + struct block_device *bdev;
>>> + struct buffer_head *bh;
>>> + struct btrfs_super_block *disk_super;
>>> + u64 devid;
>>> + int ret;
>>> +
>>> + if (device->bdev)
>>> + return -EINVAL;
>>> + if (!device->name)
>>> + return -EINVAL;
>>> +
>>> + ret = btrfs_get_bdev_and_sb(device->name->str, flags, holder, 1,
>>> + &bdev, &bh);
>>> + if (ret)
>>> + return ret;
>>> +
>>> + disk_super = (struct btrfs_super_block *)bh->b_data;
>>> + devid = btrfs_stack_device_id(&disk_super->dev_item);
>>> + if (devid != device->devid)
>>> + goto error_brelse;
>>> +
>>> + if (memcmp(device->uuid, disk_super->dev_item.uuid,
>>> + BTRFS_UUID_SIZE))
>>> + goto error_brelse;
>>> +
>>> + device->generation = btrfs_super_generation(disk_super);
>>> +
>>> + if (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_SEEDING) {
>>> + device->writeable = 0;
>>> + fs_devices->seeding = 1;
>>> + } else {
>>> + device->writeable = !bdev_read_only(bdev);
>>> + }
>>> +
>>> + q = bdev_get_queue(bdev);
>>> + if (blk_queue_discard(q))
>>> + device->can_discard = 1;
>>> + if (!blk_queue_nonrot(q))
>>> + fs_devices->rotating = 1;
>>> +
>>> + device->bdev = bdev;
>>> + device->in_fs_metadata = 0;
>>> + device->mode = flags;
>>> +
>>> + fs_devices->open_devices++;
>>> + if (device->writeable &&
>>> + device->devid != BTRFS_DEV_REPLACE_DEVID) {
>>> + fs_devices->rw_devices++;
>>> + list_add(&device->dev_alloc_list,
>>> +&fs_devices->alloc_list);
>>> + }
>>> + brelse(bh);
>>> +
>>> + return 0;
>>> +
>>> +error_brelse:
>>> + brelse(bh);
>>> + blkdev_put(bdev, flags);
>>> +
>>> + return -EINVAL;
>>> +}
>>> +
>>> /*
>>>* Add new device to list of registered devices
>>>*
>>> @@ -978,69 +1045,14 @@ static int __btrfs_open_devices(struct
>>> btrfs_fs_devices *fs_devices,
>>> flags |= FMODE_EXCL;
>>>
>>> list_for_each_entry(device, head, de
Re: [PATCH 4/4] btrfs: factor __btrfs_open_devices() to create btrfs_open_one_device()
Hi Anand,
With this patch applied, btrfs-progs/misc-test/021 error out. Is this
same for you?
Without this patch: https://asciinema.org/a/RJmE5469mHlL3S1BIOCifWVn6
With this patch:https://asciinema.org/a/1h5UX6DIFNsvvMXgLo4GiEgdE
thanks!
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
On Thu, Nov 9, 2017 at 9:15 PM, Anand Jain wrote:
> No functional changes, create btrfs_open_one_device() from
> __btrfs_open_devices(). This is a preparatory work to add dynamic
> device scan.
>
> Signed-off-by: Anand Jain
> ---
> fs/btrfs/volumes.c | 126
> +
> 1 file changed, 69 insertions(+), 57 deletions(-)
>
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 0857b580014d..d24e966ee29f 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -601,6 +601,73 @@ void btrfs_free_stale_device(struct btrfs_device
> *cur_dev)
> }
> }
>
> +static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> + struct btrfs_device *device, fmode_t flags,
> + void *holder)
> +{
> + struct request_queue *q;
> + struct block_device *bdev;
> + struct buffer_head *bh;
> + struct btrfs_super_block *disk_super;
> + u64 devid;
> + int ret;
> +
> + if (device->bdev)
> + return -EINVAL;
> + if (!device->name)
> + return -EINVAL;
> +
> + ret = btrfs_get_bdev_and_sb(device->name->str, flags, holder, 1,
> + &bdev, &bh);
> + if (ret)
> + return ret;
> +
> + disk_super = (struct btrfs_super_block *)bh->b_data;
> + devid = btrfs_stack_device_id(&disk_super->dev_item);
> + if (devid != device->devid)
> + goto error_brelse;
> +
> + if (memcmp(device->uuid, disk_super->dev_item.uuid,
> + BTRFS_UUID_SIZE))
> + goto error_brelse;
> +
> + device->generation = btrfs_super_generation(disk_super);
> +
> + if (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_SEEDING) {
> + device->writeable = 0;
> + fs_devices->seeding = 1;
> + } else {
> + device->writeable = !bdev_read_only(bdev);
> + }
> +
> + q = bdev_get_queue(bdev);
> + if (blk_queue_discard(q))
> + device->can_discard = 1;
> + if (!blk_queue_nonrot(q))
> + fs_devices->rotating = 1;
> +
> + device->bdev = bdev;
> + device->in_fs_metadata = 0;
> + device->mode = flags;
> +
> + fs_devices->open_devices++;
> + if (device->writeable &&
> + device->devid != BTRFS_DEV_REPLACE_DEVID) {
> + fs_devices->rw_devices++;
> + list_add(&device->dev_alloc_list,
> +&fs_devices->alloc_list);
> + }
> + brelse(bh);
> +
> + return 0;
> +
> +error_brelse:
> + brelse(bh);
> + blkdev_put(bdev, flags);
> +
> + return -EINVAL;
> +}
> +
> /*
> * Add new device to list of registered devices
> *
> @@ -978,69 +1045,14 @@ static int __btrfs_open_devices(struct
> btrfs_fs_devices *fs_devices,
> flags |= FMODE_EXCL;
>
> list_for_each_entry(device, head, dev_list) {
> - struct request_queue *q;
> - struct block_device *bdev;
> - struct buffer_head *bh;
> - struct btrfs_super_block *disk_super;
> - u64 devid;
> -
> - if (device->bdev)
> - continue;
> - if (!device->name)
> - continue;
> -
> /* Just open everything we can; ignore failures here */
> - if (btrfs_get_bdev_and_sb(device->name->str, flags, holder, 1,
> - &bdev, &bh))
> + ret = btrfs_open_one_device(fs_devices, device, flags,
> holder);
> + if (ret)
> continue;
>
> - disk_super = (struct btrfs_super_block *)bh->b_data;
> - devid = btrfs_stack_device_id(&disk_super->dev_item);
> - if (devid != device->devid)
> - goto error_brelse;
> -
> - if (memcmp(device->uuid, disk_super->dev_item.uuid,
> - BTRFS_UUID_SIZE))
> - goto error_brelse;
> -
> -
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
Hi Qu, I checked with David, Seems like his github/repo.or.cz repo has latest dev patches before pushing them into kernel.org repo. -- With the help of 'git bisect', managed to debug this issue further. With this patch[1] applied we are running into this issue. https://asciinema.org/a/OMLXpd69iHaej0Xfsrr2ihKiP [1] btrfs: factor __btrfs_open_devices() to create btrfs_open_one_device() https://github.com/kdave/btrfs-devel/commit/25908a59bb5c54f896c646ed3a1f73b2727ad9b5 ---- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Fri, Nov 24, 2017 at 5:35 PM, Lakshmipathi.G wrote: >>>> >>>> To make it clear, what kernel config and commit are you using when the >>>> test fails? >>>> >>> When I use attached kernel config along with tar from misc-next branch >>> (wget https://github.com/kdave/btrfs-devel/archive/misc-next.zip) this >>> issue happens. >> >> BTW, David seems to update his branch more often in his git.kernel.org >> branch than github. >> >> So it's possible that the branch from github is out dated. > > Okay, I didn't know that one, let me checkout his git.kernel.org branch and > check the results there. > >> >> For v4.14-rc7, even I enabled all the btrfs related configs, I still >> can't reproduce it. >> -- >> $ zcat /proc/config.gz | grep BTRFS >> CONFIG_BTRFS_FS=m >> CONFIG_BTRFS_FS_POSIX_ACL=y >> CONFIG_BTRFS_FS_CHECK_INTEGRITY=y >> CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y >> CONFIG_BTRFS_DEBUG=y >> CONFIG_BTRFS_ASSERT=y >> -- >> >> So there is something strange happened. >> > > Yes. something strange going on. I use the same kernel config > for misc-next, 4.14 and 4.14.2 - issue happens only with > misc-next, as you mentioned it may not be up-to-date. > > > > Cheers, > Lakshmipathi.G > http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
>>> >>> To make it clear, what kernel config and commit are you using when the >>> test fails? >>> >> When I use attached kernel config along with tar from misc-next branch >> (wget https://github.com/kdave/btrfs-devel/archive/misc-next.zip) this >> issue happens. > > BTW, David seems to update his branch more often in his git.kernel.org > branch than github. > > So it's possible that the branch from github is out dated. Okay, I didn't know that one, let me checkout his git.kernel.org branch and check the results there. > > For v4.14-rc7, even I enabled all the btrfs related configs, I still > can't reproduce it. > -- > $ zcat /proc/config.gz | grep BTRFS > CONFIG_BTRFS_FS=m > CONFIG_BTRFS_FS_POSIX_ACL=y > CONFIG_BTRFS_FS_CHECK_INTEGRITY=y > CONFIG_BTRFS_FS_RUN_SANITY_TESTS=y > CONFIG_BTRFS_DEBUG=y > CONFIG_BTRFS_ASSERT=y > -- > > So there is something strange happened. > Yes. something strange going on. I use the same kernel config for misc-next, 4.14 and 4.14.2 - issue happens only with misc-next, as you mentioned it may not be up-to-date. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
> > No, it's not about btrfs-image itself, the flag is introduced long long > long way ago. > Okay. > It's about kernel change which doesn't allow image with such flag to be > mounted AFAIK. No luck yet! It works on 4.14.0 - -bash-4.4$ sudo make TEST=021\* test-misc [TEST] misc-tests.sh [TEST/misc] 021-image-multi-devices -bash-4.4$ sudo dmesg [ 162.635824] BTRFS: device fsid f0c92f1b-c1ba-4576-8a07-8ce4f7b5bacd devid 1 transid 5 /dev/loop0 [ 162.672952] BTRFS: device fsid f0c92f1b-c1ba-4576-8a07-8ce4f7b5bacd devid 2 transid 5 /dev/loop1 [ 162.692848] BTRFS info (device loop1): disk space caching is enabled [ 162.695725] BTRFS info (device loop1): has skinny extents [ 162.698003] BTRFS info (device loop1): flagging fs with big metadata feature [ 162.701643] BTRFS info (device loop1): creating UUID tree [ 163.792851] BTRFS warning (device loop0): unrecognized super flag: 17179869184 [ 163.796350] BTRFS info (device loop0): disk space caching is enabled [ 163.807644] BTRFS info (device loop0): has skinny extents [ 163.811613] BTRFS info (device loop0): checking UUID tree [ 163.841625] BTRFS error (device loop0): 1 errors while writing supers [ 163.845457] BTRFS: error (device loop0) in write_all_supers:3709: errno=-5 IO failure (1 errors while writing supers) [ 163.851520] BTRFS info (device loop0): forced readonly [ 163.855184] BTRFS warning (device loop0): Skipping commit of aborted transaction. [ 163.859121] BTRFS: error (device loop0) in cleanup_transaction:1873: errno=-5 IO failure [ 163.862577] BTRFS info (device loop0): delayed_refs has NO entry [ 163.865623] BTRFS error (device loop0): commit super ret -5 [ 163.868303] BTRFS error (device loop0): cleaner transaction attach returned -30 -bash-4.4$ uname -a Linux ip-172-31-33-196.us-west-2.compute.internal 4.14.0 #1 SMP Fri Nov 24 09:48:11 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux - Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
Okay thanks for the details on btrfs-image. I was about to try with previous version of btrfs-image. I'll use v4.14 (https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.tar.xz) and check again. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Fri, Nov 24, 2017 at 1:36 PM, Qu Wenruo wrote: > > > On 2017年11月24日 15:47, Lakshmipathi.G wrote: >> Hi Qu, >> >> I'm using btrfs-progs devel branch. mount command is failing with misc/021. >> When I tried to manually mount it. dmesg shows "unrecognized super >> flag: 17179869184" >> and interestingly "df -h" shows 16E as available! > > That flag is meta dump flag, any image recovered from btrfs-image should > have that flag. > > 16E may also related to that, but not a big problem as btrfs-image > result should only be used by developers, and all its data is wiped out. > (Maybe one day we will not allow RW mount for such image) > > > The problem happens when you try to mount the image, there may be some > new commits doing more restrict check for rw mount in devel branch. > > Would you please try v4.14 kernel? > > Thanks, > Qu > >> >> https://asciinema.org/a/4JGYg7YGQz1PHtdBRJ6AVw0fh >> >> Let me check further. >> >> Cheers, >> Lakshmipathi.G >> http://www.giis.co.in http://www.webminal.org >> >> >> On Fri, Nov 24, 2017 at 9:55 AM, Lakshmipathi.G >> wrote: >>> Hi Qu, >>> >>> I'm using the same instance. Let me check again with and without the >>> commits list on Nov-23 >>> (https://github.com/kdave/btrfs-devel/commits/misc-next) and get back >>> with results. >>> >>> >>> Cheers, >>> Lakshmipathi.G >>> http://www.giis.co.in http://www.webminal.org >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
Hi Qu, I'm using btrfs-progs devel branch. mount command is failing with misc/021. When I tried to manually mount it. dmesg shows "unrecognized super flag: 17179869184" and interestingly "df -h" shows 16E as available! https://asciinema.org/a/4JGYg7YGQz1PHtdBRJ6AVw0fh Let me check further. ---- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Fri, Nov 24, 2017 at 9:55 AM, Lakshmipathi.G wrote: > Hi Qu, > > I'm using the same instance. Let me check again with and without the > commits list on Nov-23 > (https://github.com/kdave/btrfs-devel/commits/misc-next) and get back > with results. > > > Cheers, > Lakshmipathi.G > http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
Hi Qu, I'm using the same instance. Let me check again with and without the commits list on Nov-23 (https://github.com/kdave/btrfs-devel/commits/misc-next) and get back with results. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
Quick update: It continues to fail: https://asciinema.org/a/8HAL3uZ4MG7eb5gXnlgq3jm7q Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Sat, Nov 18, 2017 at 12:59 PM, Lakshmipathi.G wrote: > Hi. > > Few days back(Nov14th) this(misc/021-image-multi-devices) test script > used to pass and seems like its failing > now. may be due to recent commits? Logs,screen-casts available here: > https://lakshmipathi.github.io/btrfsqa/ > > > Cheers, > Lakshmipathi.G > http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: quotas: failure on removing a file via SFTP/SSH
Seems like issue is similar to this bug: https://bugzilla.kernel.org/show_bug.cgi?id=197681 Whats your kernel/brtfs-prog version? Can you upgrade them to latest if they are quite old? Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Tue, Nov 21, 2017 at 5:38 PM, ST wrote: > Hello, > > I'm trying to use quotas for a simple chrooted sftp setup, limiting > space for each user's subvolume (now for testing to 1M). > > I tried to hit the limit by uploading files and once it comes to the > limit I face following problem: if I try to free space by removing a > file via Linux sftp client (or Filezilla) - I get error: > "Couldn't delete file: Failure" > > Sometimes, but not always, if I repeat it for 3-5 times it does removes > the file at the end. > If I login as root and try to remove the file via SSH I get the error: > "rm: cannot remove 'example.txt': Disk quota exceeded" > > What is the problem? And how can I solve it? > > Thank you in advance! > > > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
misc/021-image-multi-devices fails on latest btrfs-devel/misc-next
Hi. Few days back(Nov14th) this(misc/021-image-multi-devices) test script used to pass and seems like its failing now. may be due to recent commits? Logs,screen-casts available here: https://lakshmipathi.github.io/btrfsqa/ Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs test bed
Hi, I'm trying to setup a public test bed for btrfs. Idea is to quickly test user scripts and check its valid or not for specific kernel/btrfs-progs versions. Also keep running them daily and record/save the logs. Its ran for few days and results can be view on below link[2]. If you have suggestion/idea/scripts to improve this setup, let me know. thanks! [1] Setup: https://github.com/Lakshmipathi/btrfsqa [2] Results: https://lakshmipathi.github.io/btrfsqa/ Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] btrfs-progs: Replace some BUG_ON by return
Hi Gu,
I applied these two patches:
https://patchwork.kernel.org/patch/9939879/
https://patchwork.kernel.org/patch/9939881/
Still the test "sudo make TEST=003\* test-fuzz" is failing like:
btrfs-progs/btrfs check --check-data-csum
btrfs-progs/tests/fuzz-tests/images/bko-156731.raw.restored
checking extents
cmds-check.c:5660: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1
btrfs-progs/btrfs[0x44dce4]
btrfs-progs/btrfs[0x45b396]
btrfs-progs/btrfs[0x45c2f0]
btrfs-progs/btrfs[0x45ce92]
btrfs-progs/btrfs(cmd_check+0x17ee)[0x460ee7]
btrfs-progs/btrfs(main+0x15d)[0x40b523]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7faf98a4d830]
btrfs-progs/btrfs(_start+0x29)[0x40aff9]
Checking filesystem on
btrfs-progs/tests/fuzz-tests/images/bko-156731.raw.restored
UUID: 52975385-d05a-482b-8fb8-613a86a66042
failed (ignored, ret=134): btrfs-progs/btrfs check --check-data-csum
btrfs-progs/tests/fuzz-tests/images/bko-156731.raw.restored
mayfail: returned code 134 (SIGABRT), not ignored
test failed for case 003-multi-check-unmounted
Did I miss some other patch? thanks.
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
On Wed, Sep 6, 2017 at 7:12 AM, Gu Jinxiang wrote:
> The following test failed becasuse there is no data/metadata/system
> block_group.
> Use return ret to replace BUG_ON(ret) to avoid system crash, because there is
> enough message for user to understand what happened.
>
> $sudo TEST=003\* make test-fuzz
> Unable to find block group for 0
> Unable to find block group for 0
> Unable to find block group for 0
> extent-tree.c:2693: btrfs_reserve_extent: BUG_ON `ret` triggered, value -28
> /home/fnst/btrfs/btrfs-progs/btrfs[0x419966]
> /home/fnst/btrfs/btrfs-progs/btrfs(btrfs_reserve_extent+0xb16)[0x41f500]
> /home/fnst/btrfs/btrfs-progs/btrfs(btrfs_alloc_free_block+0x55)[0x41f59b]
> /home/fnst/btrfs/btrfs-progs/btrfs[0x46a6ce]
> /home/fnst/btrfs/btrfs-progs/btrfs(cmd_check+0x1012)[0x47c885]
> /home/fnst/btrfs/btrfs-progs/btrfs(main+0x127)[0x40b055]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x2aae83e89f45]
> /home/fnst/btrfs/btrfs-progs/btrfs[0x40a939]
> Creating a new CRC tree
> Checking filesystem on
> /home/fnst/btrfs/btrfs-progs/tests/fuzz-tests/images/bko-155621-bad-block-group-offset.raw.restored
> UUID: 5cb33553-6f6d-4ce8-83fd-20af5a2f8181
> Reinitialize checksum tree
> failed (ignored, ret=134): /home/fnst/btrfs/btrfs-progs/btrfs check
> --init-csum-tree
> /home/fnst/btrfs/btrfs-progs/tests/fuzz-tests/images/bko-155621-bad-block-group-offset.raw.restored
> mayfail: returned code 134 (SIGABRT), not ignored
> test failed for case 003-multi-check-unmounted
>
> Signed-off-by: Gu Jinxiang
> ---
> extent-tree.c | 16 ++--
> transaction.c | 8 ++--
> 2 files changed, 16 insertions(+), 8 deletions(-)
>
> diff --git a/extent-tree.c b/extent-tree.c
> index eed56886..14838a5d 100644
> --- a/extent-tree.c
> +++ b/extent-tree.c
> @@ -2678,11 +2678,13 @@ int btrfs_reserve_extent(struct btrfs_trans_handle
> *trans,
> ret = do_chunk_alloc(trans, info,
> num_bytes,
> BTRFS_BLOCK_GROUP_METADATA);
> - BUG_ON(ret);
> + if (ret)
> + goto out;
> }
> ret = do_chunk_alloc(trans, info,
> num_bytes + SZ_2M, data);
> - BUG_ON(ret);
> + if (ret)
> + goto out;
> }
>
> WARN_ON(num_bytes < info->sectorsize);
> @@ -2690,9 +2692,11 @@ int btrfs_reserve_extent(struct btrfs_trans_handle
> *trans,
>search_start, search_end, hint_byte, ins,
>trans->alloc_exclude_start,
>trans->alloc_exclude_nr, data);
> - BUG_ON(ret);
> + if (ret)
> + goto out;
> clear_extent_dirty(&info->free_space_cache,
>ins->objectid, ins->objectid + ins->offset - 1);
> +out:
> return ret;
> }
>
> @@ -2761,7 +2765,8 @@ static int alloc_tree_block(struct btrfs_trans_handle
> *trans,
> int ret;
> ret = btrfs_reserve_extent(trans, root, num_bytes, empty_size,
>hint_byte, search_end, ins, 0);
> - BUG_ON(ret);
> + if (ret)
> + goto out;
>
> if (root_objectid == BTRFS_EXTENT_TREE_OBJECTID) {
> struct pending_extent_op *extent_op;
> @@ -2792,6 +2797,7 @@ static int alloc_tree_block(struct btrfs_trans_handl
Re: kernel BUG at fs/btrfs/ctree.h:3457
> Please skip that patch, as the check timing has its problem. > Okay, I'll skip the patch and run the tests hereafter. thanks. ---- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG at fs/btrfs/ctree.h:3457
Seems like output is messed by my email client: Here is formatted one: https://bugzilla.kernel.org/show_bug.cgi?id=197587#c6 Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Mon, Nov 6, 2017 at 10:47 AM, Lakshmipathi.G wrote: > Hi. > > While running xfstests/065 with 4.14.0-rc7 and latest btrfs-progs > from devel branch, it crashes the system after dumping below log: > (Note: For btrfs-devel, I'm still using SANITY_TESTS=y and > FS_INTEGRITY=y and applied patch > https://patchwork.kernel.org/patch/10038011/) > > > Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): disk space caching is enabled > Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): has skinny extents > Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): flagging fs with big metadata feature > Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): enabling ssd optimizations > Nov 06 05:08:09 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): creating UUID tree > Nov 06 05:08:10 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): dev_replace from /dev/xvdd (devid 2) to > /dev/xvdg started > Nov 06 05:08:12 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): dev_replace from /dev/xvdd (devid 2) to > /dev/xvdg finished > Nov 06 05:08:16 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 1 > transid 5 /dev/xvdc > Nov 06 05:08:16 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 2 > transid 5 /dev/xvdd > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 3 > transid 5 /dev/xvde > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS: device fsid 3ebc32b5-ba0a-405b-8acf-4785b0a4b81a devid 4 > transid 5 /dev/xvdf > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): disk space caching is enabled > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): has skinny extents > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): flagging fs with big metadata feature > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): enabling ssd optimizations > Nov 06 05:08:17 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): creating UUID tree > Nov 06 05:08:18 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): dev_replace from /dev/xvdd (devid 2) to > /dev/xvdg started > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS critical (device xvdf): corrupt leaf: root=9 block=4400742400 > slot=0, invalid nritems, have 0 should not be 0 for non-root leaf > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > BTRFS info (device xvdf): leaf 4400742400 total ptrs 0 free space > 16283 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > assertion failed: 0, file: fs/btrfs/disk-io.c, line: 540 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > [ cut here ] > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > kernel BUG at fs/btrfs/ctree.h:3457! > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > invalid opcode: [#1] SMP > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > Modules linked in: cirrus drm_kms_helper ttm drm intel_rapl sb_edac > ppdev crct10dif_pclmul crc32_pclmul crc32c_intel xen_netfront > ghash_clmulni_intel intel_rapl_perf parport_pc parport i2c_piix4 > xen_blkfront serio_raw ata_generic pata_acpi > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > CPU: 1 PID: 13413 Comm: fsstress Not tainted 4.14.0-rc7 #1 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/12/2016 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > task: 9d2331a48000 task.stack: bc0f43fd4000 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > RIP: 0010:assfail.constprop.49+0x1c/0x22 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > RSP: 0018:bc0f43fd79c0 EFLAGS: 00010282 > Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: > RAX: 0038 RBX:
kernel BUG at fs/btrfs/ctree.h:3457
ta_acpi Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: CPU: 1 PID: 13413 Comm: fsstress Tainted: G D 4.14.0-rc7 #1 Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/12/2016 Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: task: 9d2331a48000 task.stack: bc0f43fd4000 Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: RIP: 0010:do_exit+0x51/0xb40 Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: RSP: 0018:bc0f43fd7ed8 EFLAGS: 00010206 Nov 06 05:08:23 ip-172-31-32-145.us-west-2.compute.internal kernel: RAX: bc0f43fd7d40 RBX: 9d2331a48000 RCX: bc0f43fd7d50 ---- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs-progs:fuzz-tests/003-multi-check-unmounted broken?
Hi. Seems like fuzz-test/003 is broken for quite sometime now (it works on v4.7.3 but not after that). Can anyone else able to run this test successfully? If this test is not required anymore, shall we remove it from fuzz-tests? TEST=003\* tests/fuzz-tests.sh [TEST/fuzz] 003-multi-check-unmounted btrfs-progs/tests/common: line 170: 1439 Aborted (core dumped) $INSTRUMENT "$@" >> "$RESULTS" 2>&1 mayfail: returned code 134 (SIGABRT), not ignored test failed for case 003-multi-check-unmounted Tried applying these two patches https://patchwork.kernel.org/patch/9939881/ still script doesn't succeed. thanks. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: xfstests:btrfs/025, 026 fails
Okay I narrowed it down to /bin/mount -t btrfs -o context=system_u:object_r:root_t:s0 -o compress /dev/xvdc /mnt/scratch command from xfstests/btrfs/026. Cheers. On 11/4/17, Lakshmipathi.G wrote: > Hi . > > While running xfstests with 4.14.0-rc7 the test hangs on btrfs/026 > (after running 025) and produces below trace: > > [ 2134.147940] run fstests btrfs/026 at 2017-11-04 09:46:19 > [ 2134.388024] BTRFS: device fsid d3de0b02-8b7e-422a-8165-c074f834e909 > devid 1 transid 5 /dev/xvdc > [ 2134.405001] general protection fault: [#1] SMP > [ 2134.408660] Modules linked in: cirrus drm_kms_helper ttm drm ppdev > intel_rapl sb_edac parport_pc crct10dif_pclmul parport crc32_pclmul > crc32c_intel ghash_clmulni_intel xen_netfront intel_rapl_perf > i2c_piix4 xen_blkfront serio_raw ata_generic pata_acpi > [ 2134.420112] CPU: 1 PID: 8478 Comm: mount Not tainted 4.14.0-rc7 #1 > [ 2134.423711] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 > [ 2134.427557] task: 8ec43c672600 task.stack: a90fc886c000 > [ 2134.431351] RIP: 0010:btrfs_compress_str2level+0x18/0x4e > [ 2134.434764] RSP: 0018:a90fc886fa10 EFLAGS: 00010202 > [ 2134.438128] RAX: 0001 RBX: 8ec434c8 RCX: > 0004 > [ 2134.442160] RDX: 5485389ff3298000 RSI: 5485389ff3298000 RDI: > 94cdb929 > [ 2134.446266] RBP: a90fc886fac0 R08: R09: > 5485389ff3298000 > [ 2134.450252] R10: 002c R11: 94a917c0 R12: > 94d1ec8a > [ 2134.454875] R13: 8ec434d1c720 R14: R15: > > [ 2134.459141] FS: 7f9e870c8480() GS:8ec43fe4() > knlGS: > [ 2134.464051] CS: 0010 DS: ES: CR0: 80050033 > [ 2134.467519] CR2: 7f5ae9499710 CR3: 0004343e4002 CR4: > 001606e0 > [ 2134.471549] DR0: DR1: DR2: > > [ 2134.475601] DR3: DR6: fffe0ff0 DR7: > 0400 > [ 2134.480049] Call Trace: > [ 2134.482285] ? btrfs_parse_options+0x9fd/0x1020 > [ 2134.485502] open_ctree+0x1244/0x24df > [ 2134.488245] ? open_ctree+0x1244/0x24df > [ 2134.490954] btrfs_mount+0xe16/0xefc > [ 2134.493481] ? find_next_bit+0xb/0x10 > [ 2134.496095] ? cpumask_next+0x1b/0x20 > [ 2134.498643] mount_fs+0x32/0x150 > [ 2134.501001] ? __alloc_percpu+0x15/0x20 > [ 2134.504016] vfs_kern_mount.part.22+0x5d/0x120 > [ 2134.507094] vfs_kern_mount+0x13/0x20 > [ 2134.509737] btrfs_mount+0x1a8/0xefc > [ 2134.512338] ? find_next_bit+0xb/0x10 > [ 2134.514963] mount_fs+0x32/0x150 > [ 2134.517289] ? __alloc_percpu+0x15/0x20 > [ 2134.519917] vfs_kern_mount.part.22+0x5d/0x120 > [ 2134.522754] do_mount+0x5c7/0xca0 > [ 2134.525101] ? kmem_cache_alloc_trace+0x165/0x1d0 > [ 2134.528554] ? copy_mount_options+0x2c/0x220 > [ 2134.531459] SyS_mount+0x98/0xe0 > [ 2134.533880] entry_SYSCALL_64_fastpath+0x1a/0xa5 > [ 2134.536942] RIP: 0033:0x7f9e8611c28a > [ 2134.539496] RSP: 002b:7fff6cddbf68 EFLAGS: 0246 ORIG_RAX: > 00a5 > [ 2134.543632] RAX: ffda RBX: 7f9e86c9c7ef RCX: > 7f9e8611c28a > [ 2134.547646] RDX: 5562a4af82f0 RSI: 5562a4af83b0 RDI: > 5562a4af8390 > [ 2134.552254] RBP: 7f9e86eae184 R08: 5562a4af8310 R09: > 7fff6cddae68 > [ 2134.556487] R10: c0ed R11: 0246 R12: > 5562a4af8140 > [ 2134.560647] R13: 7fff6cddc288 R14: 5562a359f4a0 R15: > > [ 2134.564622] Code: ad 31 db eb a9 0f 1f 40 00 66 2e 0f 1f 84 00 00 > 00 00 00 0f 1f 44 00 00 55 48 89 fa b9 04 00 00 00 48 89 d6 48 c7 c7 > 29 b9 cd 94 a6 48 89 e5 40 0f 97 c6 0f 92 c1 31 c0 40 38 ce 75 06 > 80 7a > [ 2134.574756] RIP: btrfs_compress_str2level+0x18/0x4e RSP: > ffffa90fc886fa10 > [ 2134.578890] ---[ end trace 47a5851d0ed3c8fd ]--- > > > https://bugzilla.kernel.org/show_bug.cgi?id=197771 > > > Cheers, > Lakshmipathi.G > http://www.giis.co.in http://www.webminal.org > -- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
xfstests:btrfs/025, 026 fails
Hi . While running xfstests with 4.14.0-rc7 the test hangs on btrfs/026 (after running 025) and produces below trace: [ 2134.147940] run fstests btrfs/026 at 2017-11-04 09:46:19 [ 2134.388024] BTRFS: device fsid d3de0b02-8b7e-422a-8165-c074f834e909 devid 1 transid 5 /dev/xvdc [ 2134.405001] general protection fault: [#1] SMP [ 2134.408660] Modules linked in: cirrus drm_kms_helper ttm drm ppdev intel_rapl sb_edac parport_pc crct10dif_pclmul parport crc32_pclmul crc32c_intel ghash_clmulni_intel xen_netfront intel_rapl_perf i2c_piix4 xen_blkfront serio_raw ata_generic pata_acpi [ 2134.420112] CPU: 1 PID: 8478 Comm: mount Not tainted 4.14.0-rc7 #1 [ 2134.423711] Hardware name: Xen HVM domU, BIOS 4.2.amazon 08/24/2006 [ 2134.427557] task: 8ec43c672600 task.stack: a90fc886c000 [ 2134.431351] RIP: 0010:btrfs_compress_str2level+0x18/0x4e [ 2134.434764] RSP: 0018:a90fc886fa10 EFLAGS: 00010202 [ 2134.438128] RAX: 0001 RBX: 8ec434c8 RCX: 0004 [ 2134.442160] RDX: 5485389ff3298000 RSI: 5485389ff3298000 RDI: 94cdb929 [ 2134.446266] RBP: a90fc886fac0 R08: R09: 5485389ff3298000 [ 2134.450252] R10: 002c R11: 94a917c0 R12: 94d1ec8a [ 2134.454875] R13: 8ec434d1c720 R14: R15: [ 2134.459141] FS: 7f9e870c8480() GS:8ec43fe4() knlGS: [ 2134.464051] CS: 0010 DS: ES: CR0: 80050033 [ 2134.467519] CR2: 7f5ae9499710 CR3: 0004343e4002 CR4: 001606e0 [ 2134.471549] DR0: DR1: DR2: [ 2134.475601] DR3: DR6: fffe0ff0 DR7: 0400 [ 2134.480049] Call Trace: [ 2134.482285] ? btrfs_parse_options+0x9fd/0x1020 [ 2134.485502] open_ctree+0x1244/0x24df [ 2134.488245] ? open_ctree+0x1244/0x24df [ 2134.490954] btrfs_mount+0xe16/0xefc [ 2134.493481] ? find_next_bit+0xb/0x10 [ 2134.496095] ? cpumask_next+0x1b/0x20 [ 2134.498643] mount_fs+0x32/0x150 [ 2134.501001] ? __alloc_percpu+0x15/0x20 [ 2134.504016] vfs_kern_mount.part.22+0x5d/0x120 [ 2134.507094] vfs_kern_mount+0x13/0x20 [ 2134.509737] btrfs_mount+0x1a8/0xefc [ 2134.512338] ? find_next_bit+0xb/0x10 [ 2134.514963] mount_fs+0x32/0x150 [ 2134.517289] ? __alloc_percpu+0x15/0x20 [ 2134.519917] vfs_kern_mount.part.22+0x5d/0x120 [ 2134.522754] do_mount+0x5c7/0xca0 [ 2134.525101] ? kmem_cache_alloc_trace+0x165/0x1d0 [ 2134.528554] ? copy_mount_options+0x2c/0x220 [ 2134.531459] SyS_mount+0x98/0xe0 [ 2134.533880] entry_SYSCALL_64_fastpath+0x1a/0xa5 [ 2134.536942] RIP: 0033:0x7f9e8611c28a [ 2134.539496] RSP: 002b:7fff6cddbf68 EFLAGS: 0246 ORIG_RAX: 00a5 [ 2134.543632] RAX: ffda RBX: 7f9e86c9c7ef RCX: 7f9e8611c28a [ 2134.547646] RDX: 5562a4af82f0 RSI: 5562a4af83b0 RDI: 5562a4af8390 [ 2134.552254] RBP: 7f9e86eae184 R08: 5562a4af8310 R09: 7fff6cddae68 [ 2134.556487] R10: c0ed R11: 0246 R12: 5562a4af8140 [ 2134.560647] R13: 7fff6cddc288 R14: 5562a359f4a0 R15: [ 2134.564622] Code: ad 31 db eb a9 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 fa b9 04 00 00 00 48 89 d6 48 c7 c7 29 b9 cd 94 a6 48 89 e5 40 0f 97 c6 0f 92 c1 31 c0 40 38 ce 75 06 80 7a [ 2134.574756] RIP: btrfs_compress_str2level+0x18/0x4e RSP: a90fc886fa10 [ 2134.578890] ---[ end trace 47a5851d0ed3c8fd ]--- https://bugzilla.kernel.org/show_bug.cgi?id=197771 Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG at fs/btrfs/ctree.h:3457!
Yes, the patch works. Enabled both CONFIG_BTRFS_FS_CHECK_INTEGRITY and CONFIG_BTRFS_FS_RUN_SANITY_TESTS. And applied above patch. This method also resolved the issue. thanks. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG at fs/btrfs/ctree.h:3457!
> BTW, it would be better to see if this patch also solves your problem. > > https://patchwork.kernel.org/patch/10038011/ Okay sure, let me apply this patch and check the results. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG at fs/btrfs/ctree.h:3457!
After disabling FS_CHECK_INTEGRITY and FS_RUN_SANITY_TESTS in the config. Now the issue resolved. thanks. cast: https://asciinema.org/a/Dy6eHhhWPEIxotVbVUBWrhFeF Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Thu, Nov 2, 2017 at 12:55 PM, Lakshmipathi.G wrote: > Okay thanks, I'll disable above mentioned entry in kernel config and > run the tests. > Will get back with updated results. > > > > Cheers, > Lakshmipathi.G > http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG at fs/btrfs/ctree.h:3457!
Okay thanks, I'll disable above mentioned entry in kernel config and run the tests. Will get back with updated results. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: kernel BUG at fs/btrfs/ctree.h:3457!
> > I'll try to reproduce it with FS_CHECK_INTEGRITY enabled. > Okay thanks for the details. Here's the kernel config file which hits this issue: https://github.com/Lakshmipathi/btrfsqa/blob/master/setup/config/kernel.config#L3906 thanks! Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
kernel BUG at fs/btrfs/ctree.h:3457!
Hi. I'm constantly hitting this bug while running btrfs-progs:fsck-test:012 . Screencast: https://asciinema.org/a/wQxZjCeVvX2kVqKjVBGyR0klq Logged more details: https://bugzilla.kernel.org/show_bug.cgi?id=197587 Anyone else got this issue or something wrong with my test environment? Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building a BTRFS test machine
Last few days, made small progress on this. Though Python/bash script is working, its not yet perfect. Gladly welcome any feedback and/or scripts :-) https://github.com/Lakshmipathi/btrfsqa Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"
> I will gladly repeat this process, but I am very concerned why this > corruption happened in the first place. > Can you share these steps as simple bash script? Currently I'm running few tests, I can check your script and share the results. > The only thing I could think of is that the btrfs version that I used to mkfs > was not up to date. Is there a way to determine which version was used to > create the filesystem? > If I'm not wrong, i don't think we can get mkfs version from existing file system layout. what was your kernel version? Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH 1/1] btrfs-progs: tests/common: Display warning only after searching for btrfs kernel module
Signed-off-by: Lakshmipathi.G
---
tests/common | 8 ++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/tests/common b/tests/common
index eb525a4..773b1e5 100644
--- a/tests/common
+++ b/tests/common
@@ -428,8 +428,12 @@ run_check_umount_test_dev()
check_kernel_support()
{
if ! grep -iq 'btrfs' /proc/filesystems; then
- echo "WARNING: btrfs filesystem not listed in
/proc/filesystems, some tests might fail"
- return 1
+ run_check $SUDO_HELPER modprobe btrfs
+ if ! grep -iq 'btrfs' /proc/filesystems; then
+ echo "WARNING: btrfs filesystem not listed in \
+/proc/filesystems, some tests might fail"
+ return 1
+ fi
fi
return 0
}
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"
1. I guess you should be able to dump tree details via 'btrfs-debug-tree' and then map the extent/data (from scrub offline output) and track it back to inode-object. Store output of both btrfs-debug-tree and scrub-offline in different files and then play around with grep to extract required data. 2. I think normal scrub(online) fails to detect these csum errors for some reason,I don't have much idea about online scrub. 3. I assume, the issue is not related to hardware. Since the offline scrub able to get available (corrupted) csum. Yes, offline scrub will try to fix corruption whenever it is possible. And also you have quite lot of "all mirror(s) corrupted, can't be repaired", which will be hard to recovery. I suggest running offline scrub on all devices. Then online scrub and finally track those corrupted files with the help of extent info. Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Wed, Oct 25, 2017 at 7:22 AM, Zak Kohler wrote: > I apologize for the bad line wrapping on the last post...will be > setting up mutt soon. > > This is the final result for the offline scrub: > Doing offline scrub [O] [681/683] > Scrub result: > Tree bytes scrubbed: 5234491392 > Tree extents scrubbed: 638975 > Data bytes scrubbed: 4353723572224 > Data extents scrubbed: 374300 > Data bytes without csum: 533200896 > Read error: 0 > Verify error: 0 > Csum error: 175 > > The offline scrub apparently corrected some metadata extents while > scanning /dev/sdn > > > I also ran the online scrub directly on the /dev/sdn, "0 errors": > > $ btrfs scrub status /dev/sdn > scrub status for 88406942-e3e1-42c6-ad71-e23bb315caa7 > scrub started at Tue Oct 24 06:55:12 2017 and finished after 01:52:44 > total bytes scrubbed: 677.35GiB with 0 errors > > The csum mismatches are still missed by the online scrub when choosing > a single . Now I am doing offline scrub on the other devices > to see if they are clean. > > $ lsblk -o +SERIAL > NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT SERIAL > sdh 8:112 0 1.8T 0 disk WD-WMAZA370 > sdi 8:128 0 1.8T 0 disk WD-WCAZA569 > sdn 8:208 0 1.8T 0 disk WD-WCAZA580 > > $ btrfs scrub start --offline --progress /dev/sdh > ERROR: data at bytenr 5365456896 ... > ERROR: extent 5341712384 ... > ... > > One thing to note is that a /dev/sdh is also having csum errors > detected despite it having never been mentioned dmesg. I understand > that you may have the ability to run two offline checks at once but > the error message I get is slightly misleading. > > $ btrfs scrub start --offline --progress /dev/sdi > ERROR: cannot open device '/dev/sdn': Device or resource busy > ERROR: cannot open file system > > I get an error about sdn when the device I am trying to scan is sdi, > and the device that is currently being scanned is sdh. > > On Tue, Oct 24, 2017 at 2:00 AM, Zak Kohler wrote: >> Yes, it is finding much more than just one error. >> >> From dmesg >> [89520.441354] BTRFS warning (device sdn): csum failed ino 4708 off >> 27529216 csum 2615801759 expected csum 874979996 >> >> $ sudo btrfs scrub start --offline --progress /dev/sdn >> ERROR: data at bytenr 68431499264 mirror 1 csum mismatch, have >> 0x5aa0d40f expect 0xd4a15873 >> ERROR: extent 68431474688 len 14467072 CORRUPTED, all mirror(s) >> corrupted, can't be repaired >> ERROR: data at bytenr 83646357504 mirror 1 csum mismatch, have >> 0xfc0baabe expect 0x7f9cb681 >> ERROR: extent 83519741952 len 134217728 CORRUPTED, all mirror(s) >> corrupted, can't be repaired >> ERROR: data at bytenr 121936633856 mirror 1 csum mismatch, have >> 0x507016a5 expect 0x50609afe >> ERROR: extent 121858334720 len 134217728 CORRUPTED, all mirror(s) >> corrupted, can't be repaired >> ERROR: data at bytenr 144872591360 mirror 1 csum mismatch, have >> 0x33964d73 expect 0xf9937032 >> ERROR: extent 144822386688 len 61231104 CORRUPTED, all mirror(s) >> corrupted, can't be repaired >> ERROR: data at bytenr 167961075712 mirror 1 csum mismatch, have >> 0xf43bd0e3 expect 0x5be589bb >> ERROR: extent 167950999552 len 27537408 CORRUPTED, all mirror(s) >> corrupted, can't be repaired >> ERROR: data at bytenr 175643619328 mirror 1 csum mismatch, have >> 0x1e168ca1 expect 0xd413b1e0 >> ERROR: data at bytenr 175643754496 mirror 1 csum mismatch, have >> 0x6cfdc8ae expect 0xa6f8f5ef >> ERROR: extent 175640539136 len 6381568 CORRUPTED, all mirror(s) >> corrupted, can't be repaired >> ERROR: data at bytenr 183316750336 mirror 1
Re: btrfs send yields "ERROR: send ioctl failed with -5: Input/output error"
> Does anyone know why scrub did not catch these errors that show up in dmesg? Can you try offline scrub from this repo https://github.com/gujx2017/btrfs-progs/tree/offline_scrub and see whether it detects the issue? "btrfs scrub start --offline " Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3] btrfs-progs: RAID5: Inject data stripe corruption and verify scrub fixes it and retains correct parity.
Any review comments on this RAID5 test script? I understand it depends
on 'dump-tree' output, but I assume, right now, we have no other
choice other than this method.
I'll modify this patch to reflect coding practices as mentioned in
tests/README.md, if current method is okay.
Cheers.
Lakshmipathi.G
On Mon, Feb 20, 2017 at 11:57 PM, Lakshmipathi.G
wrote:
> Test script to create file with specific data-stripe layout.Computes stripe
> locations.
> Inject corruption into data-stripe and verify scrubbing process fixes
> corrupted block.
> It also validates the corresponding parity stripe.
>
> Signed-off-by: Lakshmipathi.G
> ---
> .../test.sh| 402
> +
> 1 file changed, 402 insertions(+)
> create mode 100755
> tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
>
> diff --git
> a/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
> b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
> new file mode 100755
> index 000..0129a75
> --- /dev/null
> +++ b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
> @@ -0,0 +1,402 @@
> +#!/bin/bash
> +#
> +# Raid5: Inject data stripe corruption and fix them using scrub.
> +#
> +# Script will perform the following:
> +# 1) Create Raid5 using 3 loopback devices.
> +# 2) Ensure file layout is created in a predictable manner.
> +#Each data stripe(64KB) should uniquely start with 'DN',
> +#where N represents the data stripe number.(ex:D0,D1 etc)
> +# 3) Once file is created with specific layout, check whether file
> +#has single extent. At the moment, script wont handle multi-extent
> +#files.
> +# 4) If file has single extent with the help of 'dump-tree' compute data and
> +#parity stripe details like devicename, position and actual on-disk data.
> +# 5) $STRIPEINFO_COMPLETE file will have all necessary data at this stage.
> +# 6) Inject corruption into given data-stripe by zero'ing out its first 4
> bytes.
> +# 7) After injecting corruption, running online-scrub is expected to fix
> +#the corrupted data stripe with the help of parity block and
> +#corresponding data stripe.
> +# 8) If scrub successful, verify the data stripe has original un-corrupted
> value.
> +# 9) If scrub successful, verify parity stripe is valid, otherwise its a
> parity bug.
> +# 10) If no issues found, cleanup files and devices. Repeat the process for
> +#different file size and data-stripe.
> +#
> +# Note: This script corrupts only data-stripe blocks.
> +# Known Limitations (will be addressed later):
> +# - Script expects even number of data-stripes in file.
> +# - Script expects the file to have single extent.
> +
> +source $TOP/tests/common
> +
> +check_prereq btrfs
> +check_prereq mkfs.btrfs
> +check_prereq btrfs-debugfs
> +
> +setup_root_helper
> +prepare_test_dev 1024M
> +
> +ndevs=3
> +declare -a devs
> +declare -a parity_offsets
> +stripe_entry=""
> +device_name=""
> +stripe_offset=""
> +stripe_content=""
> +
> +#Complete stripe layout
> +STRIPEINFO_COMPLETE=$(mktemp --tmpdir
> btrfs-progs-raid5-stripe-complete.infoXX)
> +#dump-tree output file
> +DUMPTREE_OUTPUT=$(mktemp --tmpdir btrfs-progs-raid5-tree-dump.infoXX)
> +#tmp files
> +STRIPEINFO_PARTIAL=$(mktemp --tmpdir
> btrfs-progs-raid5-stripe-partial.infoXX)
> +STRIPE_TMP=$(mktemp --tmpdir btrfs-progs-raid5-stripetmp.infoXX)
> +MULTI_EXTENT_CHECK=$(mktemp --tmpdir
> btrfs-progs-raid5-extent-check.infoXX)
> +EXTENT_WITH_SIZE=$(mktemp --tmpdir btrfs-progs-raid5-extent-size.infoXX)
> +PARITY_LOC1=$(mktemp --tmpdir btrfs-progs-raid5-parity-loc1.infoXX)
> +PARITY_LOCATION=$(mktemp --tmpdir
> btrfs-progs-raid5-parity-locations.infoXX)
> +
> +
> +prepare_devices()
> +{
> + for i in `seq $ndevs`; do
> + touch img$i
> + chmod a+rw img$i
> + truncate -s0 img$i
> + truncate -s512M img$i
> + devs[$i]=`run_check_stdout $SUDO_HELPER losetup --find --show
> img$i`
> + done
> + truncate -s0 $STRIPE_TMP
> + truncate -s0 $STRIPEINFO_PARTIAL
> + truncate -s0 $STRIPEINFO_COMPLETE
> +}
> +
> +cleanup_devices()
> +{
> + for dev in ${devs[@]}; do
> + run_check $SUDO_HELPER losetup -d $dev
> + done
> + for i in `seq $ndevs`; do
> + truncate -s0 img$i
> + done
> + run_check $SUDO_HELPER losetup --all
> +}
> +
> +test_do_mkfs()
> +{
> +
Re: btrfs-progs task tracking on github
Existing project list is good. Reg: "Project tests: Build and test coverage: pre-built images on docker hub" If we don't have pre-built images on docker hub, I'll pick up this task and start working on it. thanks. Cheers, Lakshmipathi.G On Sat, Sep 9, 2017 at 5:43 AM, Qu Wenruo wrote: > > > On 2017年09月09日 01:36, David Sterba wrote: >> >> Hi, >> >> in order to make more visible which tasks for btrfs-progs are in >> progress or desired, I've started to populate the github Projects [1] >> some time ago. I haven't fleshed out the workflow so this hasn't been >> announced yet. >> >> The aim is to help contributors to join progs development and start from >> one point with the tasks to choose from. Current practice is to either >> ask on IRC or find something on the wiki, hoping that the projects are >> still up to date. Not all of them are, wiki is not the best tool for >> task tracking. >> >> Using github issues and projects is diversion from the current model >> similar to the kernel part of btrfs, ie. mailinglist for everything. >> This does suite all the needs but I don't want to completely switch to >> github-only workflow. Leaving both options means that bug/patchset >> discussions will not be visible to both audiences directly, but web >> archives or publicly accessible github pages should be enough for >> cross-references. >> >> The exact workflow from a task to a merge is not yet defined, I hope >> we'll be able to find out and document it once the first guinea pig >> volunteers. The Projects use the kanban-style, I made a bit familiar >> with that but have no prior experience with it. >> >> There are categories for tasks and some brief descriptions. The rough >> idea is to: >> >> 1. move a task from TODO to WIP >> 2. create issue from the task, for discussions etc >> 3. work on patches, review cycle >> 4. when finished, move to next stage on the project page, and ask for >> merge >> 5. task is done, code merged and released >> >> Who does what and how exactly will have to be found on the way and >> documented eventually. >> >> [1] https://github.com/kdave/btrfs-progs/projects > > > This looks quite good. > > BTW, is that only repo owner be able to modify the status? > In fact I found that some projects like offline scrub doesn't show up in > either WIP or TODO list. > > Thanks, > Qu > >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majord...@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building a BTRFS test machine
Definitely it will be useful. For quite sometime, I was thinking about: 1. Creating test cases for each BTRFS features. (ex: https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases) 2. Automate these feature specific test scripts using bash/python while maintaining them in public git repos. 3. Launching BTRFS test machines in cloud (aws spot instance are very cheap) . 4. Run those feature specific tests and publish the results into wiki or/and bugs into bugzilla. 5. Destroy the test machines. Such setup will require quite good effort. As Qu mentioned, the challenging part will be integrating existing different tests and make it automatic. Another thought on BTRFS testing: At this moment, when someone sends a patch - i assume, they automatically pushed into patchwork.kernel.org (based on some back-end configuration I guess). On Btrfs mailing-list,we have lot of bug reports so using something like [1] we can write a program which parses mailing-list for specific keyword like "BUGREPORT:" similar to "PATCH:" and create a new bug or update the details on bugzilla.kernel.org? This will ensure we can keep track of bugs on bugzilla. I think updating bugs based on mailing-list responses will be little tricky than creating new bugs. something like that can be useful as well. I can help a bit on such testbed setup :-) Thanks! Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org On Mon, Aug 14, 2017 at 6:31 AM, Cerem Cem ASLAN wrote: > Would that be useful to build a BTRFS test machine, which will perform > both software tests (btrfs send | btrfs receive, read/write random > data etc.) and hardware tests, such as abrupt power off test, abruptly > removing a raid-X disk physically, etc. > > If it would be useful, what tests should it cover? > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 4/5] btrfs-progs: tests: fix typo in convert-tests/008-readonly-image
> if [ $? -ne 1 ]; then > echo "after convert ext2_save/image is not read-only" Though this mayn't affect test-script result, please change the typo in this location also. thanks. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-convert --no-datasum & btrfs check: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1
Looked like I missed few patch. After checking with latest branch. Its working. Cheers, Lakshmipathi.G On Sun, Jul 2, 2017 at 7:30 PM, Lakshmipathi.G wrote: > Hi. > While using btrfs-convert with '--no-datasum' option & running btrfs > check crashes: > > steps: > mkfs.ext4 tests/t1 > mount tests/t1 /data > cp -r *.c /data > umount /data > ./btrfs-convert --no-datasum tests/t1 > ./btrfs check tests/t1 > > cmds-check.c:5488: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1 > > Is this a bug or something wrong with my environment? > > More info: https://bugzilla.kernel.org/show_bug.cgi?id=196217 > > Cheers, > Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs-convert --no-datasum & btrfs check: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1
Hi. While using btrfs-convert with '--no-datasum' option & running btrfs check crashes: steps: mkfs.ext4 tests/t1 mount tests/t1 /data cp -r *.c /data umount /data ./btrfs-convert --no-datasum tests/t1 ./btrfs check tests/t1 cmds-check.c:5488: check_owner_ref: BUG_ON `rec->is_root` triggered, value 1 Is this a bug or something wrong with my environment? More info: https://bugzilla.kernel.org/show_bug.cgi?id=196217 Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] Btrfs-progs: convert: do not clear nodatasum flag in inode item
Could you please add also a test-script for this interesting issue
under tests/convert-tests
(If its not already there)?. thanks.
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
On Thu, Jun 29, 2017 at 2:59 AM, Liu Bo wrote:
> With the current btrfs-convert, if we convert a ext4 without data checksum,
> it'd not set nodatasum flag in inode item, nor create csum item, reading
> file ends up with checksum errors.
>
> Signed-off-by: Liu Bo
> ---
> convert/source-ext2.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/convert/source-ext2.c b/convert/source-ext2.c
> index 1b0576b..64b95c5 100644
> --- a/convert/source-ext2.c
> +++ b/convert/source-ext2.c
> @@ -742,7 +742,7 @@ static int ext2_check_state(struct btrfs_convert_context
> *cctx)
> static void ext2_convert_inode_flags(struct btrfs_inode_item *dst,
> struct ext2_inode *src)
> {
> - u64 flags = 0;
> + u64 flags = btrfs_stack_inode_flags(dst);
>
> COPY_ONE_EXT2_FLAG(flags, src, APPEND);
> COPY_ONE_EXT2_FLAG(flags, src, SYNC);
> --
> 2.5.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4] btrfs-progs: btrfs-convert: Add larger device support
> > - u32 free_inodes_count; > > + u64 first_data_block; > > + u64 block_count; > > + u64 inodes_count; > > + u64 free_inodes_count; > > I've split this change from the patch as it does not logically belong to > the same patch, altough the change is simple. Okay sure, thanks. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Stability status of btrfs-convert...
>Is this tool supported, at least? Can I use this tool as a part of >software upgrade to change the data filesystem to btrfs? Yes its supported. If your existing FS is too large (>22TB) we have a bug for it https://bugzilla.kernel.org/show_bug.cgi?id=194795 AFAIK, there are quite number of people converted their EXT4 to BTRFS . IMO, btrfs-convert is fairly stable, especially if you have only GB's of data Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2] btrfs-progs: Fix 'btrfs device stats --check' cli option
Bug 194961 - btrfs device stats --check does not work
https://bugzilla.kernel.org/show_bug.cgi?id=194961
Reported-by: Tomas Thiemel
Signed-off-by: Lakshmipathi.G
---
cmds-device.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/cmds-device.c b/cmds-device.c
index 5e016a7..4337eb2 100644
--- a/cmds-device.c
+++ b/cmds-device.c
@@ -400,6 +400,7 @@ static int cmd_device_stats(int argc, char **argv)
while (1) {
int c;
static const struct option long_options[] = {
+ {"check", no_argument, NULL, 'c'},
{"reset", no_argument, NULL, 'z'},
{NULL, 0, NULL, 0}
};
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] Fix 'btrfs device stats --check' cli option
Reported by Tomas Thiemel
Bug 194961 - btrfs device stats --check does not work
---
cmds-device.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/cmds-device.c b/cmds-device.c
index 5e016a7..4337eb2 100644
--- a/cmds-device.c
+++ b/cmds-device.c
@@ -400,6 +400,7 @@ static int cmd_device_stats(int argc, char **argv)
while (1) {
int c;
static const struct option long_options[] = {
+ {"check", no_argument, NULL, 'c'},
{"reset", no_argument, NULL, 'z'},
{NULL, 0, NULL, 0}
};
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v4 00/20] Btrfs-progs offline scrub
> For any one who wants to try it, it can be get from my repo: > https://github.com/adam900710/btrfs-progs/tree/offline_scrub While running single corruption script receive message like "REPARIED: corrupted data with good P/Q, repaired" along with message from script which says "Parity stripe check passed." complete output here: https://github.com/Lakshmipathi/btrfs_offline_scrub/blob/master/logs-june18/single_corruption_misc-tests-results.txt multiple-corruption: While corrupting continuous blocks received: >Filename=file256k.txt Total Stripes=4 Data Stripe to be corrupted=2,3 .. ..//debugfs-tree output ends >ERROR: full stripe 145358848 CORRUPTED: too many read error or corrupted >devices >ERROR: full stripe 145358848: tolerance: 1, missing: 0, read error: 0, csum >error: 2 While corrupting non-continuous blocks: >Filename=file512k.txt Total Stripes=8 Data Stripe to be corrupted=1,3,5 .. ..//debugfs-tree output ends >full stripe 145227776 REPARIED: corrupted data with good P/Q, repaired >full stripe 145358848 REPARIED: corrupted data with good P/Q, repaired >full stripe 145489920 REPARIED: corrupted data with good P/Q, repaired output for other combinations like : >Filename=file768k.txt Total Stripes=12 Data Stripe to be >corrupted=10,8,6,4 >Filename=file1m.txt Total Stripes=16 Data Stripe to be >corrupted=14,12,10,8,7,6 >Filename=file2m.txt Total Stripes=32 Data Stripe to be >corrupted=23,22,21,20,19,18,16 >Filename=file4m.txt Total Stripes=64 Data Stripe to be >corrupted=34,33,32,31,30,20,18,15,10,8,5 >Filename=file8m.txt Total Stripes=128 Data Stripe to be >corrupted=100,90,80,70,60,50,40,30,20,10 can be found here: https://github.com/Lakshmipathi/btrfs_offline_scrub/blob/master/logs-june18/multiple_corruptions_misc-tests-results.txt Outputs are looking fine, but more testing required with different file-types :-) thanks. Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-convert: whats minimum free space requirement?
On Fri, Jun 09, 2017 at 03:38:31PM +0800, Qu Wenruo wrote: > >I was thinking about adding a free-space check (based on total blocks/free > >block > >stat from source superblock) before btrfs-convert begins, looks like its more > >complex than i thought. > > BTW, it's already done, since it's complex, btrfs-convert will try to > calculate it early on, so even it fails to find the needed space, it will > not screw up the old fs. > > Thanks, > Qu > > oh, I didn't realize that, thanks. Will test/try with little free space and check the returning error messages. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: btrfs-convert: whats minimum free space requirement?
On Fri, Jun 09, 2017 at 01:35:12PM +0800, Qu Wenruo wrote: > >ex: Like 5% of ext3/4 free space is needed for > >btrfs-convert to succeed? > > That's hard to say. > > It's not only dependent on how much free space is here, but also how > continuous the free space is. > > In convert, it needs the following continuous space at least: > > 1) 64K for temporary btrfs super block > 2) 4M for system chunk > 3) 32M for metadata chunk. > > Further more, we want to keep chunks at reasonable size, that's to say we > want each used data space on old fs is covered by a chunk >= 32M. > > For worst case, if the free space in old fs is so scattered, either we can't > find continuous for sys/meta chunk, or used data space is so scattered that > we can't find free space after rounding up data chunks, then we can't do > convert. > > Thanks, > Qu Thanks for the details, Qu. I was thinking about adding a free-space check (based on total blocks/free block stat from source superblock) before btrfs-convert begins, looks like its more complex than i thought. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
btrfs-convert: whats minimum free space requirement?
Hi. Just wanted to check whether do we have any numbers on whats the minimum free space requirement on source file system for btrfs-convert to work? ex: Like 5% of ext3/4 free space is needed for btrfs-convert to succeed? Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4] btrfs-progs: btrfs-convert: Add larger device support
With larger file system (in this case its 22TB), ext2fs_open() returns
EXT2_ET_CANT_USE_LEGACY_BITMAPS error message with ext2fs_read_block_bitmap().
To overcome this issue, (a) we need pass EXT2_FLAG_64BITS flag with ext2fs_open.
(b) use 64-bit functions like ext2fs_get_block_bitmap_range2,
ext2fs_inode_data_blocks2,ext2fs_read_ext_attr2. (c) use 64bit types with
btrfs_convert_context fields.
bug: https://bugzilla.kernel.org/show_bug.cgi?id=194795
Signed-off-by: Lakshmipathi.G
---
convert/common.h | 8
convert/source-ext2.c | 11 ++-
2 files changed, 10 insertions(+), 9 deletions(-)
diff --git a/convert/common.h b/convert/common.h
index 0d3adea..2f4ea48 100644
--- a/convert/common.h
+++ b/convert/common.h
@@ -30,10 +30,10 @@ struct btrfs_mkfs_config;
struct btrfs_convert_context {
u32 blocksize;
- u32 first_data_block;
- u32 block_count;
- u32 inodes_count;
- u32 free_inodes_count;
+ u64 first_data_block;
+ u64 block_count;
+ u64 inodes_count;
+ u64 free_inodes_count;
u64 total_bytes;
char *volume_name;
const struct btrfs_convert_operations *convert_ops;
diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx,
const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+ int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
- ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
+ ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
btrfs_convert_context *cctx)
return -ENOMEM;
for (i = 0; i < fs->group_desc_count; i++) {
- ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+ ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle
*trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
- if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+ if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
- err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+ err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
- fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+ fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] btrfs-progs: btrfs-convert: Add larger device support
> > I think we should use the 64bit types just to be safe. > Dave, Okay made those changes and new patch sent. >If we need to modify tons of things, then please split such modification to >different patches. Qu, Seems like only minor changes needed, so used a single patch. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
> > Only recovery needs to be implemented now. > > Thanks, > Qu > Once recovery is implemented, I'll try again. Just one suggestion: Optionally, It is possible to print filename for these detected blocks. For ex, if corruption happened on a old/unwanted archived log file (/var/log/nginx_access_20160101.log) I don't need to panic, as I can live without that file. Or possibly delete that file, so that scrub doesn't report errors. -- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
Okay, I did multiple (upto 11) corruption on the same file. Seems like it says 'CORRUPTED' when we corrupt two continuous data stripes and reports 'RECOVERABLE' whenever possible.It looks fine. You can find the logs and test-scripts on below link. thanks. https://github.com/Lakshmipathi/btrfs_offline_scrub -- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
Okay sure, When I find free-time, will run few more combinations and let you know the details along with any other suggestions/thoughts on the interface. thanks. On 5/22/17, Qu Wenruo wrote: > Good to hear that. > > And you could try more combination (although I have done a lot during > the test) to see if it's good enough to be used as a better scrub > equivalent. > > And of course, any idea on the interface design (--offline under scrub > or other place) is welcomed. > > Thanks, > Qu > > At 05/22/2017 04:30 PM, Lakshmipathi.G wrote: >> Yes. It detects the injected corruption. >> >> #btrfs scrub start --offline /dev/loop6 >> ERROR: data at bytenr 145293312 mirror 0 csum mismatch, have >> 0x790f1fe1 expect 0xa30cb5c5 >> ERROR: full stripe 145227776 RECOVERABLE: Data stripes corrupted, but >> P/Q is good >> >> Scrub result: >> Tree bytes scrubbed: 131072 >> Tree extents scrubbed: 8 >> Data bytes scrubbed: 192512 >> Data extents scrubbed: 3 >> Data bytes without csum: 131072 >> Read error: 0 >> Verify error: 0 >> Csum error: 1 >> >> >> Cheers, >> Lakshmipathi.G >> >> > > > -- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
Yes. It detects the injected corruption. #btrfs scrub start --offline /dev/loop6 ERROR: data at bytenr 145293312 mirror 0 csum mismatch, have 0x790f1fe1 expect 0xa30cb5c5 ERROR: full stripe 145227776 RECOVERABLE: Data stripes corrupted, but P/Q is good Scrub result: Tree bytes scrubbed: 131072 Tree extents scrubbed: 8 Data bytes scrubbed: 192512 Data extents scrubbed: 3 Data bytes without csum: 131072 Read error: 0 Verify error: 0 Csum error: 1 Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
> > As the title said, it's *offline* scrub, while you're still using the > *online* scrub. > > To use offline scrub, you should exec "btrfs scrub start --offline > " > > And then it should detect the same error. > Output is nothing like kernel. > > Thanks, > Qu > Ah,my bad. I thought the entire binary is changed, didn't realize its an option. Will check again with 'offline' option. -- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
When I ran the script, it produces below output. From the output it looks like, it detected the error and corrected. But parity-block corruption fix patch not yet merged or I missed something? Run: (with scrub_offline repo) btrfs scrub start /home/laks/centos/laks/BTRFS/scrub_offline/tests/mnt scrub started on /home/laks/centos/laks/BTRFS/scrub_offline/tests/mnt, fsid a5cf2e54-e8da-4570-8328-2cee46678de3 (pid=5943) WARNING: errors detected during scrubbing, corrected. Scrub corrected value: D1xx ##output from testscript Scrub corrupted parity stripe. ##output from testscript (mkfs.btrfs -f -d raid5 -m raid5 /dev/loop18 /dev/loop19 /dev/loop20) >> >> Can I inject corruption with existing script [1] and expect offline >> scrub to fix it? If so, I'll give it try and let you know the results. >> >> [1] https://patchwork.kernel.org/patch/9583455/ > > No fixing yet. > > This part is focusing on error detection and report. > > Feel free to inject corruption and to see if it behave as expected. > > Thanks, > Qu > Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 2/2] btrfs-progs: test for restoring multiple devices fs into a single device
Looks fine, just couple of minor feedback. > +run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $loop1 $loop2 Please add quotation around variable in the script, as suggested in tests/README.md 'Coding style, best practices' section. This section is added recently, we are updating older scripts. > + > +# Cleanup loop devices. > +run_check $SUDO_HELPER losetup -d $loop1 > +run_check $SUDO_HELPER losetup -d $loop2 > +rm -f dev1 dev2 > + > +# Compare the file digests. > +[ $orig_md5 == $new_md5 ] || _fail "File digests do not match" "Cleanup loop devices" can be done after comparing file integrity? This way, if integrity fails, the setup will be there to debug further. thanks. -- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: tests: remove variable quotation from convert-tests
Oops, sorry, I introduced those two issues in recent patches and missed (skipped?) them while testing. With above patch, 008/009 test-cases are working fine now. thanks. On 5/16/17, Tsutomu Itoh wrote: > In btrfs-progs-v4.11-rc1, the following convert-tests failed. > > [TEST/conv] 008-readonly-image > [TEST/conv] readonly image test, btrfs defaults > failed: mke2fs -t ext4 -b 4096 -F > /Build/btrfs-progs-v4.11-rc1/tests/test.img > test failed for case 008-readonly-image > Makefile:271: recipe for target 'test-convert' failed > make: *** [test-convert] Error 1 > [TEST/conv] 009-common-inode-flags > [TEST/conv] common inode flags test, btrfs defaults > failed: mke2fs -t ext4 -b 4096 -F > /Build/btrfs-progs-v4.11-rc1/tests/test.img > test failed for case 009-common-inode-flags > Makefile:271: recipe for target 'test-convert' failed > make: *** [test-convert] Error 1 > > So, remove quotes from $default_mke2fs. > > Signed-off-by: Tsutomu Itoh > --- > tests/convert-tests/008-readonly-image/test.sh | 2 +- > tests/convert-tests/009-common-inode-flags/test.sh | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/tests/convert-tests/008-readonly-image/test.sh > b/tests/convert-tests/008-readonly-image/test.sh > index b2f1ae37..27c9373e 100755 > --- a/tests/convert-tests/008-readonly-image/test.sh > +++ b/tests/convert-tests/008-readonly-image/test.sh > @@ -10,7 +10,7 @@ check_prereq btrfs-convert > > default_mke2fs="mke2fs -t ext4 -b 4096" > convert_test_preamble '' 'readonly image test' 16k "$default_mke2fs" > -convert_test_prep_fs "$default_mke2fs" > +convert_test_prep_fs $default_mke2fs > run_check_umount_test_dev > convert_test_do_convert > run_check_mount_test_dev > diff --git a/tests/convert-tests/009-common-inode-flags/test.sh > b/tests/convert-tests/009-common-inode-flags/test.sh > index a5828790..02823e14 100755 > --- a/tests/convert-tests/009-common-inode-flags/test.sh > +++ b/tests/convert-tests/009-common-inode-flags/test.sh > @@ -11,7 +11,7 @@ check_prereq btrfs-convert > fail=0 > default_mke2fs="mke2fs -t ext4 -b 4096" > convert_test_preamble '' 'common inode flags test' 16k "$default_mke2fs" > -convert_test_prep_fs "$default_mke2fs" > +convert_test_prep_fs $default_mke2fs > > # create file with specific flags > run_check $SUDO_HELPER touch "$TEST_MNT/flag_test" > -- > 2.12.2 > > > Tsutomu Itoh t-i...@jp.fujitsu.com > -- > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- Cheers, Lakshmipathi.G http://www.giis.co.in http://www.webminal.org -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2] btrfs-progs: btrfs-convert: Add larger device support
On Mon, May 15, 2017 at 09:40:29AM +0800, Qu Wenruo wrote: > >bug: https://bugzilla.kernel.org/show_bug.cgi?id=194795 > > Errr, it seems that you forgot to update ext2_open_fs() to update how we get > cctx->block_counts. > > Without that update, we still get wrong total size of original fs, so > converted image will be corrupted. > > In this 22T case, it can't pass convert test since after conversion, > converted image can't pass e2fsck. > > Thanks, > Qu > Thanks for pointing out the issue. So we need to update common.h/cctx->block_count from u32 to u64? do we also need to change other fields like inodes_count and free_inode_count? Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3 00/19] Btrfs-progs offline scrub
> > Ping? > > Any comments? > > Thanks, > Qu Can I inject corruption with existing script [1] and expect offline scrub to fix it? If so, I'll give it try and let you know the results. [1] https://patchwork.kernel.org/patch/9583455/ ---- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v2] btrfs-progs: btrfs-convert: Add larger device support
With larger file system (in this case its 22TB), ext2fs_open() returns
EXT2_ET_CANT_USE_LEGACY_BITMAPS error message with ext2fs_read_block_bitmap().
To overcome this issue, we need pass EXT2_FLAG_64BITS flag with ext2fs_open
and also use 64-bit functions like ext2fs_get_block_bitmap_range2,
ext2fs_inode_data_blocks2,ext2fs_read_ext_attr2
bug: https://bugzilla.kernel.org/show_bug.cgi?id=194795
Signed-off-by: Lakshmipathi.G
---
convert/source-ext2.c | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx,
const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+ int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
- ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
+ ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
btrfs_convert_context *cctx)
return -ENOMEM;
for (i = 0; i < fs->group_desc_count; i++) {
- ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+ ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle
*trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
- if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+ if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
- err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+ err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
- fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+ fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
> > Does the mke2fs command exist on your system? We maybe want to use > mkfs.ext4 directly. Yes, mke2fs is there. (I'll check using mkfs.ext4 directly) # ./convert-tests/009-common-inode-flags/test.sh [TEST/conv] common inode flags test, btrfs defaults failed: mke2fs -t ext4 -b 4096 -F /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img After failure, I can run failed command manually. # mke2fs -t ext4 -b 4096 -F /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.imgmke2fs 1.42.13 (17-May-2015) Discarding device blocks: done Creating filesystem with 131072 4k blocks and 32768 inodes Filesystem UUID: 659dbb16-ea5e-48e0-8b1a-30474f54c282 Superblock backups stored on blocks: 32768, 98304 Allocating group tables: done Writing inode tables: done Creating journal (4096 blocks): done Writing superblocks and filesystem accounting information: done -- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
created a test script but it fail to detect mk2fs. Test script 009 also produces: # ./convert-tests/009-common-inode-flags/test.sh [TEST/conv] common inode flags test, btrfs defaults failed: mke2fs -t ext4 -b 4096 -F /home/laks/centos/laks/BTRFS/btrfs-progs/tests/test.img Seems like run_check fails to detect mke2fs command. Something wrong with my setup or issue with run_check? -- Cheers, Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror
okay, but I'm not sure whether it can be useful here other than the
sample cli usage. The reason, this tool relies heavily on Python/C
layer[1]. The framework works like this:
C-api(fs-progs) <--> Python/C api <--> Python program.
C-api is typical fs library (like libext2fs/libbtrfs) and C/Python
used to export every on-disk fs structure as an python-object along
with few fs-library functions. Finally in python script we can perform
stuffs like:
--
import fslib as fs
disk_object=fs.get_inode("/file/path")
#set few values finally make a call
retval = fs.corrupt_object(disk_object, offset, size, optype, fixcrc)
---
at C-python layer above python-object converted into C appropriate
data-type and passed on to C-layer.
hard-part will be troubleshooting issues, tracking them is difficult
because both gdb and pdb can help only upto certain point.
[1]: https://docs.python.org/2/extending/extending.html
Cheers.
Lakshmipathi.G
On 5/5/17, Qu Wenruo wrote:
>
>
> At 05/05/2017 12:53 AM, David Sterba wrote:
>> On Sun, Apr 23, 2017 at 01:12:42PM +0530, Lakshmipathi.G wrote:
>>> Thanks for the example and details. I understood some and need to
>>> re-read couple of more times to understand the remaining.
>>>
>>> btw, I created a corruption framework(with previous org), the sample
>>> usage and example is below. It looks similar to Btrfs corruption tool.
>>> thanks.
>>>
>>> --
>>> corrupt.py --help
>> [...]
>>
>> Interesting, can you please share the script? This is another
>> alternative that seems more plausible for rapid prototyping of various
>> corruption scenarios. The C utility (either existing btrfs-corrupt-block
>> or the proposed btrfs-modify) can become tedious to change, but can be
>> compiled and distributed without the python dependency.
>>
>> I wanted to use something python-based for tests when Hans announced the
>> python-btrfs project, but it has broader goals than just the testsuite
>> needs. So we could have our own corrupt.py, just for our internal use.
>>
>> I'm not sure if a compiled tool like btrfs-modify is really needed, but
>> why we can't have both.
>>
>>
> Python based tool is always a good idea.
>
> If python based lib can provide the same capability of current C
> facilities (btrfs_search_slot() at least) and have better encapsulation,
> I'm totally committed to python based one.
>
> Although I'm a little afraid that the python script is not using the low
> level btrfs_search_slot(), but just getting chunk layout then do
> mathematics to get stripe location.
>
> Thanks,
> Qu
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
> file is ~160MB. This sounds much better. So please write a test, using > something like > > truncate -s3T image > mkfs.ext4 image > mount && write some data > convert && rollback > > Thanks. Later we might need to add some mkfs.ext4 option coverage. Sure, will send the above test script in upcoming days, then look into mkfs.ext4 options. thanks. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
On Tue, May 02, 2017 at 05:06:01PM +0200, David Sterba wrote: > On Thu, Apr 27, 2017 at 03:35:34PM +0530, Lakshmipathi.G wrote: > > Bug: Fail to convert 22TB EXT4 to BTRFS > > https://bugzilla.kernel.org/show_bug.cgi?id=194795 > > Thanks for working on the bug, the proposed fix looks good to me judging > by the use of extended API. This would need a test though. I'm trying > this on a sparse file, 3TB occupies 50G and is slow to create. > Conversion works and hasn't finished yet, so this is not suitable for > tests run by default. > > Please write a more detailed changelog, a reference to bugzilla is not > really sufficient. Sure, will send a patch with better commit description about the bug and proposed fix. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/1] btrfs-progs: send: fail on first -ENODATA only
Hi Christian, thanks for fixing it quickly :) I don't have permission
to change the bug state to fixed, if you or Nazar have those rights, i
think this bug can be closed.
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
On Mon, May 1, 2017 at 6:39 PM, Christian Brauner
wrote:
> Hi,
>
> The original bug-reporter verified that my patch fixes the bug. See
>
> https://bugzilla.kernel.org/show_bug.cgi?id=195597
>
> Christian
>
> On Sat, Apr 29, 2017 at 11:54:05PM +0200, Christian Brauner wrote:
>> Returning -ENODATA is only considered invalid on the first run of the loop.
>>
>> Signed-off-by: Christian Brauner
>> ---
>> cmds-receive.c | 20 ++--
>> 1 file changed, 14 insertions(+), 6 deletions(-)
>>
>> diff --git a/cmds-receive.c b/cmds-receive.c
>> index b59f00e4..72e9c8f3 100644
>> --- a/cmds-receive.c
>> +++ b/cmds-receive.c
>> @@ -1091,6 +1091,7 @@ static int do_receive(struct btrfs_receive *rctx,
>> const char *tomnt,
>> char *dest_dir_full_path;
>> char root_subvol_path[PATH_MAX];
>> int end = 0;
>> + int iterations = 0;
>>
>> dest_dir_full_path = realpath(tomnt, NULL);
>> if (!dest_dir_full_path) {
>> @@ -1198,13 +1199,18 @@ static int do_receive(struct btrfs_receive *rctx,
>> const char *tomnt,
>>rctx,
>>rctx->honor_end_cmd,
>>max_errors);
>> - if (ret < 0 && ret == -ENODATA) {
>> + if (ret < 0) {
>> + if (ret != -ENODATA)
>> + goto out;
>> +
>> /* Empty stream is invalid */
>> - error("empty stream is not considered valid");
>> - ret = -EINVAL;
>> - goto out;
>> - } else if (ret < 0) {
>> - goto out;
>> + if (iterations == 0) {
>> + error("empty stream is not considered valid");
>> + ret = -EINVAL;
>> + goto out;
>> + }
>> +
>> + ret = 1;
>> }
>> if (ret > 0)
>> end = 1;
>> @@ -1213,6 +1219,8 @@ static int do_receive(struct btrfs_receive *rctx,
>> const char *tomnt,
>> ret = finish_subvol(rctx);
>> if (ret < 0)
>> goto out;
>> +
>> + iterations++;
>> }
>> ret = 0;
>>
>> --
>> 2.11.0
>>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 1/2 v2] btrfs-progs: fix btrfs send & receive with -e flag
Hi.
Adding the bug reporter, Nazar for the discussion (as I'm not familiar
with send/receive feature/code).
Cheers,
Lakshmipathi.G
http://www.giis.co.in http://www.webminal.org
On Fri, Apr 28, 2017 at 3:25 PM, Christian Brauner
wrote:
>
> Hi,
>
> On Fri, Apr 28, 2017 at 02:55:31PM +0530, Lakshmipathi.G wrote:
> > Seems like user reported an issue with this patch. please check
> > https://bugzilla.kernel.org/show_bug.cgi?id=195597
>
> I can take a look. What I'm wondering about is why it fails only in the HDD
> to SSD case. If -ENODATA is returned with this patch it should mean that there
> was no header data. So is the user sure that this doesn't indicate a valid
> error?
>
> Christian
>
> >
> >
> > Cheers,
> > Lakshmipathi.G
> >
> >
> > On Tue, Apr 4, 2017 at 1:51 AM, Christian Brauner <
> > christian.brau...@ubuntu.com> wrote:
> > > The old check here tried to ensure that empty streams are not considered
> > valid.
> > > The old check however, will always fail when only one run through the
> > while(1)
> > > loop is needed and honor_end_cmd is set. So this:
> > >
> > > btrfs send /some/subvol | btrfs receive -e /some/
> > >
> > > will consistently fail because -e causes honor_cmd_to be set and
> > > btrfs_read_and_process_send_stream() to correctly return 1. So the
> > command will
> > > be successful but btrfs receive will error out because the send - receive
> > > concluded in one run through the while(1) loop.
> > >
> > > If we want to exclude empty streams we need a way to tell the difference
> > between
> > > btrfs_read_and_process_send_stream() returning 1 because read_buf() did
> > not
> > > detect any data and read_and_process_cmd() returning 1 because
> > honor_end_cmd was
> > > set. Without introducing too many changes the best way to me seems to have
> > > btrfs_read_and_process_send_stream() return -ENODATA in the first case.
> > The rest
> > > stays the same. We can then check for -ENODATA in do_receive() and report
> > a
> > > proper error in this case. This should also be backwards compatible to
> > previous
> > > versions of btrfs receive. They will fail on empty streams because a
> > negative
> > > value is returned. The only thing that they will lack is a nice error
> > message.
> > >
> > > Signed-off-by: Christian Brauner
> > > ---
> > > Changelog: 2017-04-03
> > > - no changes
> > > ---
> > > cmds-receive.c | 13 +
> > > send-stream.c | 2 +-
> > > 2 files changed, 6 insertions(+), 9 deletions(-)
> > >
> > > diff --git a/cmds-receive.c b/cmds-receive.c
> > > index 6cf22637..b59f00e4 100644
> > > --- a/cmds-receive.c
> > > +++ b/cmds-receive.c
> > > @@ -1091,7 +1091,6 @@ static int do_receive(struct btrfs_receive *rctx,
> > const char *tomnt,
> > > char *dest_dir_full_path;
> > > char root_subvol_path[PATH_MAX];
> > > int end = 0;
> > > - int count;
> > >
> > > dest_dir_full_path = realpath(tomnt, NULL);
> > > if (!dest_dir_full_path) {
> > > @@ -1186,7 +1185,6 @@ static int do_receive(struct btrfs_receive *rctx,
> > const char *tomnt,
> > > if (ret < 0)
> > > goto out;
> > >
> > > - count = 0;
> > > while (!end) {
> > > if (rctx->cached_capabilities_len) {
> > > if (g_verbose >= 3)
> > > @@ -1200,16 +1198,15 @@ static int do_receive(struct btrfs_receive *rctx,
> > const char *tomnt,
> > > rctx,
> > >
> > rctx->honor_end_cmd,
> > > max_errors);
> > > - if (ret < 0)
> > > - goto out;
> > > - /* Empty stream is invalid */
> > > - if (ret && count == 0) {
> > > + if (ret < 0 && ret == -ENODATA) {
> > > + /* Empty stream is invalid */
> > > error("empty stream is not considered valid");
> > > ret = -EINVAL;
> > > goto out;
> > > + } else if (ret < 0) {
> > > + goto out;
>
Re: [PATCH] btrfs-progs: btrfs-convert: Add larger device support
Hi Qu,
Seems like its a known issue with older bitmaps.
http://linux-ext4.vger.kernel.narkive.com/wkNbJe0b/patch-debugfs-open-with-ext2-flag-64bits
Cheers.
Lakshmipathi.G
On Fri, Apr 28, 2017 at 09:48:10AM +0800, Qu Wenruo wrote:
>
>
> At 04/28/2017 09:27 AM, Qu Wenruo wrote:
> >At 04/27/2017 06:05 PM, Lakshmipathi.G wrote:
> >>Bug: Fail to convert 22TB EXT4 to BTRFS
> >>https://bugzilla.kernel.org/show_bug.cgi?id=194795
> >
> >Thanks for fixing it.
> >
> >While testing manually using LVM thin provision, although the convert
> >itself works, the converted image has something wrong on its size.
> >
> >The converted image is only 4T sized, not the correct size (20T) of my
> >original setup.
> >
> >This may not be the problem of your patch.
> >I'll double check the original code to ensure it works.
>
> It's the method we get blocks count wrong.
>
> We should do it just as dumpe2fs does:
> --
> static __u64 e2p_blocks_count(struct ext2_super_block *super)
> {
> return super->s_blocks_count |
> (ext2fs_has_feature_64bit(super) ?
> (__u64) super->s_blocks_count_hi << 32 : 0);
> }
> --
>
> Current super->s_blocks_count can only handle U32_MAX, higher bits are in
> super->s_blocks_count_hi.
>
> Thanks,
> Qu
>
> >
> >Thanks,
> >Qu
> >
> >>
> >>Signed-off-by: Lakshmipathi.G
> >>---
> >> convert/source-ext2.c | 11 ++-
> >> 1 file changed, 6 insertions(+), 5 deletions(-)
> >>
> >>diff --git a/convert/source-ext2.c b/convert/source-ext2.c
> >>index 1b0576b..275cb89 100644
> >>--- a/convert/source-ext2.c
> >>+++ b/convert/source-ext2.c
> >>@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context
> >>*cctx, const char *name)
> >> ext2_filsys ext2_fs;
> >> ext2_ino_t ino;
> >> u32 ro_feature;
> >>+int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
> >>-ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
> >>+ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
> >> if (ret) {
> >> fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
> >> return -1;
> >>@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
> >>btrfs_convert_context *cctx)
> >> return -ENOMEM;
> >> for (i = 0; i < fs->group_desc_count; i++) {
> >>-ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
> >>+ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
> >> block_nbytes * 8, block_bitmap);
> >> if (ret) {
> >> error("fail to get bitmap from ext2, %s",
> >>@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct
> >>btrfs_trans_handle *trans,
> >> int ret;
> >> char *pathname;
> >> u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
> >>-if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
> >>+if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
> >> btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
> >> ret = ext2_create_file_extents(trans, root, objectid,
> >> btrfs_inode, ext2_fs, ext2_ino,
> >>@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
> >>btrfs_trans_handle *trans,
> >> ret = -ENOMEM;
> >> goto out;
> >> }
> >>-err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
> >>+err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl,
> >>buffer);
> >> if (err) {
> >>-fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
> >>+fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
> >> error_message(err));
> >> ret = -1;
> >> goto out;
> >>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs-progs: btrfs-convert: Add larger device support
Bug: Fail to convert 22TB EXT4 to BTRFS
https://bugzilla.kernel.org/show_bug.cgi?id=194795
Signed-off-by: Lakshmipathi.G
---
convert/source-ext2.c | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/convert/source-ext2.c b/convert/source-ext2.c
index 1b0576b..275cb89 100644
--- a/convert/source-ext2.c
+++ b/convert/source-ext2.c
@@ -34,8 +34,9 @@ static int ext2_open_fs(struct btrfs_convert_context *cctx,
const char *name)
ext2_filsys ext2_fs;
ext2_ino_t ino;
u32 ro_feature;
+ int open_flag = EXT2_FLAG_SOFTSUPP_FEATURES | EXT2_FLAG_64BITS;
- ret = ext2fs_open(name, 0, 0, 0, unix_io_manager, &ext2_fs);
+ ret = ext2fs_open(name, open_flag, 0, 0, unix_io_manager, &ext2_fs);
if (ret) {
fprintf(stderr, "ext2fs_open: %s\n", error_message(ret));
return -1;
@@ -148,7 +149,7 @@ static int ext2_read_used_space(struct
btrfs_convert_context *cctx)
return -ENOMEM;
for (i = 0; i < fs->group_desc_count; i++) {
- ret = ext2fs_get_block_bitmap_range(fs->block_map, blk_itr,
+ ret = ext2fs_get_block_bitmap_range2(fs->block_map, blk_itr,
block_nbytes * 8, block_bitmap);
if (ret) {
error("fail to get bitmap from ext2, %s",
@@ -353,7 +354,7 @@ static int ext2_create_symlink(struct btrfs_trans_handle
*trans,
int ret;
char *pathname;
u64 inode_size = btrfs_stack_inode_size(btrfs_inode);
- if (ext2fs_inode_data_blocks(ext2_fs, ext2_inode)) {
+ if (ext2fs_inode_data_blocks2(ext2_fs, ext2_inode)) {
btrfs_set_stack_inode_size(btrfs_inode, inode_size + 1);
ret = ext2_create_file_extents(trans, root, objectid,
btrfs_inode, ext2_fs, ext2_ino,
@@ -627,9 +628,9 @@ static int ext2_copy_extended_attrs(struct
btrfs_trans_handle *trans,
ret = -ENOMEM;
goto out;
}
- err = ext2fs_read_ext_attr(ext2_fs, ext2_inode->i_file_acl, buffer);
+ err = ext2fs_read_ext_attr2(ext2_fs, ext2_inode->i_file_acl, buffer);
if (err) {
- fprintf(stderr, "ext2fs_read_ext_attr: %s\n",
+ fprintf(stderr, "ext2fs_read_ext_attr2: %s\n",
error_message(err));
ret = -1;
goto out;
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH] btrfs-progs: Introduce 'btrfs inspect-internal dump-csum' option
Example usage:
btrfs inspect-internal dump-csum /btrfs/50gbfile /dev/sda4
csum for /btrfs/50gbfile dumped to /btrfs/50gbfile.csumdump
Signed-off-by: Lakshmipathi.G
---
Makefile | 3 +-
cmds-inspect-dump-csum.c | 256 +++
cmds-inspect-dump-csum.h | 24 +
cmds-inspect.c | 3 +
4 files changed, 285 insertions(+), 1 deletion(-)
create mode 100644 cmds-inspect-dump-csum.c
create mode 100644 cmds-inspect-dump-csum.h
diff --git a/Makefile b/Makefile
index 81598df..6e63e18 100644
--- a/Makefile
+++ b/Makefile
@@ -102,7 +102,7 @@ cmds_objects = cmds-subvolume.o cmds-filesystem.o
cmds-device.o cmds-scrub.o \
cmds-restore.o cmds-rescue.o chunk-recover.o super-recover.o \
cmds-property.o cmds-fi-usage.o cmds-inspect-dump-tree.o \
cmds-inspect-dump-super.o cmds-inspect-tree-stats.o cmds-fi-du.o
\
- mkfs/common.o
+ cmds-inspect-dump-csum.o mkfs/common.o
libbtrfs_objects = send-stream.o send-utils.o kernel-lib/rbtree.o btrfs-list.o
\
kernel-lib/crc32c.o messages.o \
uuid-tree.o utils-lib.o rbtree-utils.o
@@ -191,6 +191,7 @@ btrfs_convert_cflags =
-DBTRFSCONVERT_EXT2=$(BTRFSCONVERT_EXT2)
btrfs_fragments_libs = -lgd -lpng -ljpeg -lfreetype
btrfs_debug_tree_objects = cmds-inspect-dump-tree.o
btrfs_show_super_objects = cmds-inspect-dump-super.o
+btrfs_dump_csum_objects = cmds-inspect-dump-csum.o
btrfs_calc_size_objects = cmds-inspect-tree-stats.o
# collect values of the variables above
diff --git a/cmds-inspect-dump-csum.c b/cmds-inspect-dump-csum.c
new file mode 100644
index 000..6065eb4
--- /dev/null
+++ b/cmds-inspect-dump-csum.c
@@ -0,0 +1,256 @@
+/*
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+
+#include "kerncompat.h"
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "ctree.h"
+#include "disk-io.h"
+#include "print-tree.h"
+#include "transaction.h"
+#include "list.h"
+#include "utils.h"
+#include "commands.h"
+#include "crc32c.h"
+#include "cmds-inspect-dump-csum.h"
+#include "help.h"
+#include "volumes.h"
+
+
+const char * const cmd_inspect_dump_csum_usage[] = {
+ "btrfs inspect-internal dump-csum ",
+ "Get csums for the given file.",
+ NULL
+};
+
+int btrfs_lookup_csums(struct btrfs_trans_handle *trans, struct btrfs_root
*root,
+ struct btrfs_path *path, u64 bytenr, int cow, int total_csums, FILE *fp)
+{
+ int ret;
+ int i;
+ int start_pos = 0;
+ struct btrfs_key file_key;
+ struct btrfs_key found_key;
+ struct btrfs_csum_item *item;
+ struct extent_buffer *leaf;
+ u64 csum_offset = 0;
+ u16 csum_size =
+ btrfs_super_csum_size(root->fs_info->super_copy);
+ int csums_in_item = 0;
+ unsigned int tree_csum = 0;
+ int pending_csums = total_csums;
+ static int cnt=1;
+
+ file_key.objectid = BTRFS_EXTENT_CSUM_OBJECTID;
+ file_key.offset = bytenr;
+ file_key.type = BTRFS_EXTENT_CSUM_KEY;
+ ret = btrfs_search_slot(trans, root, &file_key, path, 0, cow);
+ if (ret < 0)
+ goto fail;
+ while(1){
+ leaf = path->nodes[0];
+ if (ret > 0) {
+ ret = 1;
+ if (path->slots[0] == 0)
+ goto fail;
+ path->slots[0]--;
+ btrfs_item_key_to_cpu(leaf, &found_key, path->slots[0]);
+ if (found_key.type != BTRFS_EXTENT_CSUM_KEY){
+ fprintf(stderr, "\nInvalid key found.");
+ goto fail;
+ }
+
+ csum_offset = ((bytenr - found_key.offset) /
root->sectorsize) * csum_size;
+ csums_in_item = btrfs_item_size_nr(leaf,
path->slots[0]);
+ csums_in_item /= csum_size;
+ csums_in_item -= ( bytenr - found_key.offset ) /
root->sectorsize;
+ start_pos=csum_offset;
+
[PATCH] btrfs-progs: tests: add variable quotation to fsck-tests
Signed-off-by: Lakshmipathi.G
---
tests/fsck-tests/006-bad-root-items/test.sh| 6 +++---
tests/fsck-tests/012-leaf-corruption/test.sh | 24 +++---
tests/fsck-tests/013-extent-tree-rebuild/test.sh | 22 ++--
tests/fsck-tests/018-leaf-crossing-stripes/test.sh | 4 ++--
.../fsck-tests/019-non-skinny-false-alert/test.sh | 4 ++--
tests/fsck-tests/020-extent-ref-cases/test.sh | 4 ++--
.../021-partially-dropped-snapshot-case/test.sh| 4 ++--
tests/fsck-tests/022-qgroup-rescan-halfway/test.sh | 4 ++--
tests/fsck-tests/023-qgroup-stack-overflow/test.sh | 4 ++--
tests/fsck-tests/024-clear-space-cache/test.sh | 16 +++
tests/fsck-tests/025-file-extents/test.sh | 2 +-
tests/fsck-tests/026-check-inode-link/test.sh | 2 +-
12 files changed, 48 insertions(+), 48 deletions(-)
diff --git a/tests/fsck-tests/006-bad-root-items/test.sh
b/tests/fsck-tests/006-bad-root-items/test.sh
index 8433234..bf3ef78 100755
--- a/tests/fsck-tests/006-bad-root-items/test.sh
+++ b/tests/fsck-tests/006-bad-root-items/test.sh
@@ -1,15 +1,15 @@
#!/bin/bash
-source $TOP/tests/common
+source "$TOP/tests/common"
check_prereq btrfs
-echo "extracting image default_case.tar.xz" >> $RESULTS
+echo "extracting image default_case.tar.xz" >> "$RESULTS"
tar --no-same-owner -xJf default_case.tar.xz || \
_fail "failed to extract default_case.tar.xz"
check_image test.img
-echo "extracting image skinny_case.tar.xz" >> $RESULTS
+echo "extracting image skinny_case.tar.xz" >> "$RESULTS"
tar --no-same-owner -xJf skinny_case.tar.xz || \
_fail "failed to extract skinny_case.tar.xz"
check_image test.img
diff --git a/tests/fsck-tests/012-leaf-corruption/test.sh
b/tests/fsck-tests/012-leaf-corruption/test.sh
index a308727..43b0e6d 100755
--- a/tests/fsck-tests/012-leaf-corruption/test.sh
+++ b/tests/fsck-tests/012-leaf-corruption/test.sh
@@ -1,6 +1,6 @@
#!/bin/bash
-source $TOP/tests/common
+source "$TOP/tests/common"
check_prereq btrfs-image
@@ -37,16 +37,16 @@ leaf_no_data_ext_list=(
generate_leaf_corrupt_no_data_ext()
{
dest=$1
- echo "generating leaf_corrupt_no_data_ext.btrfs-image" >> $RESULTS
+ echo "generating leaf_corrupt_no_data_ext.btrfs-image" >> "$RESULTS"
tar --no-same-owner -xJf ./no_data_extent.tar.xz || \
_fail "failed to extract leaf_corrupt_no_data_ext.btrfs-image"
- $TOP/btrfs-image -r test.img.btrfs-image $dest || \
+ "$TOP/btrfs-image" -r test.img.btrfs-image "$dest" || \
_fail "failed to extract leaf_corrupt_no_data_ext.btrfs-image"
# leaf at 4206592 and 20905984 contains no regular data
# extent, clear its csum to corrupt the leaf.
for x in 4206592 20905984; do
- dd if=/dev/zero of=$dest bs=1 count=32 conv=notrunc seek=$x \
+ dd if=/dev/zero of="$dest" bs=1 count=32 conv=notrunc seek="$x"
\
1>/dev/null 2>&1
done
}
@@ -60,21 +60,21 @@ check_inode()
name=$5
# Check whether the inode exists
- exists=$($SUDO_HELPER find $path -inum $ino)
+ exists=$($SUDO_HELPER find "$path" -inum "$ino")
if [ -z "$exists" ]; then
_fail "inode $ino not recovered correctly"
fi
# Check inode type
- found_mode=$(printf "%o" 0x$($SUDO_HELPER stat $exists -c %f))
- if [ $found_mode -ne $mode ]; then
+ found_mode=$(printf "%o" 0x$($SUDO_HELPER stat "$exists" -c %f))
+ if [ "$found_mode" -ne "$mode" ]; then
echo "$found_mode"
_fail "inode $ino modes not recovered"
fi
# Check inode size
- found_size=$($SUDO_HELPER stat $exists -c %s)
- if [ $mode -ne 41700 -a $found_size -ne $size ]; then
+ found_size=$($SUDO_HELPER stat "$exists" -c %s)
+ if [ $mode -ne 41700 -a "$found_size" -ne "$size" ]; then
_fail "inode $ino size not recovered correctly"
fi
@@ -90,11 +90,11 @@ check_inode()
check_leaf_corrupt_no_data_ext()
{
image=$1
- $SUDO_HELPER mount -o loop $image -o ro $TEST_MNT
+ $SUDO_HELPER mount -o loop "$image" -o ro "$TEST_MNT"
i=0
while [ $i -lt ${#leaf_no_data_ext_list[@]} ]; do
- check_inode $TEST_MNT/lost+found \
+ check_inode "$TEST_MNT/lost+found" \
${leaf_no_data_ext_list[i]} \
${leaf_no_data_ext_list[i + 1]} \
${
[PATCH] btrfs-progs: tests: add variable quotation to convert-tests
Signed-off-by: Lakshmipathi.G
---
tests/convert-tests.sh | 2 +-
tests/convert-tests/001-ext2-basic/test.sh | 4 ++--
tests/convert-tests/002-ext3-basic/test.sh | 4 ++--
tests/convert-tests/003-ext4-basic/test.sh | 4 ++--
.../004-ext2-backup-superblock-ranges/test.sh | 22 +++---
.../convert-tests/005-delete-all-rollback/test.sh | 6 +++---
tests/convert-tests/006-large-hole-extent/test.sh | 6 +++---
.../007-unsupported-block-sizes/test.sh| 8
tests/convert-tests/008-readonly-image/test.sh | 8
tests/convert-tests/009-common-inode-flags/test.sh | 14 +++---
10 files changed, 39 insertions(+), 39 deletions(-)
diff --git a/tests/convert-tests.sh b/tests/convert-tests.sh
index c566336..819e7e2 100755
--- a/tests/convert-tests.sh
+++ b/tests/convert-tests.sh
@@ -37,7 +37,7 @@ run_one_test() {
./test.sh
if [ $? -ne 0 ]; then
_fail "test failed for case $testname"
- if [[ $TEST_LOG =~ dump ]]; then
+ if [[ "$TEST_LOG" =~ dump ]]; then
cat "$RESULTS"
fi
fi
diff --git a/tests/convert-tests/001-ext2-basic/test.sh
b/tests/convert-tests/001-ext2-basic/test.sh
index 8f4f935..07f1d6e 100755
--- a/tests/convert-tests/001-ext2-basic/test.sh
+++ b/tests/convert-tests/001-ext2-basic/test.sh
@@ -1,7 +1,7 @@
#!/bin/bash
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
setup_root_helper
prepare_test_dev 512M
diff --git a/tests/convert-tests/002-ext3-basic/test.sh
b/tests/convert-tests/002-ext3-basic/test.sh
index aeb111e..be8a929 100755
--- a/tests/convert-tests/002-ext3-basic/test.sh
+++ b/tests/convert-tests/002-ext3-basic/test.sh
@@ -1,7 +1,7 @@
#!/bin/bash
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
setup_root_helper
prepare_test_dev 512M
diff --git a/tests/convert-tests/003-ext4-basic/test.sh
b/tests/convert-tests/003-ext4-basic/test.sh
index 531c81b..44ee592 100755
--- a/tests/convert-tests/003-ext4-basic/test.sh
+++ b/tests/convert-tests/003-ext4-basic/test.sh
@@ -1,7 +1,7 @@
#!/bin/bash
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
setup_root_helper
prepare_test_dev 512M
diff --git a/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
b/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
index c56650b..0ce62f7 100755
--- a/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
+++ b/tests/convert-tests/004-ext2-backup-superblock-ranges/test.sh
@@ -10,7 +10,7 @@
# 4) Overlap file extents
# 5) Unable to rollback
-source $TOP/tests/common
+source "$TOP/tests/common"
check_prereq btrfs-convert
check_prereq btrfs
@@ -23,20 +23,20 @@ prepare_test_dev 512M
# override common function
function check_image() {
TEST_DEV="$1"
- run_check e2fsck -n -f $TEST_DEV
- run_check $TOP/btrfs-convert $TEST_DEV
- run_check $TOP/btrfs check $TEST_DEV
- run_check $TOP/btrfs inspect-internal dump-super $TEST_DEV
+ run_check e2fsck -n -f "$TEST_DEV"
+ run_check "$TOP/btrfs-convert" "$TEST_DEV"
+ run_check "$TOP/btrfs" check "$TEST_DEV"
+ run_check "$TOP/btrfs" inspect-internal dump-super "$TEST_DEV"
run_check_mount_test_dev
- run_check $SUDO_HELPER e2fsck -n -f $TEST_MNT/ext2_saved/image
- run_check $SUDO_HELPER umount $TEST_MNT
+ run_check $SUDO_HELPER e2fsck -n -f "$TEST_MNT/ext2_saved/image"
+ run_check $SUDO_HELPER umount "$TEST_MNT"
- run_check $TOP/btrfs check $TEST_DEV
- run_check $TOP/btrfs-convert -r $TEST_DEV
- run_check e2fsck -n -f $TEST_DEV
+ run_check "$TOP/btrfs" check "$TEST_DEV"
+ run_check "$TOP/btrfs-convert" -r "$TEST_DEV"
+ run_check e2fsck -n -f "$TEST_DEV"
- rm -f $TEST_DEV
+ rm -f "$TEST_DEV"
}
check_all_images
diff --git a/tests/convert-tests/005-delete-all-rollback/test.sh
b/tests/convert-tests/005-delete-all-rollback/test.sh
index cf576e7..c6f7466 100755
--- a/tests/convert-tests/005-delete-all-rollback/test.sh
+++ b/tests/convert-tests/005-delete-all-rollback/test.sh
@@ -2,8 +2,8 @@
# create a base image, convert to btrfs, remove all files, rollback the ext4
image
# note: ext4 only
-source $TOP/tests/common
-source $TOP/tests/common.convert
+source "$TOP/tests/common"
+source "$TOP/tests/common.convert"
Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror
Thanks for the example and details. I understood some and need to re-read couple of more times to understand the remaining. btw, I created a corruption framework(with previous org), the sample usage and example is below. It looks similar to Btrfs corruption tool. thanks. -- corrupt.py --help Usage: corrupt.py [options]...] Options: --version show program's version number and exit -h, --helpshow this help message and exit -t object, --object=object The object type to be corrupted [ choose from (inode, l_inner,l_master,l_leaf,fileinode_inner,fileinode_leaf,dirinode_in ner,dirinode_leaf,data_block,fec_block,baddr) default: inode] -p /fs/path, --path=/fs/path The file or directory path in /fs -q optype, --optype=optype The corruption operation on the object [choose from (set,xor,add) default: set] -o offset, --offset=offset The valid offset value of the object [default:0] -m mirror, --mirror=mirror The mirror number [default: 1] -s size, --size=size The offset field/byte to be corrupted [default: 1] -c fixcrc, --fixcrc=fixcrc This fixes the CRC's of the corrupted objects -n lbn, --lbn=lbn The lbn of the data block to be corrupted -b baddr, --baddr=baddr The baddr to be corrupted --log=LOG_FILEThis logs the result into the log file Example usage: corrupt.py --object=inode --offset=0 --size=4 --path=/fs/path/to/file.txt Will corrupt first 4 bytes of inode structure. corrupt.py --object=dirinode_inner --offset=0 --size=4 --path=/fs/path/to/file.txt Will corrupt first 4 bytes of directory-inode structure. corrupt.py --object=dirinode_leaf --offset=0 --size=4 --path=/fs/path/to/file.txt Will corrupt first 4 bytes of directory-inode leaf structure. (file.txt should be large enough to contain leafs). corrupt.py --object=inode --offset=0 --size=4 --path=/fs/path/to/file.txt --mirror=2 Will corrupt first 4 bytes of inode structure that reside on Node/drive-2. --optype=optype ==> default is set. It sets '0' to specified offset. add => Will try given value to existing value. For example. offset=(nlink's offset) --optype=add will increase the nlink count by 1. (running fsck, expected to find this and fix it) where as optype xor => should flip the bits at given offset. -- Cheers, Lakshmipathi.G FOSS Programmer. http://www.giis.co.in http://www.webminal.org On Thu, Apr 20, 2017 at 6:30 AM, Qu Wenruo wrote: > > > > At 04/19/2017 03:46 PM, Lakshmipathi.G wrote: >> >> Thanks. This should make easier to create corruption test scripts. >> >> when using --root-ino-offset 5,257,0 Does final entry 0 has to be default >> value ? Does it represent any value at >> this moment? > > > For case 5,257,0, it means the start point of the corruption is root 5, inode > 257 and file offset 0. > > So let's take the following file layout as an example (a little complex one, > to show off all the ability) > > Root 5 inode 257: > >\/ This is file offset > Extent 0: disk bytenr X, disk len 4K, nr_bytes 4K offset 0 plain > Extent 4K: disk bytenr Y, disk len 8K, nr_bytes 4K offset 0 plain > Extent 8K: disk bytenr Z, disk len 8K, nr_bytes 4K offset 4K plain > > And if you specified --root-ino-offset 5,257,0 and --length 12288 > > Then the following on-disk range will be corrupted: > [X, X+4K) > [Y, Y+4K) > [Z+4K, Z+8K) > > Making all the 0~12K of root5, inode257 get corrupted, while unrelated ranges > like [Z, Z+4K) and [Y+4K, Y+8K) not modified. > > While for RAID1/RAID10/DUP usage, just create a file on a new fs, then pass > --root-ino-offset 5,257,0 with --length will corrupt the whole > file for you, allowing us to test recovery. > >> >> One more thing, does btrfs-modify command has option to print out existing >> parity value (all parity of a file or specific a parity)? >> something like get_parity option? > > > 1) For print parity value > I didn't see the need to print parity value. > As if you want to verify if the parity is good or not, just use offline scrub. > It will detect parity/data corruption. > > 2) For printing/corrupting value of a file in RAID56 profile > Not as easy as one may think. > > Think about the following case: > > Root 5 ino 257: > Extent 0: disk bytenr X(rounded to stripe boundary), disk len 32K, nr_bytes > 32K, offset 0. > Extent 1: disk bytenr X+32K, disk len 64K, nr_bytes 64K, offset 0 > > On disk it will be: > > |Dev 1 | Dev 2| Dev 3 | > X X+64K X+128K X+192K &g
Re: [PATCH 9/9] btrfs-progs: modify: Introduce option to specify the pattern to fill mirror
Nice. With this new option, it should be possible to achieve something
like RAID56 corruption test-script say,
https://patchwork.kernel.org/patch/9583455/ right?
Cheers,
Lakshmipathi.G
FOSS Programmer.
http://www.giis.co.in http://www.webminal.org
On Mon, Apr 17, 2017 at 8:56 AM, Qu Wenruo wrote:
> Introduce a new option, --pattern, to specify the pattern to fill
> mirror.
>
> Signed-off-by: Qu Wenruo
> ---
> Documentation/btrfs-modify.asciidoc | 5 +
> modify/mirror.c | 11 ++-
> 2 files changed, 15 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/btrfs-modify.asciidoc
> b/Documentation/btrfs-modify.asciidoc
> index ae2ada65..d82fb0df 100644
> --- a/Documentation/btrfs-modify.asciidoc
> +++ b/Documentation/btrfs-modify.asciidoc
> @@ -41,6 +41,11 @@ stripe number 1.
> P for RAID56 1st parity stripe. Only RAID5 and RAID6 support stripe number P.
> Q for RAID6 2nd parity stripe.
>
> +--pattern
> +Specify to pattern to modify the mirror data5f536652.
> ++
> +Default value is "0x66524842" (ascii "BHRf", part of the btrfs magic
> "_BHRfS_M").
> +
> EXIT STATUS
> ---
> *btrfs-modify* returns a zero exit status if all its operations succeed.
> diff --git a/modify/mirror.c b/modify/mirror.c
> index c89927f6..dd717fbe 100644
> --- a/modify/mirror.c
> +++ b/modify/mirror.c
> @@ -373,13 +373,15 @@ int modify_mirror(int argc, char **argv)
> char *device;
> u64 length = (u64)-1;
> u64 logical = (u64)-1;
> + u32 pattern = 0x66524842;
> int stripe = STRIPE_UNINITILIZED;
> int ret;
>
> while (1) {
> int c;
> enum { GETOPT_VAL_LOGICAL = 257, GETOPT_VAL_LENGTH,
> - GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET };
> + GETOPT_VAL_STRIPE, GETOPT_VAL_ROOT_INO_OFFSET,
> + GETOPT_VAL_PATTERN };
> static const struct option long_options[] = {
> { "logical", required_argument, NULL,
> GETOPT_VAL_LOGICAL },
> @@ -388,6 +390,8 @@ int modify_mirror(int argc, char **argv)
> { "stripe", required_argument, NULL,
> GETOPT_VAL_STRIPE },
> { "root-ino-offset", required_argument, NULL,
> GETOPT_VAL_ROOT_INO_OFFSET},
> + { "pattern", required_argument, NULL,
> + GETOPT_VAL_PATTERN},
> { NULL, 0, NULL, 0 }
> };
>
> @@ -407,6 +411,9 @@ int modify_mirror(int argc, char **argv)
> case GETOPT_VAL_ROOT_INO_OFFSET:
> parse_root_ino_offset(&dest, optarg);
> break;
> + case GETOPT_VAL_PATTERN:
> + pattern = arg_strtou32(optarg);
> + break;
> case '?':
> case 'h':
> usage(modify_mirror_usage);
> @@ -439,6 +446,8 @@ int modify_mirror(int argc, char **argv)
> stripe = 0;
> }
>
> + memset(write_buf, pattern, sizeof(write_buf) / sizeof(pattern));
> +
> fs_info = open_ctree_fs_info(device, 0, 0, 0, OPEN_CTREE_WRITES);
> if (!fs_info) {
> error("failed to open btrfs on device %s\n", device);
> --
> 2.12.2
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: cli-tests: Convert non-raid filesystem to raid
On Mon, Mar 13, 2017 at 06:47:50PM +0100, David Sterba wrote: > On Sun, Jan 29, 2017 at 08:14:55PM +0530, Lakshmipathi.G wrote: > > Simple script to verify non-raid filesystem conversion. > > The cli (command line interface) tests are supposed to cover the common > usecases from the point of option combinations etc, not really verifying > the result. It would be good if you add more than just one simple test, > especially for the balance filters. Adding the helpers to prepare the > filesystem (number of devices, raid profiles) would be a good start. > Some code for that is in the mkfs tests, but this belongs to the common > scripts so we could easily use that. Ok. I'll try to add helpers to common. So that option like this can be re-used easily. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3] btrfs-progs: misc-tests: Superblock corruption and recovery using backup.
On Mon, Mar 13, 2017 at 06:15:04PM +0100, David Sterba wrote:
> On Fri, Feb 17, 2017 at 04:01:59AM +0530, Lakshmipathi.G wrote:
> > Test script to recover damaged primary superblock using backup superblock.
> >
> > Signed-off-by: Lakshmipathi.G
>
> Thanks for the test, a few comments below.
>
> > ---
> > .../019-fix-superblock-corruption/test.sh | 36
> > ++
> > 1 file changed, 36 insertions(+)
> > create mode 100755 tests/misc-tests/019-fix-superblock-corruption/test.sh
> >
> > diff --git a/tests/misc-tests/019-fix-superblock-corruption/test.sh
> > b/tests/misc-tests/019-fix-superblock-corruption/test.sh
> > new file mode 100755
> > index 000..95815e4
> > --- /dev/null
> > +++ b/tests/misc-tests/019-fix-superblock-corruption/test.sh
> > @@ -0,0 +1,36 @@
> > +#!/bin/bash
> > +#
> > +# Corrupt primary superblock and restore it using backup superblock.
> > +#
> > +
> > +source $TOP/tests/common
> > +
> > +check_prereq btrfs-select-super
> > +check_prereq btrfs
> > +
> > +setup_root_helper
> > +prepare_test_dev 512M
> > +
> > +FIRST_SUPERBLOCK_OFFSET=65536
> > +
> > +test_superblock_restore()
> > +{
> > + run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $TEST_DEV
> > +
> > + # Corrupt superblock checksum
> > +dd if=/dev/zero of=$TEST_DEV seek=$FIRST_SUPERBLOCK_OFFSET bs=1 \
> > +count=4 conv=notrunc &> /dev/null
> > + # Run btrfs check to detect corruption
> > + $TOP/btrfs check $TEST_DEV >& /dev/null && \
> > + _fail "btrfs check should detect corruption"
>
> No run_check? In general all test output is desired and should be in the
> test log so you can use 'run_mayfail'. Also, please keep the entier
> output and don't redirect it to /dev/null .
>
> > + # Copy backup superblock to primary
> > + run_check $TOP/btrfs-select-super -s 1 $TEST_DEV
> > +
> > + echo "Performing btrfs check" &>> $RESULTS
>
> Use _log for that
>
> > + $TOP/btrfs check $TEST_DEV &>> $RESULTS
>
> Use run_check
>
> > +if [ $? -ne 0 ]; then
> > + _fail "Failed to fix superblock."
>
> And this would be unnecessary.
>
> I've added a section to tests/README.md to describe the common coding
> practices. Feel free to read/update/fix. Older tests could violate the
> recommendations so at least new tests are in line and we'll update the
> rest incrementally.
Just sent a new patch(v4), addressing above review comments.thanks.
Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4] btrfs-progs: misc-tests: Superblock corruption and recovery using backup
Signed-off-by: Lakshmipathi.G
---
.../020-fix-superblock-corruption/test.sh | 35 ++
1 file changed, 35 insertions(+)
create mode 100755 tests/misc-tests/020-fix-superblock-corruption/test.sh
diff --git a/tests/misc-tests/020-fix-superblock-corruption/test.sh
b/tests/misc-tests/020-fix-superblock-corruption/test.sh
new file mode 100755
index 000..0faebad
--- /dev/null
+++ b/tests/misc-tests/020-fix-superblock-corruption/test.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+#
+# Corrupt primary superblock and restore it using backup superblock.
+#
+
+source $TOP/tests/common
+
+check_prereq btrfs-select-super
+check_prereq btrfs
+
+setup_root_helper
+prepare_test_dev
+
+FIRST_SUPERBLOCK_OFFSET=65536
+
+test_superblock_restore()
+{
+ run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$TEST_DEV"
+
+ # Corrupt superblock checksum
+run_check $SUDO_HELPER dd if=/dev/zero of="$TEST_DEV" \
+ seek="$FIRST_SUPERBLOCK_OFFSET" bs=1 count=4 conv=notrunc
+
+ # Run btrfs check to detect corruption
+ run_mayfail "$TOP/btrfs" check "$TEST_DEV" && \
+ _fail "btrfs check should detect corruption"
+
+ # Copy backup superblock to primary
+ run_check "$TOP/btrfs-select-super" -s 1 "$TEST_DEV"
+
+ #Performing btrfs check
+ run_check "$TOP/btrfs" check "$TEST_DEV"
+}
+
+test_superblock_restore
--
2.7.4
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v4] btrfs-progs: fsck-tests: verify 'btrfs check --repair' fixes corrupted nlink field
commit 5cdcc57b8d599f03fe692e0285ee9e17f32267fa
Author: Lakshmipathi.G
Date: Sat Apr 15 14:26:20 2017 +0530
btrfs-progs: fsck-tests: verify 'btrfs check --repair' fixes corrupted
nlink field
Signed-off-by: Lakshmipathi.G
diff --git a/tests/fsck-tests/026-check-inode-link/test.sh
b/tests/fsck-tests/026-check-inode-link/test.sh
new file mode 100755
index 000..3a7a3c1
--- /dev/null
+++ b/tests/fsck-tests/026-check-inode-link/test.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+# verify that 'btrfs check --repair' fixes corrupted inode nlink field
+
+source $TOP/tests/common
+
+check_prereq btrfs-corrupt-block
+check_prereq mkfs.btrfs
+
+setup_root_helper
+prepare_test_dev
+
+test_inode_nlink_field()
+{
+ run_check $SUDO_HELPER "$TOP/mkfs.btrfs" -f "$TEST_DEV"
+
+ run_check_mount_test_dev
+ run_check $SUDO_HELPER touch "$TEST_MNT/test_nlink.txt"
+
+ # find inode_number
+ inode_number=`stat -c%i "$TEST_MNT/test_nlink.txt"`
+ run_check_umount_test_dev
+
+ # corrupt nlink field of inode object
+run_check $SUDO_HELPER "$TOP/btrfs-corrupt-block" -i "$inode_number" \
+ -f nlink "$TEST_DEV"
+
+ check_image "$TEST_DEV"
+
+ # verify link count
+ run_check_mount_test_dev
+ link_count=`stat -c%h "$TEST_MNT/test_nlink.txt"`
+ run_check_umount_test_dev
+ if [ "$link_count" != 1 ];then
+ _fail "btrfs check --repair: Unable to fix nlink corruption."
+ fi
+}
+
+test_inode_nlink_field
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v3] btrfs-progs: fsck-tests: verify 'btrfs check --repair' fixes corrupted nlink field
On Thu, Mar 30, 2017 at 05:52:11PM +0200, David Sterba wrote:
> On Thu, Feb 02, 2017 at 01:35:02PM +0530, Lakshmipathi.G wrote:
> > Signed-off-by: Lakshmipathi.G
> > ---
> > tests/fsck-tests/026-check-inode-link/test.sh | 30
> > +++
> > 1 file changed, 30 insertions(+)
> > create mode 100755 tests/fsck-tests/026-check-inode-link/test.sh
> >
> > diff --git a/tests/fsck-tests/026-check-inode-link/test.sh
> > b/tests/fsck-tests/026-check-inode-link/test.sh
> > new file mode 100755
> > index 000..6822ee2
> > --- /dev/null
> > +++ b/tests/fsck-tests/026-check-inode-link/test.sh
> > @@ -0,0 +1,30 @@
> > +#!/bin/bash
> > +# verify that 'btrfs check --repair' fixes corrupted inode nlink field
> > +
> > +source $TOP/tests/common
> > +
> > +check_prereq btrfs-corrupt-block
> > +check_prereq mkfs.btrfs
> > +
> > +setup_root_helper
> > +prepare_test_dev 512M
>
> Please use default size unless you really need 512M for the test
> purposes.
>
> > +
> > +test_inode_nlink_field()
> > +{
> > + run_check $SUDO_HELPER $TOP/mkfs.btrfs -f $TEST_DEV
> > +
> > + run_check_mount_test_dev
> > + run_check $SUDO_HELPER touch $TEST_MNT/test_nlink.txt
> > +
> > + # find inode_number
> > + inode_number=`stat -c%i $TEST_MNT/test_nlink.txt`
> > + run_check_umount_test_dev
> > +
> > + # corrupt nlink field of inode object
> > +run_check $SUDO_HELPER $TOP/btrfs-corrupt-block -i $inode_number \
> > + -f nlink $TEST_DEV
> > +
> > + check_image $TEST_DEV
>
> The wrapper runs a pre-check that must detect errors, then runs --repair
> and then again a plain check. Is this intended here?
>
Yes its intentional. The test should fail, if its unable to detect the
corruption.
This is to ensure btrfs-corrupt-block -f nlink works as expected.
> I think we should try to mount the fixed image again and read the file
> or check link count.
>
Right, I missed the verification part, will added this and change
prepare_test_dev
to default size.
> Please add shell quotation around all variables.
Now quotation is added.
Cheers.
Lakshmipathi.G
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: List of corruption cases for scrub
On Mon, Feb 27, 2017 at 11:40:31AM +0800, Qu Wenruo wrote: > > > At 02/24/2017 10:32 AM, Lakshmipathi.G wrote: > >Hi. > > > >I tried to a create list of corruption test scenarios for scrubbing process > >with RAID5. > >Here's the list: > >https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases > > > >If similar list already exists in btrfs wiki, let me know, I'll move this > >data > >to there. Feel free to add/update/delete entries or suggest new corrupt > >ideas! > > What about adding testing result for latest kernel release? > > Thanks, > Qu True, Ideally, the wiki should have list-corruption cases along with result for each latest kernel version. Planning to add more cases, gradually. For example, I think inode object with RAID5 can be replicated across drives. If thats correct, can scrub fix those corruption too? cases like this may be more important than other to-do items like raid5 with features like snapshot/quota etc I think it should be possible to add test-results for few cases but list has repeated-cases with different sizes,drives etc for these cases we need some kind of automated scripts. Hope to extend the cases first and then start updating results with manual or automated tests. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: List of corruption cases for scrub
On Fri, Feb 24, 2017 at 06:39:20PM +0100, Goffredo Baroncelli wrote: > Hi, > On 2017-02-24 03:32, Lakshmipathi.G wrote: > > Hi. > > > > I tried to a create list of corruption test scenarios for scrubbing process > > with RAID5. > > Here's the list: > > https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases > > > > If similar list already exists in btrfs wiki, let me know, I'll move this > > data > > to there. Feel free to add/update/delete entries or suggest new corrupt > > ideas! > > Thanks for doing that. About the test "Device-corruption", I suppose that you > want to test a scenario where one disk fails. If so you do not have to always > corrupt "D0" block, but the first block of the file which is stored in the > failing disk (which could be different from the D0). > Yes, that's right. Updated (Device-corruption) the list now. > Another scenario which should be tested, is the reshaping of the raid: add a > drive (or remove a drive) then re-balance the system (with or without some > corruption). IIRC this use the same scrub code. > Added this as 'Rebalance-corruption' on the wiki. thanks! Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
List of corruption cases for scrub
Hi. I tried to a create list of corruption test scenarios for scrubbing process with RAID5. Here's the list: https://btrfs.wiki.kernel.org/index.php/Scrub_corruption_cases If similar list already exists in btrfs wiki, let me know, I'll move this data to there. Feel free to add/update/delete entries or suggest new corrupt ideas! Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large no.of extents for a single small file with a fresh RAID5 setup
> > >I'm using Ubuntu(16.04) desktop version (not server) running Xorg and > >others. May be its possible its > >flushing data to disk at constant time. If you like to give it a try again > >with servers, below is the > >exact script along with its timing & multiextent output. > > Archlinux here, with my btrfs vm and browser and music player, still one > extent. > looks like I have better setup to produce single-extent files than yours :-) > Are you under heavy memory usage? > no, not as per top and free. 'top' shows mem.usage as: KiB Mem : 7092516 total, 5961796 free, 794836 used, 335884 buff/cache where 'compiz' process uses around 2.5% of memory and 'free' command reports almost 5gb as free. May be some 'sysctl' or others needs to be adjusted on my setup? > > > >one more curious thing, is it fine to have extents with size 4096 on RAID5 > >setup? > > > > Sure, why not? > > Any size aligned to sector size is valid, no matter the profile. > > For 4K extent, RAID5/6 will do read-modify-write, which is very common. > > Thanks, > Qu > While looking into mapping from logical->physical chunks on RAID5. I was wondering how these 4KB extents will fit into 64KB data-stripe? I didn't have idea about read-modify-write, just found few wiki links on it, will check. thanks Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] btrfs-progs: RAID5:Inject data stripe corruption and verify scrub fixes it.
> > > >Looked into patch description: > > > >After scrubbing dev3 only: > >0xcdcd (Good) | 0xcdcd | 0xcdcd (Bad) > >(D1) (D2)(P) > > > >So the Parity stripe (P) always get replaced by exact content of D1/D2 > >(data-stripe) > >or by random data? > > Neither. it's just XOR result of D2(never changed, 0xcdcd) and old D1(wrong, > 0x). > 0xcdcd XOR 0x = 0xcdcd > > So you got 0xcdcd, bad result. > > If you corrupt D1 with random data, then parity will be random then. > > >If it always get replaced by exact value from either > >D1 or D2. I think current script can be modified to detect that bug. If > >parity gets > >replaced by random value, then it will the make task more difficult. > > Not hard to detect. > As the content is completely under your control, you know the correct parity > value, and you can verify it very easy. > version-3 of this script calculates exact data/parity location, instead of dumping data and searching locations. Tested with upto 8MB file, from the output all 128 data-stripes and 64 parity stripe location seems fine. It constantly hit the parity bug with the script. If the script gets accepted, will add slightly other corruption variants likes: - corrupt all even stripe (D2,D4..) - corrupt all odd stripe (D1,D3..) - corrupt all parity stripes - corrupt all both data stripe (D0 & D1) and expect error message (Cover above cases for RAID6) thanks. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Large no.of extents for a single small file with a fresh RAID5 setup
> >>How did you create the 2m file?
> >
> >Yes,this is the problem. script invokes 'dd' with 'notrunc' for each
> >byte. If the file is 1kb, 'dd' invoked 1024 times with notrunc.
> >This seems to be creating the issue.
>
> IIRC that's not the direct cause though.
>
> Since kernel is using delayed allocation, each time we trigger buffered
> write, kernel just info fs to do accounting and copy the data into page
> cache, no real write is triggered.
>
> Only sync/fsync and memory pressure will make us to write pages into disc,
> and until then we allocate space for them.
>
> So unless you're trying such operation on a busy server which triggered
> several sync/fsync/memory pressure during the several seconds of dd, it
> won't cause so many fragments.
>
> I also tried your dd bs=1 method, no problem and still one single extent.
>
> Thanks,
> Qu
I'm using Ubuntu(16.04) desktop version (not server) running Xorg and others.
May be its possible its
flushing data to disk at constant time. If you like to give it a try again with
servers, below is the
exact script along with its timing & multiextent output.
one more curious thing, is it fine to have extents with size 4096 on RAID5
setup?
Cheers.
Lakshmipathi.G
#time /root/check_extents.sh f2MB 32
real1m30.284s
user1m22.796s
sys 0m39.324s
laks/btrfs-progs# ./btrfs-debugfs -f tests/mnt/f2MB
(263 0): ram 110592 disk 151015424 disk_size 114688
(263 110592): ram 4096 disk 147103744 disk_size 4096
(263 114688): ram 110592 disk 151130112 disk_size 114688
(263 225280): ram 696320 disk 151257088 disk_size 700416
(263 921600): ram 4096 disk 147107840 disk_size 4096
(263 925696): ram 700416 disk 152043520 disk_size 704512
(263 1626112): ram 4096 disk 147111936 disk_size 4096
(263 1630208): ram 344064 disk 153223168 disk_size 348160
(263 1974272): ram 4096 disk 147116032 disk_size 4096
(263 1978368): ram 118784 disk 152748032 disk_size 118784
file: tests/mnt/f2MB extents 10 disk size 2117632 logical size 2097152 ratio
0.99
--
$cat /root/check_extents.sh
#!/bin/bash
#$1 Filename
#$2 Expected no.of data stripes for the file.
create_layout(){
fname=$1
size=$(( $2 * 65536 ))
n=0
bs_value=1
stripe=0
while (( $n < $size ))
do
if [ $(( $n % 65536 )) -eq 0 ]; then
val='D'$stripe
echo -n $val
stripe=$(( $stripe+1 ))
# ensure proper value
bs_value=`echo "${#val}"`
else
echo -n 'x'
bs_value=1
fi
n=$(( $n+$bs_value ))
done | dd of="/home/laks/btrfs-progs/tests/mnt/$fname" bs=$bs_value
conv=notrunc &> /dev/null
##EDIT above hard-coded path
}
create_layout $1 $2
--
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[PATCH v3] btrfs-progs: RAID5: Inject data stripe corruption and verify scrub fixes it and retains correct parity.
Test script to create file with specific data-stripe layout.Computes stripe
locations.
Inject corruption into data-stripe and verify scrubbing process fixes corrupted
block.
It also validates the corresponding parity stripe.
Signed-off-by: Lakshmipathi.G
---
.../test.sh| 402 +
1 file changed, 402 insertions(+)
create mode 100755
tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
diff --git
a/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
new file mode 100755
index 000..0129a75
--- /dev/null
+++ b/tests/misc-tests/020-raid5-datastripe-corruption-parity-check/test.sh
@@ -0,0 +1,402 @@
+#!/bin/bash
+#
+# Raid5: Inject data stripe corruption and fix them using scrub.
+#
+# Script will perform the following:
+# 1) Create Raid5 using 3 loopback devices.
+# 2) Ensure file layout is created in a predictable manner.
+#Each data stripe(64KB) should uniquely start with 'DN',
+#where N represents the data stripe number.(ex:D0,D1 etc)
+# 3) Once file is created with specific layout, check whether file
+#has single extent. At the moment, script wont handle multi-extent
+#files.
+# 4) If file has single extent with the help of 'dump-tree' compute data and
+#parity stripe details like devicename, position and actual on-disk data.
+# 5) $STRIPEINFO_COMPLETE file will have all necessary data at this stage.
+# 6) Inject corruption into given data-stripe by zero'ing out its first 4
bytes.
+# 7) After injecting corruption, running online-scrub is expected to fix
+#the corrupted data stripe with the help of parity block and
+#corresponding data stripe.
+# 8) If scrub successful, verify the data stripe has original un-corrupted
value.
+# 9) If scrub successful, verify parity stripe is valid, otherwise its a
parity bug.
+# 10) If no issues found, cleanup files and devices. Repeat the process for
+#different file size and data-stripe.
+#
+# Note: This script corrupts only data-stripe blocks.
+# Known Limitations (will be addressed later):
+# - Script expects even number of data-stripes in file.
+# - Script expects the file to have single extent.
+
+source $TOP/tests/common
+
+check_prereq btrfs
+check_prereq mkfs.btrfs
+check_prereq btrfs-debugfs
+
+setup_root_helper
+prepare_test_dev 1024M
+
+ndevs=3
+declare -a devs
+declare -a parity_offsets
+stripe_entry=""
+device_name=""
+stripe_offset=""
+stripe_content=""
+
+#Complete stripe layout
+STRIPEINFO_COMPLETE=$(mktemp --tmpdir
btrfs-progs-raid5-stripe-complete.infoXX)
+#dump-tree output file
+DUMPTREE_OUTPUT=$(mktemp --tmpdir btrfs-progs-raid5-tree-dump.infoXX)
+#tmp files
+STRIPEINFO_PARTIAL=$(mktemp --tmpdir
btrfs-progs-raid5-stripe-partial.infoXX)
+STRIPE_TMP=$(mktemp --tmpdir btrfs-progs-raid5-stripetmp.infoXX)
+MULTI_EXTENT_CHECK=$(mktemp --tmpdir btrfs-progs-raid5-extent-check.infoXX)
+EXTENT_WITH_SIZE=$(mktemp --tmpdir btrfs-progs-raid5-extent-size.infoXX)
+PARITY_LOC1=$(mktemp --tmpdir btrfs-progs-raid5-parity-loc1.infoXX)
+PARITY_LOCATION=$(mktemp --tmpdir
btrfs-progs-raid5-parity-locations.infoXX)
+
+
+prepare_devices()
+{
+ for i in `seq $ndevs`; do
+ touch img$i
+ chmod a+rw img$i
+ truncate -s0 img$i
+ truncate -s512M img$i
+ devs[$i]=`run_check_stdout $SUDO_HELPER losetup --find --show
img$i`
+ done
+ truncate -s0 $STRIPE_TMP
+ truncate -s0 $STRIPEINFO_PARTIAL
+ truncate -s0 $STRIPEINFO_COMPLETE
+}
+
+cleanup_devices()
+{
+ for dev in ${devs[@]}; do
+ run_check $SUDO_HELPER losetup -d $dev
+ done
+ for i in `seq $ndevs`; do
+ truncate -s0 img$i
+ done
+ run_check $SUDO_HELPER losetup --all
+}
+
+test_do_mkfs()
+{
+ run_check $SUDO_HELPER $TOP/mkfs.btrfs -f \
+ $@
+}
+
+test_mkfs_multi()
+{
+ test_do_mkfs $@ ${devs[@]}
+}
+
+#$1 Filename
+#$2 Expected no.of data stripes for the file.
+create_layout(){
+ fname=$1
+ size=$(( $2 * 65536 ))
+ n=0
+ bs_value=1
+ stripe=0
+ while (( $n < $size ))
+ do
+ if [ $(( $n % 65536 )) -eq 0 ]; then
+ val='D'$stripe
+ echo -n $val
+ stripe=$(( $stripe+1 ))
+ # ensure proper value
+ bs_value=`echo "${#val}"`
+ else
+ echo -n 'x'
+ bs_value=1
+ fi
+n=$(( $n+$bs_value ))
+ done | dd of=/tmp/$fname bs=$bs_value conv=notrunc &> /dev/null
+ cp /tmp/$fname $TEST_MNT
+ rm -f /t
Re: Large no.of extents for a single small file with a fresh RAID5 setup
On Mon, Feb 20, 2017 at 03:53:15PM +0800, Qu Wenruo wrote: > > > At 02/18/2017 10:07 AM, Lakshmipathi.G wrote: > >Hi, > > > >With RAID5, using 3 loop-back devices each has 4GB. On a fresh setup, > >created a 2mb file. > >While checking no.of extents,it shows like: > > > ># ./btrfs-debugfs -f /mnt/file2m.txt > >(257 0): ram 110592 disk 145227776 disk_size 114688 > >(257 110592): ram 4096 disk 145461248 disk_size 4096 > >(257 114688): ram 114688 disk 145342464 disk_size 118784 > >(257 229376): ram 712704 disk 145620992 disk_size 716800 > >(257 942080): ram 4096 disk 145096704 disk_size 4096 > >(257 946176): ram 737280 disk 146407424 disk_size 741376 > >(257 1683456): ram 413696 disk 147193856 disk_size 413696 > >file: /mnt/file2m.txt extents 7 disk size 2113536 logical size 2097152 ratio > >0.99 > > > >Is this expected behaviour? Is there any mount option to minimize no.of > >extents for > >a smaller file? > > What mount option did you use? > Are you using compression? Used the default mount options (mount /dev/loop0 /mnt) and compression not enabled. > How did you create the 2m file? Yes,this is the problem. script invokes 'dd' with 'notrunc' for each byte. If the file is 1kb, 'dd' invoked 1024 times with notrunc. This seems to be creating the issue. Tried using open-fd (exec FD<>/path/f, write finally close FD) method even this runs into possible sync and creates multiple extents. so finally found a simple trick, created the file outside btrfs then copy it into btrfs-mnt-pt :) this workaround helps to solve the script file-creation issue. > > This seems quite strange, if you go through normal buffered write without > compression, 2m can easily be embedded into one file extent. > > And according to extent at offset 0, 112K, 224K, its data extent is larger > than its file extent, which means it goes through CoW. > > Either you did fsync/sync between writes or you goes through Direct IO. > > Anyway, the way you create the 2M file is quite important here now. > > Thanks, > Qu > > thanks for the help. Cheers. Lakshmipathi.G -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
