Re: [lustre-discuss] lfsck namespace doesn't stop and I cancel it
Thank you Fan. It works!! Sorry for the question. This option is not documented in the lustre manual (pdf or html manuals), but I see it in the man pages of mount.lustre Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 26 jun 2019, a las 18:02, Yong, Fan escribió: > > If you have no other way to stop the lfsck, then you can try re-mount the MDT > with option "-o skip_lfsck". > > -- > Cheers, > Nasf > > -Original Message- > From: lustre-discuss [mailto:lustre-discuss-boun...@lists.lustre.org] On > Behalf Of Fernando Perez > Sent: Wednesday, June 26, 2019 7:59 PM > To: lustre-discuss@lists.lustre.org > Subject: [lustre-discuss] lfsck namespace doesn't stop and I cancel it > > Dear lustre users, > > I tried to run a lfsck namespace in our lustre filesystem, lustre 2.10.7, and > the lfsck never ends and I can't stop it using the lctl lfsck_stop command. > > I tried to kill the lfsck process, stop lustre, reboot the metadata server, > and when I mount the combined mdt + mgs the lfsck starts again automatically. > > How can I stop the lfsck namespace process? > > The commands that I runned are the following: > > Start the lfsck namespace: > > lctl lfsck_query -t namespace -M lustre-MDT > > Try to stop the lfsck because it never ends: > > lctl lfsck_stop -M lustre-MDT > > Regards. > > -- > = > Fernando Pérez > Institut de Ciències del Mar (CSIC) > Departament Oceanografía Física i Tecnològica Passeig Marítim de la > Barceloneta,37-49 > 08003 Barcelona > Phone: (+34) 93 230 96 35 > = > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Stop writes for users
Dear lustre experts, Is there a way to stop file writes for all users or for groups without using quotes? We have a lustre filesystem with corrupted quotes and I need to stop the write for all users (or for some users). Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] lfsck repair quota
> You can try... > unmount the filesystem and the run a force e2fsck on all OSTs and MDT. > > e2fsck -f -p /dev/ost... > > Regargs. Thank you Mahmoud. The problem is that e2fsck in the MDT runs very very slowly…10 inodes per second for more than 100 million inodes. >> Are there a lot of inodes moved to lost+found by the fsck, which contribute >> to the occupied quota now? Thank you Martin. I can’t finish the e2fsck in the MDT. > Correct, LFSCK will repair the OST object ownership, but does not *directly* > repair quota > files. That said, if an object is owned by the wrong owner for some reason, > changing the > ownership _should_ transfer the quota usage properly to the new owner, so in > that regard > the quota usage should be indirectly repaired by an LFSCK run. Thank you Andreas. I can confirm that lfsck layout does not repair the wrong quotes. I’m running lfsck namespace in the MDT to try other way. > Note the "tune2fs -O ^quota" to repair the quota accounting is a bit of a > "cargo cult" behaviour. > > Running "e2fsck -fp" is the proper way to repair the quota files, since it > not only recalculates the quota accounting using the same code as "tune2fs -O > quota" does, but it also ensures that the files themselves are valid in the > first place. They should take about the same time, except in the case your > filesystem is corrupted, in which case you'd want e2fsck to repair the > filesystem anyway. I ran this in the lfsck MDT and lfsck OSTs. After do that the quotes for all users were more corrupted than before (for example for a user with 27 TB the quota reports 900 MB). Then I ran the e2fsck in the lfsck OSTs. I tried to do the same in the lfsck MDT but it ran very very slow….and I canceled it. I suppose that the correct way to repair quota is to run the e2fsck in the MDT, but the problem is the system downtime. I copy the lfsck stats at the end of this message: > lctl get_param -n mdd.lustre-MDT.lfsck_layout > name: lfsck_layout > magic: 0xb17371b9 > version: 2 > status: partial > flags: > param: > last_completed_time: 1555393553 > time_since_last_completed: 50098 seconds > latest_start_time: 1555339300 > time_since_latest_start: 104351 seconds > last_checkpoint_time: 1555393553 > time_since_last_checkpoint: 50098 seconds > latest_start_position: 77 > last_checkpoint_position: 102983584 > first_failure_position: 21342296 > success_count: 1 > repaired_dangling: 2470024 > repaired_unmatched_pair: 7487028 > repaired_multiple_referenced: 0 > repaired_orphan: 0 > repaired_inconsistent_owner: 18950288 > repaired_others: 0 > skipped: 0 > failed_phase1: 1622485 > failed_phase2: 0 > checked_phase1: 119083557 > checked_phase2: 0 > run_time_phase1: 31777 seconds > run_time_phase2: 22423 seconds > average_speed_phase1: 3747 items/sec > average_speed_phase2: 0 objs/sec > real-time_speed_phase1: N/A > real-time_speed_phase2: N/A > current_position: N/A > lctl lfsck_query -t namespace -M lustre-MDT > namespace_mdts_init: 0 > namespace_mdts_scanning-phase1: 0 > namespace_mdts_scanning-phase2: 1 > namespace_mdts_completed: 0 > namespace_mdts_failed: 0 > namespace_mdts_stopped: 0 > namespace_mdts_paused: 0 > namespace_mdts_crashed: 0 > namespace_mdts_partial: 0 > namespace_mdts_co-failed: 0 > namespace_mdts_co-stopped: 0 > namespace_mdts_co-paused: 0 > namespace_mdts_unknown: 0 > namespace_osts_init: 0 > namespace_osts_scanning-phase1: 0 > namespace_osts_scanning-phase2: 0 > namespace_osts_completed: 0 > namespace_osts_failed: 0 > namespace_osts_stopped: 0 > namespace_osts_paused: 0 > namespace_osts_crashed: 0 > namespace_osts_partial: 0 > namespace_osts_co-failed: 0 > namespace_osts_co-stopped: 0 > namespace_osts_co-paused: 0 > namespace_osts_unknown: 0 > namespace_repaired: 4550034 = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 16 abr 2019, a las 20:43, Martin Hecht escribió: > > Are there a lot of inodes moved to lost+found by the fsck, which contribute > to the occupied quota now? > > - Ursprüngliche Mail - > Von: Fernando Pérez > An: lustre-discuss@lists.lustre.org > Gesendet: Tue, 16 Apr 2019 16:24:13 +0200 (CEST) > Betreff: Re: [lustre-discuss] lfsck repair quota > > Thank you Rick. > > I followed these steps for the ldiskfs OSTs and MDT, but the quotes for all > users is more corrupted than before. > > I tried to run e2fsck in ldiskfs OSTs MDT, but the prob
Re: [lustre-discuss] lfsck repair quota
Thank you Rick. Then, does anyone know how repair corrupted quotes? Regards. Fernando Pérez Institut de Ciències del Mar (CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 > El 16 abr 2019, a las 17:22, Mohr Jr, Richard Frank (Rick Mohr) > escribió: > > >> On Apr 16, 2019, at 10:24 AM, Fernando Pérez wrote: >> >> According to the lustre wiki I though that the lfsck could repair corrupted >> quotes: >> >> http://wiki.lustre.org/Lustre_Quota_Troubleshooting > > Keep in mind that page is a few years old, but I assume they were referring > to LFSCK Phase 2 > (http://wiki.lustre.org/LFSCK_Phase_2_-_MDT-OST_Consistency_Solution_Architecture) > which maintains consistency between the MDTs and OSTs. One of the things > that lfsck will do is make sure that the ownership info for a file’s MDT > object matches the ownership of the OST objects for that file. This is > necessary to ensure that quota information reported by Lustre is accurate, > but I don’t believe it is meant to fix any corruption in the quota files > themselves. > > -- > Rick Mohr > Senior HPC System Administrator > National Institute for Computational Sciences > http://www.nics.tennessee.edu > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] lfsck repair quota
Thank you Rick. I followed these steps for the ldiskfs OSTs and MDT, but the quotes for all users is more corrupted than before. I tried to run e2fsck in ldiskfs OSTs MDT, but the problem was the MDT e2fsck ran very slow ( 10 inodes per second for more than 100 million inodes). According to the lustre wiki I though that the lfsck could repair corrupted quotes: http://wiki.lustre.org/Lustre_Quota_Troubleshooting Regards. Fernando Pérez Institut de Ciències del Mar (CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 > El 16 abr 2019, a las 15:34, Mohr Jr, Richard Frank (Rick Mohr) > escribió: > > >> On Apr 15, 2019, at 10:54 AM, Fernando Perez wrote: >> >> Could anyone confirm me that the correct way to repair wrong quotes in a >> ldiskfs mdt is lctl lfsck_start -t layout -A? > > As far as I know, lfsck doesn’t repair quota info. It only fixes internal > consistency within Lustre. > > Whenever I have had to repair quotas, I just follow the procedure you did > (unmount everything, run “tune2fs -O ^quota ”, run “tune2fs -O quota > ”, and then remount). But all my systems used ldiskfs, so I don’t know > if the ZFS OSTs introduce any sort of complication. (Actually, I am not even > sure if/how you can regenerate quota info for ZFS.) > > -- > Rick Mohr > Senior HPC System Administrator > National Institute for Computational Sciences > http://www.nics.tennessee.edu > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Migrate MGS to ZFS
Dear lustre experts. Whats is the best way to migrate a MGS device to ZFS? Copy the CONFIGS/filesystem_name-* files from the old ldiskfs device to the new ZFS MGS device? Currently we have a combined MDT/MGT under ldiskfs with lustre 2.10.4. We want to upgrade to lustre 2.12.0 and then separate the combined MDT/MGT and migrate MDT and MGT to separate ZFS devices. Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] stop lustre operations for a moment
Dear all, Is there any way to stop all lustre operations for a moment to reboot this switch and avoid stop all jobs in our systems and unmount all clients? We have a problem with our core lustre switch and we need to reboot it. Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] OST doomed after e2fsck
I had the same problem in the past with 2.4 release. I solved the problem upgrading the e2fsprogs to the its latest release and running again e2fsck in the failed OST. Regards. Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 > El 31 may 2018, a las 5:36, Riccardo Veraldi > escribió: > > Hello, > > after a power outage I had one of my OSTs (total of 60) in an unhappy state. > > Lustre version 2.4.1 > > I ran then a FS check and here follows: > > e2fsck 1.42.7.wc1 (12-Apr-2013) > Pass 1: Checking inodes, blocks, and sizes > Pass 2: Checking directory structure > Pass 3: Checking directory connectivity > Pass 4: Checking reference counts > Unattached inode 25793 > Connect to /lost+found? yes > Inode 25793 ref count is 2, should be 1. Fix? yes > Unattached inode 29096 > Connect to /lost+found? yes > Inode 29096 ref count is 2, should be 1. Fix? yes > Unattached inode 29745 > Connect to /lost+found? yes > Inode 29745 ref count is 2, should be 1. Fix? yes > Unattached inode 29821 > Connect to /lost+found? yes > Inode 29821 ref count is 2, should be 1. Fix? yes > yPass 5: Checking group summary information > Inode bitmap differences: +23902 +29082 +29096 +29130 +29459 +29497 > -29530 +29552 +29566 +29596 +(29643--29644) +29655 +29668 +29675 +29696 > +29701 +29720 +29736 +29739 +29745 +29751 +29778 +29787 -29795 +29808 > +29821 > Fix? yes > Free inodes count wrong for group #70 (1, counted=0). > Fix? yes > Free inodes count wrong for group #76 (1, counted=0). > Fix? yes > Free inodes count wrong for group #90 (1, counted=0). > Fix? yes > Free inodes count wrong for group #93 (3, counted=2). > Fix? yes > Free inodes count wrong for group #100 (2, counted=0). > Fix? yes > Free inodes count wrong for group #101 (1, counted=0). > Fix? yes > Free inodes count wrong for group #113 (5, counted=2). > Fix? yes > Free inodes count wrong for group #114 (1, counted=0). > Fix? yes > Free inodes count wrong for group #115 (13, counted=4). > Fix? yes > Free inodes count wrong for group #116 (149, counted=140). > Fix? yes > Free inodes count wrong (30493545, counted=30493516). > Fix? yes > [QUOTA WARNING] Usage inconsistent for ID 0:actual (2083721216, 841) != > expected (2082398208, 678) > [QUOTA WARNING] Usage inconsistent for ID 9997:actual (1095815659520, > 19800) != expected (664375967744, 19791) > [QUOTA WARNING] Usage inconsistent for ID -1597706240:actual (0, 0) != > expected (90112, 1) > [QUOTA WARNING] Usage inconsistent for ID -1428439040:actual (0, 0) != > expected (126976, 1) > [QUOTA WARNING] Usage inconsistent for ID -1936064512:actual (0, 0) != > expected (12288, 1) > [QUOTA WARNING] Usage inconsistent for ID -1684783104:actual (0, 0) != > expected (28672, 1) > [QUOTA WARNING] Usage inconsistent for ID -2131947520:actual (0, 0) != > expected (4096, 1) > [QUOTA WARNING] Usage inconsistent for ID 963263424:actual (957718528, > 49) != expected (957628416, 48) > [QUOTA WARNING] Usage inconsistent for ID 987173056:actual (1364516864, > 158) != expected (1364426752, 157) > [QUOTA WARNING] Usage inconsistent for ID -1537871872:actual (0, 0) != > expected (73728, 1) > [QUOTA WARNING] Usage inconsistent for ID -2105077760:actual (0, 0) != > expected (49152, 1) > [QUOTA WARNING] Usage inconsistent for ID -2145202176:actual (0, 0) != > expected (24576, 1) > [QUOTA WARNING] Usage inconsistent for ID -1422704640:actual (0, 0) != > expected (65536, 1) > Update quota info for quota type 0? yes > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (472). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (507). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (170). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (435). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (89). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (5). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (130). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it should (435). > [ERROR] quotaio_tree.c:357:free_dqentry:: Quota structure has offset to > other block (0) than it shou
Re: [lustre-discuss] Migrate mdt ldiskfs to zfs
Sorry, I have already seen in the lustre documentation that it’s possible migrate between ldiskfs and zfs since the 2.11 lustre release. I appreciate the stability and I think that the 2.10.4 release is the best option for our systems. Any experiencies using ldiskfs under zvol? Do you think that it's better than ldiskfs under mdadm + lvm? Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 26 may 2018, a las 16:12, Fernando Pérez <fpe...@icm.csic.es> escribió: > > Dear all, > > We have a lustre filesystem recently migrated from 2.8 to 2.10.4 release with > a combined mdt + mgs under ldiskfs. I want to migrate it to a new external > jbod with ssds. > > Is it possible to migrate it to zfs? > > I think that the best two options are the following: > > 1. Migrate it to zfs, but I think that it was not possible in lustre previous > releases > 2. Migrate it to ldiskfs under a zfs pool. > > What is the best way? > > Thank you very much in advance. > > Regards. > > = > Fernando Pérez > Institut de Ciències del Mar (CMIMA-CSIC) > Departament Oceanografía Física i Tecnològica > Passeig Marítim de la Barceloneta,37-49 > 08003 Barcelona > Phone: (+34) 93 230 96 35 > = > > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Migrate mdt ldiskfs to zfs
Dear all, We have a lustre filesystem recently migrated from 2.8 to 2.10.4 release with a combined mdt + mgs under ldiskfs. I want to migrate it to a new external jbod with ssds. Is it possible to migrate it to zfs? I think that the best two options are the following: 1. Migrate it to zfs, but I think that it was not possible in lustre previous releases 2. Migrate it to ldiskfs under a zfs pool. What is the best way? Thank you very much in advance. Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Do 2.9 Client and 2.8 Server play nice
I have had problems with clients 2.9 against 2.8 servers. Please check this: https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg13225.html <https://www.mail-archive.com/lustre-discuss@lists.lustre.org/msg13225.html> Regards. P.D.: Sorry, I could not open a JIRA ticket because I want to do more testing in our systems before. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 24 ene 2017, a las 17:35, E.S. Rosenberg <esr+lus...@mail.hebrew.edu> > escribió: > > Hi everyone, > > In the past 2.8 Client + 2.7 (or maybe it was 2.5.3) server gave us issues > (servers crashing). > > We would like to be able to move beyond kernel 4.2 on our clients but doing a > server upgrade is not really what I want to do right now. > > Does anyone know/is anyone running 2.9.0 clients against 2.8.0 servers? > Thanks, > Eli > ___ > lustre-discuss mailing list > lustre-discuss@lists.lustre.org > http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] Permanently remove osts: tunefs writeconf on zfs osts
Dear all. I have a lustre filesystem with one combined mdt / mgs, some old ldiskfs osts and other new zfs osts. I need to remove some old ldiskfs osts. Currently these osts are empty (I deactivated these osts and then I migrated all the files before). What is the correct way to permanent deactivate and remove these osts? Deactivate them using lctl and then regenerate lustre configuration files logs using tunefs according to chapter 14 in the lustre documentation? My question is due to the zfs osts, Do I need to run tunefs.lustre —writeconf on these zfs osts? Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
[lustre-discuss] lustre 2.8 problem: lfs_migrate runs very very slow
Hi. I have migrated our lustre filesystem from 2.4 to 2.8 release two weeks ago and all works very well. Now I need to migrate one OST due to hardware problems. I’m using lfs_migrate and it runs very very slow. There is no problems in the OSTs. I try to copy in and outside the osts files and all works very fast. I used lfs_migrate several times in the past and it always runned very fast. According to the manual seems that the current lfs_migrate release runs in background inside the lustre filesystem, because the files are not copied to the clients that executed it, and in the past releases, for example 2.4,it did a simple rsync. What could be the problem? Thanks in advance. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] lustre-discuss Digest, Vol 122, Issue 9
Hi Andreas. The latest that I have seen in the lustre repository: 1.42.13.wc4-7 Regards = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 7 may 2016, a las 0:02, Dilger, Andreas <andreas.dil...@intel.com> > escribió: > > On 2016/05/06, 15:48, "lustre-discuss on behalf of Fernando Pérez" > <lustre-discuss-boun...@lists.lustre.org > <mailto:lustre-discuss-boun...@lists.lustre.org> on behalf of > fpe...@icm.csic.es <mailto:fpe...@icm.csic.es>> > wrote: > >> Thank you Mark. >> >> Finally I have killed the e2fsck. After restart again our lustre >> filesystem it seems all works OK. >> >> We are using two 300 GB RAID 1 10K SAS drives for the combined mdt / mgs. >> >> I tried to run the e2fsck -fy because the -fn finish in 2 hoursŠI think >> there is a problem in the latest e2fsprogs because the e2fsck returned >> that it was repairing more inodes than our filesystem has. > > Which specific version of e2fsprogs are you using? > > Cheers, Andreas > >> >> Regards. >> = >> Fernando Pérez >> Institut de Ciències del Mar (CMIMA-CSIC) >> Departament Oceanografía Física i Tecnològica >> Passeig Marítim de la Barceloneta,37-49 >> 08003 Barcelona >> Phone: (+34) 93 230 96 35 >> = >> >>> El 6 may 2016, a las 17:57, Mark Hahn <h...@mcmaster.ca> escribió: >>> >>>> More information about our lustre system: combined mds / mdt has 189 >>>> GB and >>>> 8.9 GB used. It was formatted with the default options. >>> >>> fsck time is more about the number of files (inodes), rather than >>> the size. but either you have quite slow storage, or something is >>> wrong. >>> >>> as a comparison point, I can do a full/force fsck on one of our MDS/MDT >>> that has 143G or 3.3T in use (313M inodes) in about 2 hours. it is a MD >>> raid10 on 16x 10K SAS drives, admittedly. >>> >>> if your hardware is conventional (locally-attached multi-disk RAID), >>> it might make sense to look at its configuration. for instance, fsck >>> is largely seek-limited, but doing too much readahead, or using large >>> RAID block sizes (for R5/6) can be disadvantageous. having plenty of >>> RAM helps in some phases. >>> >>> regards, mark hahn. >> >> ___ >> lustre-discuss mailing list >> lustre-discuss@lists.lustre.org <mailto:lustre-discuss@lists.lustre.org> >> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org >> <http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org> >> > > > Cheers, Andreas > -- > Andreas Dilger > > Lustre Principal Architect > Intel High Performance Data Division ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] lustre-discuss Digest, Vol 122, Issue 9
Thank you Mark. Finally I have killed the e2fsck. After restart again our lustre filesystem it seems all works OK. We are using two 300 GB RAID 1 10K SAS drives for the combined mdt / mgs. I tried to run the e2fsck -fy because the -fn finish in 2 hours…I think there is a problem in the latest e2fsprogs because the e2fsck returned that it was repairing more inodes than our filesystem has. Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 6 may 2016, a las 17:57, Mark Hahn <h...@mcmaster.ca> escribió: > >> More information about our lustre system: combined mds / mdt has 189 GB and >> 8.9 GB used. It was formatted with the default options. > > fsck time is more about the number of files (inodes), rather than > the size. but either you have quite slow storage, or something is wrong. > > as a comparison point, I can do a full/force fsck on one of our MDS/MDT > that has 143G or 3.3T in use (313M inodes) in about 2 hours. it is a MD > raid10 on 16x 10K SAS drives, admittedly. > > if your hardware is conventional (locally-attached multi-disk RAID), > it might make sense to look at its configuration. for instance, fsck > is largely seek-limited, but doing too much readahead, or using large > RAID block sizes (for R5/6) can be disadvantageous. having plenty of RAM > helps in some phases. > > regards, mark hahn. ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] lustre-discuss Digest, Vol 122, Issue 9
Thank you Chris. I'm running e2fsk, the last e2fsprogs version, in the combined mds / mdt and I think ll_recover is for OSTs. Am I wrong? Currently my main problem is the downtime. After three days e2fsck is still running and I see old messages in the list about e2fsck running for weeks...I can't wait weeks to finish the e2fsck More information about our lustre system: combined mds / mdt has 189 GB and 8.9 GB used. It was formatted with the default options. Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 6 may 2016, a las 15:13, Chris Hunter <chunt...@gmail.com> escribió: > > On 05/05/2016 04:05 PM, lustre-discuss-requ...@lists.lustre.org wrote: > Hi Fernando, > > When e2fsck has finished, you should run "ll_recover_lost_found_objs" > command. This tool can recover files from OST lost+found directory using file > extended attributes. > > https://build.hpdd.intel.com/job/lustre-manual/lastSuccessfulBuild/artifact/lustre_manual.xhtml#dbdoclet.50438219_44971 > > > FYI, e2fsck is part of e2fsprogs rpm package. You can update e2fsprogs > independently from lustre version. > > -- > regards, > chris hunter > chunt...@gmail.com > >> Date: Thu, 5 May 2016 15:37:13 +0200 >> From: Fernando Perez <fpe...@icm.csic.es> >> To: "lustre-discuss@lists.lustre.org" >><lustre-discuss@lists.lustre.org> >> Subject: [lustre-discuss] Problems running e2fsck mdt/mds: long time >>running it and a lot of errors >> Message-ID: <572b4c89.9060...@icm.csic.es> >> Content-Type: text/plain; charset=utf-8; format=flowed >> >> Hi, >> >> We have a lustre 2.4.1 filesystem with 250 TB, 158 TB used, and a >> combined MDT/MDS with ldiskfs and the latest e2fsprogs >> >> Due to problems mounting an OST and other hardware problems in the past >> with other OSTs, we have started to run a e2fsck -fy in the combined MDT >> / MDS two days ago because after run e2fsck -fn I saw a lot of errors. >> Previously I made a file level mdt / mds backup. >> >> It seems that the e2fsck is repairing all the inodes. Currently is >> running pass 4 and returns for each inode this kind of messages: >> >>> Unattached inode 26977505 >>> Connect to /lost+found? yes >>> >>> Inode 26977505 ref count is 2, should be 1. Fix? yes >> >> Is it normal? What time can I expect to finish the e2fsck? >> >> Regards. > > > ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
Re: [lustre-discuss] Questions about migrate OSTs from ldiskfs to zfs
Thanks Andreas. I will follow your recommendations. Regards. = Fernando Pérez Institut de Ciències del Mar (CMIMA-CSIC) Departament Oceanografía Física i Tecnològica Passeig Marítim de la Barceloneta,37-49 08003 Barcelona Phone: (+34) 93 230 96 35 = > El 27 feb 2016, a las 1:36, Dilger, Andreas <andreas.dil...@intel.com> > escribió: > > On Feb 23, 2016, at 05:24, Fernando Perez <fpe...@icm.csic.es> wrote: >> >> Hi all. >> >> We have a 230 TB small lustre system. We are using lustre 2.4.1 with zfs >> 0.6.2 installed in the OSSs. The lustre architecture is the following: >> >> 1 MDS +1 MDT in the same server + 3 OSS with 15 ldiskfs OSTs (external >> disks some with fibre controllers and SAS disks and others Coraid aoe >> cabinet with standard SATA disks). >> >> We need to replace 9 OSTs by 2 zfs OSTs due to hardware problems: replace >> Coraid storage by a Supermicro SAS JBOD with twenty disks, each disk has 8 >> TB. >> >> I have some questions that I hope you can help me to answer: >> >> - What can I do with the inactive osts numbers? Acording to lustre manual I >> suppose that I can't erase them, > > If you are replacing the old OSTs with a different number of new OSTs, and > because the OSTs have a different back-end storage type, you will need to do > file migration at the Lustre level. > > That means you will have to add at least some of the new OSTs before removing > the old ones, or completely empty at least one of the old OSTs if you want to > add a new one in its place. The more of the new ones you have online when > doing the migration the faster it will finish, so this is a bit of a trade > off vs. having gaps in your OST config. With some careful juggling of OST > indices you could minimize the number of unused indices. > > Note that you can totally remove old OSTs from your config by doing a > writeconf to completely regenerate your config. There will still be gaps in > your OST index if you didn't avoid this during reconfiguration, but the > removed OSTs will no longer be listed. If you are planning to add more new > OSTs to replace the removed ones in the near future then this may not be > worthwhile. > > We are looking at fixing this to allow complete removal of OSTs from the > config, since permanently inactive OSTs is a common complaint. > >> - I don't need to restore OST configuration files to replace ldiskfs OSTs by >> zfs OSTs, do I? > > If you have a newer Lustre version with "mkfs.lustre --replace" you don't > need to do anything like this. > >> - Dou you recommend to do a lustre update before replace the OSTs by the new >> zfs OSTs? > > Lustre 2.4.1 is very old. It makes sense to use a newer version than this, > especially if you are using ZFS. > > However, it is generally bad sysadmin practice to do major hardware and > software updates at the same time, since it becomes very difficult to isolate > any problems that appear afterward. I would recommend to upgrade to a newer > version of Lustre (2.5.3, or what is recommended from your support provider) > at least on the servers and run that for a week or two before doing the > hardware upgrade. > >> - I have read in the list that there are problem with the last zfsonlinux >> release and lustre only works with zfsonlinux 0.6.3. Is this right? > > Newer versions of Lustre work well with ZFS 0.6.4.3, I don't remember anymore > what ZFS versions were tested with 2.4.1. We had some problems under heavy > load with 0.6.5.3 and have moved back to 0.6.4.3 for the Lustre 2.8.0 release > until that is fixed. > > Cheers, Andreas ___ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
