We do not test or plan for version/protocol interoperability between MDS nodes in the same filesystem. The MDS nodes of a single filesystem should always be upgraded at the same time.
The MDS will check the peer MDS versions at connection time, and should refuse to connect to an MDS with a significantly different release number, with a message like "refused the connection from different version MDS" printed on the console of both MDS nodes. This is what caused the "????" for some directory operations, because they were on the other MDT with a different release version (> 0.0.3 difference, which is about 1 year apart). The MDS version check is enforced to avoid potential filesystem issues between MDS versions that are not tested with each other. While I don't know of specific MDS interoperability issues, this isn't something that is tested. Conversely, a lot of effort is put into protocol compatibility between different client and server versions. There are many production systems that run with mismatched client versions that may be several years apart. In large compute environments that cross-mount filesystems between multiple supercomputers and multiple storage clusters it is not possible to upgrade all of these resources at the same time. Cheers, Andreas > On Feb 7, 2026, at 21:29, Huijun Wu <[email protected]> wrote: > > Andreas, thanks for your reply. > The MDTs were originally created using Lustre 2.15.4. In my case, I > accidentally mounted one of the MDTs with Lustre 2.16.1. The system has two > MDTs in total: one remained untouched by 2.16.1, while the other was mounted > once with 2.16.1. > After that, I switched back to Lustre 2.15.4 and mounted both MDTs. At that > point, some files and directories appeared as “????”. It seemed that some > kind of inconsistency between the states of the two MDTs caused the issue. > Later, I mounted the second MDT (the one that had not been touched before) > with 2.16.1 as well, and then switched back to 2.15.4 to mount both MDTs. > After this, the problem was resolved. > I’m curious about what kinds of inconsistencies between the MDTs might have > led to this behavior. > Thanks in advance for your insights. > > Cheers， > Huijun > > > Original > From: Andreas Dilger <[email protected]> > Date: Feb 7, 2026 08:10 > To: Huijun Wu <[email protected]> > Cc: lustre-discuss < > > > Subject: Re: [lustre-discuss] Issues regarding corss-version mounting of MDT > > On Feb 3, 2026, at 06:01, Huijun Wu <[email protected]> wrote: > > I recently encountered an issue related to cross-version mounting of an MDT. > > After an MDS crash, I rebooted the MDS and accidentally mounted an MDT that > > was originally created and used with Lustre 2.15.4 using Lustre 2.16.1. > > After this, I noticed that many files and directories were shown as “????” > > on the client side. > > I then ran lfsck on this MDT, and found that some directories that > > originally belonged to this MDT disappeared entirely. > > Any recommendations on how to analyze, recover, or mitigate this issue? > > There definitely shouldn't be any issues upgrading from a 2.15 MDT to a 2.16 > MDT, and this is tested regularly during normal Lustre development. > > Did you have any unusual features enabled in Lustre or ldiskfs? Was the > filesystem originally formatted with 2.15, or was it formatted with an > older version and upgraded already? Were there error messages on the MDS > or client console when accessing the problematic files? > > Cheers, Andreas > --- > Andreas Dilger > Principal Lustre Architect > [email protected] > > > > > _______________________________________________ lustre-discuss mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
