Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Andrew Elwell]

On Tue, 26 Feb 2019 at 23:25, Andreas Dilger  wrote:
> I agree that having an option that creates the OSTs as inactive might be 
> helpful, though I wouldn't want that to be the default as I'd imagine it 
> would also cause problems for the majority users that wouldn't know that they 
> need to enable the OSTs after they are mounted.

> Could you file a feature request for this in Jira?
Done https://jira.whamcloud.com/browse/LU-12036
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Patrick Farrell]

This is very good advice, and you can also vary it to aid in removing old OSTs 
(thinking of the previous message) - simply take the old ones you wish to 
remove out of the pool, then new files will not be created there.  Makes 
migration easier.

One thing though:
Setting a default layout everywhere may be prohibitively slow for a large fs.

If you set a default layout on the root of the file system, it is automatically 
used as the default for all directories that do not have another default set.

So if you have not previously set a default layout on any directories, there is 
no need to go through the fs changing them like this.  (And perhaps if you 
have, you can find those directories and handle them manually, rather than 
setting a pool on every directory.)

- Patrick

From: lustre-discuss  on behalf of 
Jongwoo Han 
Sent: Thursday, February 28, 2019 6:09:18 AM
To: Stephane Thiell
Cc: lustre-discuss
Subject: Re: [lustre-discuss] Suspended jobs and rebooting lustre servers


My strategy for adding new OSTs on live filesystem is to define a pool with 
currently running OST and apply pool stripe (lfs setstripe -p [live-ost-pool]) 
on all existing directories. It is better when it is done at first filesystem 
creation.

After that, you can safely add new OSTs without newly created files filling in 
like flood - newly added OST will remain silently until you add them to pool.

Try failover tests with new OSTs and OSSes while it do not store files. After 
the failover/restart test is done on new OSS and OSTs, you can add new OSTs to 
the pool then they will start to store files shortly after.

If you did not create a pool, create a pool with old OSTs and

# lfs find  -type d | while read DIR ; do echo "processing :" $DIR; 
lfs setstripe -p  $DIR ; done

will mark all subdirectories on the pool, so newly added OSTs are safe from 
files coming in until these new OSTs are added to the pool.

I always expand live filesystem in this manner, not to worry about heavily 
loaded situation.

On Thu, Feb 28, 2019 at 1:02 AM Stephane Thiell 
mailto:sthi...@stanford.edu>> wrote:
On one of our filesystem, we add a few new OSTs almost every month with no 
downtime, this is very convenient. The only thing that I would recommend is to 
avoid doing that during a peak of I/Os on your filesystem (we usually do it as 
early as possible in the morning), as the added OSTs will immediately an heavy 
I/O load, likely because they are empty.

Best,

Stephane


> On Feb 22, 2019, at 2:03 PM, Andreas Dilger 
> mailto:adil...@whamcloud.com>> wrote:
>
> This is not really correct.
>
> Lustre clients can handle the addition of OSTs to a running filesystem. The 
> MGS will register the new OSTs, and the clients will be notified by the MGS 
> that the OSTs have been added, so no need to unmount the clients during this 
> process.
>
>
> Cheers, Andreas
>
> On Feb 21, 2019, at 19:23, Raj 
> mailto:rajgau...@gmail.com>> wrote:
>
>> Hello Raj,
>> It’s best and safe to unmount from all the clients and then do the upgrade. 
>> Your FS is getting more OSTs and changing conf in the existing ones, your 
>> client needs to get the new layout by remounting it.
>> Also you mentioned about client eviction, during eviction the client has to 
>> drop it’s dirty pages and all the open file descriptors in the FS will be 
>> gone.
>>
>> On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam 
>> mailto:ans...@gmail.com>> wrote:
>> What can I expect to happen to the jobs that are suspended during the file 
>> system restart?
>> Will the processes holding an open file handle die when I unsuspend them 
>> after the filesystem restart?
>>
>> Thanks!
>> -Raj
>>
>>
>> On Thu, Feb 21, 2019 at 12:52 PM Colin Faber 
>> mailto:cfa...@gmail.com>> wrote:
>> Ah yes,
>>
>> If you're adding to an existing OSS, then you will need to reconfigure the 
>> file system which requires writeconf event.
>>
>> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
>> mailto:ans...@gmail.com>> wrote:
>> The new OST's will be added to the existing file system (the OSS nodes are 
>> already part of the filesystem), I will have to re-configure the current HA 
>> resource configuration to tell it about the 4 new OST's.
>> Our exascaler's HA monitors the individual OST and I need to re-configure 
>> the HA on the existing filesystem.
>>
>> Our vendor support has confirmed that we would have to restart the 
>> filesystem if we want to regenerate the HA configs to include the new OST's.
>>
>> Thanks,
>> -Raj
>>
>>
>> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber 
>> mailto:cfa...@gmail.com>> wrote:
>&

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Jongwoo Han]

My strategy for adding new OSTs on live filesystem is to define a pool with
currently running OST and apply pool stripe (lfs setstripe -p
[live-ost-pool]) on all existing directories. It is better when it is done
at first filesystem creation.

After that, you can safely add new OSTs without newly created files filling
in like flood - newly added OST will remain silently until you add them to
pool.

Try failover tests with new OSTs and OSSes while it do not store files.
After the failover/restart test is done on new OSS and OSTs, you can add
new OSTs to the pool then they will start to store files shortly after.

If you did not create a pool, create a pool with old OSTs and

# lfs find  -type d | while read DIR ; do echo "processing :"
$DIR; lfs setstripe -p  $DIR ; done

will mark all subdirectories on the pool, so newly added OSTs are safe from
files coming in until these new OSTs are added to the pool.

I always expand live filesystem in this manner, not to worry about heavily
loaded situation.

On Thu, Feb 28, 2019 at 1:02 AM Stephane Thiell 
wrote:

> On one of our filesystem, we add a few new OSTs almost every month with no
> downtime, this is very convenient. The only thing that I would recommend is
> to avoid doing that during a peak of I/Os on your filesystem (we usually do
> it as early as possible in the morning), as the added OSTs will immediately
> an heavy I/O load, likely because they are empty.
>
> Best,
>
> Stephane
>
>
> > On Feb 22, 2019, at 2:03 PM, Andreas Dilger 
> wrote:
> >
> > This is not really correct.
> >
> > Lustre clients can handle the addition of OSTs to a running filesystem.
> The MGS will register the new OSTs, and the clients will be notified by the
> MGS that the OSTs have been added, so no need to unmount the clients during
> this process.
> >
> >
> > Cheers, Andreas
> >
> > On Feb 21, 2019, at 19:23, Raj  wrote:
> >
> >> Hello Raj,
> >> It’s best and safe to unmount from all the clients and then do the
> upgrade. Your FS is getting more OSTs and changing conf in the existing
> ones, your client needs to get the new layout by remounting it.
> >> Also you mentioned about client eviction, during eviction the client
> has to drop it’s dirty pages and all the open file descriptors in the FS
> will be gone.
> >>
> >> On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam 
> wrote:
> >> What can I expect to happen to the jobs that are suspended during the
> file system restart?
> >> Will the processes holding an open file handle die when I unsuspend
> them after the filesystem restart?
> >>
> >> Thanks!
> >> -Raj
> >>
> >>
> >> On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:
> >> Ah yes,
> >>
> >> If you're adding to an existing OSS, then you will need to reconfigure
> the file system which requires writeconf event.
> >>
> >> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
> wrote:
> >> The new OST's will be added to the existing file system (the OSS nodes
> are already part of the filesystem), I will have to re-configure the
> current HA resource configuration to tell it about the 4 new OST's.
> >> Our exascaler's HA monitors the individual OST and I need to
> re-configure the HA on the existing filesystem.
> >>
> >> Our vendor support has confirmed that we would have to restart the
> filesystem if we want to regenerate the HA configs to include the new OST's.
> >>
> >> Thanks,
> >> -Raj
> >>
> >>
> >> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:
> >> It seems to me that steps may still be missing?
> >>
> >> You're going to rack/stack and provision the OSS nodes with new OSTs'.
> >>
> >> Then you're going to introduce failover options somewhere? new osts?
> existing system? etc?
> >>
> >> If you're introducing failover with the new OST's and leaving the
> existing system in place, you should be able to accomplish this without
> bringing the system offline.
> >>
> >> If you're going to be introducing failover to your existing system then
> you will need to reconfigure the file system to accommodate the new
> failover settings (failover nides, etc.)
> >>
> >> -cf
> >>
> >>
> >> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
> wrote:
> >> Our upgrade strategy is as follows:
> >>
> >> 1) Load all disks into the storage array.
> >> 2) Create RAID pools and virtual disks.
> >> 3) Create lustre file system using mkfs.lustre command. (I still have
> to figure out all the parameters used on the existing OSTs).
> >> 4) Create mount points on all OSSs.
> >> 5) Mount the lustre OSTs.
> >> 6) Maybe rebalance the filesystem.
> >> My understanding is that the above can be done without bringing the
> filesystem down. I want to create the HA configuration (corosync and
> pacemaker) for the new OSTs. This step requires the filesystem to be down.
> I want to know what would happen to the suspended processes across the
> cluster when I bring the filesystem down to re-generate the HA configs.
> >>
> >> Thanks,
> >> -Raj
> >>
> >> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Stephane Thiell]

On one of our filesystem, we add a few new OSTs almost every month with no 
downtime, this is very convenient. The only thing that I would recommend is to 
avoid doing that during a peak of I/Os on your filesystem (we usually do it as 
early as possible in the morning), as the added OSTs will immediately an heavy 
I/O load, likely because they are empty.

Best,

Stephane


> On Feb 22, 2019, at 2:03 PM, Andreas Dilger  wrote:
> 
> This is not really correct. 
> 
> Lustre clients can handle the addition of OSTs to a running filesystem. The 
> MGS will register the new OSTs, and the clients will be notified by the MGS 
> that the OSTs have been added, so no need to unmount the clients during this 
> process. 
> 
> 
> Cheers, Andreas
> 
> On Feb 21, 2019, at 19:23, Raj  wrote:
> 
>> Hello Raj,
>> It’s best and safe to unmount from all the clients and then do the upgrade. 
>> Your FS is getting more OSTs and changing conf in the existing ones, your 
>> client needs to get the new layout by remounting it. 
>> Also you mentioned about client eviction, during eviction the client has to 
>> drop it’s dirty pages and all the open file descriptors in the FS will be 
>> gone. 
>> 
>> On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam  wrote:
>> What can I expect to happen to the jobs that are suspended during the file 
>> system restart?
>> Will the processes holding an open file handle die when I unsuspend them 
>> after the filesystem restart?
>> 
>> Thanks!
>> -Raj
>> 
>> 
>> On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:
>> Ah yes, 
>> 
>> If you're adding to an existing OSS, then you will need to reconfigure the 
>> file system which requires writeconf event.
>> 
>> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam  wrote:
>> The new OST's will be added to the existing file system (the OSS nodes are 
>> already part of the filesystem), I will have to re-configure the current HA 
>> resource configuration to tell it about the 4 new OST's.
>> Our exascaler's HA monitors the individual OST and I need to re-configure 
>> the HA on the existing filesystem.
>> 
>> Our vendor support has confirmed that we would have to restart the 
>> filesystem if we want to regenerate the HA configs to include the new OST's.
>> 
>> Thanks,
>> -Raj
>> 
>> 
>> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:
>> It seems to me that steps may still be missing?
>> 
>> You're going to rack/stack and provision the OSS nodes with new OSTs'. 
>> 
>> Then you're going to introduce failover options somewhere? new osts? 
>> existing system? etc?
>> 
>> If you're introducing failover with the new OST's and leaving the existing 
>> system in place, you should be able to accomplish this without bringing the 
>> system offline. 
>> 
>> If you're going to be introducing failover to your existing system then you 
>> will need to reconfigure the file system to accommodate the new failover 
>> settings (failover nides, etc.)
>> 
>> -cf
>> 
>> 
>> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam  wrote:
>> Our upgrade strategy is as follows:
>> 
>> 1) Load all disks into the storage array.
>> 2) Create RAID pools and virtual disks.
>> 3) Create lustre file system using mkfs.lustre command. (I still have to 
>> figure out all the parameters used on the existing OSTs).
>> 4) Create mount points on all OSSs.
>> 5) Mount the lustre OSTs.
>> 6) Maybe rebalance the filesystem.
>> My understanding is that the above can be done without bringing the 
>> filesystem down. I want to create the HA configuration (corosync and 
>> pacemaker) for the new OSTs. This step requires the filesystem to be down. I 
>> want to know what would happen to the suspended processes across the cluster 
>> when I bring the filesystem down to re-generate the HA configs.
>> 
>> Thanks,
>> -Raj
>> 
>> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:
>> Can you provide more details on your upgrade strategy? In some cases 
>> expanding your storage shouldn't impact client / job activity at all.
>> 
>> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam  wrote:
>> Hello,
>> 
>> We are planning on expanding our storage by adding more OSTs to our lustre 
>> file system. It looks like it would be easier to expand if we bring the 
>> filesystem down and perform the necessary operations. We are planning to 
>> suspend all the jobs running on the cluster. We originally planned to add 
>> new OSTs to the live filesystem.
>> 
>> We are trying to determine the potential impact to the suspended jobs if we 
>> bring down the filesystem for the upgrade.
>> One of the questions we have is what would happen to the suspended processes 
>> that hold an open file handle in the lustre file system when the filesystem 
>> is brought down for the upgrade?
>> Will they recover from the client eviction?
>> 
>> We do have vendor support and have engaged them. I wanted to ask the 
>> community and get some feedback.
>> 
>> Thanks,
>> -Raj
>> ___
>> lustre-discuss 

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Andreas Dilger]

This is not really correct.

Lustre clients can handle the addition of OSTs to a running filesystem. The MGS 
will register the new OSTs, and the clients will be notified by the MGS that 
the OSTs have been added, so no need to unmount the clients during this process.


Cheers, Andreas

On Feb 21, 2019, at 19:23, Raj 
mailto:rajgau...@gmail.com>> wrote:

Hello Raj,
It’s best and safe to unmount from all the clients and then do the upgrade. 
Your FS is getting more OSTs and changing conf in the existing ones, your 
client needs to get the new layout by remounting it.
Also you mentioned about client eviction, during eviction the client has to 
drop it’s dirty pages and all the open file descriptors in the FS will be gone.

On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam 
mailto:ans...@gmail.com>> wrote:
What can I expect to happen to the jobs that are suspended during the file 
system restart?
Will the processes holding an open file handle die when I unsuspend them after 
the filesystem restart?

Thanks!
-Raj


On Thu, Feb 21, 2019 at 12:52 PM Colin Faber 
mailto:cfa...@gmail.com>> wrote:
Ah yes,

If you're adding to an existing OSS, then you will need to reconfigure the file 
system which requires writeconf event.

On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
mailto:ans...@gmail.com>> wrote:
The new OST's will be added to the existing file system (the OSS nodes are 
already part of the filesystem), I will have to re-configure the current HA 
resource configuration to tell it about the 4 new OST's.
Our exascaler's HA monitors the individual OST and I need to re-configure the 
HA on the existing filesystem.

Our vendor support has confirmed that we would have to restart the filesystem 
if we want to regenerate the HA configs to include the new OST's.

Thanks,
-Raj


On Thu, Feb 21, 2019 at 11:23 AM Colin Faber 
mailto:cfa...@gmail.com>> wrote:
It seems to me that steps may still be missing?

You're going to rack/stack and provision the OSS nodes with new OSTs'.

Then you're going to introduce failover options somewhere? new osts? existing 
system? etc?

If you're introducing failover with the new OST's and leaving the existing 
system in place, you should be able to accomplish this without bringing the 
system offline.

If you're going to be introducing failover to your existing system then you 
will need to reconfigure the file system to accommodate the new failover 
settings (failover nides, etc.)

-cf


On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
mailto:ans...@gmail.com>> wrote:
Our upgrade strategy is as follows:

1) Load all disks into the storage array.
2) Create RAID pools and virtual disks.
3) Create lustre file system using mkfs.lustre command. (I still have to figure 
out all the parameters used on the existing OSTs).
4) Create mount points on all OSSs.
5) Mount the lustre OSTs.
6) Maybe rebalance the filesystem.
My understanding is that the above can be done without bringing the filesystem 
down. I want to create the HA configuration (corosync and pacemaker) for the 
new OSTs. This step requires the filesystem to be down. I want to know what 
would happen to the suspended processes across the cluster when I bring the 
filesystem down to re-generate the HA configs.

Thanks,
-Raj

On Thu, Feb 21, 2019 at 12:59 AM Colin Faber 
mailto:cfa...@gmail.com>> wrote:
Can you provide more details on your upgrade strategy? In some cases expanding 
your storage shouldn't impact client / job activity at all.

On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
mailto:ans...@gmail.com>> wrote:
Hello,

We are planning on expanding our storage by adding more OSTs to our lustre file 
system. It looks like it would be easier to expand if we bring the filesystem 
down and perform the necessary operations. We are planning to suspend all the 
jobs running on the cluster. We originally planned to add new OSTs to the live 
filesystem.

We are trying to determine the potential impact to the suspended jobs if we 
bring down the filesystem for the upgrade.
One of the questions we have is what would happen to the suspended processes 
that hold an open file handle in the lustre file system when the filesystem is 
brought down for the upgrade?
Will they recover from the client eviction?

We do have vendor support and have engaged them. I wanted to ask the community 
and get some feedback.

Thanks,
-Raj
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj Ayyampalayam]

Got it. I rather be safe than sorry. This is my first time doing a lustre
configuration change.

Raj

On Thu, Feb 21, 2019, 11:55 PM Raj  wrote:

> I also agree with Colin's comment.
> If the current OSTs are not touched, and you are only adding new OSTs to
> existing OSS nodes and adding new ost-mount resources in your existing
> (already running) Pacemaker configuration, you can achieve the upgrade with
> no downtime. If your Corosync-Pacemaker configuration is working correctly,
> you can failover and failback and take turn to reboot each OSS nodes. But,
> chances of human error is too high in doing this.
>
> On Thu, Feb 21, 2019 at 10:30 PM Raj Ayyampalayam 
> wrote:
>
>> Hi Raj,
>>
>> Thanks for the explanation. We will have to rethink our upgrade process.
>>
>> Thanks again.
>> Raj
>>
>> On Thu, Feb 21, 2019, 10:23 PM Raj  wrote:
>>
>>> Hello Raj,
>>> It’s best and safe to unmount from all the clients and then do the
>>> upgrade. Your FS is getting more OSTs and changing conf in the existing
>>> ones, your client needs to get the new layout by remounting it.
>>> Also you mentioned about client eviction, during eviction the client has
>>> to drop it’s dirty pages and all the open file descriptors in the FS will
>>> be gone.
>>>
>>> On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam 
>>> wrote:
>>>
 What can I expect to happen to the jobs that are suspended during the
 file system restart?
 Will the processes holding an open file handle die when I unsuspend
 them after the filesystem restart?

 Thanks!
 -Raj


 On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:

> Ah yes,
>
> If you're adding to an existing OSS, then you will need to reconfigure
> the file system which requires writeconf event.
>

> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
> wrote:
>
>> The new OST's will be added to the existing file system (the OSS
>> nodes are already part of the filesystem), I will have to re-configure 
>> the
>> current HA resource configuration to tell it about the 4 new OST's.
>> Our exascaler's HA monitors the individual OST and I need to
>> re-configure the HA on the existing filesystem.
>>
>> Our vendor support has confirmed that we would have to restart the
>> filesystem if we want to regenerate the HA configs to include the new 
>> OST's.
>>
>> Thanks,
>> -Raj
>>
>>
>> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber 
>> wrote:
>>
>>> It seems to me that steps may still be missing?
>>>
>>> You're going to rack/stack and provision the OSS nodes with new
>>> OSTs'.
>>>
>>> Then you're going to introduce failover options somewhere? new osts?
>>> existing system? etc?
>>>
>>> If you're introducing failover with the new OST's and leaving the
>>> existing system in place, you should be able to accomplish this without
>>> bringing the system offline.
>>>
>>> If you're going to be introducing failover to your existing system
>>> then you will need to reconfigure the file system to accommodate the new
>>> failover settings (failover nides, etc.)
>>>
>>> -cf
>>>
>>>
>>> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
>>> wrote:
>>>
 Our upgrade strategy is as follows:

 1) Load all disks into the storage array.
 2) Create RAID pools and virtual disks.
 3) Create lustre file system using mkfs.lustre command. (I still
 have to figure out all the parameters used on the existing OSTs).
 4) Create mount points on all OSSs.
 5) Mount the lustre OSTs.
 6) Maybe rebalance the filesystem.
 My understanding is that the above can be done without bringing the
 filesystem down. I want to create the HA configuration (corosync and
 pacemaker) for the new OSTs. This step requires the filesystem to be 
 down.
 I want to know what would happen to the suspended processes across the
 cluster when I bring the filesystem down to re-generate the HA configs.

 Thanks,
 -Raj

 On Thu, Feb 21, 2019 at 12:59 AM Colin Faber 
 wrote:

> Can you provide more details on your upgrade strategy? In some
> cases expanding your storage shouldn't impact client / job activity 
> at all.
>
> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
> wrote:
>
>> Hello,
>>
>> We are planning on expanding our storage by adding more OSTs to
>> our lustre file system. It looks like it would be easier to expand 
>> if we
>> bring the filesystem down and perform the necessary operations. We 
>> are
>> planning to suspend all the jobs running on the cluster. We 
>> originally
>> planned to add new OSTs to the live 

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj]

I also agree with Colin's comment.
If the current OSTs are not touched, and you are only adding new OSTs to
existing OSS nodes and adding new ost-mount resources in your existing
(already running) Pacemaker configuration, you can achieve the upgrade with
no downtime. If your Corosync-Pacemaker configuration is working correctly,
you can failover and failback and take turn to reboot each OSS nodes. But,
chances of human error is too high in doing this.

On Thu, Feb 21, 2019 at 10:30 PM Raj Ayyampalayam  wrote:

> Hi Raj,
>
> Thanks for the explanation. We will have to rethink our upgrade process.
>
> Thanks again.
> Raj
>
> On Thu, Feb 21, 2019, 10:23 PM Raj  wrote:
>
>> Hello Raj,
>> It’s best and safe to unmount from all the clients and then do the
>> upgrade. Your FS is getting more OSTs and changing conf in the existing
>> ones, your client needs to get the new layout by remounting it.
>> Also you mentioned about client eviction, during eviction the client has
>> to drop it’s dirty pages and all the open file descriptors in the FS will
>> be gone.
>>
>> On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam 
>> wrote:
>>
>>> What can I expect to happen to the jobs that are suspended during the
>>> file system restart?
>>> Will the processes holding an open file handle die when I unsuspend them
>>> after the filesystem restart?
>>>
>>> Thanks!
>>> -Raj
>>>
>>>
>>> On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:
>>>
 Ah yes,

 If you're adding to an existing OSS, then you will need to reconfigure
 the file system which requires writeconf event.

>>>
 On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
 wrote:

> The new OST's will be added to the existing file system (the OSS nodes
> are already part of the filesystem), I will have to re-configure the
> current HA resource configuration to tell it about the 4 new OST's.
> Our exascaler's HA monitors the individual OST and I need to
> re-configure the HA on the existing filesystem.
>
> Our vendor support has confirmed that we would have to restart the
> filesystem if we want to regenerate the HA configs to include the new 
> OST's.
>
> Thanks,
> -Raj
>
>
> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:
>
>> It seems to me that steps may still be missing?
>>
>> You're going to rack/stack and provision the OSS nodes with new
>> OSTs'.
>>
>> Then you're going to introduce failover options somewhere? new osts?
>> existing system? etc?
>>
>> If you're introducing failover with the new OST's and leaving the
>> existing system in place, you should be able to accomplish this without
>> bringing the system offline.
>>
>> If you're going to be introducing failover to your existing system
>> then you will need to reconfigure the file system to accommodate the new
>> failover settings (failover nides, etc.)
>>
>> -cf
>>
>>
>> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
>> wrote:
>>
>>> Our upgrade strategy is as follows:
>>>
>>> 1) Load all disks into the storage array.
>>> 2) Create RAID pools and virtual disks.
>>> 3) Create lustre file system using mkfs.lustre command. (I still
>>> have to figure out all the parameters used on the existing OSTs).
>>> 4) Create mount points on all OSSs.
>>> 5) Mount the lustre OSTs.
>>> 6) Maybe rebalance the filesystem.
>>> My understanding is that the above can be done without bringing the
>>> filesystem down. I want to create the HA configuration (corosync and
>>> pacemaker) for the new OSTs. This step requires the filesystem to be 
>>> down.
>>> I want to know what would happen to the suspended processes across the
>>> cluster when I bring the filesystem down to re-generate the HA configs.
>>>
>>> Thanks,
>>> -Raj
>>>
>>> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber 
>>> wrote:
>>>
 Can you provide more details on your upgrade strategy? In some
 cases expanding your storage shouldn't impact client / job activity at 
 all.

 On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
 wrote:

> Hello,
>
> We are planning on expanding our storage by adding more OSTs to
> our lustre file system. It looks like it would be easier to expand if 
> we
> bring the filesystem down and perform the necessary operations. We are
> planning to suspend all the jobs running on the cluster. We originally
> planned to add new OSTs to the live filesystem.
>
> We are trying to determine the potential impact to the suspended
> jobs if we bring down the filesystem for the upgrade.
> One of the questions we have is what would happen to the suspended
> processes that hold an open file handle in the lustre file system 
> 

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj Ayyampalayam]

Hi Raj,

Thanks for the explanation. We will have to rethink our upgrade process.

Thanks again.
Raj

On Thu, Feb 21, 2019, 10:23 PM Raj  wrote:

> Hello Raj,
> It’s best and safe to unmount from all the clients and then do the
> upgrade. Your FS is getting more OSTs and changing conf in the existing
> ones, your client needs to get the new layout by remounting it.
> Also you mentioned about client eviction, during eviction the client has
> to drop it’s dirty pages and all the open file descriptors in the FS will
> be gone.
>
> On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam 
> wrote:
>
>> What can I expect to happen to the jobs that are suspended during the
>> file system restart?
>> Will the processes holding an open file handle die when I unsuspend them
>> after the filesystem restart?
>>
>> Thanks!
>> -Raj
>>
>>
>> On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:
>>
>>> Ah yes,
>>>
>>> If you're adding to an existing OSS, then you will need to reconfigure
>>> the file system which requires writeconf event.
>>>
>>
>>> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
>>> wrote:
>>>
 The new OST's will be added to the existing file system (the OSS nodes
 are already part of the filesystem), I will have to re-configure the
 current HA resource configuration to tell it about the 4 new OST's.
 Our exascaler's HA monitors the individual OST and I need to
 re-configure the HA on the existing filesystem.

 Our vendor support has confirmed that we would have to restart the
 filesystem if we want to regenerate the HA configs to include the new 
 OST's.

 Thanks,
 -Raj


 On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:

> It seems to me that steps may still be missing?
>
> You're going to rack/stack and provision the OSS nodes with new OSTs'.
>
> Then you're going to introduce failover options somewhere? new osts?
> existing system? etc?
>
> If you're introducing failover with the new OST's and leaving the
> existing system in place, you should be able to accomplish this without
> bringing the system offline.
>
> If you're going to be introducing failover to your existing system
> then you will need to reconfigure the file system to accommodate the new
> failover settings (failover nides, etc.)
>
> -cf
>
>
> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
> wrote:
>
>> Our upgrade strategy is as follows:
>>
>> 1) Load all disks into the storage array.
>> 2) Create RAID pools and virtual disks.
>> 3) Create lustre file system using mkfs.lustre command. (I still have
>> to figure out all the parameters used on the existing OSTs).
>> 4) Create mount points on all OSSs.
>> 5) Mount the lustre OSTs.
>> 6) Maybe rebalance the filesystem.
>> My understanding is that the above can be done without bringing the
>> filesystem down. I want to create the HA configuration (corosync and
>> pacemaker) for the new OSTs. This step requires the filesystem to be 
>> down.
>> I want to know what would happen to the suspended processes across the
>> cluster when I bring the filesystem down to re-generate the HA configs.
>>
>> Thanks,
>> -Raj
>>
>> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber 
>> wrote:
>>
>>> Can you provide more details on your upgrade strategy? In some cases
>>> expanding your storage shouldn't impact client / job activity at all.
>>>
>>> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
>>> wrote:
>>>
 Hello,

 We are planning on expanding our storage by adding more OSTs to our
 lustre file system. It looks like it would be easier to expand if we 
 bring
 the filesystem down and perform the necessary operations. We are 
 planning
 to suspend all the jobs running on the cluster. We originally planned 
 to
 add new OSTs to the live filesystem.

 We are trying to determine the potential impact to the suspended
 jobs if we bring down the filesystem for the upgrade.
 One of the questions we have is what would happen to the suspended
 processes that hold an open file handle in the lustre file system when 
 the
 filesystem is brought down for the upgrade?
 Will they recover from the client eviction?

 We do have vendor support and have engaged them. I wanted to ask
 the community and get some feedback.

 Thanks,
 -Raj

>>> ___
 lustre-discuss mailing list
 lustre-discuss@lists.lustre.org
 http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

>>> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> 

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj]

Hello Raj,
It’s best and safe to unmount from all the clients and then do the upgrade.
Your FS is getting more OSTs and changing conf in the existing ones, your
client needs to get the new layout by remounting it.
Also you mentioned about client eviction, during eviction the client has to
drop it’s dirty pages and all the open file descriptors in the FS will be
gone.

On Thu, Feb 21, 2019 at 12:25 PM Raj Ayyampalayam  wrote:

> What can I expect to happen to the jobs that are suspended during the file
> system restart?
> Will the processes holding an open file handle die when I unsuspend them
> after the filesystem restart?
>
> Thanks!
> -Raj
>
>
> On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:
>
>> Ah yes,
>>
>> If you're adding to an existing OSS, then you will need to reconfigure
>> the file system which requires writeconf event.
>>
>
>> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
>> wrote:
>>
>>> The new OST's will be added to the existing file system (the OSS nodes
>>> are already part of the filesystem), I will have to re-configure the
>>> current HA resource configuration to tell it about the 4 new OST's.
>>> Our exascaler's HA monitors the individual OST and I need to
>>> re-configure the HA on the existing filesystem.
>>>
>>> Our vendor support has confirmed that we would have to restart the
>>> filesystem if we want to regenerate the HA configs to include the new OST's.
>>>
>>> Thanks,
>>> -Raj
>>>
>>>
>>> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:
>>>
 It seems to me that steps may still be missing?

 You're going to rack/stack and provision the OSS nodes with new OSTs'.

 Then you're going to introduce failover options somewhere? new osts?
 existing system? etc?

 If you're introducing failover with the new OST's and leaving the
 existing system in place, you should be able to accomplish this without
 bringing the system offline.

 If you're going to be introducing failover to your existing system then
 you will need to reconfigure the file system to accommodate the new
 failover settings (failover nides, etc.)

 -cf


 On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
 wrote:

> Our upgrade strategy is as follows:
>
> 1) Load all disks into the storage array.
> 2) Create RAID pools and virtual disks.
> 3) Create lustre file system using mkfs.lustre command. (I still have
> to figure out all the parameters used on the existing OSTs).
> 4) Create mount points on all OSSs.
> 5) Mount the lustre OSTs.
> 6) Maybe rebalance the filesystem.
> My understanding is that the above can be done without bringing the
> filesystem down. I want to create the HA configuration (corosync and
> pacemaker) for the new OSTs. This step requires the filesystem to be down.
> I want to know what would happen to the suspended processes across the
> cluster when I bring the filesystem down to re-generate the HA configs.
>
> Thanks,
> -Raj
>
> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:
>
>> Can you provide more details on your upgrade strategy? In some cases
>> expanding your storage shouldn't impact client / job activity at all.
>>
>> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
>> wrote:
>>
>>> Hello,
>>>
>>> We are planning on expanding our storage by adding more OSTs to our
>>> lustre file system. It looks like it would be easier to expand if we 
>>> bring
>>> the filesystem down and perform the necessary operations. We are 
>>> planning
>>> to suspend all the jobs running on the cluster. We originally planned to
>>> add new OSTs to the live filesystem.
>>>
>>> We are trying to determine the potential impact to the suspended
>>> jobs if we bring down the filesystem for the upgrade.
>>> One of the questions we have is what would happen to the suspended
>>> processes that hold an open file handle in the lustre file system when 
>>> the
>>> filesystem is brought down for the upgrade?
>>> Will they recover from the client eviction?
>>>
>>> We do have vendor support and have engaged them. I wanted to ask the
>>> community and get some feedback.
>>>
>>> Thanks,
>>> -Raj
>>>
>> ___
>>> lustre-discuss mailing list
>>> lustre-discuss@lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj Ayyampalayam]

What can I expect to happen to the jobs that are suspended during the file
system restart?
Will the processes holding an open file handle die when I unsuspend them
after the filesystem restart?

Thanks!
-Raj


On Thu, Feb 21, 2019 at 12:52 PM Colin Faber  wrote:

> Ah yes,
>
> If you're adding to an existing OSS, then you will need to reconfigure the
> file system which requires writeconf event.
>
> On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam 
> wrote:
>
>> The new OST's will be added to the existing file system (the OSS nodes
>> are already part of the filesystem), I will have to re-configure the
>> current HA resource configuration to tell it about the 4 new OST's.
>> Our exascaler's HA monitors the individual OST and I need to re-configure
>> the HA on the existing filesystem.
>>
>> Our vendor support has confirmed that we would have to restart the
>> filesystem if we want to regenerate the HA configs to include the new OST's.
>>
>> Thanks,
>> -Raj
>>
>>
>> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:
>>
>>> It seems to me that steps may still be missing?
>>>
>>> You're going to rack/stack and provision the OSS nodes with new OSTs'.
>>>
>>> Then you're going to introduce failover options somewhere? new osts?
>>> existing system? etc?
>>>
>>> If you're introducing failover with the new OST's and leaving the
>>> existing system in place, you should be able to accomplish this without
>>> bringing the system offline.
>>>
>>> If you're going to be introducing failover to your existing system then
>>> you will need to reconfigure the file system to accommodate the new
>>> failover settings (failover nides, etc.)
>>>
>>> -cf
>>>
>>>
>>> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
>>> wrote:
>>>
 Our upgrade strategy is as follows:

 1) Load all disks into the storage array.
 2) Create RAID pools and virtual disks.
 3) Create lustre file system using mkfs.lustre command. (I still have
 to figure out all the parameters used on the existing OSTs).
 4) Create mount points on all OSSs.
 5) Mount the lustre OSTs.
 6) Maybe rebalance the filesystem.
 My understanding is that the above can be done without bringing the
 filesystem down. I want to create the HA configuration (corosync and
 pacemaker) for the new OSTs. This step requires the filesystem to be down.
 I want to know what would happen to the suspended processes across the
 cluster when I bring the filesystem down to re-generate the HA configs.

 Thanks,
 -Raj

 On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:

> Can you provide more details on your upgrade strategy? In some cases
> expanding your storage shouldn't impact client / job activity at all.
>
> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
> wrote:
>
>> Hello,
>>
>> We are planning on expanding our storage by adding more OSTs to our
>> lustre file system. It looks like it would be easier to expand if we 
>> bring
>> the filesystem down and perform the necessary operations. We are planning
>> to suspend all the jobs running on the cluster. We originally planned to
>> add new OSTs to the live filesystem.
>>
>> We are trying to determine the potential impact to the suspended jobs
>> if we bring down the filesystem for the upgrade.
>> One of the questions we have is what would happen to the suspended
>> processes that hold an open file handle in the lustre file system when 
>> the
>> filesystem is brought down for the upgrade?
>> Will they recover from the client eviction?
>>
>> We do have vendor support and have engaged them. I wanted to ask the
>> community and get some feedback.
>>
>> Thanks,
>> -Raj
>>
> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Colin Faber]

Ah yes,

If you're adding to an existing OSS, then you will need to reconfigure the
file system which requires writeconf event.

On Thu, Feb 21, 2019 at 10:00 AM Raj Ayyampalayam  wrote:

> The new OST's will be added to the existing file system (the OSS nodes are
> already part of the filesystem), I will have to re-configure the current HA
> resource configuration to tell it about the 4 new OST's.
> Our exascaler's HA monitors the individual OST and I need to re-configure
> the HA on the existing filesystem.
>
> Our vendor support has confirmed that we would have to restart the
> filesystem if we want to regenerate the HA configs to include the new OST's.
>
> Thanks,
> -Raj
>
>
> On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:
>
>> It seems to me that steps may still be missing?
>>
>> You're going to rack/stack and provision the OSS nodes with new OSTs'.
>>
>> Then you're going to introduce failover options somewhere? new osts?
>> existing system? etc?
>>
>> If you're introducing failover with the new OST's and leaving the
>> existing system in place, you should be able to accomplish this without
>> bringing the system offline.
>>
>> If you're going to be introducing failover to your existing system then
>> you will need to reconfigure the file system to accommodate the new
>> failover settings (failover nides, etc.)
>>
>> -cf
>>
>>
>> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam 
>> wrote:
>>
>>> Our upgrade strategy is as follows:
>>>
>>> 1) Load all disks into the storage array.
>>> 2) Create RAID pools and virtual disks.
>>> 3) Create lustre file system using mkfs.lustre command. (I still have to
>>> figure out all the parameters used on the existing OSTs).
>>> 4) Create mount points on all OSSs.
>>> 5) Mount the lustre OSTs.
>>> 6) Maybe rebalance the filesystem.
>>> My understanding is that the above can be done without bringing the
>>> filesystem down. I want to create the HA configuration (corosync and
>>> pacemaker) for the new OSTs. This step requires the filesystem to be down.
>>> I want to know what would happen to the suspended processes across the
>>> cluster when I bring the filesystem down to re-generate the HA configs.
>>>
>>> Thanks,
>>> -Raj
>>>
>>> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:
>>>
 Can you provide more details on your upgrade strategy? In some cases
 expanding your storage shouldn't impact client / job activity at all.

 On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
 wrote:

> Hello,
>
> We are planning on expanding our storage by adding more OSTs to our
> lustre file system. It looks like it would be easier to expand if we bring
> the filesystem down and perform the necessary operations. We are planning
> to suspend all the jobs running on the cluster. We originally planned to
> add new OSTs to the live filesystem.
>
> We are trying to determine the potential impact to the suspended jobs
> if we bring down the filesystem for the upgrade.
> One of the questions we have is what would happen to the suspended
> processes that hold an open file handle in the lustre file system when the
> filesystem is brought down for the upgrade?
> Will they recover from the client eviction?
>
> We do have vendor support and have engaged them. I wanted to ask the
> community and get some feedback.
>
> Thanks,
> -Raj
>
 ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj Ayyampalayam]

The new OST's will be added to the existing file system (the OSS nodes are
already part of the filesystem), I will have to re-configure the current HA
resource configuration to tell it about the 4 new OST's.
Our exascaler's HA monitors the individual OST and I need to re-configure
the HA on the existing filesystem.

Our vendor support has confirmed that we would have to restart the
filesystem if we want to regenerate the HA configs to include the new OST's.

Thanks,
-Raj


On Thu, Feb 21, 2019 at 11:23 AM Colin Faber  wrote:

> It seems to me that steps may still be missing?
>
> You're going to rack/stack and provision the OSS nodes with new OSTs'.
>
> Then you're going to introduce failover options somewhere? new osts?
> existing system? etc?
>
> If you're introducing failover with the new OST's and leaving the existing
> system in place, you should be able to accomplish this without bringing the
> system offline.
>
> If you're going to be introducing failover to your existing system then
> you will need to reconfigure the file system to accommodate the new
> failover settings (failover nides, etc.)
>
> -cf
>
>
> On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam  wrote:
>
>> Our upgrade strategy is as follows:
>>
>> 1) Load all disks into the storage array.
>> 2) Create RAID pools and virtual disks.
>> 3) Create lustre file system using mkfs.lustre command. (I still have to
>> figure out all the parameters used on the existing OSTs).
>> 4) Create mount points on all OSSs.
>> 5) Mount the lustre OSTs.
>> 6) Maybe rebalance the filesystem.
>> My understanding is that the above can be done without bringing the
>> filesystem down. I want to create the HA configuration (corosync and
>> pacemaker) for the new OSTs. This step requires the filesystem to be down.
>> I want to know what would happen to the suspended processes across the
>> cluster when I bring the filesystem down to re-generate the HA configs.
>>
>> Thanks,
>> -Raj
>>
>> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:
>>
>>> Can you provide more details on your upgrade strategy? In some cases
>>> expanding your storage shouldn't impact client / job activity at all.
>>>
>>> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam 
>>> wrote:
>>>
 Hello,

 We are planning on expanding our storage by adding more OSTs to our
 lustre file system. It looks like it would be easier to expand if we bring
 the filesystem down and perform the necessary operations. We are planning
 to suspend all the jobs running on the cluster. We originally planned to
 add new OSTs to the live filesystem.

 We are trying to determine the potential impact to the suspended jobs
 if we bring down the filesystem for the upgrade.
 One of the questions we have is what would happen to the suspended
 processes that hold an open file handle in the lustre file system when the
 filesystem is brought down for the upgrade?
 Will they recover from the client eviction?

 We do have vendor support and have engaged them. I wanted to ask the
 community and get some feedback.

 Thanks,
 -Raj

>>> ___
 lustre-discuss mailing list
 lustre-discuss@lists.lustre.org
 http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

>>>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Colin Faber]

It seems to me that steps may still be missing?

You're going to rack/stack and provision the OSS nodes with new OSTs'.

Then you're going to introduce failover options somewhere? new osts?
existing system? etc?

If you're introducing failover with the new OST's and leaving the existing
system in place, you should be able to accomplish this without bringing the
system offline.

If you're going to be introducing failover to your existing system then you
will need to reconfigure the file system to accommodate the new failover
settings (failover nides, etc.)

-cf


On Thu, Feb 21, 2019 at 9:13 AM Raj Ayyampalayam  wrote:

> Our upgrade strategy is as follows:
>
> 1) Load all disks into the storage array.
> 2) Create RAID pools and virtual disks.
> 3) Create lustre file system using mkfs.lustre command. (I still have to
> figure out all the parameters used on the existing OSTs).
> 4) Create mount points on all OSSs.
> 5) Mount the lustre OSTs.
> 6) Maybe rebalance the filesystem.
> My understanding is that the above can be done without bringing the
> filesystem down. I want to create the HA configuration (corosync and
> pacemaker) for the new OSTs. This step requires the filesystem to be down.
> I want to know what would happen to the suspended processes across the
> cluster when I bring the filesystem down to re-generate the HA configs.
>
> Thanks,
> -Raj
>
> On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:
>
>> Can you provide more details on your upgrade strategy? In some cases
>> expanding your storage shouldn't impact client / job activity at all.
>>
>> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam  wrote:
>>
>>> Hello,
>>>
>>> We are planning on expanding our storage by adding more OSTs to our
>>> lustre file system. It looks like it would be easier to expand if we bring
>>> the filesystem down and perform the necessary operations. We are planning
>>> to suspend all the jobs running on the cluster. We originally planned to
>>> add new OSTs to the live filesystem.
>>>
>>> We are trying to determine the potential impact to the suspended jobs if
>>> we bring down the filesystem for the upgrade.
>>> One of the questions we have is what would happen to the suspended
>>> processes that hold an open file handle in the lustre file system when the
>>> filesystem is brought down for the upgrade?
>>> Will they recover from the client eviction?
>>>
>>> We do have vendor support and have engaged them. I wanted to ask the
>>> community and get some feedback.
>>>
>>> Thanks,
>>> -Raj
>>>
>> ___
>>> lustre-discuss mailing list
>>> lustre-discuss@lists.lustre.org
>>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>>
>>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj Ayyampalayam]

Our upgrade strategy is as follows:

1) Load all disks into the storage array.
2) Create RAID pools and virtual disks.
3) Create lustre file system using mkfs.lustre command. (I still have to
figure out all the parameters used on the existing OSTs).
4) Create mount points on all OSSs.
5) Mount the lustre OSTs.
6) Maybe rebalance the filesystem.
My understanding is that the above can be done without bringing the
filesystem down. I want to create the HA configuration (corosync and
pacemaker) for the new OSTs. This step requires the filesystem to be down.
I want to know what would happen to the suspended processes across the
cluster when I bring the filesystem down to re-generate the HA configs.

Thanks,
-Raj

On Thu, Feb 21, 2019 at 12:59 AM Colin Faber  wrote:

> Can you provide more details on your upgrade strategy? In some cases
> expanding your storage shouldn't impact client / job activity at all.
>
> On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam  wrote:
>
>> Hello,
>>
>> We are planning on expanding our storage by adding more OSTs to our
>> lustre file system. It looks like it would be easier to expand if we bring
>> the filesystem down and perform the necessary operations. We are planning
>> to suspend all the jobs running on the cluster. We originally planned to
>> add new OSTs to the live filesystem.
>>
>> We are trying to determine the potential impact to the suspended jobs if
>> we bring down the filesystem for the upgrade.
>> One of the questions we have is what would happen to the suspended
>> processes that hold an open file handle in the lustre file system when the
>> filesystem is brought down for the upgrade?
>> Will they recover from the client eviction?
>>
>> We do have vendor support and have engaged them. I wanted to ask the
>> community and get some feedback.
>>
>> Thanks,
>> -Raj
>>
> ___
>> lustre-discuss mailing list
>> lustre-discuss@lists.lustre.org
>> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>>
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Colin Faber]

Can you provide more details on your upgrade strategy? In some cases
expanding your storage shouldn't impact client / job activity at all.

On Wed, Feb 20, 2019, 11:09 AM Raj Ayyampalayam  wrote:

> Hello,
>
> We are planning on expanding our storage by adding more OSTs to our lustre
> file system. It looks like it would be easier to expand if we bring the
> filesystem down and perform the necessary operations. We are planning to
> suspend all the jobs running on the cluster. We originally planned to add
> new OSTs to the live filesystem.
>
> We are trying to determine the potential impact to the suspended jobs if
> we bring down the filesystem for the upgrade.
> One of the questions we have is what would happen to the suspended
> processes that hold an open file handle in the lustre file system when the
> filesystem is brought down for the upgrade?
> Will they recover from the client eviction?
>
> We do have vendor support and have engaged them. I wanted to ask the
> community and get some feedback.
>
> Thanks,
> -Raj
> ___
> lustre-discuss mailing list
> lustre-discuss@lists.lustre.org
> http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] Suspended jobs and rebooting lustre servers

[Gin Tan]

Hi Raj,

You can add the OSTs online, we have been doing it but if you are expanding
the storage array, you might want to think about what is involved such as
cabling etc depends on the recommendation from your storage vendor. We
added an expansion on every Dell storage array last year and because of the
physical location of these storage, we needed to do a full shutdown. It
means we created a maintenance reservation and performed a full filesystem
shutdown.

In many occasions when we perform Lustre maintenance, we have suspended
jobs but that was when we know the filesystem will stay online, some
clients might get evicted during the failover but they will reconnect when
jobs were resumed.

In your case, if you want to do a full filesystem shutdown, you will have
to unmount all the Lustre clients, it means the jobs will need to be killed
in order to unmount the filesystem. We always use cat /proc/sys/lnet/peers
or lshowmount to make sure that there are no other clients connected before
doing the full FS shut down.

Hope it helps.

-- 

*Gin Tan*
MASSIVE support and consulting services

*Monash eResearch Centre*
Monash University

15 Innovation Walk
Ground Floor, Room G22
Clayton Campus
Wellington Road
Clayton VIC 3800
Australia

T: +61 3 9902 0245
E: gin@monash.edu
Z: https://monash.zoom.us/my/gintan
www.monash.edu.au/eresearch


>
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

[lustre-discuss] Suspended jobs and rebooting lustre servers

[Raj Ayyampalayam]

Hello,

We are planning on expanding our storage by adding more OSTs to our lustre
file system. It looks like it would be easier to expand if we bring the
filesystem down and perform the necessary operations. We are planning to
suspend all the jobs running on the cluster. We originally planned to add
new OSTs to the live filesystem.

We are trying to determine the potential impact to the suspended jobs if we
bring down the filesystem for the upgrade.
One of the questions we have is what would happen to the suspended
processes that hold an open file handle in the lustre file system when the
filesystem is brought down for the upgrade?
Will they recover from the client eviction?

We do have vendor support and have engaged them. I wanted to ask the
community and get some feedback.

Thanks,
-Raj
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org