Re: [lustre-discuss] LMT 3.2 - MDT display

2016-04-13 Thread Crowe, Tom 
Thanks for the explanation Olaf.

We updated LMT on a 2.7 file system, which only has a single MDT, so that 
explains why nothing “new” was showing up in the [ mdt list ] section.

Thanks,
Tom Crowe


On Apr 13, 2016, at 5:40 PM, Faaland, Olaf P. 
> wrote:

Reposting to the correct mailing list.

To: Crowe, Tom; 
lustre-de...@lists.lustre.org
Subject: Re: [lustre-devel] LMT 3.2 - MDT display

Hi Tom,

It sounds like maybe you see the summary and the OST list, but not the MDT list.

The ltop display has 3 different components:
[ summary ]
[ mdt list ]
[ ost list ]

The summary at the top has all the fields displayed in the mdt list; so if you 
have only 1 MDT, then it doesn't waste the screen space used by an MDT list, 
and just shows
[summary]
[ ost list]

Does that explain it?

thanks,
Olaf P. Faaland
Livermore Computing
phone : 925-422-2263

From: lustre-devel 
[lustre-devel-boun...@lists.lustre.org]
 on behalf of Crowe, Tom [thcr...@iu.edu]
Sent: Wednesday, April 13, 2016 1:45 PM
To: lustre-de...@lists.lustre.org
Subject: [lustre-devel] LMT 3.2 - MDT display

Greetings,

After seeing a presentation at LUG16 
(http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_LMT-Only-a-Flesh-Wound_Faaland.pdf),
 I was excited to try the updated version of LMT (3.2) as it appeared detailed 
MDT/MDS information is included in ltop output.

I have successfully built and installed LMT 3.2, but there is no display of MDT 
info.

Wondering how I might troubleshoot this further. Everything is working, its 
just not giving me the MDT/MDS info as I expected it might.

Thanks,
Tom Crowe

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] LMT 3.2 - MDT display

2016-04-13 Thread Faaland, Olaf P. 
Reposting to the correct mailing list.

To: Crowe, Tom; lustre-de...@lists.lustre.org
Subject: Re: [lustre-devel] LMT 3.2 - MDT display

Hi Tim,

It sounds like maybe you see the summary and the OST list, but not the MDT list.

The ltop display has 3 different components:
[ summary ]
[ mdt list ]
[ ost list ]

The summary at the top has all the fields displayed in the mdt list; so if you 
have only 1 MDT, then it doesn't waste the screen space used by an MDT list, 
and just shows
[summary]
[ ost list]

Does that explain it?

thanks,
Olaf P. Faaland
Livermore Computing
phone : 925-422-2263

From: lustre-devel [lustre-devel-boun...@lists.lustre.org] on behalf of Crowe, 
Tom [thcr...@iu.edu]
Sent: Wednesday, April 13, 2016 1:45 PM
To: lustre-de...@lists.lustre.org
Subject: [lustre-devel] LMT 3.2 - MDT display

Greetings,

After seeing a presentation at LUG16 
(http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_LMT-Only-a-Flesh-Wound_Faaland.pdf),
 I was excited to try the updated version of LMT (3.2) as it appeared detailed 
MDT/MDS information is included in ltop output.

I have successfully built and installed LMT 3.2, but there is no display of MDT 
info.

Wondering how I might troubleshoot this further. Everything is working, its 
just not giving me the MDT/MDS info as I expected it might.

Thanks,
Tom Crowe
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] [lustre-devel] LMT 3.2 - MDT display

2016-04-13 Thread Christopher J. Morrone 
This is more of a lustre-discuss topic, so I'm taking my reply there.

Can you describe how you installed lmt?  Most likely there is some
problem with the cerebro setup, and the data isn't getting to the node
on which you are running ltop.

There are some instructions here:

https://github.com/LLNL/lmt/wiki/Installation

There are some commands on that page that might help debug where the
problem is arising, for instance lmtmetric and cerebro-stat.

What version of Lustre are you using?

Chris

On 04/13/2016 01:45 PM, Crowe, Tom wrote:
> Greetings,
> 
> After seeing a presentation at LUG16
> (http://cdn.opensfs.org/wp-content/uploads/2016/04/LUG2016D1_LMT-Only-a-Flesh-Wound_Faaland.pdf),
> I was excited to try the updated version of LMT (3.2) as it appeared
> detailed MDT/MDS information is included in ltop output.
> 
> I have successfully built and installed LMT 3.2, but there is no display
> of MDT info. 
> 
> Wondering how I might troubleshoot this further. Everything is working,
> its just not giving me the MDT/MDS info as I expected it might.
> 
> Thanks,
> Tom Crowe
> 
> 
> ___
> lustre-devel mailing list
> lustre-de...@lists.lustre.org
> http://secure-web.cisco.com/1FcGncnFF6G41PbPKJ5EAhWJttOY4qhP58rzz-u_-ftqGBZ1-ZxXEySe7VjKLD7zDlemrLB_pMHnnYB_2hUu2x5CsPw7dozHOTdz31tZUbRroqzV9JLjAhHHZWrzUrs4CHZ-NGJF-w9q2UAZTwrAiwE88K17UtBVYaZQ3UH4ncdNi5GXqtxNpM7gDDBwedga3-G-VZqCOsGZDiGO-znc1ivUfWn46Gwdb_50p7tijvDxHnY-fhCqi6-Nti3TmRhoKGaq1yKU2aFkL1HgkFybh2tqn6SJlDwjCaxTKCRIJSqW2Ba2broz7iqoHVLPrKxLOe9VMm7snfnL0gTUjXiJnxVBQP-ypdA1oDwpDjPWiC2A/http%3A%2F%2Flists.lustre.org%2Flistinfo.cgi%2Flustre-devel-lustre.org
> 

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

2016-04-13 Thread Mohr Jr, Richard Frank (Rick Mohr) 

> On Apr 13, 2016, at 2:53 PM, Mark Hahn  wrote:
> thanks, we'll be trying the LU-5726 patch and cpu_npartitions things.
> it's quite a long thread - do I understand correctly that periodic
> vm.drop_caches=1 can postpone the issue?

Not really.  I was periodically dropping the caches as a way to monitor how 
fast the memory was leaking.  I was trying to distinguish between normal cache 
usage (which allows memory to be reclaimed) and some other non-reclaimable 
usage.  The rate at which the memory leak grows depends upon how many file 
unlinks are happening (and based on my testing, sometimes the pattern in which 
they are unlinked).  But in general, the memory usage will just continue to 
gradually grow.

> It seems odd that if this is purely a memory balance problem,
> it manifests as a 0xdeadbeef panic, rather than OOM.  While I understand
> that the oom-killer path itself need memory to operate, does this also imply 
> that some allocation in the kernel or filesystem is not checking a return 
> value?

Not sure about that one, but in my experience when nodes start to run out of 
memory, they can fail in all sorts of new and exciting ways.  IIRC, SDSC also 
had an issue with LU-5726 but the symptoms they saw were not identical to mine. 
 So maybe you are seeing the same problem manifest itself in a different way.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

2016-04-13 Thread Mark Hahn 

We had to use lustre-2.5.3.90 on the MDS servers because of memory leak.

https://jira.hpdd.intel.com/browse/LU-5726


Mark,

If you don?t have the patch for LU-5726, then you should definitely try to get that 
one.  If nothing else, reading through the bug report might be useful.  It details 
some of the MDS OOM problems I had and mentions setting vm.zone_reclaim_mode=0.  It 
also has Robin Humble?s suggestion of setting "options libcfs 
cpu_npartitions=1? (which is something that I started doing as well).


thanks, we'll be trying the LU-5726 patch and cpu_npartitions things.
it's quite a long thread - do I understand correctly that periodic
vm.drop_caches=1 can postpone the issue?  I can replicate the warning
signs mentioned in the thread (growth of Inactive(file) to dizzying 
heights when doing a lot of unlinks).


It seems odd that if this is purely a memory balance problem,
it manifests as a 0xdeadbeef panic, rather than OOM.  While I understand
that the oom-killer path itself need memory to operate, does this 
also imply that some allocation in the kernel or filesystem is not 
checking a return value?


thanks, mark hahn.
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

2016-04-13 Thread Mohr Jr, Richard Frank (Rick Mohr) 

> On Apr 13, 2016, at 8:02 AM, Tommi T  wrote:
> 
> We had to use lustre-2.5.3.90 on the MDS servers because of memory leak.
> 
> https://jira.hpdd.intel.com/browse/LU-5726

Mark,

If you don’t have the patch for LU-5726, then you should definitely try to get 
that one.  If nothing else, reading through the bug report might be useful.  It 
details some of the MDS OOM problems I had and mentions setting 
vm.zone_reclaim_mode=0.  It also has Robin Humble’s suggestion of setting 
"options libcfs cpu_npartitions=1” (which is something that I started doing as 
well).

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

2016-04-13 Thread Mohr Jr, Richard Frank (Rick Mohr) 

> On Apr 12, 2016, at 6:46 PM, Mark Hahn  wrote:
> 
> all our existing Lustre MDSes run happily with vm.zone_reclaim_mode=0,
> and making this one consistent appears to have resolved a problem
> (in which one family of lustre kernel threads would appear to spin,
> "perf top" showing nearly all time spent in spinlock_irq.  iirc.)
> 
> might your system have had a *lot* of memory?  ours tend to be fairly modest 
> (32-64G, dual-socket intel.)

I have 64 GB on my servers.

--
Rick Mohr
Senior HPC System Administrator
National Institute for Computational Sciences
http://www.nics.tennessee.edu

___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

2016-04-13 Thread Robin Humble 
Hi Mark,

On Tue, Apr 12, 2016 at 04:49:10PM -0400, Mark Hahn wrote:
>One of our MDSs is crashing with the following:
>
>BUG: unable to handle kernel paging request at deadbeef
>IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
>PGD 0
>Oops: 0002 [#1] SMP
>
>The MDS is running 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
>with about 2k clients ranging from 1.8.8 to 2.6.0

I saw an identical crash in Sep 2014 when the MDS was put under memory
pressure.

>to be related to vm.zone_reclaim_mode=1.  We also enabled quotas

zone_reclaim_mode should always be 0. 1 is broken. hung processes
perpetually 'scanning' in one zone in /proc/zoneinfo whilst plenty of
pages are free in another zone is a sure sign of this issue.

however if you have vm.zone_reclaim_mode=0 now and are still seeing the
issue, then I would suspect that lustre's overly agresssive memory
affinity code is partially to blame. at the very least it is most
likely stopping you from making use of half your MDS ram.

see
  https://jira.hpdd.intel.com/browse/LU-5050

set
  options libcfs cpu_npartitions=1
to fix it. that's what I use on OSS and MDS for all my clusters.

cheers,
robin
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org

Re: [lustre-discuss] MDS crashing: unable to handle kernel paging request at 00000000deadbeef (iam_container_init+0x18/0x70)

2016-04-13 Thread Tommi T 
Hi,

We had to use lustre-2.5.3.90 on the MDS servers because of memory leak.

https://jira.hpdd.intel.com/browse/LU-5726

I'm not sure if it's related but worthwhile to check.

BR,
Tommi



- Original Message -
From: Mark Hahn 
To: lustre-discuss@lists.lustre.org
Sent: Tuesday, April 12, 2016 11:49 PM
Subject: [lustre-discuss] MDS crashing: unable to handle kernel paging request 
at deadbeef (iam_container_init+0x18/0x70)

One of our MDSs is crashing with the following:

BUG: unable to handle kernel paging request at deadbeef
IP: [] iam_container_init+0x18/0x70 [osd_ldiskfs]
PGD 0
Oops: 0002 [#1] SMP

The MDS is running 2.5.3-RC1--PRISTINE-2.6.32-431.23.3.el6_lustre.x86_64
with about 2k clients ranging from 1.8.8 to 2.6.0

I'd appreciate any comments on where to point fingers: google doesn't
provide anything suggestive about iam_container_init.

Our problem seems to correlate with an unintentional creation of a tree 
of >500M files.  Some of the crashes we've had since then appeared
to be related to vm.zone_reclaim_mode=1.  We also enabled quotas 
right after the 500M file thing, and were thinking that inconsistent
quota records might cause this sort of crash.

But 0xdeadbeef is usually added as a canary for allocation issues;
is it used this way in Lustre?

thanks,
Mark Hahn | SHARCnet Sysadmin | h...@sharcnet.ca | http://www.sharcnet.ca
   | McMaster RHPCS| h...@mcmaster.ca | 905 525 9140 x24687
   | Compute/Calcul Canada| http://www.computecanada.ca
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
___
lustre-discuss mailing list
lustre-discuss@lists.lustre.org
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org