scale.v5r04.doc/bl1ins_gatewaynodefailureafm.htm
>
> ~Venkat (vpuvv...@in.ibm.com)
>
>
>
> From: Matt Weil
> To: gpfsug-discuss@spectrumscale.org
> Date: 03/25/2020 10:34 PM
> Subject: [EXTERNAL] Re: [gpfsug-discuss] AFM gateway node scaling
> Sent by:
per gateway). AFM gateway nodes are
> licensed as server nodes.
>
>
> ~Venkat (vpuvv...@in.ibm.com)
>
>
>
> From: Matt Weil
> To: gpfsug-discuss@spectrumscale.org
> Date: 03/23/2020 11:39 PM
> Subject: [EXTERNAL] [gpf
Hello all,
Is there any guide and or recommendation as to how to scale this.
filesets per gateway node? Is it necessary to separate NSD server and
gateway roles. Are dedicated gateway nodes licensed as clients?
Thanks for any guidance.
Matt
___
any input on this
Thanks
On 7/5/17 10:51 AM, Matt Weil wrote:
> Hello all,
>
> Question on the requirements on pmcollector node/s for a 500+ node
> cluster. Is there a sizing guide? What specifics should we scale?
> CPU Disks memory?
>
Hello all,
what happens if we set socketMaxListenConnections to a larger number
than we have clients? more memory used?
Thanks
Matt
The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive
yes it tells you that when you attempt to make the node a gateway and is does
not have a server license designation.
On 4/12/17 4:53 AM, Venkateswara R Puvvada wrote:
Gateway node requires server license.
~Venkat (vpuvv...@in.ibm.com<mailto:vpuvv...@in.ibm.com>)
From: Matt We
o:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Matt Weil
Sent: Thursday, March 23, 2017 10:24 AM
To: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale.org>
Subject: Re: [gpfsug-discuss] CES node slow to respond
FYI all
We also ran into this after bumping maxFilesToC
also running version 4.2.2.2.
On 3/24/17 2:57 PM, Matt Weil wrote:
On 3/24/17 1:13 PM, Bryan Banister wrote:
Hi Vipul,
Hmm… interesting. We have dedicated systems running CES and nothing else, so
the only thing opening files on GPFS is ganesha. IBM Support recommended we
massively
Message-
From:
gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>
[mailto:gpfsug-discuss-boun...@spectrumscale.org] On Behalf Of Matt Weil
Sent: Thursday, March 23, 2017 10:24 AM
To: gpfsug-discuss@spectrumscale.org<mailto:gpfsug-discuss@spectrumscale
ver to the right networks.
>
> On Thu, Mar 23, 2017 at 10:35:30AM -0500, Matt Weil wrote:
>> Hello all,
>>
>> Are there any issues with connecting CES nodes to multiple networks?
The materials in this message are private and may contain P
Hello all,
Are there any issues with connecting CES nodes to multiple networks?
Thanks
Matt
The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature. If you are not the
intended
ug-discuss-boun...@spectrumscale.org] On Behalf Of Matt Weil
> Sent: Wednesday, March 22, 2017 11:43 AM
> To: gpfsug main discussion list <gpfsug-discuss@spectrumscale.org>
> Subject: [gpfsug-discuss] CES node slow to respond
>
> All,
>
> We had an indecent yesterday where
All,
We had an indecent yesterday where one of our CES nodes slowed to a
crawl. GPFS waiters showed pre fetch threads going after inodes.
iohist also showed lots of inode fetching. Then we noticed that the CES
host had 5.4 million files open.
The change I made was to set maxStatCache=DEFAULT
Hello all
Is this necessary any more?
numastat -p mmfsd
seems to spread it out without it.
Thanks
Matt
The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature. If you are not the
Hello all,
Some how we misplaced the password for our dev instance. Is there any
way to reset it?
Thanks
Matt
The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature. If you are
lue of max sectors of the block device.
-jf
On Sat, Feb 11, 2017 at 7:32 PM, Matt Weil
<mw...@wustl.edu<mailto:mw...@wustl.edu>> wrote:
https://access.redhat.com/solutions/2437991
I ran into this issue the other day even with the echo "4096" >
/sys/block/$ii/queu
https://access.redhat.com/solutions/2437991
I ran into this issue the other day even with the echo "4096" >
/sys/block/$ii/queue/max_sectors_kb; in place. I have always made that
larger to get to the 2M IO size. So I never really seen this issue
until the other day. I may have triggered it
running cnfs
# rpm -qa | grep gpfs
gpfs.gpl-4.1.1-7.noarch
gpfs.base-4.1.1-7.x86_64
gpfs.docs-4.1.1-7.noarch
gpfs.gplbin-3.10.0-327.18.2.el7.x86_64-4.1.1-7.x86_64
pcp-pmda-gpfs-3.10.6-2.el7.x86_64
gpfs.ext-4.1.1-7.x86_64
gpfs.gskit-8.0.50-47.x86_64
gpfs.msg.en_US-4.1.1-7.noarch
=== mmdiag:
I still see small 4k IO's going to the nvme device after changing the
max_sectors_kb. Writes did increase from 64 to 512. Is that a nvme
limitation.
> [root@ces1 system]# cat /sys/block/nvme0n1/queue/read_ahead_kb
> 8192
> [root@ces1 system]# cat /sys/block/nvme0n1/queue/nr_requests
> 512
>
100% utilized are bursts above 200,000 IO's. Any way to tell ganesha.nfsd to
cache more?
On 1/25/17 3:51 PM, Matt Weil wrote:
[ces1,ces2,ces3]
maxStatCache 8
worker1Threads 2000
maxFilesToCache 50
pagepool 100G
maxStatCache 8
lrocData no
378G system memory.
On 1/25/17 3:29
off
also did you increase maxstatcache so LROC actually has some compact objects to
use ?
if you send value for maxfilestocache,maxfilestocache,workerthreads and
available memory of the node i can provide a start point.
On Wed, Jan 25, 2017 at 10:20 PM Matt Weil
<mw...@wustl.edu<mai
way more than they ever where before. I guess we
will need another nvme.
sven
On Wed, Jan 25, 2017 at 9:50 PM Matt Weil
<mw...@wustl.edu<mailto:mw...@wustl.edu>> wrote:
Hello all,
We are having an issue where the LROC on a CES node gets overrun 100%
utilized. Processes then sta
Hello all,
We are having an issue where the LROC on a CES node gets overrun 100%
utilized. Processes then start to backup waiting for the LROC to
return data. Any way to have the GPFS client go direct if LROC gets to
busy?
Thanks
Matt
The materials in this
All,
I typically turn Hyper threading off on storage nodes. So I did on our
CES nodes as well. Now they are running at a load of over 100 and have
25% cpu idle. With two 8 cores I am now wondering if hyper threading
would help or did we just under size them :-(.
These are nfs v3 servers only
drew Beattie
> Software Defined Storage - IT Specialist
> Phone: 614-2133-7927
> E-mail: abeat...@au1.ibm.com <mailto:abeat...@au1.ibm.com>
>
>
>
> - Original message -
> From: Matt Weil <mw...@wustl.edu>
>
> again.
>
> My clients environment is currently deployed on Centos 7.
> Andrew Beattie
> Software Defined Storage - IT Specialist
> Phone: 614-2133-7927
> E-mail: abeat...@au1.ibm.com <mailto:abeat...@au1.ibm.com>
>
>
>
> - Original message ---
this follows the IP what ever node the ip lands on. the ganesha.nfsd
process seems to stop working. any ideas? there is nothing helpful in
the logs.
time mount ces200:/vol/aggr14/temp403 /mnt/test
mount.nfs: mount system call failed
real1m0.000s
user0m0.000s
sys 0m0.010s
MB, currently in use: 0 MB
> Statistics from: Thu Dec 29 10:35:32 2016
>
> Total objects stored 0 (0 MB) recalled 0 (0 MB)
> objects failed to store 0 failed to recall 0 failed to inval 0
> objects queried 0 (0 MB) not found 0 = 0.00 %
> objects invalidated 0 (
rom: Thu Dec 29 10:08:58 2016
It is not caching however. I will restart gpfs to see if that makes it
start working.
On 12/29/16 10:18 AM, Matt Weil wrote:
>
>
>
> On 12/29/16 10:09 AM, Sven Oehme wrote:
>> i agree that is a very long name , given this is a nvme device it
>
osName = Linux ]]
> then
> : # Add function to discover disks in the Linux environment.
> for luns in `ls /dev/disk/by-id | grep nvme`
> do
> all_luns=disk/by-id/$luns
> echo $all_luns dmm
> done
>
> fi
>
I will try tha
avage "well there's yer problem". Are you
> perhaps running a version of GPFS 4.1 older than 4.1.1.9? Looks like
> there was an LROC related assert fixed in 4.1.1.9 but I can't find
> details on it.
>
>
>
> *From:*Matt Weil
> *Sent:* 12/28/16, 5:21 PM
> *To:
yes
> Wed Dec 28 16:17:07.507 2016: [X] *** Assert exp(ssd->state !=
> ssdActive) in line 427 of file
> /project/sprelbmd1/build/rbmd11027d/src/avs/fs/mmfs/ts/flea/fs_agent_gpfs.C
> Wed Dec 28 16:17:07.508 2016: [E] *** Traceback:
> Wed Dec 28 16:17:07.509 2016: [E] 2:0x7FF1604F39B5
>
cache metadata or also data associated to the
> files ?
>
> sven
>
>
>
> On Tue, Dec 20, 2016 at 5:35 PM Matt Weil <mw...@wustl.edu
> <mailto:mw...@wustl.edu>> wrote:
>
>
> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Gen
Hello all,
Thanks for sharing that. I am setting this up on our CES nodes. In this
example the nvme devices are not persistent. RHEL's default udev rules put
them in /dev/disk/by-id/ persistently by serial number so I modified
mmdevdiscover to look for them there. What are others doing?
I always suspend the disk then use mmrestripefs -m to remove the data. Then
delete the disk with mmdeldisk.
‐m
Migrates all critical data off of any suspended
disk in this file system. Critical data is all
data that would be lost if
http://www.emc.com/collateral/TechnicalDocument/docu5128.pdf
page 219
this is the default in rhel.
device {
vendor "EMC"
product "SYMMETRIX"
path_grouping_policy multibus
getuid_callout "/lib/udev/scsi_id --page=pre-spc3-83 --whitelisted
--device=/dev/%n"
> #manual install protocal nodes
> yum install nfs-ganesha-2.3.2-0.ibm24_2.el7.x86_64
> nfs-ganesha-gpfs-2.3.2-0.ibm24_2.el7.x86_64
> nfs-ganesha-utils-2.3.2-0.ibm24_2.el7.x86_64
> gpfs.smb-4.3.11_gpfs_21-8.el7.x86_64 spectrum-scale-object-4.2.1-1.noarch
>
> there is a dependancy problem with
anybody no what this means?
Wed Oct 26 20:08:29.619 2016: [D] Leave protocol detail info: LA: 75
LFLG: 24409951 LFLG delta: 75
The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature.
All,
What exactly happens that makes the clients hang when a file set inodes
are increased?
The materials in this message are private and may contain Protected Healthcare
Information or other information of a sensitive nature. If you are not the
intended
Hello all,
Just brain storming here mainly but want to know how you are all
approaching this. Do you replicate using GPFS and forget about backups?
> https://www.ibm.com/support/knowledgecenter/STXKQY_4.2.1/com.ibm.spectrum.scale.v4r21.doc/bl1adv_osbackup.htm
This seems good for a full recovery
On 7/24/16 5:27 AM, Stef Coene wrote:
> Hi,
>
> Like the subject says, I'm new to Spectrum Scale.
>
> We are considering GPFS as back end for CommVault back-up data.
> Back-end storage will be iSCSI (300 TB) and V5000 SAS (100 TB).
> I created a 2 node cluster (RHEL) with 2 protocol nodes and 1
Hello all,
> https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/General%20Parallel%20File%20System%20(GPFS)/page/Sizing%20Guidance%20for%20Protocol%20Node
>
> Is there any more guidance on this as one socket can be a lot of cores and
> memory today.
>
> Thanks
>
a limited set
> of nodes.
>
> Bob Oesterlin
> Sr Storage Engineer, Nuance HPC Grid
> 507-269-0413
>
>
> From: <gpfsug-discuss-boun...@spectrumscale.org
> <mailto:gpfsug-discuss-boun...@spectrumscale.org>> on behalf of Matt
> Weil <mw...@genome.wustl.edu <mailto:m
Fri Mar 18 11:50:43 CDT 2016: mmcesop: /vol/system/ found but is not on
a GPFS filesystem
On 3/18/16 11:39 AM, Matt Weil wrote:
> upgrading to 4.2.2 fixed the dependency issue. I now get Unable to
> access CES shared root.
>
> # /usr/lpp/mmfs/bin/mmlsconfig | grep 'cesSharedRoot'
>
>
>
> From:
> <gpfsug-discuss-boun...@spectrumscale.org<mailto:gpfsug-discuss-boun...@spectrumscale.org>>
> on behalf of Matt Weil
> <mw...@genome.wustl.edu<mailto:mw...@genome.wustl.edu>>
> Reply-To: gpfsug main discussion list
> <gpf
pfsug-discuss-boun...@spectrumscale.org
> [gpfsug-discuss-boun...@spectrumscale.org] on behalf of Matt Weil
> [mw...@genome.wustl.edu]
> Sent: 16 March 2016 19:37
> To: gpfsug main discussion list
> Subject: Re: [gpfsug-discuss] 4.2 installer
>
> any help here?
> ~]# yum -d0
thernet?
>
> I have seen problems that look like yours in the past with single-network
> Ethernet setups.
>
> Regards,
>
> Vic
>
> Sent from my iPhone
>
>> On 2 Mar 2016, at 20:54, Matt Weil <mw...@genome.wustl.edu> wrote:
>>
>> Can you share
Paul,
We currently run netbackup to push about 1.3PB of real data to tape.
This using 1 nb master and a single media server that is also a GPFS
client. The media server uses the spare file system space as a staging
area before writing to tape. We have recently invested into a TSM
server
All,
Just made a new cluster added 4 OSD and the pgs seem to be stuck on
creation.
cluster 9cccdd9d-8845-4f4f-b8f4-d2f85469f958
health HEALTH_WARN
320 pgs stuck inactive
320 pgs stuck unclean
monmap e1: 2 mons at
Is working now thanks.
On 8/17/15 2:54 PM, Matt Weil wrote:
All,
Just made a new cluster added 4 OSD and the pgs seem to be stuck on
creation.
cluster 9cccdd9d-8845-4f4f-b8f4-d2f85469f958
health HEALTH_WARN
320 pgs stuck inactive
320 pgs stuck unclean
Hello all,
Is this possible?
This email message is a private communication. The information transmitted,
including attachments, is intended only for the person or entity to which it is
addressed and may contain confidential, privileged, and/or proprietary
material. Any review,
Message-
From: ceph-devel-ow...@vger.kernel.org [mailto:ceph-devel-
ow...@vger.kernel.org] On Behalf Of Matt Weil
Sent: Wednesday, August 12, 2015 10:26 PM
To: Ceph Development
Subject: create OSD on IBM's GPFS file system
Hello all,
Is this possible?
This email message is a private
I have a system with a bunch or ram that I want to remain the active MDS
but still have a backup.
This config doesn't seem to be working. I can make linuscs92 the active
by stopping and starting the mds on linuscs95. It would be nice for
linuscs92 to be the active from the start.
Hi Matt,
Can you post your ceph config? Once you startup your ceph cluster, you
see that linuscs92 is the standby and linuscs95 is the active? How are
you starting your cluster?
service ceph -a start
and yes linuscs95 comes out as active.
[global]
; enable secure authentication
Is it possible to filter disks and show only things under /dev/mapper/
Thanks
Matt
--
The Windows 8 Center - In partnership with Sourceforge
Your idea - your app - 30 days.
Get started!
be cost prohibitive.
On Tue, Aug 28, 2012 at 11:44 AM, Joe Topjian j...@topjian.net wrote:
Hi Matt,
On Tue, Aug 28, 2012 at 9:29 AM, Matt Weil mw...@genome.wustl.edu wrote:
Since we are on the subject of hardware what would be the perfect fit for
a gluster brick. We where looking at a PowerEdge
where bonded together. Are
there any cheap 10 gig switch alternatives that anyone would suggest?
Matt
On 8/24/12 4:28 PM, Brian Candler wrote:
On Fri, Aug 24, 2012 at 10:51:24AM -0500, Matt Weil wrote:
I am curious what is used typically for the file system replication
and how do you make sure
Is it possible to get IO sizes 1024k and larger?
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On 7/13/12 3:02 PM, Tommi Virtanen wrote:
On Fri, Jul 13, 2012 at 12:07 PM, Matt Weil mw...@genome.wustl.edu wrote:
Is it possible to get IO sizes 1024k and larger?
Can you be a bit more explicit in what you're asking?
Are you talking about submitting IO to RBD (via the kernel module?
inside
CXXlibosdc_la-Objecter.lo
CXXlibosdc_la-ObjectCacher.lo
CXXlibosdc_la-Filer.lo
osdc/ObjectCacher.cc: In member function 'int
ObjectCacher::_readx(ObjectCacher::OSDRead*, ObjectCacher::ObjectSet*,
Context*, bool)':
osdc/ObjectCacher.cc:1043: error: 'INT_MAX' was not declared in
build/obj/util.do: Compiling dynamic object
src/simplexml.c:27:27: error: libxml/parser.h: No such file or directory
src/simplexml.c:48: error: expected '=', ',', ';', 'asm' or '__attribute__'
before 'saxGetEntity'
cc1: warnings being treated as errors
src/simplexml.c:56: error: type defaults
I recreated the file system and got this on the client until it was
rebooted.
Is there a step I missed?
libceph: bad fsid, had 02f7ef57-25e0-475f-8948-5c562a4d370c got
6089a28f-8ff4-42f5-8825-1a929f5770bd
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of
How do you do this type of config (specifying multiple devices per
server) on xfs not btrfs?
[osd.1]
host = linuscs92
btrfs devs = /dev/cciss/c1d2
[osd.2]
host = linuscs92
btrfs devs = /dev/cciss/c1d3
[osd.3]
host = linuscs92
btrfs devs =
Hello all:
given six osd servers does it make any difference of each one also runs
a mds or not?
How many mds's is to many. How does this effect metadata performance.
Thanks
Matt
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to
On 12/20/11 5:39 PM, Gregory Farnum wrote:
On Tue, Dec 20, 2011 at 3:30 PM, Matt Weilmw...@genome.wustl.edu wrote:
Hello all:
given six osd servers does it make any difference of each one also runs a
mds or not?
How many mds's is to many. How does this effect metadata performance.
Right
another butter bug?
btrfs: open_ctree failed
[ cut here ]
WARNING: at fs/btrfs/inode.c:2194 btrfs_orphan_commit_root+0xb0/0xc0
[btrfs]()
Hardware name: ProLiant DL380 G5
Modules linked in: btrfs zlib_deflate nfsd exportfs autofs4 nfs lockd
fscache auth_rpcgss
All,
Is this normal? Can this be corrected?
Thanks in advance for your responses.
[2011-12-06 17:56:59.48153] I
[glusterd-rpc-ops.c:1243:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC
from uuid: fc5e6659-a90a-4e25-a3a7-11de9a7de81d
[2011-12-06 17:56:59.48811] I
Just some simply iozone testing failed due to folder permissions. The
tmp folder is created by iozone.
six nodes with stripe 6 and underlying EXT4 file system.
The ext4 filesystems where not mounted with the -o acl option.
Any Ideas?
in both cases it created a folder with no permissions.
On 12/7/11 12:53 PM, Matt Weil wrote:
Just some simply iozone testing failed due to folder permissions. The
tmp folder is created by iozone.
six nodes with stripe 6 and underlying EXT4 file system.
The ext4 filesystems where not mounted with out the -o acl option.
I meant with out the -o
are there any options to have glusterd use syslog?
Would like the logs to go to a central server.
Thanks
Matt
___
Gluster-users mailing list
Gluster-users@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
On 04/15/2011 12:47 PM, Noah Watkins wrote:
On Apr 14, 2011, at 8:47 PM, Qian Wei wrote:
Hello everyonel:
Recently, I have put up a simple test platform. Now I want to test
the MDS cluster's performance,but I don't how to get it. Can anyone
give me some advices or some tools may help
.
see attached.
Yehuda
On Thu, Jun 10, 2010 at 8:58 AM, Matt Weil mw...@watson.wustl.edu wrote:
anyone seen this?
ceph client
Jun 10 08:45:22 linuscs102 kernel: [174034.263762] [81144e05] ?
get_write_access+0x45/0x70
Jun 10 08:45:22 linuscs102 kernel: [174034.263762
anyone seen this?
ceph client
inline: ceph_linuscs102.gif
73 matches
Mail list logo
