Re: [OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Volker A. Brandt]

Hi Dan!


Thanks for all the work you're doing on OmniOS!

> > On Oct 22, 2015, at 1:11 PM, Dan McDonald 
> > wrote:
> >
> > "pkg update" followed by "svcadm restart ntp" as a safety measure
> > should be sufficient.  No rebooting is needed.
>
> I made a small mistake here.  The "svcadm restart..." is not
> necessary, the IPS package does the right thing here.

Well, no, it doesn't. :-)  That's due to a design flaw in the interaction
between IPS and SMF (IMHO).  Even though the manifest object in the package
is properly tagged with restart_fmri, the service is never restarted,
because the manifest is not touched during the "pkg update", as it has not
changed since the last package version.

So if you change an SMF method in a package and want an "automatic"
restart, you  need to also physically modify the SMF manifest.  I do that
by just incrementing a version counter or a timestamp, and noting the
fact in an XML comment in the manifest.  Otherwise, you need to manually
restart the service.


Unrelated, when I updated my local copy of the r151014 repo in preparation
of the pkg udpate for ntp, I got this:

  Processing packages for publisher omnios ...
  Retrieving and evaluating 6161 package(s)...
  Download Manifests ( 907/6161) -pkgrecv: http protocol error: code: 404 
reason: Not Found
  URL: 
'http://pkg.omniti.com/omnios/r151014/omnios/manifest/0/developer%2Fillumos-tools@11%2C5.11-0.151014%3A20151016T122410Z'
 (happened 4 times)

  Processing packages for publisher omnios ...
  Retrieving and evaluating 1030 package(s)...
  PROCESS ITEMSGET (MB)   SEND (MB)
  Completed 1/1 2.4/2.4 4.9/4.9


So the recent addition of the illumos-tools pkg broke something in your
repo.  I worked around that by specifying the service/network/ntp pkg
in the pkgrecv invocation.


Regards -- Volker
-- 

Volker A. Brandt   Consulting and Support for Oracle Solaris
Brandt & Brandt Computer GmbH   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANYEmail: v...@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513  Schuhgröße: 46
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt

"When logic and proportion have fallen sloppy dead"
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Min Kim]

Are you using the same record size of 4K on your zfs pool as you used with your 
linux test system?

If the record size for the zpool and slog is set at the default value of 128K, 
it will greatly reduce the measured IOPS relative to that measured with a 
recordsize of 4K.

Min Kim




> On Oct 22, 2015, at 12:26 PM, Matej Zerovnik  wrote:
> 
> Interesting…
> 
> Although, I’m not sure if this is really the problem.
> 
> For test, I booted up linux and put both ZeusRAM to raid1 software raid and 
> repeated the test. I got full 48kIOPS in the test, meaning there was 96kIOPS 
> sent to JBOD (48k IOPS for each drive).
> 
> On the OmniOS test bed, there are 28k IOPS sent to ZIL and X amount to 
> spindles when flushing write cache, but no more then 1000 IOPS (100 
> iops/drive * 10). Comparing that to the case above, IOPS shouldn’t be a limit.
> 
> Maybe I could try building my pools with hard drives that aren’t near ZIL 
> drive, which is in bay 0. I could take hard drives from bays 4-15, which 
> probably use different SAS lanes.
> 
> lp, Matej
> 
> 
>> On 22 Oct 2015, at 21:10, Min Kim > > wrote:
>> 
>> I believe this is an known issue with SAS expanders.
>> 
>> Please see here:
>> 
>> http://serverfault.com/questions/242336/sas-expanders-vs-direct-attached-sas 
>> 
>> 
>> When you are stress-testing the Zeusram by itself, all the IOPs and 
>> bandwidth of the expander are allocated to that device alone.  Once you add 
>> all the other drives, you lose some of that as you have to share it with the 
>> other disks.
>> 
>> Min Kim
>> 
>> 
>> 
>>> On Oct 22, 2015, at 12:02 PM, Matej Zerovnik >> > wrote:
>>> 
>>> Hello,
>>> 
>>> I'm building a new system and I'm having a bit of a performance problem. 
>>> Well, its either that or I'm not getting the whole ZIL idea:)
>>> 
>>> My system is following:
>>> - IBM xServer 3550 M4 server (dual CPU with 160GB memory)
>>> - LSI 9207 HBA (P19 firmware)
>>> - Supermicro JBOD with SAS expander
>>> - 4TB SAS3 drives
>>> - ZeusRAM for ZIL
>>> - LTS Omnios (all patches applied)
>>> 
>>> If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 48k 
>>> IOPS out of it, no problem there.
>>> 
>>> If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for 
>>> ZIL and set sync=always, I can only squeeze 14k IOPS out of the system.
>>> Is that normal or should I be getting 48k IOPS on the 2nd pool as well, 
>>> since this is the performance ZeusRAM can deliver?
>>> 
>>> I'm testing with fio:
>>> fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers 
>>> --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 
>>> --numjobs=16 --runtime=60 --group_reporting --name=4ktest
>>> 
>>> thanks, Matej___
>>> OmniOS-discuss mailing list
>>> OmniOS-discuss@lists.omniti.com 
>>> http://lists.omniti.com/mailman/listinfo/omnios-discuss 
>>> 
>> 
> 

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Matej Zerovnik]

Interesting…

Although, I’m not sure if this is really the problem.

For test, I booted up linux and put both ZeusRAM to raid1 software raid and 
repeated the test. I got full 48kIOPS in the test, meaning there was 96kIOPS 
sent to JBOD (48k IOPS for each drive).

On the OmniOS test bed, there are 28k IOPS sent to ZIL and X amount to spindles 
when flushing write cache, but no more then 1000 IOPS (100 iops/drive * 10). 
Comparing that to the case above, IOPS shouldn’t be a limit.

Maybe I could try building my pools with hard drives that aren’t near ZIL 
drive, which is in bay 0. I could take hard drives from bays 4-15, which 
probably use different SAS lanes.

lp, Matej


> On 22 Oct 2015, at 21:10, Min Kim  wrote:
> 
> I believe this is an known issue with SAS expanders.
> 
> Please see here:
> 
> http://serverfault.com/questions/242336/sas-expanders-vs-direct-attached-sas 
> 
> 
> When you are stress-testing the Zeusram by itself, all the IOPs and bandwidth 
> of the expander are allocated to that device alone.  Once you add all the 
> other drives, you lose some of that as you have to share it with the other 
> disks.
> 
> Min Kim
> 
> 
> 
>> On Oct 22, 2015, at 12:02 PM, Matej Zerovnik > > wrote:
>> 
>> Hello,
>> 
>> I'm building a new system and I'm having a bit of a performance problem. 
>> Well, its either that or I'm not getting the whole ZIL idea:)
>> 
>> My system is following:
>> - IBM xServer 3550 M4 server (dual CPU with 160GB memory)
>> - LSI 9207 HBA (P19 firmware)
>> - Supermicro JBOD with SAS expander
>> - 4TB SAS3 drives
>> - ZeusRAM for ZIL
>> - LTS Omnios (all patches applied)
>> 
>> If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 48k 
>> IOPS out of it, no problem there.
>> 
>> If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for ZIL 
>> and set sync=always, I can only squeeze 14k IOPS out of the system.
>> Is that normal or should I be getting 48k IOPS on the 2nd pool as well, 
>> since this is the performance ZeusRAM can deliver?
>> 
>> I'm testing with fio:
>> fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers 
>> --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 
>> --numjobs=16 --runtime=60 --group_reporting --name=4ktest
>> 
>> thanks, Matej___
>> OmniOS-discuss mailing list
>> OmniOS-discuss@lists.omniti.com 
>> http://lists.omniti.com/mailman/listinfo/omnios-discuss
> 



smime.p7s
Description: S/MIME cryptographic signature
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] ILB memory leak?

[Al Slater]

On 21/10/2015 17:35, Dan McDonald wrote:



On Oct 21, 2015, at 6:08 AM, Al Slater 
wrote:

Hi,

I am running omnios r151014 on a couple of machines with a couple
of zones each.  1 zone runs apache as an SSL reverse proxy, the
other runs ILB for load balancing web to app tier connections.

I noticed that in the ILB zone, the ilbd process memory grows to
about 2Gb.   Restarting ILB releases the memory, and then the
memory usage gradually increases again, with each memory increase
approximately 2 * the size of the previous one.  I run a cronjob
twice a day ( 8am and 8pm) which restarts the ilb service and
releases the memory.

A graph of memory usage is available at
https://www.dropbox.com/s/zaz51apxslnivlq/ILB_Memory_2_days.png?dl=0


>> There are currently 62 rules in the load balancer, with a
>> total

of 664 server/port pairs.

Is there anything I can provide that would help track this down?


You can use svccfg(1M) to enable user-level memory debugging on ilb.
 It may cause the ilb daemon to dump core.  (And you're just noticing
 this in the process, not kernel memory consumption, correct?)


I am seeing kernel memory consumption increasing as well, but that may 
be a different issue.  The ilbd process memory is definitely growing.



As root:

svcadm disable -t ilb svccfg -s ilb setenv LD_PRELOAD libumem.so
svccfg -s ilb setenv UMEM_DEBUG default svccfg -s ilb refresh svcadm
 enable ilb

That should enable user-level memory debugging.  If you get a
coredump, save it and share it.  If you don't and the ilb daemon
keeps running, eventually please:

gcore `pgrep ilbd`

and share THAT corefile.  You can also do this by youself:

mdb  > ::findleaks

and share ::findleaks.

Once you're done generating corefiles, repeat the steps above, but
use "unsetenv LD_PRELOAD" and "unsetenv UMEM_DEBUG" instead of the
setenv lines.


Thanks Dan.  As we are talking about production boxes here, I will have 
to try and reproduce on another box and then I will give the process 
above a go and see what we come up with.


--
Al Slater

Technical Director
SCL

Phone : +44 (0)1273 07
Fax   : +44 (0)1273 01
email : al.sla...@scluk.com

Stanton Consultancy Ltd

Park Gate, 161 Preston Road, Brighton, East Sussex, BN1 6AU

Registered in England Company number: 1957652 VAT number: GB 760 2433 55

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Schweiss, Chip]

The ZIL on log devices suffer a bit from not filling queues well.   In
order to get the queues to fill more, try running your test to several zfs
folders on the pool simultaneously and measure your total I/O.

As I understand it, ff you're writing to only one zfs folder, your queue
depth will stay at 1 on the log device and you be come latency bound.

-Chip

On Thu, Oct 22, 2015 at 2:02 PM, Matej Zerovnik  wrote:

> Hello,
>
> I'm building a new system and I'm having a bit of a performance problem.
> Well, its either that or I'm not getting the whole ZIL idea:)
>
> My system is following:
> - IBM xServer 3550 M4 server (dual CPU with 160GB memory)
> - LSI 9207 HBA (P19 firmware)
> - Supermicro JBOD with SAS expander
> - 4TB SAS3 drives
> - ZeusRAM for ZIL
> - LTS Omnios (all patches applied)
>
> If I benchmark ZeusRAM on its own with random 4k sync writes, I can get
> 48k IOPS out of it, no problem there.
>
> If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for
> ZIL and set sync=always, I can only squeeze 14k IOPS out of the system.
> Is that normal or should I be getting 48k IOPS on the 2nd pool as well,
> since this is the performance ZeusRAM can deliver?
>
> I'm testing with fio:
> fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers
> --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16
> --numjobs=16 --runtime=60 --group_reporting --name=4ktest
>
> thanks, Matej
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss
>
>
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Bob Friesenhahn]

On Thu, 22 Oct 2015, Matej Zerovnik wrote:


If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for ZIL 
and set sync=always, I can only squeeze 14k IOPS out of the system.
Is that normal or should I be getting 48k IOPS on the 2nd pool as well, since 
this is the performance ZeusRAM can deliver?


Is your zfs filesystem using 4k blocks?  Random writes may also 
require random reads due to COW.  If the data is not perfectly aligned 
and perfectly fill the underlying zfs block, and the existing data is 
not already cached in ARC, then it needs to be read from underlying 
store so existing data can be modified during the write.


I do see that you are using asynchronous I/O, which may add more 
factors.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] OmniOS backup box hanging regularly

[Lauri Tirkkonen]

On Thu, Oct 22 2015 20:51:32 +0200, Jim Klimov wrote:
> Did anyone run into issues with many zfs-auto-snapshots (e.g.
> thousands - many datasets and many snaps until they are killed to keep
> some 200gb free) on a small nujber of spindles?

Not with that number of snapshots, but we had several thousand
filesystems with dozens (I think about 70) of snapshots per fs, and it
turned out to be a really bad idea due to the memory requirements:
things slowed down *a lot*. We didn't see any hangs though.

I have a pool with around four thousand snapshots total at home and that
box is performing just fine, and it's just two spinning disks + two SSDs
for cache/slog.

-- 
Lauri Tirkkonen | lotheac @ IRCnet
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

[OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Dan McDonald]

The NTP software was updated to 4.2.8p4 yesterday.  I've pushed out updates for 
r151006, r151014, and bloody.  As I mentioned earlier, r151012 users should 
update to r151014.

"pkg update" followed by "svcadm restart ntp" as a safety measure should be 
sufficient.  No rebooting is needed.

NTP's update on this patch is here:


http://support.ntp.org/bin/view/Main/SecurityNotice#Recent_Vulnerabilities

Thanks,
Dan

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] OmniOS backup box hanging regularly

[Yavor Tomov]

Hi Tovarishch Jim,

I had similar issue with my box and it was related to the NFS locks. I
assume you are using it due to the Linux backups. The solution was posted
by Chip on the mailing list. Copy of his solution below:

"I've seen issues like this when you run out of NFS locks.   NFSv3 in
Illumos is really slow at releasing locks.

On all my NFS servers I do:

sharectl set -p lockd_listen_backlog=256 nfs
sharectl set -p lockd_servers=2048 nfs

Everywhere I can, I use NFSv4 instead of v3.   It handles lock much better."

All the Best
Yavor



On Thu, Oct 22, 2015 at 11:59 AM, Jim Klimov  wrote:

> Hello all,
>
> I have this HP-Z400 workstation with 16Gb ECC(should be) RAM running
> OmniOS bloody, which acts as a backup server for our production systems
> (regularly rsync'ing large files off Linux boxes, and rotating ZFS
> auto-snapshots to keep its space free). Sometimes it also runs replicas of
> infrastructure (DHCP, DNS) and was set up as a VirtualBox + phpVirtualBox
> host to test that out, but no VMs running.
>
> So the essential loads are ZFS snapshots and ZFS scrubs :)
>
> And it freezes roughly every week. Stops responding to ping, attempts to
> log in via SSH or physical console - it processes keypresses on the latter,
> but does not present a login prompt. It used to be stable, and such regular
> hangs began around summertime.
>
>
>
> My primary guess would be for flaky disks, maybe timing out under load or
> going to sleep or whatever... But I have yet to prove it, or any other
> theory. Maybe just CPU is overheating due to regular near-100% load with
> disk I/O... At least I want to rule out OS errors and rule out (or point
> out) operator/box errors as much as possible - which is something I can
> change to try and fix ;)
>
> Before I proceed to TL;DR screenshots, I'd overview what I see:
>
> * In the "top" output, processes owned by zfssnap lead most of the time...
> But even the SSH shell is noticeably slow to respond (1 sec per line when
> just pressing enter to clear the screen to prepare nice screenshots).
>
> * SMART was not enabled on 3TB mirrored "pool" SATA disks (is now, long
> tests initiated), but was in place on the "rpool" SAS disk where it logged
> some corrected ECC errors - but none uncorrected.
>
> Maybe the cabling should be reseated.
>
> * iostat shows disks are generally not busy (they don't audibly rattle nor
> visibly blink all the time, either)
>
> * zpool scrubs return clean
>
> * there are partitions of the system rpool disk (10K RPM SAS) used as log
> and cache devices for the main data pool on 3TB SATA disks. The system disk
> is fast and underutilized, so what the heck ;) And it was not a problem for
> the first year of this system's honest and stable workouts. These devices
> are pretty empty at the moment.
>
>
>
> I have enabled deadman panics according to Wiki, but none have happened so
> far:
>
> # cat /etc/system  | egrep -v '(^\*|^$)'
> set snooping=1
> set pcplusmp:apic_panic_on_nmi=1
> set apix:apic_panic_on_nmi = 1
>
>
>
>
>
> In the "top" output, processes owned by zfssnap lead most of the time:
>
>
>
> last pid: 22599;  load avg:  12.9,  12.2,  11.2;   up
> 0+09:52:11
> 18:34:41
> 140 processes: 125 sleeping, 13 running, 2 on cpu
> CPU states:  0.0% idle, 22.9% user, 77.1% kernel,  0.0% iowait,  0.0% swap
> Memory: 16G phys mem, 1765M free mem, 2048M total swap, 2048M free swap
> Seconds to delay:
>PID USERNAME LWP PRI NICE  SIZE   RES STATETIMECPU COMMAND
>  21389 zfssnap1  432  863M  860M run  5:04 35.61% zfs
>  22360 zfssnap1  522  118M  115M run  0:37 16.50% zfs
>  21778 zfssnap1  522  563M  560M run  3:15 13.17% zfs
>  21278 zfssnap1  522  947M  944M run  5:32  6.91% zfs
>  21881 zfssnap1  432  433M  431M run  2:31  5.41% zfs
>  21852 zfssnap1  522  459M  456M run  2:39  5.16% zfs
>  21266 zfssnap1  432  906M  903M run  5:18  3.95% zfs
>  21757 zfssnap1  432  597M  594M run  3:26  2.91% zfs
>  21274 zfssnap1  522  930M  927M cpu/05:27  2.78% zfs
>  22588 zfssnap1  432   30M   27M run  0:08  2.48% zfs
>  22580 zfssnap1  522   49M   46M run  0:14  0.71% zfs
>  22038 root   1  590 5312K 3816K cpu/10:01  0.10% top
>  22014 root   1  590 8020K 4988K sleep0:00  0.02% sshd
>
>
>
> Average "iostats" are not that busy:
>
>
>
> # zpool iostat -Td 5
> Thu Oct 22 18:24:59 CEST 2015
>capacity operationsbandwidth
> poolalloc   free   read  write   read  write
> --  -  -  -  -  -  -
> pool2.52T   207G802116  28.3M   840K
> rpool   33.0G   118G  0  4  4.52K  58.7K
> --  -  -  -  -  -  -
>
> Thu Oct 22 18:25:04 CEST 2015
> pool2.52T   207G  0  0  0  0
> rpool   33.0G   118G  0 10  0  97.9K
> --  -  -  

[OmniOS-discuss] Slow performance with ZeusRAM?

[Matej Zerovnik]

Hello,

I'm building a new system and I'm having a bit of a performance problem. Well, 
its either that or I'm not getting the whole ZIL idea:)

My system is following:
- IBM xServer 3550 M4 server (dual CPU with 160GB memory)
- LSI 9207 HBA (P19 firmware)
- Supermicro JBOD with SAS expander
- 4TB SAS3 drives
- ZeusRAM for ZIL
- LTS Omnios (all patches applied)

If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 48k 
IOPS out of it, no problem there.

If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for ZIL 
and set sync=always, I can only squeeze 14k IOPS out of the system.
Is that normal or should I be getting 48k IOPS on the 2nd pool as well, since 
this is the performance ZeusRAM can deliver?

I'm testing with fio:
fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers 
--norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 
--numjobs=16 --runtime=60 --group_reporting --name=4ktest

thanks, Matej

smime.p7s
Description: S/MIME cryptographic signature
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Min Kim]

I believe this is an known issue with SAS expanders.

Please see here:

http://serverfault.com/questions/242336/sas-expanders-vs-direct-attached-sas 


When you are stress-testing the Zeusram by itself, all the IOPs and bandwidth 
of the expander are allocated to that device alone.  Once you add all the other 
drives, you lose some of that as you have to share it with the other disks.

Min Kim

> On Oct 22, 2015, at 12:02 PM, Matej Zerovnik  wrote:
> 
> Hello,
> 
> I'm building a new system and I'm having a bit of a performance problem. 
> Well, its either that or I'm not getting the whole ZIL idea:)
> 
> My system is following:
> - IBM xServer 3550 M4 server (dual CPU with 160GB memory)
> - LSI 9207 HBA (P19 firmware)
> - Supermicro JBOD with SAS expander
> - 4TB SAS3 drives
> - ZeusRAM for ZIL
> - LTS Omnios (all patches applied)
> 
> If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 48k 
> IOPS out of it, no problem there.
> 
> If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for ZIL 
> and set sync=always, I can only squeeze 14k IOPS out of the system.
> Is that normal or should I be getting 48k IOPS on the 2nd pool as well, since 
> this is the performance ZeusRAM can deliver?
> 
> I'm testing with fio:
> fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers 
> --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 
> --numjobs=16 --runtime=60 --group_reporting --name=4ktest
> 
> thanks, Matej___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Lauri Tirkkonen]

On Thu, Oct 22 2015 20:57:45 +0200, Volker A. Brandt wrote:
> Thanks for all the work you're doing on OmniOS!
> 
> > > On Oct 22, 2015, at 1:11 PM, Dan McDonald 
> > > wrote:
> > >
> > > "pkg update" followed by "svcadm restart ntp" as a safety measure
> > > should be sufficient.  No rebooting is needed.
> >
> > I made a small mistake here.  The "svcadm restart..." is not
> > necessary, the IPS package does the right thing here.
> 
> Well, no, it doesn't. :-)  That's due to a design flaw in the interaction
> between IPS and SMF (IMHO).  Even though the manifest object in the package
> is properly tagged with restart_fmri, the service is never restarted,
> because the manifest is not touched during the "pkg update", as it has not
> changed since the last package version.
> 
> So if you change an SMF method in a package and want an "automatic"
> restart, you  need to also physically modify the SMF manifest.  I do that
> by just incrementing a version counter or a timestamp, and noting the
> fact in an XML comment in the manifest.  Otherwise, you need to manually
> restart the service.

Well, that's not a design flaw. Actuators are executed only when the
action (eg. file) specifying them changes -- in other words, the
packager should include restart_fmri actuators in file actions that are
relevant for the service in question (eg. the ntpd binary at minimum).
This ntp package does not contain *any* restart_fmri actuators for the
ntp service:

% pkg contents -mr 
pkg://omnios/service/network/ntp@4.2.8.4-0.151014:20151022T170026Z|grep 
restart_fmri
file cb84fc718d7aa637c12641aed4405107b5659ab8 
chash=8758e80d1b9738c35f2d29cdebceee2930cdfa3b group=bin mode=0444 owner=root 
path=lib/svc/manifest/network/ntp.xml pkg.csize=1649 pkg.size=4681 
restart_fmri=svc:/system/manifest-import:default

(sidebar: the manifest-import service is what imports service manifests
into the SMF repository, which you need to do when you install a new
service)

If you wanted to restart ntp when any files in the ntp package change on
update, you would need an actuator like
'restart_fmri=svc:/network/ntp:default' on *all* file actions delivered
by the package.

-- 
Lauri Tirkkonen | lotheac @ IRCnet
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Dan McDonald]

> On Oct 22, 2015, at 1:11 PM, Dan McDonald  wrote:
> 
> "pkg update" followed by "svcadm restart ntp" as a safety measure should be 
> sufficient.  No rebooting is needed.

I made a small mistake here.  The "svcadm restart..." is not necessary, the IPS 
package does the right thing here.

Sorry for the confusion,
Dan

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] OmniOS backup box hanging regularly

[Jim Klimov]

22 октября 2015 г. 19:36:56 CEST, Yavor Tomov  пишет:
>Hi Tovarishch Jim,
>
>I had similar issue with my box and it was related to the NFS locks. I
>assume you are using it due to the Linux backups. The solution was
>posted
>by Chip on the mailing list. Copy of his solution below:
>
>"I've seen issues like this when you run out of NFS locks.   NFSv3 in
>Illumos is really slow at releasing locks.
>
>On all my NFS servers I do:
>
>sharectl set -p lockd_listen_backlog=256 nfs
>sharectl set -p lockd_servers=2048 nfs
>
>Everywhere I can, I use NFSv4 instead of v3.   It handles lock much
>better."
>
>All the Best
>Yavor
>
>
>
>On Thu, Oct 22, 2015 at 11:59 AM, Jim Klimov  wrote:
>
>> Hello all,
>>
>> I have this HP-Z400 workstation with 16Gb ECC(should be) RAM running
>> OmniOS bloody, which acts as a backup server for our production
>systems
>> (regularly rsync'ing large files off Linux boxes, and rotating ZFS
>> auto-snapshots to keep its space free). Sometimes it also runs
>replicas of
>> infrastructure (DHCP, DNS) and was set up as a VirtualBox +
>phpVirtualBox
>> host to test that out, but no VMs running.
>>
>> So the essential loads are ZFS snapshots and ZFS scrubs :)
>>
>> And it freezes roughly every week. Stops responding to ping, attempts
>to
>> log in via SSH or physical console - it processes keypresses on the
>latter,
>> but does not present a login prompt. It used to be stable, and such
>regular
>> hangs began around summertime.
>>
>>
>>
>> My primary guess would be for flaky disks, maybe timing out under
>load or
>> going to sleep or whatever... But I have yet to prove it, or any
>other
>> theory. Maybe just CPU is overheating due to regular near-100% load
>with
>> disk I/O... At least I want to rule out OS errors and rule out (or
>point
>> out) operator/box errors as much as possible - which is something I
>can
>> change to try and fix ;)
>>
>> Before I proceed to TL;DR screenshots, I'd overview what I see:
>>
>> * In the "top" output, processes owned by zfssnap lead most of the
>time...
>> But even the SSH shell is noticeably slow to respond (1 sec per line
>when
>> just pressing enter to clear the screen to prepare nice screenshots).
>>
>> * SMART was not enabled on 3TB mirrored "pool" SATA disks (is now,
>long
>> tests initiated), but was in place on the "rpool" SAS disk where it
>logged
>> some corrected ECC errors - but none uncorrected.
>>
>> Maybe the cabling should be reseated.
>>
>> * iostat shows disks are generally not busy (they don't audibly
>rattle nor
>> visibly blink all the time, either)
>>
>> * zpool scrubs return clean
>>
>> * there are partitions of the system rpool disk (10K RPM SAS) used as
>log
>> and cache devices for the main data pool on 3TB SATA disks. The
>system disk
>> is fast and underutilized, so what the heck ;) And it was not a
>problem for
>> the first year of this system's honest and stable workouts. These
>devices
>> are pretty empty at the moment.
>>
>>
>>
>> I have enabled deadman panics according to Wiki, but none have
>happened so
>> far:
>>
>> # cat /etc/system  | egrep -v '(^\*|^$)'
>> set snooping=1
>> set pcplusmp:apic_panic_on_nmi=1
>> set apix:apic_panic_on_nmi = 1
>>
>>
>>
>>
>>
>> In the "top" output, processes owned by zfssnap lead most of the
>time:
>>
>>
>>
>> last pid: 22599;  load avg:  12.9,  12.2,  11.2;   up
>> 0+09:52:11
>> 18:34:41
>> 140 processes: 125 sleeping, 13 running, 2 on cpu
>> CPU states:  0.0% idle, 22.9% user, 77.1% kernel,  0.0% iowait,  0.0%
>swap
>> Memory: 16G phys mem, 1765M free mem, 2048M total swap, 2048M free
>swap
>> Seconds to delay:
>>PID USERNAME LWP PRI NICE  SIZE   RES STATETIMECPU COMMAND
>>  21389 zfssnap1  432  863M  860M run  5:04 35.61% zfs
>>  22360 zfssnap1  522  118M  115M run  0:37 16.50% zfs
>>  21778 zfssnap1  522  563M  560M run  3:15 13.17% zfs
>>  21278 zfssnap1  522  947M  944M run  5:32  6.91% zfs
>>  21881 zfssnap1  432  433M  431M run  2:31  5.41% zfs
>>  21852 zfssnap1  522  459M  456M run  2:39  5.16% zfs
>>  21266 zfssnap1  432  906M  903M run  5:18  3.95% zfs
>>  21757 zfssnap1  432  597M  594M run  3:26  2.91% zfs
>>  21274 zfssnap1  522  930M  927M cpu/05:27  2.78% zfs
>>  22588 zfssnap1  432   30M   27M run  0:08  2.48% zfs
>>  22580 zfssnap1  522   49M   46M run  0:14  0.71% zfs
>>  22038 root   1  590 5312K 3816K cpu/10:01  0.10% top
>>  22014 root   1  590 8020K 4988K sleep0:00  0.02% sshd
>>
>>
>>
>> Average "iostats" are not that busy:
>>
>>
>>
>> # zpool iostat -Td 5
>> Thu Oct 22 18:24:59 CEST 2015
>>capacity operationsbandwidth
>> poolalloc   free   read  write   read  write
>> --  -  -  -  -  -  -
>> pool2.52T   207G802116  28.3M   840K
>> rpool   33.0G   118G  0  4  4.52K  58.7K
>> 

Re: [OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Eric Sproul]

On Thu, Oct 22, 2015 at 2:57 PM, Volker A. Brandt  wrote:
>> I made a small mistake here.  The "svcadm restart..." is not
>> necessary, the IPS package does the right thing here.
>
> Well, no, it doesn't. :-)  That's due to a design flaw in the interaction
> between IPS and SMF (IMHO).  Even though the manifest object in the package
> is properly tagged with restart_fmri, the service is never restarted,
> because the manifest is not touched during the "pkg update", as it has not
> changed since the last package version.

The service has restarted correctly for me on both 006 and 014 with
this update.  I'm not sure why that is though, because you're correct
that the ntp.xml file has not changed in all of the '014 versions
published.  I was under the impression that the restart_fmri actuator
would only fire when the associated action was triggered.

However, if we really *do* want to restart ntp when the *daemon*
updates, then we could add a restart_fmri actuator on the
usr/lib/inet/ntpd file.  Thus, whenever that file is updated,
svc:/network/ntp:default could be restarted.

Eric
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Matej Zerovnik]

I’m using the default value of 128K on linux and OmniOS. I tried with 
recordsize=4k, but there is no different in iops…

Matej


> On 22 Oct 2015, at 21:36, Min Kim  wrote:
> 
> Are you using the same record size of 4K on your zfs pool as you used with 
> your linux test system?
> 
> If the record size for the zpool and slog is set at the default value of 
> 128K, it will greatly reduce the measured IOPS relative to that measured with 
> a recordsize of 4K.
> 
> Min Kim
> 
> 
> 
> 
>> On Oct 22, 2015, at 12:26 PM, Matej Zerovnik > > wrote:
>> 
>> Interesting…
>> 
>> Although, I’m not sure if this is really the problem.
>> 
>> For test, I booted up linux and put both ZeusRAM to raid1 software raid and 
>> repeated the test. I got full 48kIOPS in the test, meaning there was 96kIOPS 
>> sent to JBOD (48k IOPS for each drive).
>> 
>> On the OmniOS test bed, there are 28k IOPS sent to ZIL and X amount to 
>> spindles when flushing write cache, but no more then 1000 IOPS (100 
>> iops/drive * 10). Comparing that to the case above, IOPS shouldn’t be a 
>> limit.
>> 
>> Maybe I could try building my pools with hard drives that aren’t near ZIL 
>> drive, which is in bay 0. I could take hard drives from bays 4-15, which 
>> probably use different SAS lanes.
>> 
>> lp, Matej
>> 
>> 
>>> On 22 Oct 2015, at 21:10, Min Kim >> > wrote:
>>> 
>>> I believe this is an known issue with SAS expanders.
>>> 
>>> Please see here:
>>> 
>>> http://serverfault.com/questions/242336/sas-expanders-vs-direct-attached-sas
>>>  
>>> 
>>> 
>>> When you are stress-testing the Zeusram by itself, all the IOPs and 
>>> bandwidth of the expander are allocated to that device alone.  Once you add 
>>> all the other drives, you lose some of that as you have to share it with 
>>> the other disks.
>>> 
>>> Min Kim
>>> 
>>> 
>>> 
 On Oct 22, 2015, at 12:02 PM, Matej Zerovnik > wrote:
 
 Hello,
 
 I'm building a new system and I'm having a bit of a performance problem. 
 Well, its either that or I'm not getting the whole ZIL idea:)
 
 My system is following:
 - IBM xServer 3550 M4 server (dual CPU with 160GB memory)
 - LSI 9207 HBA (P19 firmware)
 - Supermicro JBOD with SAS expander
 - 4TB SAS3 drives
 - ZeusRAM for ZIL
 - LTS Omnios (all patches applied)
 
 If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 
 48k IOPS out of it, no problem there.
 
 If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for 
 ZIL and set sync=always, I can only squeeze 14k IOPS out of the system.
 Is that normal or should I be getting 48k IOPS on the 2nd pool as well, 
 since this is the performance ZeusRAM can deliver?
 
 I'm testing with fio:
 fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers 
 --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 
 --numjobs=16 --runtime=60 --group_reporting --name=4ktest
 
 thanks, Matej___
 OmniOS-discuss mailing list
 OmniOS-discuss@lists.omniti.com 
 http://lists.omniti.com/mailman/listinfo/omnios-discuss 
 
>>> 
>> 
> 
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com
> http://lists.omniti.com/mailman/listinfo/omnios-discuss



smime.p7s
Description: S/MIME cryptographic signature
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Bob Friesenhahn]

On Thu, 22 Oct 2015, Matej Zerovnik wrote:


I’m using the default value of 128K on linux and OmniOS. I tried with 
recordsize=4k, but there is no different in iops…


There should be a large difference unless the file data is already 
cached in the ARC.  Even with caching, a block size of 128k means that 
128k is written to underlying store, although a useful purpose of your 
ZIL device is that an aggregation of multiple writes during the TXG 
interval to the same 128k block may be written as one write at the 
next TXG sync interval (rather than immediately).


Try umounting and re-mounting your zfs filesystem (or 'zfs destroy' 
followed by 'zfs create') to see how performance differs on a freshly 
mounted filesystem.  The zfs ARC caching will be purged when the 
filesystem is unmounted.


Do you have compression enabled for this filesystem?

Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Matej Zerovnik]

Chip:
I tried running fio on multiple folders and it’s a little better

When I run 7x fio (iodepth=4 threads=4), I get 28k IOPS on average
When I run 7x fio (iodepth=4 threads=16), I get 35k IOPS on average. iostat 
shows that there is transfer rate of 140-220MB/s with average request size of 
35kB
When I run 7x fio (iodepth=1 threads=1), I get 24k IOPS on average

There are still at least 10k IOPS left to use I guess:)

Bob:
Yes, my ZFS is ashift=12, since all my drives report 4k blocks (is that what 
you ment?). The pool is completly empty, so there is enough place for writes, 
so write speed should not be limited because of COW. Looking at iostat, there 
are no reads on the drives at all.
I’m not sure where fio gets its data, probably from /dev/zero or somewhere?

I will try sync engine instead of solarisaio to see if there is any difference.

I don’t have compression enabled, since I want to test raw performance. I also 
disabled ARC (primarycache=metadata), just so my read tests are also as real as 
possible (so I don’t need to run tests with 1TB test file).

> Try umounting and re-mounting your zfs filesystem (or 'zfs destroy' followed 
> by 'zfs create') to see how performance differs on a freshly mounted 
> filesystem.  The zfs ARC caching will be purged when the filesystem is 
> unmounted.


If I understand you correctly, you are saying I should destroy my folders, set 
recordsize=4k on my pool and then create zfs folders?

thanks, Matej


> On 22 Oct 2015, at 21:47, Schweiss, Chip  wrote:
> 
> The ZIL on log devices suffer a bit from not filling queues well.   In order 
> to get the queues to fill more, try running your test to several zfs folders 
> on the pool simultaneously and measure your total I/O.  
> 
> As I understand it, ff you're writing to only one zfs folder, your queue 
> depth will stay at 1 on the log device and you be come latency bound.
> 
> -Chip
> 
> On Thu, Oct 22, 2015 at 2:02 PM, Matej Zerovnik  > wrote:
> Hello,
> 
> I'm building a new system and I'm having a bit of a performance problem. 
> Well, its either that or I'm not getting the whole ZIL idea:)
> 
> My system is following:
> - IBM xServer 3550 M4 server (dual CPU with 160GB memory)
> - LSI 9207 HBA (P19 firmware)
> - Supermicro JBOD with SAS expander
> - 4TB SAS3 drives
> - ZeusRAM for ZIL
> - LTS Omnios (all patches applied)
> 
> If I benchmark ZeusRAM on its own with random 4k sync writes, I can get 48k 
> IOPS out of it, no problem there.
> 
> If I create a new raidz2 pool with 10 hard drives, mirrored ZeusRAMs for ZIL 
> and set sync=always, I can only squeeze 14k IOPS out of the system.
> Is that normal or should I be getting 48k IOPS on the 2nd pool as well, since 
> this is the performance ZeusRAM can deliver?
> 
> I'm testing with fio:
> fio --filename=/pool0/test01 --size=5g --rw=randwrite --refill_buffers 
> --norandommap --randrepeat=0 --ioengine=solarisaio --bs=4k --iodepth=16 
> --numjobs=16 --runtime=60 --group_reporting --name=4ktest
> 
> thanks, Matej
> ___
> OmniOS-discuss mailing list
> OmniOS-discuss@lists.omniti.com 
> http://lists.omniti.com/mailman/listinfo/omnios-discuss 
> 
> 
> 



smime.p7s
Description: S/MIME cryptographic signature
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] Slow performance with ZeusRAM?

[Bob Friesenhahn]

On Thu, 22 Oct 2015, Matej Zerovnik wrote:

Bob:
Yes, my ZFS is ashift=12, since all my drives report 4k blocks (is that what 
you ment?). The pool is completly empty, so there is enough place for
writes, so write speed should not be limited because of COW. Looking at iostat, 
there are no reads on the drives at all.
I’m not sure where fio gets its data, probably from /dev/zero or somewhere?


To be clear, zfs does not overwrite blocks.  Instead zfs modifies (in 
memory) any prior data from a block, and then it writes the block data 
to a new location.  This is called "copy on write".  If the prior data 
would not be entirely overwritten and is not already cached in memory, 
then it needs to be read from underlying disk.


It is interesting that you say there are no reads on the drives at 
all.



  Try umounting and re-mounting your zfs filesystem (or 'zfs destroy' 
followed by 'zfs create') to see how performance differs on a freshly
  mounted filesystem.  The zfs ARC caching will be purged when the 
filesystem is unmounted.

If I understand you correctly, you are saying I should destroy my folders, set 
recordsize=4k on my pool and then create zfs folders?


It should suffice to delete the file and set recordsize=4k on the 
filesystem, and then use fio to create a new file.  The file retains 
its original recordsize after it has been created so you would need to 
create a new file.


There is maximum performance for random write if the write blocksize 
matches the filesystem blocksize.  There is still a catch though if 
the random write is truely random because the writes may still not 
match up perfectly to underlying blocks.  For example (top is logical 
write data and bottom is file block data):


aligned "random write" data:

XX  XX
XX  XX

vs unaligned "random write" data:

   XX   XX
XX  XX  XX

The writes aligned to the start of the underlying block will be much 
faster.


A major benefit of your ZIL device is to help turn random writes into 
fewer random writes or even sequential writes when the TXG is written 
to your data drives.


It is very difficult to test raw hardware performance through zfs 
filesytem access.


Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer,http://www.GraphicsMagick.org/___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Volker A. Brandt]

Lauri Tirkkonen writes:
> > Well, no, it doesn't. :-) That's due to a design flaw in the
> > interaction between IPS and SMF (IMHO).
[...]
 
> Well, that's not a design flaw. Actuators are executed only when the
> action (eg. file) specifying them changes

Yes, this is how IPS does it.  IPS does not really know that the
manifest-import service is special.  There should have been an explicit
"re-import this manifest now" actuator, much like users or groups are 
created.

[...]
> If you wanted to restart ntp when any files in the ntp package
> change on update, you would need an actuator like
> 'restart_fmri=svc:/network/ntp:default' on *all* file actions
> delivered by the package.

I know what you mean.  That might work, but that is normally not what 
you do when you deliver an SMF manifest in your package.  You just drop
it and restart manifest-import, and hope that manifest-import will see
your new manifest.  This is quite a different thing.

Also, what you wrote is not quite true.  What you wanted to write was 
"you would need an actuator on *at least one* file action *that has a
different file hash*".  If nothing is different, the action is not
executed, and the attached actuator does not fire. 

And it gets worse when you remove a package that contains a manifest for
a running SMF service, because it is impossible to call the stop method
of the service before removing the package.  Lots of fun. :-)


Viele Grüße -- Volker A. Brandt
-- 

Volker A. Brandt   Consulting and Support for Oracle Solaris
Brandt & Brandt Computer GmbH   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANYEmail: v...@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513  Schuhgröße: 46
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt

"When logic and proportion have fallen sloppy dead"

___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] UPDATE NOW --> ntp to 4.2.8p4

[Volker A. Brandt]

Eric Sproul writes:
> The service has restarted correctly for me on both 006 and 014 with
> this update.  I'm not sure why that is though, because you're
> correct that the ntp.xml file has not changed in all of the '014
> versions published.  I was under the impression that the
> restart_fmri actuator would only fire when the associated action was
> triggered.

Yes, that's why it did not restart on my 014 box, hence I noticed.
Maybe you were on an earlier rev where the manifest really did change?
I guess it also restarted when Dan tested, or else he would not have
mentioned that pkg DTRT.
 
> However, if we really *do* want to restart ntp when the *daemon*
> updates, then we could add a restart_fmri actuator on the
> usr/lib/inet/ntpd file.  Thus, whenever that file is updated,
> svc:/network/ntp:default could be restarted.

It's been a while since I last tried, but I think this will not work,
at least not in some corner cases, e.g. when the pkg is not installed
at all, and the svc:/network/ntp:default service does not exist when
the pkg is installed.  ISTR that pkg install will error out.  Hmmm, 
need to test that again sometimes. :-)


Regards -- Volker
-- 

Volker A. Brandt   Consulting and Support for Oracle Solaris
Brandt & Brandt Computer GmbH   WWW: http://www.bb-c.de/
Am Wiesenpfad 6, 53340 Meckenheim, GERMANYEmail: v...@bb-c.de
Handelsregister: Amtsgericht Bonn, HRB 10513  Schuhgröße: 46
Geschäftsführer: Rainer J.H. Brandt und Volker A. Brandt

"When logic and proportion have fallen sloppy dead"
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

Re: [OmniOS-discuss] OmniOS backup box hanging regularly

[Hildebrandt, Bill]

Chip was responding to an issue that we have been having (since 9/18) with our 
online systems becoming unresponsive to NFS and CIFS.  During these hangs we 
also see that doing an “ls” works, but an “ls –l” hangs.  NFS cannot be 
restarted, and we are forced to reboot.  I was hopeful that the NFS settings 
would work, but on Tuesday of this week, our offline replication target 
experienced the same issue.  We were about to perform a scrub, when we noticed 
in our Napp-it interface that “ZFS Filesystems” was not displaying.  After 
SSHing to the system, we saw that “ls –l” did not respond.  This was a pure 
replication target with no NFS access, so I don’t see how the NFS lock tuning 
could be the solution.  A “reboot –d” was performed, and we have a 3.9GB dump.  
If you have a preferred location to  receive such dumps, I would be more than 
happy to share.

I should note that we just started using OmniOS this summer, so we have always 
been at the r151014 version.  These were newly created units that have 
performed perfectly for 2.5 months, and now we are having hangs every 1-2 
weeks.  Here is a timeline that I shared with Guenther Alka:
(I know it’s not best practice, but we had to use “export” as the name of our 
data pool)

9/14 – notified of the L2Arc issue – removed the L2Arc device from the export 
pool and stopped nightly replication jobs.  Ran a scrub on the replicated NAS 
appliance, and found no issues

9/15 – ran “zdb –bbccsv export” on the replicated unit and it came up clean

9/16 – updated OmniOS and Napp-it on the replicated unit; re-added the L2Arc 
device; re-enabled the replication jobs

9/18 –  in the morning, we were notified that the main NAS unit had stopped 
responding to CIFS and “ls –l” was not working – NFS continued to function.  So 
this was prior to any rebuild of the system disk, upgrade of OmniOS, or 
re-enabling the L2Arc device.  That night the system disk was rebuilt from 
scratch from the original CD, with a pkg update performed prior to importing 
the data pool.

9/19-9/21 – everything appeared to be working well.  The replication jobs that 
run at 1, 2, and 3am this morning completed just fine.  Around 5:40am this 
morning is when we were notified that NFS had stopped serving up files.  After 
logging in, we found that the “ls –l” issue had returned, and CIFS was 
non-functional as well.  Also, the “Pools” tab in Napp-it worked, but the “ZFS 
Filesystems” tab did not.  I find it interesting that an “ls –l” failed while I 
was in /root – the rpool is brand new, with updated OS, and no L2Arc device.

I have performed scrubs and “zdb –bbccvs” on both units shortly after this, and 
no errors were found.  For the system that hung on Tuesday, a new scrub was 
clean, but the zdb showed leaks, and over 33k checksum errors (full disclosure 
– several replication jobs were ran during the zdb).  A subsequent scrub and 
“zpool status” showed no problem.  I performed another round of replications 
and all completed without errors.  I have now exported that pool, and am 
performing another zdb, which is still running but showing clean so far.

Just in case I have done something insanely stupid with regard to the system 
configuration, here is the config (both units are identical except for the OS – 
if the config seems strange, these were conversions of two old CORAID chassis):

OS: omnios-f090f73  (the online unit is at 
omnios-c65)
Mobo: Supermicro H8DG6-H8DGi
RAM: 128GB ECC (Hynix, which is really Samsung I think)
Controllers:2x LSI 9201-16i with SAS-SATA break-out cables going to a 
SAS846TQ backplane
System disk:   mirrored Innodisk 128GB SATADOMs (SATADOM-MV 3ME) off mobo
ZIL: mirrored Samsung SSD 845D (400GB) off mobo
L2ARC:  SAMSUNG MZHPU512 (512GB PCIe)
NICs: Intel 10G (X520)
Data disks:  24x WD 1TB Enterprise SATA (WDC WD1002FBYS)

Any help/insight is greatly appreciated.

Thanks,
Bill


From: OmniOS-discuss [mailto:omnios-discuss-boun...@lists.omniti.com] On Behalf 
Of Yavor Tomov
Sent: Thursday, October 22, 2015 12:37 PM
To: jimkli...@cos.ru
Cc: OmniOS-discuss
Subject: Re: [OmniOS-discuss] OmniOS backup box hanging regularly

Hi Tovarishch Jim,

I had similar issue with my box and it was related to the NFS locks. I assume 
you are using it due to the Linux backups. The solution was posted by Chip on 
the mailing list. Copy of his solution below:

"I've seen issues like this when you run out of NFS locks.   NFSv3 in Illumos 
is really slow at releasing locks.

On all my NFS servers I do:

sharectl set -p lockd_listen_backlog=256 nfs
sharectl set -p lockd_servers=2048 nfs

Everywhere I can, I use NFSv4 instead of v3.   It handles lock much better."

All the Best
Yavor



On Thu, Oct 22, 2015 at 11:59 AM, Jim Klimov > 
wrote:

Hello all,

I have this HP-Z400 workstation with 16Gb ECC(should be) RAM 

Re: [OmniOS-discuss] ILB memory leak?

[Ryan Zezeski]

Al Slater writes:

> On 21/10/2015 17:35, Dan McDonald wrote:
>>
>> That should enable user-level memory debugging.  If you get a
>> coredump, save it and share it.  If you don't and the ilb daemon
>> keeps running, eventually please:
>>
>> gcore `pgrep ilbd`
>>
>> and share THAT corefile.  You can also do this by youself:
>>
>> mdb  > ::findleaks
>>
>> and share ::findleaks.
>>
>> Once you're done generating corefiles, repeat the steps above, but
>> use "unsetenv LD_PRELOAD" and "unsetenv UMEM_DEBUG" instead of the
>> setenv lines.
>
> Thanks Dan.  As we are talking about production boxes here, I will have 
> to try and reproduce on another box and then I will give the process 
> above a go and see what we come up with.

You can also use the DTrace pid provider to grab the user stack on every
malloc(3C) call, and the syscall provider to track mmap(2) calls. That
poses no harm to production and might make the cause of memory usage
obvious.

Something like:

dtrace -qn 'pid$target::malloc:entry { @[ustack()] = count(); }
syscall::mmap*:entry /pid == $target/ { @[ustack()] = count(); }' -p 

Let that run for a while as the memory grows, then Ctrl-C.

-Z
___
OmniOS-discuss mailing list
OmniOS-discuss@lists.omniti.com
http://lists.omniti.com/mailman/listinfo/omnios-discuss

[OmniOS-discuss] OmniOS backup box hanging regularly

[Jim Klimov]

Hello all,
I have this HP-Z400 workstation with 16Gb ECC(should be) RAM running OmniOS 
bloody, which acts as a backup server for our production systems (regularly 
rsync'ing large files off Linux boxes, and rotating ZFS auto-snapshots to keep 
its space free). Sometimes it also runs replicas of infrastructure (DHCP, DNS) 
and was set up as a VirtualBox + phpVirtualBox host to test that out, but no 
VMs running.
So the essential loads are ZFS snapshots and ZFS scrubs :)
And it freezes roughly every week. Stops responding to ping, attempts to log in 
via SSH or physical console - it processes keypresses on the latter, but does 
not present a login prompt. It used to be stable, and such regular hangs began 
around summertime.
 
My primary guess would be for flaky disks, maybe timing out under load or going 
to sleep or whatever... But I have yet to prove it, or any other theory. Maybe 
just CPU is overheating due to regular near-100% load with disk I/O... At least 
I want to rule out OS errors and rule out (or point out) operator/box errors as 
much as possible - which is something I can change to try and fix ;)
Before I proceed to TL;DR screenshots, I'd overview what I see:
* In the "top" output, processes owned by zfssnap lead most of the time... But 
even the SSH shell is noticeably slow to respond (1 sec per line when just 
pressing enter to clear the screen to prepare nice screenshots).
* SMART was not enabled on 3TB mirrored "pool" SATA disks (is now, long tests 
initiated), but was in place on the "rpool" SAS disk where it logged some 
corrected ECC errors - but none uncorrected.
Maybe the cabling should be reseated.
* iostat shows disks are generally not busy (they don't audibly rattle nor 
visibly blink all the time, either)
* zpool scrubs return clean
* there are partitions of the system rpool disk (10K RPM SAS) used as log and 
cache devices for the main data pool on 3TB SATA disks. The system disk is fast 
and underutilized, so what the heck ;) And it was not a problem for the first 
year of this system's honest and stable workouts. These devices are pretty 
empty at the moment.
 
I have enabled deadman panics according to Wiki, but none have happened so far:
# cat /etc/system  | egrep -v '(^\*|^$)'
set snooping=1
set pcplusmp:apic_panic_on_nmi=1
set apix:apic_panic_on_nmi = 1

 
 
In the "top" output, processes owned by zfssnap lead most of the time:
 
last pid: 22599;  load avg:  12.9,  12.2,  11.2;   up 0+09:52:11
  18:34:41
140 processes: 125 sleeping, 13 running, 2 on cpu
CPU states:  0.0% idle, 22.9% user, 77.1% kernel,  0.0% iowait,  0.0% swap
Memory: 16G phys mem, 1765M free mem, 2048M total swap, 2048M free swap
Seconds to delay:
   PID USERNAME LWP PRI NICE  SIZE   RES STATETIMECPU COMMAND
 21389 zfssnap1  432  863M  860M run  5:04 35.61% zfs
 22360 zfssnap1  522  118M  115M run  0:37 16.50% zfs
 21778 zfssnap1  522  563M  560M run  3:15 13.17% zfs
 21278 zfssnap1  522  947M  944M run  5:32  6.91% zfs
 21881 zfssnap1  432  433M  431M run  2:31  5.41% zfs
 21852 zfssnap1  522  459M  456M run  2:39  5.16% zfs
 21266 zfssnap1  432  906M  903M run  5:18  3.95% zfs
 21757 zfssnap1  432  597M  594M run  3:26  2.91% zfs
 21274 zfssnap1  522  930M  927M cpu/05:27  2.78% zfs
 22588 zfssnap1  432   30M   27M run  0:08  2.48% zfs
 22580 zfssnap1  522   49M   46M run  0:14  0.71% zfs
 22038 root   1  590 5312K 3816K cpu/10:01  0.10% top
 22014 root   1  590 8020K 4988K sleep0:00  0.02% sshd

 
Average "iostats" are not that busy:
 
# zpool iostat -Td 5
Thu Oct 22 18:24:59 CEST 2015
   capacity operationsbandwidth
poolalloc   free   read  write   read  write
--  -  -  -  -  -  -
pool2.52T   207G802116  28.3M   840K
rpool   33.0G   118G  0  4  4.52K  58.7K
--  -  -  -  -  -  -
Thu Oct 22 18:25:04 CEST 2015
pool2.52T   207G  0  0  0  0
rpool   33.0G   118G  0 10  0  97.9K
--  -  -  -  -  -  -
Thu Oct 22 18:25:09 CEST 2015
pool2.52T   207G  0  0  0  0
rpool   33.0G   118G  0  0  0  0
--  -  -  -  -  -  -
Thu Oct 22 18:25:14 CEST 2015
pool2.52T   207G  0  0  0  0
rpool   33.0G   118G  0  9  0  93.5K
--  -  -  -  -  -  -
Thu Oct 22 18:25:19 CEST 2015
pool2.52T   207G  0  0  0  0
rpool   33.0G   118G  0  0  0  0
--  -  -  -  -  -  -
Thu Oct 22 18:25:24 CEST 2015
pool2.52T   207G  0  0  0  0
rpool   33.0G   118G  0  0  0