date:20110803

Re: ZFS directory with a large number of files

2011-08-03 Thread Peter Jeremy

On 2011-Aug-02 08:39:03 +0100, seanr...@gmail.com seanr...@gmail.com wrote:
On my FreeBSD 8.2-S machine (built circa 12th June), I created a
directory and populated it over the course of 3 weeks with about 2
million individual files. As you might imagine, a 'ls' of this
directory took quite some time.

The files were conveniently named with a timestamp in the filename
(still images from a security camera, once per second) so I've since
moved them all to timestamped directories (/MM/dd/hh/mm). What I
found though was the original directory the images were in is still
very slow to ls -- and it only has 1 file in it, another directory.

I've also seen this behaviour on Solaris 10 after cleaning out a
directory with a large number of files (though not as pathological as
your case).  I tried creating and deleting entries in an unsuccessful
effort to trigger directory compaction.  I wound up moving the
remaining contents into a new directory, deleting the original one
and renaming the new directory.

It would appear te be a garbage collection bug in ZFS.

On 2011-Aug-02 13:10:27 +0300, Daniel Kalchev dan...@digsys.bg wrote:
On 02.08.11 12:46, Daniel O'Connor wrote:
 I am pretty sure UFS does not have this problem. i.e. once you 
 delete/move the files out of the directory its performance would be 
 good again. 

UFS would be the classic example of poor performance if you do this.

Traditional UFS (including Solaris) behave badly in this scenario but
4.4BSD derivatives will release unused space at the end of a directory
and have smarts to more efficiently skip unused entries at the start
of a directory.

-- 
Peter Jeremy


pgpmdeH6w8Ny5.pgp
Description: PGP signature

Re: dtrace ustack kernel panic

2011-08-03 Thread Andriy Gapon

on 03/08/2011 01:39 maestro something said the following:
 hm userland and CTF not that I know. I only recompiled the kernel with CTF. 
 Do I
 have to recompile userland (and would that be make world?) If so can i just 
 pass
 the same WITH_CTF=1 option to make?

I _think_ (as in: not sure) that for ustack() to work correctly it needs CTF
information for userland bits.

I've never done this myself, so be warned, but I guess that, yes, you need to
recompile/reinstall world with WITH_CTF=1.
Also, I seem to recall that there were some reports that some things were
getting broken because of WITH_CTF=1, so be cautious, plan a way to recover your
system from non-working world.
-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: dtrace ustack kernel panic

2011-08-03 Thread Andriy Gapon

on 03/08/2011 01:46 maestro something said the following:
 What do I have to recompile for the patch to be picked up?
 make libproc in /usr/src/lib

Correct way is:
$ cd /usr/src/lib/libproc
$ make obj
$ make depend
$ make
$ make install

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: dtrace ustack kernel panic

2011-08-03 Thread Andriy Gapon

on 03/08/2011 03:47 maestro something said the following:
 now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost 
 works.
 
 Almost because dtrace still exits (this time without an error message) The
 process for which the ustack should be genereated is terminated
 Here are the specifics:
 
 fb90i386# dtrace -n 'syscall::accept:return / execname == nc / { ustack();}'
 dtrace: description 'syscall::accept:return ' matched 1 probe
 CPU IDFUNCTION:NAME
   0  46457accept:return
   libc.so.7`__sys_accept+0x7
   0x3
   0xed96e824
 
 
 This alsmost looks like a stack trace I'd say
 However, dtrace quits once I connect to nc
 
 nc quits with the following error message
 
 fb90i386# nc -vl 
 nc: Polling Error: Interrupted system call

This could be a combination of several factors.
First, for some reason nc gets EINTR in one of its system calls (poll(2)).
Second, nc treats EINTR as a serious error and exits.
Third, this is what happens on the dtrace side:
 30495 100974 dtrace   CALL  write(0x1,0x8035c2000,0x1a)
 30495 100974 dtrace   GIO   fd 1 wrote 26 bytes
   libc.so.7`__sys_accept+0xc
 30495 100974 dtrace   RET   write 26/0x1a
 30495 100974 dtrace   CALL  write(0x1,0x8035c2000,0x1)
 30495 100974 dtrace   GIO   fd 1 wrote 1 byte
   
   
 30495 100974 dtrace   RET   write 1
 30495 100974 dtrace   CALL  thr_kill(0x1c129,SIGUSR1)
 30495 100974 dtrace   RET   thr_kill 0
 30495 100974 dtrace   CALL  ptrace(PT_IO,0x771e,0x7fffbb10,0)
 30495 100974 dtrace   RET   ptrace 0
 30495 100974 dtrace   CALL  ptrace(PT_IO,0x771e,0x7fffbb10,0)
 30495 114985 dtrace   RET   _umtx_op -1 errno 4 Interrupted system call
 30495 100974 dtrace   RET   ptrace 0
 30495 114985 dtrace   PSIG  SIGUSR1 caught handler=0x80085d610 mask=0x5ffefedf
code=0x10007
 30495 100974 dtrace   CALL  _umtx_op(0x802007668,0x15,0x1,0,0)
 30495 114985 dtrace   CALL  sigprocmask(SIG_SETMASK,0x7fbfd9fc,0)
 30495 100974 dtrace   RET   _umtx_op 0
 30495 114985 dtrace   RET   sigprocmask 0
 30495 100974 dtrace   CALL  _umtx_op(0x800a71778,0xf,0,0,0)
 30495 114985 dtrace   CALL  sigreturn(0x7fbfd630)
 30495 114985 dtrace   RET   sigreturn JUSTRETURN
 30495 114985 dtrace   CALL  ptrace(PT_CONTINUE,0x771e,0x1,0)
 30495 114985 dtrace   RET   ptrace 0
 30495 114985 dtrace   CALL  ptrace(PT_IO,0x771e,0x7fbfdee0,0)
 30495 114985 dtrace   RET   ptrace -1 errno 16 Device busy
 30495 114985 dtrace   CALL  _umtx_op(0x80200d668,0x15,0x1,0,0)
 30495 114985 dtrace   RET   _umtx_op 0
 30495 114985 dtrace   CALL  madvise(0x803c52000,0x11000,MADV_FREE)
 30495 114985 dtrace   RET   madvise 0
 30495 114985 dtrace   CALL  madvise(0x803c1d000,0x14000,MADV_FREE)
 30495 114985 dtrace   RET   madvise 0
 30495 114985 dtrace   CALL  madvise(0x803c07000,0x1000,MADV_FREE)
 30495 114985 dtrace   RET   madvise 0
 30495 114985 dtrace   CALL  madvise(0x8037fc000,0x2000,MADV_FREE)
 30495 114985 dtrace   RET   madvise 0
 30495 114985 dtrace   CALL  madvise(0x8037f2000,0x8000,MADV_FREE)
 30495 114985 dtrace   RET   madvise 0
 30495 114985 dtrace   CALL  madvise(0x8037f,0x1000,MADV_FREE)
 30495 100974 dtrace   RET   _umtx_op 0
 30495 114985 dtrace   RET   madvise 0
 30495 100974 dtrace   CALL  ptrace(PT_DETACH,0x771e,0,0)
 30495 114985 dtrace   CALL  madvise(0x8037e,0x9000,MADV_FREE)
 30495 100974 dtrace   RET   ptrace -1 errno 3 No such process

Looks like something fishy happens when that SIGUSR1 is generated.
I wish a dtrace-for-userland expert and/or a ptrace(2) expert could help us 
here.

-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: dtrace ustack kernel panic

2011-08-03 Thread Andriy Gapon

on 03/08/2011 11:42 Andriy Gapon said the following:
 on 03/08/2011 03:47 maestro something said the following:
 now it seems that the ustack() action on i386 and FreeBSD-9.0BETA1 almost 
 works.

 Almost because dtrace still exits (this time without an error message) The
 process for which the ustack should be genereated is terminated
 Here are the specifics:

 fb90i386# dtrace -n 'syscall::accept:return / execname == nc / { 
 ustack();}'
 dtrace: description 'syscall::accept:return ' matched 1 probe
 CPU IDFUNCTION:NAME
   0  46457accept:return
   libc.so.7`__sys_accept+0x7
   0x3
   0xed96e824


 This alsmost looks like a stack trace I'd say
 However, dtrace quits once I connect to nc

 nc quits with the following error message

 fb90i386# nc -vl 
 nc: Polling Error: Interrupted system call
 
 This could be a combination of several factors.
 First, for some reason nc gets EINTR in one of its system calls (poll(2)).
 Second, nc treats EINTR as a serious error and exits.

Looks like nc gets EINTR when dtrace does ptrace(PT_CONTINUE).
So this is the culprit for your test case.

 Third, this is what happens on the dtrace side:

Recorded dtrace behavior seems to be correct.

  30495 100974 dtrace   CALL  write(0x1,0x8035c2000,0x1a)
  30495 100974 dtrace   GIO   fd 1 wrote 26 bytes
libc.so.7`__sys_accept+0xc
  30495 100974 dtrace   RET   write 26/0x1a
  30495 100974 dtrace   CALL  write(0x1,0x8035c2000,0x1)
  30495 100974 dtrace   GIO   fd 1 wrote 1 byte


  30495 100974 dtrace   RET   write 1
  30495 100974 dtrace   CALL  thr_kill(0x1c129,SIGUSR1)
  30495 100974 dtrace   RET   thr_kill 0
  30495 100974 dtrace   CALL  ptrace(PT_IO,0x771e,0x7fffbb10,0)
  30495 100974 dtrace   RET   ptrace 0
  30495 100974 dtrace   CALL  ptrace(PT_IO,0x771e,0x7fffbb10,0)
  30495 114985 dtrace   RET   _umtx_op -1 errno 4 Interrupted system call
  30495 100974 dtrace   RET   ptrace 0
  30495 114985 dtrace   PSIG  SIGUSR1 caught handler=0x80085d610 
 mask=0x5ffefedf
 code=0x10007
  30495 100974 dtrace   CALL  _umtx_op(0x802007668,0x15,0x1,0,0)
  30495 114985 dtrace   CALL  sigprocmask(SIG_SETMASK,0x7fbfd9fc,0)
  30495 100974 dtrace   RET   _umtx_op 0
  30495 114985 dtrace   RET   sigprocmask 0
  30495 100974 dtrace   CALL  _umtx_op(0x800a71778,0xf,0,0,0)
  30495 114985 dtrace   CALL  sigreturn(0x7fbfd630)
  30495 114985 dtrace   RET   sigreturn JUSTRETURN
  30495 114985 dtrace   CALL  ptrace(PT_CONTINUE,0x771e,0x1,0)
  30495 114985 dtrace   RET   ptrace 0
  30495 114985 dtrace   CALL  ptrace(PT_IO,0x771e,0x7fbfdee0,0)
  30495 114985 dtrace   RET   ptrace -1 errno 16 Device busy
  30495 114985 dtrace   CALL  _umtx_op(0x80200d668,0x15,0x1,0,0)
  30495 114985 dtrace   RET   _umtx_op 0
  30495 114985 dtrace   CALL  madvise(0x803c52000,0x11000,MADV_FREE)
  30495 114985 dtrace   RET   madvise 0
  30495 114985 dtrace   CALL  madvise(0x803c1d000,0x14000,MADV_FREE)
  30495 114985 dtrace   RET   madvise 0
  30495 114985 dtrace   CALL  madvise(0x803c07000,0x1000,MADV_FREE)
  30495 114985 dtrace   RET   madvise 0
  30495 114985 dtrace   CALL  madvise(0x8037fc000,0x2000,MADV_FREE)
  30495 114985 dtrace   RET   madvise 0
  30495 114985 dtrace   CALL  madvise(0x8037f2000,0x8000,MADV_FREE)
  30495 114985 dtrace   RET   madvise 0
  30495 114985 dtrace   CALL  madvise(0x8037f,0x1000,MADV_FREE)
  30495 100974 dtrace   RET   _umtx_op 0
  30495 114985 dtrace   RET   madvise 0
  30495 100974 dtrace   CALL  ptrace(PT_DETACH,0x771e,0,0)
  30495 114985 dtrace   CALL  madvise(0x8037e,0x9000,MADV_FREE)
  30495 100974 dtrace   RET   ptrace -1 errno 3 No such process
 
 Looks like something fishy happens when that SIGUSR1 is generated.
 I wish a dtrace-for-userland expert and/or a ptrace(2) expert could help us 
 here.
 


-- 
Andriy Gapon
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: 9.0 B1 Panic

2011-08-03 Thread John Baldwin

On Tuesday, August 02, 2011 5:11:58 pm Martin Wilke wrote:
 
 On Aug 3, 2011, at 12:56 AM, John Baldwin wrote:
 
  On Tuesday, August 02, 2011 8:58:10 am Martin Wilke wrote:
  
  On Aug 2, 2011, at 6:58 PM, Lystopad Olexandr wrote:
  
  Hello, Martin Wilke!
  
  On Tue, Aug 02, 2011 at 12:41:29PM +0800
  m...@freebsd.org wrote about 9.0 B1 Panic:
  9.0 Beta1 Panic
  
  Hi guys,
  
  I just downloaded and install  9.0 BETA1 but it panics on ACPI. Please 
  view attached screenshot for the error. If you need more information, do 
let 
  us know.
  
  There no attachments in your mail.
  
  
  Erms Sorry forgot about that.
  
  http://people.freebsd.org/~miwi/90b1.png
  
  Can you get a stack trace?
 
 Hi,
 unfortunately no, because the system freeze few sec after the panic.

Can you set the loader tunable to force an automatic trace on panic?

-- 
John Baldwin
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org

Re: ZFS directory with a large number of files

Re: dtrace ustack kernel panic

Re: dtrace ustack kernel panic

Re: dtrace ustack kernel panic

Re: dtrace ustack kernel panic

Re: 9.0 B1 Panic

6 matches

Site Navigation

Mail list logo

Footer information