This never did get to the mailist so I'll resend.
----------------------------- Original Message -----------------------------
Subject: Re: [pkg-discuss] pkg.depotd hangs after uploading large number of
packages
From: "Dennis Clarke" <[EMAIL PROTECTED]>
Date: Wed, February 20, 2008 17:01
To: "Angelo Rajadurai" <[EMAIL PROTECTED]>
Cc: "Bart Smaalders" <[EMAIL PROTECTED]>
"Danek Duvall" <[EMAIL PROTECTED]>
----------------------------------------------------------------------------
> Hey Bart/Danek:
>
> This happened again. This time the pkg.depotd and the
> solaris.py were running on two different systems.
>
> On both the systems there are only a few connections
> in CLOSE_WAIT state.
>
> This time I was able to load 1545 pkgs without any
> problem and then the hang happened.
>
> One more clue that may help!
>
> I see the "Number of in-flight transsactions" increasing
> every 30 mins or so.
>
> The solaris.py is also hung so no idea why this number is
> increasing.
>
> Also I can install pkgs off the repo when the hang happens.
> So it looks like the "receive" side is stuck while the "send"
> side of pkg.depotd is doing ok.
>
> Let me know if you need me to do any analysis on the hung
> process
It may help to know that this is a pkg.depotd running on Solaris 10 that I
back ported.
$ cat /etc/release
Solaris 10 8/07 s10x_u4wos_12b X86
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007
$ pkginfo -l SUNWipkg
PKGINST: SUNWipkg
NAME: Image Packaging System
CATEGORY: system
ARCH: i386
VERSION: 0.0.0
BASEDIR: /
PSTAMP: blast0120080220000240
INSTDATE: Feb 20 2008 00:13
STATUS: completely installed
FILES: 136 installed pathnames
15 shared pathnames
26 directories
4 executables
1760 blocks used (approx)
Yes, the INSTDATE says early this morning. I have done the same thing on
Sparc but have not tested it under load yet.
# uname -a
SunOS pkg 5.10 Generic_120011-14 sun4u sparc SUNW,Sun-Fire-V210
# cat /etc/release
Solaris 10 8/07 s10s_u4wos_12b SPARC
Copyright 2007 Sun Microsystems, Inc. All Rights Reserved.
Use is subject to license terms.
Assembled 16 August 2007
#
# zonename
pkg
# pkginfo -l SUNWipkg
PKGINST: SUNWipkg
NAME: Image Packaging System
CATEGORY: system
ARCH: sparc
VERSION: 0.0.0
BASEDIR: /
PSTAMP: pkg20080220134239
INSTDATE: Feb 20 2008 13:43
STATUS: completely installed
FILES: 136 installed pathnames
14 shared pathnames
26 directories
4 executables
1775 blocks used (approx)
It may be interesting to note that there are 452000+ lines in the pkg
service log :
[EMAIL PROTECTED] wc -l /var/svc/log/application-pkg-server:default.log
452404 /var/svc/log/application-pkg-server:default.log
The repo update logs show activity since early this morning :
# ls -lap updatelog/
total 400
drwxr-xr-x 2 root root 512 Feb 20 20:00 ./
drwxr-xr-x 7 root root 512 Feb 20 00:20 ../
-rw-r--r-- 1 root root 5158 Feb 20 04:58 2008022004
-rw-r--r-- 1 root root 6255 Feb 20 05:59 2008022005
-rw-r--r-- 1 root root 6662 Feb 20 06:55 2008022006
-rw-r--r-- 1 root root 4794 Feb 20 07:58 2008022007
-rw-r--r-- 1 root root 13574 Feb 20 08:59 2008022008
-rw-r--r-- 1 root root 7018 Feb 20 09:47 2008022009
-rw-r--r-- 1 root root 6215 Feb 20 10:57 2008022010
-rw-r--r-- 1 root root 7065 Feb 20 11:58 2008022011
-rw-r--r-- 1 root root 10112 Feb 20 12:59 2008022012
-rw-r--r-- 1 root root 6978 Feb 20 13:59 2008022013
-rw-r--r-- 1 root root 8003 Feb 20 14:54 2008022014
-rw-r--r-- 1 root root 9351 Feb 20 15:55 2008022015
-rw-r--r-- 1 root root 8515 Feb 20 16:56 2008022016
-rw-r--r-- 1 root root 10427 Feb 20 17:59 2008022017
-rw-r--r-- 1 root root 8990 Feb 20 18:59 2008022018
-rw-r--r-- 1 root root 4668 Feb 20 19:58 2008022019
-rw-r--r-- 1 root root 3551 Feb 20 20:32 2008022020
They show that 1545 packages were uploaded :
# wc -l updatelog/*
64 updatelog/2008022004
77 updatelog/2008022005
81 updatelog/2008022006
59 updatelog/2008022007
163 updatelog/2008022008
85 updatelog/2008022009
75 updatelog/2008022010
85 updatelog/2008022011
122 updatelog/2008022012
84 updatelog/2008022013
96 updatelog/2008022014
114 updatelog/2008022015
104 updatelog/2008022016
127 updatelog/2008022017
109 updatelog/2008022018
57 updatelog/2008022019
43 updatelog/2008022020
1545 total
#
The pkg server seems to agree :
Statistics
Number of packages: 1545
Number of in-flight transactions: 6
Number of catalogs served: 8
Number of manifests served: 55
Number of files served: 0
Number of flists requested: 316
Number of files served by flist: 13476
Number of packages renamed: 0
So everything looks normal and healthy. But it hangs during upload.
Oh .. one other thing .. search.pag looks insane in terms of size :
# ls -lap
total 572096
drwxr-xr-x 7 root root 512 Feb 20 00:20 ./
drwxr-xr-x 4 root root 512 Feb 20 00:19 ../
drwxr-xr-x 2 root root 512 Feb 20 04:28 catalog/
drwxr-xr-x 258 root root 3584 Feb 20 04:32 file/
drwxr-xr-x 1554 root root 34816 Feb 20 22:15 pkg/
-rw-r--r-- 1 root root 133353472 Feb 20 20:33 search.dir
-rw-r--r-- 1 root root 1092409779200 Feb 20 20:33 search.pag
drwxr-xr-x 9 root root 1024 Feb 20 22:14 trans/
drwxr-xr-x 2 root root 512 Feb 20 20:00 updatelog/
I really hope that is sparse because there isn't 1TB of space.
Dennis
_______________________________________________
pkg-discuss mailing list
[email protected]
http://mail.opensolaris.org/mailman/listinfo/pkg-discuss
