Re: linux-image-amd64: kernel fails to find all nvme SSDs

[Jeffrey Mark Siskind]

Supermicro provided a workaround: boot with the kernel command line
parameter pci=realloc=off.

As an side, Rocky 9.2 does not have this issue even though it boots
without that kernel command line parameter.

Jeff (http://engineering.purdue.edu/~qobi)

linux-image-amd64: kernel fails to find all nvme SSDs

[Jeffrey Mark Siskind]

I purchased a new server: Supermicro AS-8125GS-TNHR. It has 17 NVME
drives installed:

1x Micron 7450
   12x Micron 9300
4x Micron 9400

Upon boot, /dev/nvme* only shows 10 drives: the Micron 7450, 8 of the
Micron 9300s, and 1 of the Micron 9400s. Before I plugged in the 12x
Micron 9300, /dev/nvme* only showed 2 drives: the Micron 7450 and 1 of
the Micron 9400s.

I run bookworm stable. Upon first install, it ran kernel
6.1.0-12. After an apt upgrade it ran 6.1.0-13. I also installed
6.4.0-0-deb12.2 from bookworm backports. All 3 exhibit the same
issue. the only difference is that under 6.1.0-13 the 10 drives that do
appear appear as /de/nvme{0,1,2,3,4,5,6,7,8,9} while under 6.4.0-0-deb12.2
the 10 drives that do appear appear as different numbers with some missing.

The 1x 7450 has 3 partitions: EFI, / formatted as btrfs, and swap.
The 12x 9300s are all formatted with 1 partition. There are 6 pairs of
2, Each pair has a btrfs raid1 file system. The 9400s are not yet formatted.

lspci shows all 17 drives. But only 10 show up in hwinfo.

Thanks,
Jeff (http: //engineering.purdue.edu/~qobi)

linux-image-amd64: failed to bring up processor 192

[Jeffrey Mark Siskind]

I purchased a new server based on the Supermicro AS-8125GS-TNHR. It
has two EPYC 9654 processors. Each processor has 96 cores and 192 hyperthreads.
Thus the system as a whole has 192 cores and 384 hyperthreads. I run
Debian stable bookworm. When first installed, it ran kernel
6.1.0.12. That kernel found all 384 "CPUs". All were reported in /proc/cpuinfo.
I subsequently did an apt upgrade which upgraded to 6.1.0.13. Upon
boot, dmesg -lerr reports:

   root@poto:~# dmesg -lerr
   [   11.080833] smpboot: do_boot_cpu failed(-1) to wakeup CPU#192

Further, /proc/cpuinfo is missing processor 192.

I subsequently installed 6.4.0-0-deb12.2-amd64 from bookwork
backports. The issue still occurs. If I boot 6.1.0.12 I do not get the issue.

There are other reasons I cannot run 6.1.0.12 that I will describe in
a different bug report.

Jeff (http: //engineering.purdue.edu/~qobi)

Re: exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

   > I just changed /etc/mailname on all of my machines to contain:
   > 
   > qobi@upplysingaoflun>cat /etc/mailname
   > purdue.edu

   You know it makes sense to do this. :)

I had never manually edited /etc/mailname. I took whatever what put there by
the debian installer perhaps as modified by whatever packages I installed and
by running dpkg-reconfigure exim4-config.

As per another post, on thakaa running wheezy I have

   qobi@thakaa>cat /etc/mailname 
   thakaa.ecn.purdue.edu
   qobi@thakaa>tail -n 1 /etc/aliases
   root: q...@purdue.edu
   qobi@thakaa>

and it does correctly forward mail to root to q...@purdue.edu.

On upplysingaoflun running jessie I have

   qobi@upplysingaoflun>cat /etc/mailname
   ecn.purdue.edu
   qobi@upplysingaoflun>tail -n 1 /etc/aliases
   root: q...@purdue.edu
   qobi@upplysingaoflun>

and it does not. I have tried putting each of
   upplysingaoflun.ecn.purdue.edu
   ecn.purdue.edu
   purdue.edu
in upplysingaoflun:/etc/mailname and none of them work. The debian installer,
perhaps as modified by dpkg-reconfigure exim4-config put
upplysingaoflun.ecn.purdue.edu in upplysingaoflun:/etc/mailname

   But it does qualify a bare qobi or root correctly.

Neither thakaa (wheezy) nor upplysingaoflun (jessie) will qualify a bare qobi.
I don't care about that. It would be nice to fix (i.e. send to @purdue.edu)
but is not crucial.

Thakaa (whezy) but not upplysingaoflun (jessie) will ultimately get mail
addressed to a bare root to q...@purdue.edu. That is what I would like to fix
on upplysingaoflun (and all of my other machines running jessie).

Thanks for everyone's help on this.

Jeff (http://engineering.purdue.edu/~qobi)

Re: exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

   I would bet you do not recall what you answered to the questions regarding
   mail config when installing wheezy.

I kept records of what I answered to those questions when installing sarge,
etch, lenny, squeeze, and wheezy.

   If you have a backup copy of /etc/exim4 you could diff and see.

All my machines save two have been upgraded to jessie. One (thakaa) is running
wheezy and one is runnign squeeze. I just tried on the machine running wheezy.

I just tried thakaa (wheezy) and it did correctly send mail that was addressed
to root to q...@purdue.edu. But it did not on upplysingaoflun (jessie).

I don't understand the files in /etc/exim4. I didn't create them. I just ran
dpkg-reconfigure exim4-config and edited /etc/aliases and
/etc/exim4/passwd.client (and more recently /etc/mailname on upplysingaoflun;
it has been unchanged on thakaa since install).

Any help on how to fix this would be appreciated.

Jeff (http://engineering.purdue.edu/~qobi)
---
qobi@thakaa>diff -r -q /tmp/{upplysingaoflun,thakaa}/
Files /tmp/upplysingaoflun/exim4/conf.d/auth/30_exim4-config_examples and 
/tmp/thakaa/exim4/conf.d/auth/30_exim4-config_examples differ
Files /tmp/upplysingaoflun/exim4/conf.d/router/200_exim4-config_primary and 
/tmp/thakaa/exim4/conf.d/router/200_exim4-config_primary differ
Files /tmp/upplysingaoflun/exim4/conf.d/router/500_exim4-config_hubuser and 
/tmp/thakaa/exim4/conf.d/router/500_exim4-config_hubuser differ
Files /tmp/upplysingaoflun/exim4/conf.d/transport/30_exim4-config_maildrop_pipe 
and /tmp/thakaa/exim4/conf.d/transport/30_exim4-config_maildrop_pipe differ
Files /tmp/upplysingaoflun/exim4/conf.d/transport/30_exim4-config_remote_smtp 
and /tmp/thakaa/exim4/conf.d/transport/30_exim4-config_remote_smtp differ
Files 
/tmp/upplysingaoflun/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost
 and /tmp/thakaa/exim4/conf.d/transport/30_exim4-config_remote_smtp_smarthost 
differ
Files /tmp/upplysingaoflun/exim4/exim4.conf.template and 
/tmp/thakaa/exim4/exim4.conf.template differ
qobi@thakaa>

qobi@thakaa>diff -r /tmp/{upplysingaoflun,thakaa}/
diff -r /tmp/upplysingaoflun/exim4/conf.d/auth/30_exim4-config_examples 
/tmp/thakaa/exim4/conf.d/auth/30_exim4-config_examples
39c39
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
54c54
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
80c80
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
91c91
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
100c100
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
109c109
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
127c127
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
136c136
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
155c155
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
170c170
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
191c191
< #   server_advertise_condition = ${if eq{$tls_in_cipher}{}{}{*}}
---
> #   server_advertise_condition = ${if eq{$tls_cipher}{}{}{*}}
225c225
<   client_send = "<; ${if !eq{$tls_out_cipher}{}\
---
>   client_send = "<; ${if !eq{$tls_cipher}{}\
241c241
<   {!eq{$tls_out_cipher}{}}\
---
>   {!eq{$tls_cipher}{}}\



diff -r /tmp/upplysingaoflun/exim4/conf.d/router/200_exim4-config_primary 
/tmp/thakaa/exim4/conf.d/router/200_exim4-config_primary
82c82
<   host_find_failed = ignore
---
>   host_find_failed = defer



diff -r /tmp/upplysingaoflun/exim4/conf.d/router/500_exim4-config_hubuser 
/tmp/thakaa/exim4/conf.d/router/500_exim4-config_hubuser
26c26
<   host_find_failed = ignore
---
>   host_find_failed = defer



diff -r 
/tmp/upplysingaoflun/exim4/conf.d/transport/30_exim4-config_maildrop_pipe 
/tmp/thakaa/exim4/conf.d/transport/30_exim4-config_maildrop_pipe
7,8d6
<   message_prefix =
<   message_suffix =



diff -r /tmp/upplysingaoflun/exim4/conf.d/transport/30_exim4-config_remote_smtp 
/tmp/thakaa/exim4/conf.d/transport/30_exim4-config_remote_smtp
42,47d41
< .ifdef REMOTE_SMTP_TLS_CERTIFICATE
< tls_certificate = REMOTE_SMTP_TLS_CERTIFICATE
< .endif
< .ifdef REMOTE_SMTP_PRIVATEKEY
< 

Re: exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

as received at Wed, 17 Aug 2016 18:10:18 -0400
from upplysingaoflun.ecn.purdue.edu [128.46.115.209]

   - The following addresses had permanent fatal errors -
<q...@upplysingaoflun.ecn.purdue.edu>

   - Transcript of session follows -
554 5.0.0 MX list for upplysingaoflun.ecn.purdue.edu. points back to 
smtp.ecn.purdue.edu
554 5.3.5 Local configuration error

--u7HMAIDg019691.1471471818/smtp.ecn.purdue.edu
Content-Type: message/delivery-status

Reporting-MTA: dns; smtp.ecn.purdue.edu
Received-From-MTA: DNS; upplysingaoflun.ecn.purdue.edu
Arrival-Date: Wed, 17 Aug 2016 18:10:18 -0400

Final-Recipient: RFC822; q...@upplysingaoflun.ecn.purdue.edu
Action: failed
Status: 5.5.0
Remote-MTA: DNS; upplysingaoflun.ecn.purdue.edu
Last-Attempt-Date: Wed, 17 Aug 2016 18:10:18 -0400

--u7HMAIDg019691.1471471818/smtp.ecn.purdue.edu
Content-Type: message/rfc822

Return-Path: <q...@purdue.edu>
Received: from upplysingaoflun (upplysingaoflun.ecn.purdue.edu [128.46.115.209])
(authenticated bits=0)
by smtp.ecn.purdue.edu (8.14.4/8.14.4) with ESMTP id u7HMAIDh019678
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 
verify=NOT)
for <q...@upplysingaoflun.ecn.purdue.edu>; Wed, 17 Aug 2016 18:10:18 
-0400
Received: from qobi by upplysingaoflun with local (Exim 4.84_2)
(envelope-from <q...@upplysingaoflun.ecn.purdue.edu>)
id 1ba92P-0002nV-R3
for q...@upplysingaoflun.ecn.purdue.edu; Wed, 17 Aug 2016 18:10:17 -0400
To: q...@upplysingaoflun.ecn.purdue.edu
Subject: to qobi from qobi
Message-Id: <E1ba92P-0002nV-R3@upplysingaoflun>
From: Jeffrey Mark Siskind <q...@purdue.edu>
Date: Wed, 17 Aug 2016 18:10:17 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ecn.purdue.edu;
 h=to:subject:message-id:from:date; s=ecn20150812; bh=XiKIQSwHBw
9kEkCLW0qiVUXMLU++52GkT0SGis6u6l4=; b=IkYwJlFHVBxvFw66XeyJWnjSNg
pbgsadt9rjwjOEOIKMMO7AY7MqzURPjm6AUe7Ho2QJnZ1U5j4bVqTLAxNdOI8Guw
jGoyhQDZb7eeul+EufYBbAuKyS3cB1eABIxRPiVD1EKGtldcXLy2YvAXjTYyuFZR
ZhUGZkpzDsR4wJVOU=
X-ECN-MailServer-SpamScanAdvice: DoNotScan (-6.9/5.0)
X-ECN-MailServer-VirusScanned: by ecn-av-milter
X-ECN-MailServer-Origination: upplysingaoflun.ecn.purdue.edu 128.46.115.209

to qobi from qobi

--u7HMAIDg019691.1471471818/smtp.ecn.purdue.edu--

From mailer-dae...@ecn.purdue.edu  Wed Aug 17 18:10:39 2016
Return-Path: <mailer-dae...@ecn.purdue.edu>
Received: from mx05.ecn.purdue.edu (mx05.ecn.purdue.edu [128.46.4.13])
by dynamo.ecn.purdue.edu (8.14.4/8.14.4) with ESMTP id u7HMAdrh029449
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 
verify=NOT)
for <q...@dynamo.ecn.purdue.edu>; Wed, 17 Aug 2016 18:10:39 -0400
Received: from mailhub179.itcs.purdue.edu (mailhub179.itcs.purdue.edu 
[128.210.5.179])
by mx05.ecn.purdue.edu (8.14.4/8.14.4) with ESMTP id u7HMAcW7019713
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
for <q...@ecn.purdue.edu>; Wed, 17 Aug 2016 18:10:38 -0400
Received: from mailhub128.itcs.purdue.edu (mailhub128.itcs.purdue.edu 
[128.210.5.128])
by mailhub179.itcs.purdue.edu (8.14.4/8.14.4/spamscan) with ESMTP id 
u7HMAcLl028586
for <q...@purdue.edu>; Wed, 17 Aug 2016 18:10:38 -0400
Received: from smtp.ecn.purdue.edu (mx03.ecn.purdue.edu [128.46.177.4])
by mailhub128.itcs.purdue.edu (8.14.4/8.14.4/mta.smtp.purdue.edu) with 
ESMTP id u7HMAbFr018763
(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT)
for <q...@purdue.edu>; Wed, 17 Aug 2016 18:10:37 -0400
Received: from localhost (localhost)
by smtp.ecn.purdue.edu (8.14.4/8.14.4) id u7HMAbH2005473;
Wed, 17 Aug 2016 18:10:37 -0400
Date: Wed, 17 Aug 2016 18:10:37 -0400
From: Mail Delivery Subsystem <mailer-dae...@ecn.purdue.edu>
Message-Id: <201608172210.u7hmabh2005...@smtp.ecn.purdue.edu>
To: <q...@purdue.edu>
To: postmas...@ecn.purdue.edu
MIME-Version: 1.0
Content-Type: multipart/report; report-type=delivery-status;
boundary="u7HMAbH2005473.1471471837/smtp.ecn.purdue.edu"
Subject: Returned mail: see transcript for details
Auto-Submitted: auto-generated (failure)
X-PMX-Version: 6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 
2016.8.17.220618
X-PMX-Version: 6.0.2.2308539
X-PerlMx-URL-Scanned: Yes
X-PerlMx-Virus-Scanned: Yes
X-PerlMx-Spam: Gauge=, Probability=8%, Report='
 RCVD_FROM_IP_DATE 0.1, HTML_00_01 0.05, HTML_00_10 0.05, 
BODYTEXTP_SIZE_3000_LESS 0, BODYTEXTP_SIZE_400_LESS 0, BODY_SIZE_2000_2999 0, 
BODY_SIZE_5000_LESS 0, BODY_SIZE_7000_LESS 0, BOUNCE_ENVELOPE 0, BOUNCE_GENERIC 
0, BOUNCE_NDR 0, DATE_TZ_NA 0, FROM_EDU_TLD 0, FROM_NAME_PHRASE 0, 
FROM_SAME_AS_TO_DOMAIN 0, NO_URI_HTTPS 0, __ANY_URI 0, __ATTACHMENT_SIZE_0_10K 
0, __BOUNCE_NDR_CT_REPORT 0, __BOUNCE_NDR_FROM 0, __BOUNCE_NDR_SUBJECT_CONTAINS 
0, __CT 0, __

Re: exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

   > qobi@upplysingaoflun>all+n cat /etc/mailname
   > tlamachilistli:
   > tlamachilistli.ecn.purdue.edu
   > zhineng:
   > zhineng.ecn.purdue.edu

   This "all+n" intrigues me. What is it ?

One of several simple scripts I wrote that run a command on different subsets
of the machines I maintain.

Jeff (http://engineering.purdue.edu/~qobi)

Re: exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

  > What it does under jessie:
  > 
  >   (1) and (3) still work. (2) does not. I have never seen any mail to 
root but
  >   the people who maintain smtp.ecn.purdue.edu claim that mail to root 
on my
  >   machines gets sent to root@empty.

^^^ meaning precisely?
  (ie using quotes if necessary).

   I'd have to get back to you on this. I had asked this question of our
   sysadmins and they gave me the above answer. I'll ask them for a more precise
   answer.

Here is their reply:

It was literally "root@empty", according to the mail server logs. The 
mail servers could not resolve "empty" as a host name and rejected the 
message.

Jeff (http://engineering.purdue.edu/~qobi)

Re: exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

Thanks for your help.

   >   # mail sent by smarthost; received via SMTP or fetchmail
   >   # default hostname
 
   >   # default 127.0.0.1 ; ::1
   >   # blank
   >   # blank
   >   # smtp.ecn.purdue.edu
   >   # yes
   >   # purdue.edu
   >   # no
   >   # mbox format in /var/mail
   >   # no

   What do you actually have here?
   (ie which ends up in /etc/mailname)

Note that I didn't change this. I accepted the default when the question was
asked.

qobi@upplysingaoflun>all+n cat /etc/mailname
tlamachilistli:
tlamachilistli.ecn.purdue.edu
zhineng:
zhineng.ecn.purdue.edu
chino:
chino.ecn.purdue.edu
buddhi:
buddhi.ecn.purdue.edu
seykhl:
seykhl.ecn.purdue.edu
maniishaa:
maniishaa.ecn.purdue.edu
alykkyys:
alykkyys.ecn.purdue.edu
mohio:
mohio.ecn.purdue.edu
seulki:
seulki.ecn.purdue.edu
rongovosai:
rongovosai.ecn.purdue.edu
faisneis:
faisneis.ecn.purdue.edu
jalitusteabe:
jalitusteabe.ecn.purdue.edu
cuddwybodaeth:
cuddwybodaeth.ecn.purdue.edu
istihbarat:
istihbarat.ecn.purdue.edu
wywiad:
wywiad.ecn.purdue.edu
upplysingaoflun:
upplysingaoflun.ecn.purdue.edu
verstand:
verstand.ecn.purdue.edu
arivu:
arivu.ecn.purdue.edu
perisikan:
perisikan.ecn.purdue.edu
aruco:
aruco.ecn.purdue.edu
save:
save.ecn.purdue.edu
akili:
akili.ecn.purdue.edu
aql:
aql.ecn.purdue.edu
qobi@upplysingaoflun>

   > What it does under jessie:
   > 
   >   (1) and (3) still work. (2) does not. I have never seen any mail to root 
but
   >   the people who maintain smtp.ecn.purdue.edu claim that mail to root on my
   >   machines gets sent to root@empty.

 ^^^ meaning precisely?
   (ie using quotes if necessary).

I'd have to get back to you on this. I had asked this question of our
sysadmins and they gave me the above answer. I'll ask them for a more precise
answer.

(The Purdue maintained machines run Windows/MacOS/RHEL. This includes
smtp.ecn.purdue.edu and the whole email infrastructure. The Purdue sysadmins
don't know Debian. They are kind enough to let me maintain my own machines.)

   > What has changed from wheezy to jessie? As far as I can tell I am 
configuring
   > everything the same.
   > 
   > How do I get mail to root to forward to q...@purdue.edu?

   Is there anything significant in /etc/mailnames ?

I have no file named /etc/mailnames
I only have a file named /etc/mailname
The full contents of this file on each of my machines is shown above.

Jeff (http://engineering.purdue.edu/~qobi)

exim4 under jessie vs wheezy

[Jeffrey Mark Siskind]

Appologies for the repost. I sent this yesterday and it hasn't yet appeared.
It may have been filtered out for some reason.

I would appreciate some help configuring exim4.

What I want:

I have multiple machines that I maintain and have root access on. They have
domain ecn.purdue.edu and names like upplysingaoflun.ecn.purdue.edu and
tlamachilistli.ecn.purdue.edu. They have static IP addresses over
ethernet. Outgoing mail should go through smtp.ecn.purdue.edu which is a
machine that I don't maintain and don't have root access to. Incoming mail is
addressed to q...@purdue.edu (and alternatively q...@ecn.purdue.edu) and comes
to q...@dynamo.ecn.purdue.edu which is a machine that I have a user account on
but don't maintain and don't have root access on. I'd like to read it on, say,
upplysingaoflun.ecn.purdue.edu as user qobi. When I send mail from any machine
that I maintain, I'd like it to appear with a from: header of
q...@purdue.edu. Since my published email address is q...@purdue.edu (which
forwards to q...@ecn.purdue.edu and then to q...@dynamo.ecn.purdue.edu by
mechanisms that I have no control over). I have a .procmailrc file on
dynamo.ecn.purdue.edu that I want all email to go through. I also want all
mail to root on all of the machines that I maintain to go to q...@purdue.edu.

What I have been doing until now:

I have been running Debian on the machines that I maintain for a decade. I
recently upgraded from wheezy to jessie. I set up jessie with the same
configuration I had been using in wheezy and prior.

All of the machines that I maintain have:

  root@tlamachilistli:~# cat /etc/exim4/update-exim4.conf.conf 
  # /etc/exim4/update-exim4.conf.conf
  #
  # Edit this file and /etc/mailname by hand and execute update-exim4.conf
  # yourself or use 'dpkg-reconfigure exim4-config'
  #
  # Please note that this is _not_ a dpkg-conffile and that automatic changes
  # to this file might happen. The code handling this will honor your local
  # changes, so this is usually fine, but will break local schemes that mess
  # around with multiple versions of the file.
  #
  # update-exim4.conf uses this file to determine variable values to generate
  # exim configuration macros for the configuration file.
  #
  # Most settings found in here do have corresponding questions in the
  # Debconf configuration, but not all of them.
  #
  # This is a Debian specific file

  dc_eximconfig_configtype='smarthost'
  dc_other_hostnames=''
  dc_local_interfaces='127.0.0.1 ; ::1'
  dc_readhost='purdue.edu'
  dc_relay_domains=''
  dc_minimaldns='false'
  dc_relay_nets=''
  dc_smarthost='smtp.ecn.purdue.edu'
  CFILEMODE='644'
  dc_use_split_config='false'
  dc_hide_mailname='true'
  dc_mailname_in_oh='true'
  dc_localdelivery='mail_spool'
  root@tlamachilistli:~# 

The was obtained by running dpkg-reconfigure exim4-config and answering the
questions with the following:

  # mail sent by smarthost; received via SMTP or fetchmail
  # default hostname
  # default 127.0.0.1 ; ::1
  # blank
  # blank
  # smtp.ecn.purdue.edu
  # yes
  # purdue.edu
  # no
  # mbox format in /var/mail
  # no

I set up /etc/exim4/passwd.client to have the password needed for
smtp.ecn.purdue.edu.

I also set up /etc/aliases to forward root mail to q...@purdue.edu.

  root@tlamachilistli:~# cat /etc/aliases
  # /etc/aliases
  mailer-daemon: postmaster
  postmaster: root
  nobody: root
  hostmaster: root
  usenet: root
  news: root
  webmaster: root
  www: root
  ftp: root
  abuse: root
  noc: root
  security: root
  root: q...@purdue.edu
  root@tlamachilistli:~# 

I run fetchmail as user qobi on upplysingaoflun to fetch mail from
dynamo.ecn.purdue.edu.

Everything else is left at the default.

What I do now: the same as I have done before.

What it used to do under wheezy and before:

 1. Mail from user qobi on any machine that I maintained would get a from:
header q...@purdue.edu.
 2. Mail to root on any machine that I maintained would get sent to
q...@purdue.edu.
 3. All mail to q...@purdue.edu gets put in /var/spool/mail/qobi on
upplysingaoflun.ecn.purdue.edu.

What it does under jessie:

  (1) and (3) still work. (2) does not. I have never seen any mail to root but
  the people who maintain smtp.ecn.purdue.edu claim that mail to root on my
  machines gets sent to root@empty.

What has changed from wheezy to jessie? As far as I can tell I am configuring
everything the same.

How do I get mail to root to forward to q...@purdue.edu?

Thanks,
Jeff (http://engineering.purdue.edu/~qobi)

Re: machine checks on Dell R815 under jessie

[Jeffrey Mark Siskind]

   From: Ritesh Raj Sarraf 

   I (still) have MCE errors on my new laptop [1]. But so far, hasn't created
   any problem.

It causes my servers to halt.

Jeff (http://engineering.purdue.edu/~qobi)

machine checks on Dell R815 under jessie

[Jeffrey Mark Siskind]

I upgraded four Dell R815s from wheezy to jessie a few weeks ago. Prior to the
upgrade, they were running reliably for about 5 years. Since the upgrade, two
machines have been getting periodic machine checks. The machines boot fine and
run for a day or more. The machine checks appear to happen sporadically. I
can't determine a correlation with anything in particular.

The front panel on the first machine says the machine check was on CPU #4. The
front panel on the second machine said the first machine check was on CPU #1
and the second machine check was on CPU #2.

I am suspicious that this is really a hardware problem. Three CPUs begin
exhibiting machine checks within a few weeks of each other, all immediately
after upgrading wheezy to jessie, after working reliably for five years.

Has anybody else encountered this issue? Any suggestions on how to debug and
fix?

Thanks,
Jeff (http://engineering.purdue.edu/~qobi)
---
root@arivu:~# ipmitool sel elist
   1 | 08/05/2016 | 00:12:47 | Event Logging Disabled SEL | Log area 
reset/cleared | Asserted
   2 | 08/06/2016 | 11:35:17 | Processor CPU Machine Chk | Transition to 
Non-recoverable | Asserted
   3 | 08/06/2016 | 11:35:17 | Unknown #0x28 |  | Asserted
   4 | 08/06/2016 | 11:35:18 | Unknown #0x28 |  | Asserted
   5 | 08/06/2016 | 11:35:18 | Unknown #0x28 |  | Asserted
   6 | 08/06/2016 | 11:35:18 | Unknown #0x28 |  | Asserted
   7 | 08/06/2016 | 11:35:18 | Unknown #0x28 |  | Asserted
   8 | 08/06/2016 | 11:35:19 | Unknown #0x28 |  | Asserted
   9 | 08/06/2016 | 11:35:19 | Unknown #0x28 |  | Asserted
   a | 08/06/2016 | 11:35:19 | Unknown #0x28 |  | Asserted
root@arivu:~# 

root@perisikan:~# ipmitool sel elist
[...]
  1c | 08/08/2016 | 12:23:02 | Processor CPU Machine Chk | Transition to 
Non-recoverable | Asserted
  1d | 08/08/2016 | 12:23:03 | Unknown #0x28 |  | Asserted
  1e | 08/08/2016 | 12:23:03 | Unknown #0x28 |  | Asserted
  1f | 08/08/2016 | 12:23:03 | Unknown #0x28 |  | Asserted
  20 | 08/08/2016 | 12:23:03 | Unknown #0x28 |  | Asserted
  21 | 08/08/2016 | 12:23:03 | Unknown #0x28 |  | Asserted
  22 | 08/08/2016 | 12:23:04 | Unknown #0x28 |  | Asserted
  23 | 08/08/2016 | 12:23:04 | Unknown #0x28 |  | Asserted
  24 | 08/08/2016 | 12:23:04 | Unknown #0x28 |  | Asserted
  25 | 08/09/2016 | 18:37:46 | Processor CPU Machine Chk | Transition to 
Non-recoverable | Asserted
  26 | 08/09/2016 | 18:37:46 | Unknown #0x28 |  | Asserted
  27 | 08/09/2016 | 18:37:47 | Unknown #0x28 |  | Asserted
  28 | 08/09/2016 | 18:37:47 | Unknown #0x28 |  | Asserted
  29 | 08/09/2016 | 18:37:47 | Unknown #0x28 |  | Asserted
  2a | 08/09/2016 | 18:37:47 | Unknown #0x28 |  | Asserted
  2b | 08/09/2016 | 18:37:48 | Unknown #0x28 |  | Asserted
  2c | 08/09/2016 | 18:37:48 | Unknown #0x28 |  | Asserted
  2d | 08/09/2016 | 18:37:48 | Unknown #0x28 |  | Asserted
root@perisikan:~#

GLX, (xserver-)xorg(-dev), and nvidia-driver under jessie(-backports)

[Jeffrey Mark Siskind]

I have installed jessie. Among other packages installed, I have

   apt-get -y install gnome
   apt-get -y install xorg
   apt-get -y install xserver-xorg-dev
   apt-get -y install afni
   apt-get -y -t jessie-backports install nvidia-driver
   apt-get -y -t jessie-backports install nvidia-cuda-toolkit

I installed skype with:
   # https://wiki.debian.org/skype#Debian_7_.22Wheezy.22
   # wget -O /tmp/usb/skype-install.deb 
http://www.skype.com/go/getskype-linux-deb
   dpkg --add-architecture i386
   apt-get update
   dpkg -i /tmp/usb/skype-install.deb
   apt-get -y -f install
   apt-get clean

There appear to be two (possibly related) problems with GLX. One is that I run

   xvfb-run \
 --auto-servernum \
 --server-num=20 \
 -s "-screen 0 1024x768x24 -ac +extension GLX +render -noreset" \
 align_epi_anat.py ...

This used to work under wheezy. Now I get.

 3dSkullStrip -orig_vol -input ./__tt_subject01-anat+orig -prefix 
./__tt_subject01-anat_ns
   Xlib:  extension "GLX" missing on display ":20".
   freeglut (3dSkullStrip): OpenGL GLX extension not supported by display ':20'
   ** ERROR: Could not strip skull

   ** ERROR - script failed

The second is that I run

   qobi@tlamachilistli>ldd `which skype`|fgrep not
  libGL.so.1 => not found
libGL.so.1 => not found
   qobi@tlamachilistli>

Despite the fact that the problems manifest when running 3dSkullStrip (afni)
and skype, I believe that the problems do not lie in those packages. They lie
in the way that nvidia-driver changes the alternatives for xorg and
xserver-xorg-dev.

Note that I did not manually select alternatives. All I did was the above
apt-get and dpkg commands.

Any suggestions for how to properly fix would be appreciated.

Jeff (http://engineering.purdue.edu/~qobi)

qobi@tlamachilistli>ls -l /usr/lib/x86_64-linux-gnu/libGL.so
lrwxrwxrwx 1 root root 48 Jun 16 16:16 /usr/lib/x86_64-linux-gnu/libGL.so -> 
/etc/alternatives/glx--libGL.so-x86_64-linux-gnu
qobi@tlamachilistli>ls -l /etc/alternatives/glx--libGL.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 48 Jun 16 16:16 
/etc/alternatives/glx--libGL.so-x86_64-linux-gnu -> 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so
lrwxrwxrwx 1 root root 10 Jun 16 16:16 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so -> libGL.so.1
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1
lrwxrwxrwx 1 root root 14 Aug 19  2015 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1 -> libGL.so.1.2.0
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2.0
-rw-r--r-- 1 root root 627320 Aug 19  2015 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2.0
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/i386-linux-gnu/libGL.so.1
lrwxrwxrwx 1 root root 14 Aug 19  2015 
/usr/lib/mesa-diverted/i386-linux-gnu/libGL.so.1 -> libGL.so.1.2.0
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/i386-linux-gnu/libGL.so.1.2.0
-rw-r--r-- 1 root root 695836 Aug 19  2015 
/usr/lib/mesa-diverted/i386-linux-gnu/libGL.so.1.2.0
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/libGL.so-master
lrwxrwxrwx 1 root root 33 Jun 16 16:16 /usr/lib/mesa-diverted/libGL.so-master 
-> /etc/alternatives/libGL.so-master/
qobi@tlamachilistli>ls -l /etc/alternatives/libGL.so-master/
total 12
drwxr-xr-x 2 root root 4096 Apr 19 17:54 arm-linux-gnueabihf/
drwxr-xr-x 2 root root 4096 Jun 16 16:23 i386-linux-gnu/
lrwxrwxrwx 1 root root   33 Jun 16 16:16 libGL.so-master -> 
/etc/alternatives/libGL.so-master/
drwxr-xr-x 2 root root 4096 Jun 16 16:16 x86_64-linux-gnu/
qobi@tlamachilistli>ls -l /usr/lib/x86_64-linux-gnu/libGL.so
lrwxrwxrwx 1 root root 48 Jun 16 16:16 /usr/lib/x86_64-linux-gnu/libGL.so -> 
/etc/alternatives/glx--libGL.so-x86_64-linux-gnu
qobi@tlamachilistli>ls -l /etc/alternatives/glx--libGL.so-x86_64-linux-gnu
lrwxrwxrwx 1 root root 48 Jun 16 16:16 
/etc/alternatives/glx--libGL.so-x86_64-linux-gnu -> 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so
lrwxrwxrwx 1 root root 10 Jun 16 16:16 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so -> libGL.so.1
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1
lrwxrwxrwx 1 root root 14 Aug 19  2015 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1 -> libGL.so.1.2.0
qobi@tlamachilistli>ls -l /usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2.0
-rw-r--r-- 1 root root 627320 Aug 19  2015 
/usr/lib/mesa-diverted/x86_64-linux-gnu/libGL.so.1.2.0
qobi@tlamachilistli>ls -l /usr/lib/x86_64-linux-gnu/nvidia/libGL.so.1
lrwxrwxrwx 1 root root 53 Jun 16 16:16 
/usr/lib/x86_64-linux-gnu/nvidia/libGL.so.1 -> 
/etc/alternatives/nvidia--libGL.so.1-x86_64-linux-gnu
qobi@tlamachilistli>ls -l /etc/alternatives/nvidia--libGL.so.1-x86_64-linux-gnu
lrwxrwxrwx 1 root root 51 Jun 16 16:19 

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

   I had big issues with mptsas and 3.16 in jessie, so I am still using
   3.2.0-4-rt-amd64

Will jessie run with 3.2.0-4-rt-amd64? If so, where do I get it and how do I
install it on a fresh jessie install that wasn't dist-upgraded from wheezy?

Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

   The non-determinism in which identifiers are shown might be a bug in the
   installer, or it might be caused by failure of ID commands to the
   drives.

   I think most of the problems you're still having must be caused by a
   bug in the RAID driver, mpt2sas (or its firmware, if that's not
   embedded in the BIOS).

Thanks. Please let me know how I can report the potential bug(s) and what I
can do to help track them down.

Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

I'd like to thank everyone for helping out.

Here is an update on installing jessie on R815s.

I succeeded in installing on three of my four R815s. But I am holding off on
the last because it is my file server and there are still issues. Please read
on. I don't believe that the problem is solved and there may be a bug lurking
that can lead to data loss.

Here is what I did.

 1. Before the install, while still running wheezy, I upgraded the BIOS.
  R815_BIOS_JF8YH_LN_3.2.2.BIN
This seemed to alleviate the problem of the jessie installer failing to
find the ISO. More on this later.

 2. Before the install, while still running wheezy, I reduced the number of
components of md0 from 6 to 4. This was in response to Steve' suggestion.
  mdadm /dev/md0 --fail /dev/sdf1
  mdadm /dev/md0 --fail /dev/sde1
  mdadm /dev/md0 --remove /dev/sdf1
  mdadm /dev/md0 --remove /dev/sde1

 3. I did a fresh USB install of jessie. More on this later.

 4. When it asked about which devices to install grub, I answered "manual" and
then typed /dev/sdb. More on this later.

 5. After the fresh install, I rebooted, and in grub, I added rootdelay=20.
This was in response to Don's suggestion.

 6. After the reboot, I ran my standard post-install script. Among other
things, this installs numerous packages, makes a small number of mods to
/etc, and does a dpkg-reconfigure grub-pc. When it did that, I specified
only the 4 drives with active components of md0 and added rootdelay=20.

 7. I rebooted. More on this later.

Now for the issues.

 A. Even after the BIOS upgrade, when it no longer fails to find the ISO,
during the installer phase where it searches for an ISO, I notice
nondetermininstic behavior. Sometimes it searchs sdb{1,2,3}, sdc{1,2,3},
sdd{1,2,3}, sde{1,2,3}, sdf{1,2,3}, sdg{1,2,3}, sd{a,b,c,d,e,f,g} and
eventually finds an ISO (sda is the USB dongle). Sometimes it finds the
ISO right away without any searching. This doesn't cause problems but I
believe that it is symptomatic of other problems.

 B. I'm not sure that reducing the number of components of md0 to 4 and/or
adding rootdelay=20 really solved the problem. I think it just reduced the
likelihood of occurrence. On one of the machines (arivu), during the
reboot in step (7), at an early phase of the boot, the machine first
reported that it found all 4 components of md0 and all 6 components of md1.
Then at  a later phase it reported that there were errors on 3 of the 4
components. After the machine came up, md0 had only one component. Three
of the four components were in failed (F) state. I did mdadm --remove to
them and then mdadm --add to them. This doesn't happen all of the time. But
it happens some of the time.


  qobi@upplysingaoflun>all-n-3g dmesg --level=err
  upplysingaoflun:
  verstand:
  arivu:
  [   28.012558] mpt2sas0: fault_state(0x265d)!
  [   29.231355] end_request: I/O error, dev sdb, sector 2056
  [   29.231600] end_request: I/O error, dev sdc, sector 2056
  [   29.231773] end_request: I/O error, dev sde, sector 2056
  [   29.232020] end_request: I/O error, dev sda, sector 2056
  perisikan:
  [   13.035132] mpt2sas0: fault_state(0x265d)!
  [   28.600099] mpt2sas0: fault_state(0x265d)!
  qobi@upplysingaoflun>

  qobi@upplysingaoflun>all-n-3g "dmesg --level=warn|fgrep -i error|fgrep -v 
ACPI"
  upplysingaoflun:
  verstand:
  arivu:
  [   29.231430] md: super_written gets error=-5, uptodate=0
  [   29.231670] md: super_written gets error=-5, uptodate=0
  [   29.231869] md: super_written gets error=-5, uptodate=0
  [   29.232117] md: super_written gets error=-5, uptodate=0
  perisikan:
  qobi@upplysingaoflun>

(These are my four R815s. upplysingaflun is the file server that has not
been updated. The other three have.) Note that one machine reports no
"mpt2sas0: fault_state(0x265d)" errors, one machine reports one, and one
machine reports two. Note that the machine that dropped three components
of md0 during boot reported I/O errors on all 4 disks with the 4
components of md0. I don't believe that there really are faulty disks.
Whenever I observe any of the behavior reported in this email, it is
almost always associated with dmesg reporting the same error on the same
sector 2056 (sometimes 2058 or 2062). Given the dozens of attempted
reinstalls and reboots, at this point, I have seen this on almost all, if
not all, of the six disks on each of the four machines. I don't believe
that 24 disks all have the same bad sectors.

 C. In step (3), sometimes, but not always, during the install, I get a screen
that says that some partition failed. If offers a menu of two options. I
select "retry". Sometimes, but not always, this causes md0 to drop
components in the installer, which I fix by going to ctrl-alt-f2 

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

Please note that bootint with rootdelay=20 does not solve the problem. It only
masks it.

 1. If I attempt a fresh USB install of jessie, when md0 is correctly built
before the install, the process of doing the fresh install breaks
md0. When it gets to grub install, components of md0 are missing (even
though all six components were present before the install). And
grub-install fails. At this point it is impossible to complete the install
and produce a bootable system.

 2. If I do a fresh minimal USB install of wheezy, rebuilding md0 in the
process, and then do a dist-upgrade to jessie, I can manually add
rootdelay=20 in grub and boot into jessie with all six components of md0
present. But if I do so, then after boot, if I do dpkg-reconfigure pc-grub,
doing that gives errors, drops components of md0, precludes me from adding
them back, fails to install grub, and leaves the machine in an unbootable
state.

I fear that there is a problem writing to disk. Even if I boot with
rootdelay=20, unless the kind of writes that dpkg-reconfigure pc-grub does are
different, doing ordinary writes to disk may also corrupt the disk.

Please let me know what new information you would like me to gather.

Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

   >Are you certain that there isn't a PERC H700 in this machine? [Sort of
   >odd that mpt2sas is triggering a state error in your screenshot if there
   >actually isn't one.]
   > 
   > There could be one. But I probably don't use it. I use software RAID. Dell
   > wouldn't sell an R815 without an OS. I think I purchased it with RHEL which
   > may have needed the PERC H700. But I never even booted RHEL. The first
   > thing I did was a fresh install of squeeze, or maybe wheezy.

   We definitely sell PowerEdge systems without an OS and have for quite a
   while. However, we do limit configuration for higher end systems to include
   hardware RAID.

My appologies. I may misremember. I purchased the machines (twelve T5500s,
four R815s, and four C6145s) about 5 years ago and don't remember precisely
the arrangements. I'd have to check archived email to know for sure.

The machines were purchased through ECN (Purdue's Engineering IT services). I'm
a lowly professor. But I software-maintain my own machines. I definitely
didn't spec out a hardware RAID controller. The mechanisms by which one was
included are unclear at this point.

   There's definitely a PERC controller in there based on 

   "05:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 
PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)"

   I'm not seeing the subvendor/subsystem ID's there but it's presumably the
   PERC 6/i. If you're really not using it at all, you might be able to pull
   it out if the driver for it is causing problems. However, I suspect you
   need it to connect to the drive backplane. Stuart (CCed) may be able to
   offer some more insight into driver issues you might see.

   The SATA controller should only really be in use by the optical drive if
   present. Some of the mid-tier systems of that generation support SATA
   drives connected directly to a controller on the motherboard, but support
   for that under Linux was spotty from my recollection.

My T5500s have optical drives. But neither my R815s nor my C6145s have optical
drives. All my machines have SATA drives. The R815s in question each have
six ST9500530NS drives. They have been running squeeze and then wheezy with
software RAID for 5 years since purchase.

Now that I have someone from Dell on the line who appears to be
Debian-friendly, it would be nice if you made firmware upgrades
Debian-friendly. I have been able to apply

  R815_BIOS_JF8YH_LN_3.2.2.BIN

but have not been able to apply

  ESM_Firmware_7N76T_LN32_1.07_A00.BIN
  ESM_Firmware_J7YYK_LN32_2.85_A00.BIN
  SATA_FRMW_LX_R300994.BIN

(I don't even know if either of the ESM upgrades are for my hardware. But the
shell scripts don't run.)

Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

I conjecture that there may be two to five separate issues.

 1. Setting up md0 upon boot takes a long time. rootdelay=20 fixes this.
 2. There is a problem writing to disk. Perhaps just writing to certain blocks.
Because even when the machine boots with rootdelay=20, and md0 has all 6
components, grub-install fails and causes md0 to drop some/most of its
components.

Both of these are observed with a dist path-upgrade from a fresh USB install
of wheezy to jessie. Separate from this, there are two other errors observed
with a direct fresh USB install of jessie.

 3. Can't find the ISO.
 4. grub-install
This may be the same as (2) above.

This is yet distinct from the fact that

 5. a fresh direct USB install of jessie on the Dell Poweredge C6145s takes a
really long time (an hour) for each hardware probe (three times, once
before finding the ISO, once before partitioning, and once before grub
install).

Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

   > and attempted
   > 
   >mdadm /dev/md0 --add /dev/sda1
   >mdadm /dev/md0 --add /dev/sdb1
   >mdadm /dev/md0 --add /dev/sdc1
   >mdadm /dev/md0 --add /dev/sdd1
   >mdadm /dev/md0 --add /dev/sde1
   >mdadm /dev/md0 --add /dev/sdf1
   > 
   > but these all failed.

   This is the wrong command; it should be mdadm --assemble /dev/md0
   /dev/sd[abcdef]1;

   And that should only be done if the md0 device doesn't show up in the
   initrd when you cat /proc/mdstat.

   What's happened is that the raid1 device now has 12 drives instead of 6,
   which basically isn't going to work at all.

You can see from the transcript that md0 is there and has only 6 drives. Just
that 5 of the six are marked as failed. And you can see that it refused to do
the mdadm --add.

   http://upplysingaoflun.ecn.purdue.edu/~qobi/upgrade-jessie2.script

   root@verstand:~# cat /proc/mdstat
   Personalities : [raid1] [raid6] [raid5] [raid4] 
   md1 : active raid5 sda2[0] sdf2[5] sdd2[4] sdc2[3] sde2[2] sdb2[1]
 1953118720 blocks super 1.2 level 5, 512k chunk, algorithm 2 [6/6] 
[UU]

   md0 : active raid1 sda1[6](F) sdd1[8](F) sdb1[7](F) sde1[9](F) sdc1[10] 
sdf1[11](F)
 39157688 blocks super 1.2 [6/1] [__U___]

   unused devices: 
   root@verstand:~# mdadm --add/dev/md0 --add /defv/sda1
   mdadm: Cannot open /dev/sda1: Device or resource busy
   root@verstand:~# mdadm /dev/md0 --add /dev/sda1b1
   mdadm: Cannot open /dev/sdb1: Device or resource busy
   root@verstand:~# mdadm /dev/md0 --add /dev/sdb1d1
   mdadm: Cannot open /dev/sdd1: Device or resource busy
   root@verstand:~# mdadm /dev/md0 --add /dev/sdd1e1
   mdadm: Cannot open /dev/sde1: Device or resource busy
   root@verstand:~# mdadm /dev/md0 --add /dev/sde1f1
   mdadm: Cannot open /dev/sdf1: Device or resource busy
   root@verstand:~# mdadm /dev/md0 --add /dev/sdf11c1
   mdadm: Cannot open /dev/sdc1: Device or resource busy

   You should be able to just directly reinstall jessie on this machine;

In earlier posts I explained how this fails. If I do a direct install from
USB, I observe two kinds of errors.

 1. Sometimes, but not every time, (it is nondeterministic) after the first 3
questions, the installer complains that it can't find the ISO.
 2. Whenever it does find the ISO, the install progresses without error all
the way to the grub install and then complains that it can't install grub.
I've tried several different things. Sometimes, I just answer sda to the
grub install question. (Actually sometimes sdb, because if I plug the USB
into the front port, the USB gets sdg and the drives get sd[a-f] but if I
plug the USB into the back port, the USB gets sda and the drives get
sd[b-g].) But this always fails. Sometimes, I go into ctrl-alt-f2 and do
  chroot target
  grub-install /dev/sda
  ...
  grub-install /dev/sdf
  (or b-g as appropriate)
but this also fails. At that point, I have no way to install grub. (If I
abort the install, the machine is unbootable.) Whenever I'm in this state
I do cat /proc/mdstat and it shows that some components of md0 are failed
or missing. Some are present. This is nondeterministic. Which components
are present and which are missing changes each time I attempt this. If I
attempt to do mdadm --add I get errors. If I reinstall fresh wheezy from
USB and then in wheezy do mdadm --add, it works and rebuilds the
array. When it is done it has all 6 components. And then I immediately do
a fresh install of jessie from USB and the same problem happens.

   I'd also zero out the superblocks on the devices in /dev/md0,

What command?

Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

   Are you certain that there isn't a PERC H700 in this machine? [Sort of
   odd that mpt2sas is triggering a state error in your screenshot if there
   actually isn't one.]

There could be one. But I probably don't use it. I use software RAID. Dell
wouldn't sell an R815 without an OS. I think I purchased it with RHEL which
may have needed the PERC H700. But I never even booted RHEL. The first thing I
did was a fresh install of squeeze, or maybe wheezy.

   OK. This:

   > 00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI 
SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode]

   makes me think that the SATA controller is in IDE/Legacy mode instead of
   AHCI. In theory, this shouldn't matter, but it's possible that this is
   also a problem. I'd try switching it in the bios and see what happens.

I'll do that in a bit. Before I got your current post, I tried some things in
response to your previous post. I'll report on that here and then go back and
try the new things.

Here is what I did.

I had a fresh minimal USB install of wheezy running. That install was done
with debian-wheezy-DI-b1-amd64-netinst.iso from Jul 15  2012. I also put the
non-free firmware on the USB. When I did that, I unchecked all of the boxes
during the install for any extra packages. The only thing that I installed
after that was

   apt-get install less

I then did

   nano /etc/apt/source.list
   (change all wheezy to jessie)
   apt-get update
   apt-get dist-upgrade

I answered all of the defaults.

(default) all
(default) no
(default) cron

I captured this with

   script -t 2>upgrade-jessie1 time -a ~/upgrade-jessie1.script

(My mistake. I forgot a period between upgrade-jessie1 and time.)

   http://upplysingaoflun.ecn.purdue.edu/~qobi/time
   http://upplysingaoflun.ecn.purdue.edu/~qobi/upgrade-jessie1

You can see that it all worked.

You can see that at the end I did

   apt-get install firmware-linux

   dpkg-reconfigure grub-pc
   # default
   # default
   # check all /dev/sd?

and it all worked.

You can also see that at the end I did

   cat /proc/mdstat

and all 6 components of both md0 and md1 were there.

Then I did and

   /sbin/reboot

The first reboot failed. It gave a similar screen as to the one that you
already saw.

Then I did a second reboot, with delay=20. That did the same.

Then I did a third reboot, with rootdelay=20. That worked. I got a login
prompt, logged in, and got a root shell.

At that point, I did a 

   cat /proc/mdstat

and all 6 components of both md0 and md1 were there.

Then I did a

   dpkg-reconfigure grub-pc

My intent was to add rootdelay=20 to the command line. But I got lots of
errors while doing so. I realized that I should have done this under script.
So I did

   script -t 2>upgrade-jessie2.time -a ~/upgrade-jessie2.script

(this time with the period) and redid

   dpkg-reconfigure grub-pc

and also did

   cat /proc/mdstat

and attempted

   mdadm /dev/md0 --add /dev/sda1
   mdadm /dev/md0 --add /dev/sdb1
   mdadm /dev/md0 --add /dev/sdc1
   mdadm /dev/md0 --add /dev/sdd1
   mdadm /dev/md0 --add /dev/sde1
   mdadm /dev/md0 --add /dev/sdf1

but these all failed.

   http://upplysingaoflun.ecn.purdue.edu/~qobi/upgrade-jessie2.script
   http://upplysingaoflun.ecn.purdue.edu/~qobi/upgrade-jessie2.time

The machine is now in the state left at the end of the above script. If you
want me to do some more things in this state, let me know. Or I can do a fresh
USB install of wheezy and rebuild md0.

   >What does the kernel output while it is detecting the disks and
   >partitions?

   Remove the quiet option from the kernel command line by editing it in grub.

I will do this next time.

   > Do all of the drives show up properly?

   echo /dev/sd*; should give you an idea of what is there in the initramfs.

I will do this next time.

   >When the boot fails, can you read from the underlying block
   >devices?

   more /dev/sda; should work, I believe.

I will do this next time.

   > I don't know what one can do in at the initramfs command prompt. If you 
give
   > me some commands, I will try them out and post the output.
   > 
   >Does specifying delay=20 or similar result in a successful boot?

   > I will try this.

   This should actually be rootdelay=20; sorry.

Done. See above.

   > I will try to get this info. It will require me to redo the exercise
   > of a fresh jessie install from USB. I'll have to take and post screen
   > pictures because I have no way to capture the console output.

   I believe the R815 still has a serial port; you can just plug in a
   serial cable and append an appropriate serial tty option to the kernel
   command line to get output as text.

I figured out how to use script. That will work for most situations.

   What I'm trying to do is get enough information so that the error is
   obvious.

Thanks. Let me know what you want me to try next. Do you still wish me to do
the following?

   >What does the kernel output while it 

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

Thanks for your help.

   > Here is a screen picture.

   Could you upload this to an image paste site or send it along (or use a
   serial console to get it as text?)

http://upplysingaoflun.ecn.purdue.edu/~qobi/20160619_140357.jpg

(The other screen picture of a machine (not an R815) that does boot but that
takes a really long time to bring up the network is at

http://upplysingaoflun.ecn.purdue.edu/~qobi/IMG-20160609-WA.jpeg

)

   > I conjecture that the jessie kernel has difficulty accessing the MD
   > array on disk. The same problem occurs when I attempt a direct fresh
   > install of jessie with the installer.

   Which add-in card are you using on the R815s?

I don't believe that I have any add-in cards. The machine was purchased
straight from Dell. It has six SATA disks and 4 gigabit ethernet ports. It has
four 12-core AMD CPUs and 128GB RAM. The output of lspci on an indentical
machin purchased at the same time that is still running wheezy is enclosed
below.

   What does the kernel
   output while it is detecting the disks and partitions? Do all of the
   drives show up properly? Are the blocksizes correct for the partitions?

I don't know how to get this info when in the initramfs after boot. If you
tell me what commands I should give I will redo this exercise. Right now, I
have a fresh minimal wheezy reinstalled. But after the reinstall of wheezy,
everything works. I did not repartition either during the (re)install of
jessie or during the (re)install of wheezy. I go back and forth. The
(re)install of wheezy works and the (re)install of jessie does not.

   When the boot fails, can you read from the underlying block devices? Do
   the block devices get detected after the boot fails?

I don't know what one can do in at the initramfs command prompt. If you give
me some commands, I will try them out and post the output.

   Does specifying delay=20 or similar result in a successful boot?

I will try this.

 I made the dongle
   > as follows:
   > 
   ># cd /tmp
   ># wget 
http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/current/images/hd-media/boot.img.gz
   ># wget 
http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/8.5.0+nonfree/amd64/iso-cd/firmware-8.5.0-amd64-netinst.iso
   ># zcat boot.img.gz >/dev/sdf
   ># mount /dev/sdf /mnt
   ># cp firmware-8.5.0-amd64-netinst.iso /mnt/.

   You can actually just cat firmware-8.5.0-amd64-netinst.iso > /dev/sdf;

Please see my other post to debian-user

subject: how to make bootable live wheezy USB that doesn't use isohybrid

One of the exercises I tried was when the machine failed to boot after a fresh
USB-install of jessie, I tried to boot a live wheezy from USB by using a USB
dongle that I made by catting the isohybrid live wheezy ISO to the USB. But
the BIOS failed to detect the USB as bootable. I haven't tried to do that with
the netinst ISO but I suspect that it also won't be detected as bootable. But
when I build the USB dongle as per above it is detected by the BIOS as bootable.

   > Every time so far, md1 has all 6 components. But md0 has only some of
   > the components, sometimes 5/6, sometimes 4/6, and sometimes 1/6. And
   > every time it is a different set of components. Even though, just a
   > few minutes earlier, I was running wheezy and md0 had all 6
   > components. I do
   > 
   > mdadm /dev/md0 --add 
   > 
   > but it refuses. I forget the error.

   The error would be useful to know. Most likely one or more of them
   dropped out of the array for some reason and you're booting off of one
   which has a lower event count and it won't assemble.

   But it could be any number of things.

   The output of mdadm --examine /dev/sd[abcdef]1; when md0 fails to
   assemble would also be useful.

I will try to get this info. It will require me to redo the exercise of a
fresh jessie install from USB. I'll have to take and post screen pictures
because I have no way to capture the console output. (I guess that I could use
iDRAC but I don't know how to and would have to learn.) If you let me know all
of the info you would like me to collect, I will try to collect it all in the
same retry of the fresh install.

But again note, that I do not believe that there are any disk hardware
errors. And I do not believe that there are any data errors in the layout of
the ext3 file system, the layout of the md0 raid array, or the partition
tables. The reason is that after the failed jessie install, I reinstall a
fressh wheezy from USB. I don't repartition. And I don't rebuild md1 and don't
rebuild /aux. But I do rebuild md0 and / as part of the fresh install. And it
works. I have done this over and over, switching between wheezy and jessie,
about a half dozen times. Each time, the jessie install leaves a different
collection of md0 components out. And each time, as part of the wheezy
install, I add them back in.

Thanks for your help.
Jeff (http://engineering.purdue.edu/~qobi)

Re: jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

I get no
errors during the upgrade. And after the upgrade, before reboot, all 6
components of md0 are there. (That is still running the wheezy kernel.) All I
do is /sbin/reboot and then it comes up in the initfs. And if I then do a
fresh reinstall of wheezy, I need to rebuild md0.

So it seems to me that something in the jessie kernel is broken, probably
related to the disk driver.

Also note that I upgraded to the latest BIOS. But the same exact problems
occurred both before the BIOS upgrade and after.

   booting jessie also takes hours to do systemd
   > configuration of the network

FYI, here is a screen picture where it takes minutes for systemd to bring up
the network. Note that I am not using DHCP. As per the enclosed, each host has
a fixed IPv4 address. There are fixed DNS servers. I am at a university and IT
services maintains the network for thousands of machines. I do not observe
issues bringing up the network when running wheezy.

Jeff (http://engineering.purdue.edu/~qobi)

default Install
default English
default United States
default American English
Go Back
default Configure network manually
128.46.115.211
default netmask
default gateway
128.210.11.57 128.210.11.5 128.46.154.76
default hostname
default domain name
root password
root password
Jeffrey Mark Siskind
qobi
password
password
default Eastern
Manual
RAID1 #1
Ext3 journaling file system
Format the partition: yes, format it
Mount point: /
Done setting up the partition
RAID5 #1
Ext3 journaling file system
default Format the partition: no, keep existing data
Mount point: /aux
Done setting up the partition
Finish partitioning and write changes to disk
Yes
default United States
default ftp.us.debian.org
default blank
Yes
uncheck all
Yes
/dev/sda
Continue
---
Disk /dev/sda: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0080

   Device Boot  Start End  Blocks   Id  System
/dev/sda1   *20487831961539158784   fd  Linux raid autodetect
/dev/sda278319616   859570175   390625280   fd  Linux raid autodetect
/dev/sda3   859570176   97677107158600448   82  Linux swap / Solaris

jessie won't install/boot on a Dell Poweredge R815

[Jeffrey Mark Siskind]

I am attempting to install jessie on a Dell Poweredge R815. It has been
running wheezy reliably for years. And running squeeze reliably for years
before that. But no matter what I try it won't install or boot.

I have tried two ways.

 1. I attempt a fresh install from a USB dongle. It gets all the way to
installing grub and then fails.

 2. I do a fresh install of wheezy from a USB dongle. It boots wheezy just fine.
I do nothing but

  nano /etc/apt/sources.list
  (change all instances of wheezy to jessie, save, and exit)
  apt-get update
  apt-get dist-upgrade
  (It upgrades without error. I answer the default to all questions.)
  /sbin/reboot

Then it fails to reboot and goes into the initramfs. I have a picture of
the screen if anybody wishes.

I can reliably install and run wheezy over and over. I have not been able to
install or boot jessie despite numerous attempts.

Any suggestions?

Jeff (http://engineering.purdue.edu/~qobi)

Re: how to make bootable live wheezy USB that doesn't use isohybrid

[Jeffrey Mark Siskind]

   > I'd like to make a live wheezy USB dongle. I followed the
   > instructions on
   > 
   >https://www.debian.org/CD/faq/#write-usb
   > 
   > to make a live USB from
   > 
   >
http://cdimage.debian.org/mirror/cdimage/archive/7.11.0-live/amd64/iso-hybrid/debian-live-7.11.0-amd64-standard.iso
   > 
   > but apparently my machine, an older Dell Poweredge R815, won't boot
   > from isohybrid. Can someone point me to a howto for making a bootable
   > live wheezy USB dongle that doesn't use isohybrid?

   Not familiar with that model.  Will it boot from the USB
   port?  I'm assuming it will.  Seems new enough for that feature.
   So check the drive boot order and change as required.

It will boot MSDOS from a fat32 formatted USB dongle. But apparently, it won't
boot an isohybrid formatted USB dongle. It has nothing to do with drive boot
order. If I create a USB dongle with

   # cd /tmp
   # wget 
http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/current/images/hd-media/boot.img.gz
   # wget 
http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/8.5.0+nonfree/amd64/iso-cd/firmware-8.5.0-amd64-netinst.iso
   # zcat boot.img.gz >/dev/sdf
   # mount /dev/sdf /mnt
   # cp firmware-8.5.0-amd64-netinst.iso /mnt/.
   # umount /mnt

the R815 will detect and boot from the resulting USB dongle. If I create a USB
dongle with

   # cd /tmp
   # wget 
http://cdimage.debian.org/mirror/cdimage/archive/7.11.0-live/amd64/iso-hybrid/debian-live-7.11.0-amd64-standard.iso
   # cp debian-live-7.11.0-amd64-standard.iso /def/sdf

the R815 will not detect and not boot from the resulting USB dongle. If I do

   # cd /tmp
   # wget 
http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/current/images/hd-media/boot.img.gz
   # wget 
http://cdimage.debian.org/mirror/cdimage/archive/7.11.0-live/amd64/iso-hybrid/debian-live-7.11.0-amd64-standard.iso
   # zcat boot.img.gz >/dev/sdf
   # mount /dev/sdf /mnt
   # cp debian-live-7.11.0-amd64-standard.iso /def/sdf
   # umount /mnt

will this be able to boot a live wheezy?

Jeff (http://engineering.purdue.edu/~qobi)

Re: how to make bootable live wheezy USB that doesn't use isohybrid

[Jeffrey Mark Siskind]

   > I'd like to make a live wheezy USB dongle. I followed the instructions on

   Why wheezy ... it's old.

I have been running wheezy for about 5 years without any problem. The machine
has two partitions / and /aux. I attempted a fresh install of jessie in /.
But the install failed and the machine won't boot. It appears that the kernel
in jessie tickles something in the hardware that wasn't tickled by wheezy. I
will likely upgrade the firmware. (Dell Poweredge machines have a lot of
distinct firmware components besides the BIOS.) But before I do, I want to get
the data off the /aux partition. So I want to boot something that I know works.

   >https://www.debian.org/CD/faq/#write-usb
   > 
   > to make a live USB from
   > 
   >
http://cdimage.debian.org/mirror/cdimage/archive/7.11.0-live/amd64/iso-hybrid/debian-live-7.11.0-amd64-standard.iso

   There are many ways to make a bootable HDD image if this can be booted.

   By using aufs or newer overlayfs, you can choose to make OS persistent
   while your running system writing on RAM(tmpfs).

https://wiki.debian.org/DebianEeePC/HowTo/InstallOnSDcardOrUsbStick
I have not tested but this may give you hint.

   I also use kvm to install system on USB connected HDD from my working
   Debian system.

   many ways ...

I don't need it to be persistent. I just need to get enough of a root shell to
run tar|ssh to get the data off of /aux. /aux is /dev/md1 which is RAID5
across /dev/sd[a-f]2. So I need enough to be able to mount that read-only.
Then when that is done I will upgrade the firmware and reattempt a jessie
install.

So I'd appreciate help getting the easiest way to get a root shell with tar,
ssh, and mdadm.

The issue is that the machine is too old to boot isohybrid from USB. It needs
to boot msdos from fat32 on USB. See other emails in this thread.

Jeff (http://engineering.purdue.edu/~qobi)

how to make bootable live wheezy USB that doesn't use isohybrid

[Jeffrey Mark Siskind]

I'd like to make a live wheezy USB dongle. I followed the instructions on

   https://www.debian.org/CD/faq/#write-usb

to make a live USB from

   
http://cdimage.debian.org/mirror/cdimage/archive/7.11.0-live/amd64/iso-hybrid/debian-live-7.11.0-amd64-standard.iso

but apparently my machine, an older Dell Poweredge R815, won't boot from
isohybrid. Can someone point me to a howto for making a bootable live wheezy
USB dongle that doesn't use isohybrid?

Thanks,
Jeff (http://engineering.purdue.edu/~qobi)

problems installing jessie on Dell R815 and C6145

[Jeffrey Mark Siskind]

I have a Debian farm of 24 machines running wheezy that I am upgrading to
jessie. Four of the machines are identical Dell R815 servers, each with 6
disks. (The others are Dell T5500 with 4 disks each (12 machines), Dell C6145
with 4 disks each (4 machines), or HP Proliant DL165 G5p with 3 disks each (4
machines).) All of the machines of each type are identical. Upgrading the
T5500, C6154, and DL165 is progressing with only minor issues that I will get
into if it matters. But I am having difficulty with the R815. So far, I have
only attempted to upgrade one machine. The other 3 are still running wheezy.
All have been running wheezy reliably for about 5 years.

I have made a USB flash drive with:

   
http://cdimage.debian.org/cdimage/unofficial/non-free/cd-including-firmware/8.5.0+nonfree/amd64/iso-cd/firmware-8.5.0-amd64-netinst.iso
   
http://ftp.nl.debian.org/debian/dists/jessie/main/installer-amd64/current/images/hd-media/boot.img.gz

I have used this to successfully install jessie on six of the T5500, one of
the DL165, and one of the C6145. But on the R815, I notice nondeterministic
behavior. I have tried the install about 7 times. All 7 succesfully boot into
the install screen. I select

  Install (default)
  English (default)
  United States (default)
  American English (default)

It then searches for hardware and an ISO image. Four of the seven times, I get
a red screen that claims that it can't find an ISO image. Three of the seven
times, it finds one, proceeds with the install, but fails during the grub
install. Each of those three times, it failed in different ways. I can
describe the precise ways in followup email if someone can help. I'm not
including the details here to avoid clutter.

The machine was running wheezy reliably until a few days ago.

The identical install media (a USB flash drive) has successfully been used to
install jessie on the T5500, DL165, and C6145, both before and after the
attempts on the R815. So I believe that the media is OK.

Because of the nondeterministic behavior, and the fact that the machine was
running reliably, I believe that there are no hardware problems, at least not
ones that were tickled by wheezy. Perhaps there are hardware problems that are
tickled by jessie. I conjecture that there may be kernel or installer changes
that no longer work reliably on R815 hardware.

What also may help in diagnosing the issues is the following observation.
While the installs on six T5500s and one C6145 were all successful, the
install on the C6145 took about 3 hours, while each install on a T5500 took
about 15 minutes. The T5500s and C6145 both have the same number of disks of
the same size. And all are partitioned and configured identically. So one
would not expect much difference in install time. Yet when installing on the
C6145, the steps that involve probing hardware take hours while those steps on
on the T5500 are instantaneous. This suggests that there have been software
issues introduced into the installer and/or kernel that affect the R815 and
C6145 platforms.

FWIW, the c-a-f4 vt during the install on the C6145 has a huge number of
messages:

   reset high-speed usb device number 6 using ehci-pc
   reset high-speed usb device number 7 using ehci-pc

Even after the successful install, dmesg on that machine gives a huge number
of

[18660.329918] usb 1-5.2: reset high-speed USB device number 5 using ehci-pci

I have nothing plugged into USB on that machine.

That machine was running wheezy reliably for 5 years prior to the upgrade to
jessie.

Jeff (http://engineering.purdue.edu/~qobi)

audio input Etch SN25P

[Jeffrey Mark Siskind]

I'm running Etch i386 2.6.15-1-k7-smp on a Shuttle SN25P with an FX-60. As of
last night, I did an apt-get update and an apt-get upgrade so I am current.
The module snd_ice1724 and its dependents are loaded. alsamixer reports that
the hardware is an SN25P with a VIA Technologies VIA1617A.

Audio output works with everything that I've tried (plaympeg, bplay, skype).
But audio input doesn't work with anything that I've tried (brec, skype).
I've tried every possible setting with xmixer and alsamixer to no avail. I've
plugged a headset/microphone to the headphone and microphone jacks in the
front of the machine. The headphones work but not the microphone. I've also
plugged a microphone into the line-in jack at the rear of the machine. Again,
not audio input.

Can somebody help me? What are the correct xmixer or alsamixer setting to get
audio input to work? Is there an idiot-proof program that will set the mixer
correctly to enable audio input? How can I go about diagnosing and fixing this
problem?

Much appreciation in advance,
Jeff (http://www.ece.purdue.edu/~qobi)
---
[EMAIL PROTECTED]dpkg --get-selections '*'
abiword purge
abiword-common  install
abiword-gnome   install
acpiinstall
acpid   install
adduser install
akregator   install
alsa-base   install
alsa-ossinstall
alsa-utils  install
alsaplayer-alsa install
alsaplayer-common   install
alsaplayer-gtk  install
amorinstall
antiwordinstall
antlr   install
apelinstall
apt install
apt-fileinstall
apt-utils   install
aptitudeinstall
ark install
artsinstall
artsbuilder install
at  install
atlantikinstall
atlantikdesignerinstall
base-files  install
base-passwd install
bashinstall
bc  install
bin86   install
bind9-host  install
binutilsinstall
bison   install
blinken install
bplay   install
bsdmainutilsinstall
bsdutilsinstall
bsh install
bug-buddy   install
busybox install
bzip2   install
ca-certificates install
capplets-data   install
cddbinstall
cdrecordinstall
chicken-bin install
cl-asdf install
common-lisp-controller  install
console-common  install
console-datainstall
console-tools   install
coreutils   install
cpioinstall
cpp install
cpp-4.0 install
cpp-4.1 install
croninstall
cu  install
cupsys  install
cupsys-bsd  install
cupsys-client   install
cvs install
dbusinstall
dc  install
dcoprss install
debconf