Re: OpenSSH Oddity
Sorry disregard, I should learn to read the whole thread first when I am behind on my reading... :-) jrw Jeremy Warren [EMAIL PROTECTED] Sent by: Linux on 390 Port LINUX-390@VM.MARIST.EDU 09/26/2006 08:09 AM Please respond to Linux on 390 Port LINUX-390@VM.MARIST.EDU To LINUX-390@VM.MARIST.EDU cc Subject Re: [LINUX-390] OpenSSH Oddity Any chance the other system is using a different PAM stack/version?. I opened a bug with SuSE/Novell regarding PAM_SUCCEED_IF.SO causing the same disconnect message and a segfault). Post, Mark K [EMAIL PROTECTED] Sent by: Linux on 390 Port LINUX-390@VM.MARIST.EDU 09/23/2006 05:20 PM Please respond to Linux on 390 Port LINUX-390@VM.MARIST.EDU To LINUX-390@VM.MARIST.EDU cc Subject Re: [LINUX-390] OpenSSH Oddity One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Any chance the other system is using a different PAM stack/version?. I opened a bug with SuSE/Novell regarding PAM_SUCCEED_IF.SO causing the same disconnect message and a segfault). Post, Mark K [EMAIL PROTECTED] Sent by: Linux on 390 Port LINUX-390@VM.MARIST.EDU 09/23/2006 05:20 PM Please respond to Linux on 390 Port LINUX-390@VM.MARIST.EDU To LINUX-390@VM.MARIST.EDU cc Subject Re: [LINUX-390] OpenSSH Oddity One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Yes, that fixed it. Now if I could just fix my typing. Sheesh. Of course I couldn't just rebuild glibc. First I had to apply the fix for http://sources.redhat.com/ml/libc-hacker/2005-12/msg00015.html, and then resurrect the fix for http://www.cygwin.com/bugzilla/show_bug.cgi?id=398, and _then_ rebuild glibc. That last one was at least a little satisfying. Almost two years after the glibc maintainers said they wouldn't fix the bug, they did. I still don't know why the bug was being tripped on some systems and not others. At this point, I just wish I had some weekend left to enjoy after fixing the problem. Thanks to everyone that offered help. I appreciate it. Mark Post -Original Message- From: Post, Mark K Sent: Sunday, September 24, 2006 11:07 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity All, I think I may have found the answer here: http://sources.redhat.com/ml/libc-hacker/2005-02/msg5.html I try to rebuilt glibc and see if it helps. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
I'm using /etc/shadow to authenticate passwords, what else? Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of John Summerfield Sent: Sunday, September 24, 2006 7:19 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity -snip- If you're not using PAM, then what mechanism are you using to check passwords? -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
I'm using /etc/shadow to authenticate passwords, what else? There are systems, even with shadow passwords enabled, where 'pwconv' has yet to be run. Doesn't seem like THAT would trip path_open(), but who can tell? -- R; -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Mark FWIW, On my Intel based Linux server I had a very similar situation with two different resolutions 1. did end up being a bad symlink to a lib, in my case even the symlink did exist what was linked to did not so it caused this problem 2, This was really weird. and it's so windows like. i removed and reinstalled my SSH client being sure all sessions were deleted as well, rebuilt my session and it started working fine. so something is the client got screwed up One test might be to do a ssh localhost and see if you can ssh in locally with password. i.e. it may be client related and this might tell you. William 'Doug' Carroll Mainframe Systems Eng Sr I Global Technology Infrastructure ECS Mainframe Operating System Services Explore IT, build IT, exploit IT - Creating excellence through teamwork. Office: (614) 213-4954 Pager: [EMAIL PROTECTED] Cell: (614) 209-0649 Fax: (614) 244-9897 http://www.jpmchase.com Post, Mark K [EMAIL PROTECTED] m To Sent by: Linux on LINUX-390@VM.MARIST.EDU 390 Port cc [EMAIL PROTECTED] IST.EDU Subject Re: OpenSSH Oddity 09/25/2006 02:17 AM Please respond to Linux on 390 Port [EMAIL PROTECTED] IST.EDU I'm using /etc/shadow to authenticate passwords, what else? Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of John Summerfield Sent: Sunday, September 24, 2006 7:19 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity -snip- If you're not using PAM, then what mechanism are you using to check passwords? -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 - This transmission may contain information that is privileged, confidential, legally privileged, and/or exempt from disclosure under applicable law. If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution, or use of the information contained herein (including any reliance thereon) is STRICTLY PROHIBITED. Although this transmission and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened, it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by JPMorgan Chase Co., its subsidiaries and affiliates, as applicable, for any loss or damage arising in any way from its use. If you received this transmission in error, please immediately contact the sender and destroy the material in its entirety, whether in electronic or hard copy format. Thank you. -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Post, Mark K wrote: Ulrich, All, Using GDB, I figured out that the S0C4 is coming from the path_open routine of elf/dl-load.c in glibc. Is there any way to figure out which assembler instructions belong to wchich lines of C source code? I don't presume to be able to debug glibc, but at least I can try to get the glibc developers to look at the problem. I'd look at the output from the bt command to see what's calling glibc; it may be something's passing some garbage. I'm thinking back to QSAM on early OS/VS (and before), where reading/writing an unopened file could result in 0Cx abends arounf h'48' h'4e', near h'4800' and, on one occasiion, a loop when the instruction down there was a bcr 15,12 and my base register was 12. -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/ Please do not reply off-list -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Looking at arch/s390/kernel/traps.c, do_trap(), I bet interruption code 0x4 corresponds to EINTR 'interrupted system call'. An strace of the sshd process while you attempt to connect may help identify which syscall is failing, if any. I used: # ps aux | grep sshd root 1267 0.0 0.9 5224 2344 ?Ss Sep16 0:00 /usr/sbin/sshd # strace -p 1267 -o /tmp/strace.out Process 1267 attached - interrupt to quit ((connect with client)) ^c Process 1267 detached I wonder if anything is returning -1.. Also, does it make a difference if you run 'ldconfig' on the target system? I tend to blame glibc for syscall errors, but maybe it's not related. On Sat, 2006-09-23 at 17:46 -0400, Post, Mark K wrote: Nope, no differences in sshd_config, since that is delivered as part of the package, and Pat doesn't do PAM, so no PAM configuration files at all. Thanks for the suggestions to check, Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Marcy Cortes Sent: Saturday, September 23, 2006 5:35 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Is you sshd_config the same? How about the /etc/pam.d stuff? Marcy Cortes This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Post, Mark K Sent: Saturday, September 23, 2006 2:21 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] OpenSSH Oddity One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Brad, Thanks for the suggestion. It didn't provide any answers (yet), but it certainly raised a whole bunch more questions. Running ldconfig (for the umpteenth time) didn't help. Here's the tail end of the strace output. What really strikes me is that it shows process 3142 trying to open a lot of files and getting -1 ENOENT as the return code, when all those files and directories really do exist on my system (except for the /usr/lib/libnss_dns.so.2 one). Then it generates a SIGSEGV, not the EINTR you thought might be the case. (What mainframers call a S0C4 abend.) This whole problem is getting stranger and stranger. [pid 3142] ... select resumed ) = 1 (in [3]) [pid 3142] read(3, \372\276\211\263{\22\223\201\240\301\326c#\342\\277\313..., 8192) = 96 [pid 3142] write(2, debug1: userauth-request for use..., 93debug1: userauth-request for user markkp service ssh-connection method keyboard-interactive ) = 93 [pid 3142] write(2, debug1: attempt 1 failures 1\r\n, 30debug1: attempt 1 failures 1 ) = 30 [pid 3142] write(2, debug1: keyboard-interactive dev..., 36debug1: keyboard-interactive devs ) = 36 [pid 3142] write(2, debug1: auth2_challenge: user=ma..., 44debug1: auth2_challenge: user=markkp devs= ) = 44 [pid 3142] write(2, debug1: kbdint_alloc: devices \'\'..., 34debug1: kbdint_alloc: devices '' ) = 34 [pid 3142] write(2, Failed keyboard-interactive for ..., 70Failed keyboard-interactive for markkp from 127.0.0.1 port 1128 ssh2 ) = 70 [pid 3142] getpeername(3, {sa_family=AF_INET6, sin6_port=htons(1128), inet_pton(AF_INET6, :::127.0.0.1, sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0 [pid 3142] open(/etc/protocols, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] getsockopt(3, SOL_IP, IP_OPTIONS, , [0]) = 0 [pid 3142] socket(PF_FILE, SOCK_STREAM, 0) = 5 [pid 3142] fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR) [pid 3142] fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 [pid 3142] connect(5, {sa_family=AF_FILE, path=/var/run/nscd/socket}, 110) = -1 ENOENT (No such file or directory) [pid 3142] close(5)= 0 [pid 3142] socket(PF_FILE, SOCK_STREAM, 0) = 5 [pid 3142] fcntl64(5, F_GETFL) = 0x2 (flags O_RDWR) [pid 3142] fcntl64(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 [pid 3142] connect(5, {sa_family=AF_FILE, path=/var/run/nscd/socket}, 110) = -1 ENOENT (No such file or directory) [pid 3142] close(5)= 0 [pid 3142] gettimeofday({1159112979, 455645}, NULL) = 0 [pid 3142] getpid()= 3142 [pid 3142] open(/etc/resolv.conf, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] uname({sys=Linux, node=linwiki, ...}) = 0 [pid 3142] open(/etc/host.conf, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] open(/etc/hosts, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] open(/etc/ld.so.cache, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] open(/lib/libnss_dns.so.2, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] stat64(/lib, 0x7fffcff8) = -1 ENOENT (No such file or directory) [pid 3142] open(/usr/lib/libnss_dns.so.2, O_RDONLY) = -1 ENOENT (No such file or directory) [pid 3142] stat64(/usr/lib, 0x7fffcff8) = -1 ENOENT (No such file or directory) [pid 3142] --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 3142 detached ... read resumed \177\377\356X\177\377\355\350\177\377\356X, 4) = -512 --- SIGCHLD (Child exited) @ 0 (0) --- read(5, , 4) = 0 write(2, debug1: do_cleanup\r\n, 20debug1: do_cleanup ) = 20 _exit(255) = ? ls -ld /etc/resolv.conf /etc/host.conf /etc/hosts /etc/ld.so.cache /lib/libnss_dns.so.2 /lib /usr/lib/libnss_dns.so.2 /usr/lib -rw-r--r-- 1 root root27 1994-07-07 11:39 /etc/host.conf -rw-r--r-- 1 root root 664 2006-09-22 19:00 /etc/hosts -rw-r--r-- 1 root root 18503 2006-09-23 14:57 /etc/ld.so.cache -rw-r--r-- 1 root root71 2006-09-22 15:33 /etc/resolv.conf drwxr-xr-x 4 root root 4096 2006-09-22 18:05 /lib/ lrwxrwxrwx 1 root root19 2006-09-22 15:15 /lib/libnss_dns.so.2 - libnss_dns-2.3.4.so* drwxr-xr-x 18 root root 12288 2006-09-23 14:57 /usr/lib/ Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Brad Hinson Sent: Sunday, September 24, 2006 3:44 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Looking at arlibnss_dns.so.2ch/s390/kernel/traps.c, do_trap(), I bet interruption code 0x4 corresponds to EINTR 'interrupted system call'. An strace of the sshd process while you attempt to connect may help identify which syscall is failing, if any. I used: # ps aux | grep sshd root 1267 0.0 0.9 5224 2344 ?Ss Sep16 0:00 /usr/sbin/sshd # strace -p 1267 -o /tmp/strace.out Process 1267 attached - interrupt to quit ((connect with client)) ^c Process 1267 detached I wonder if anything is returning -1.. Also, does it make a difference if you run 'ldconfig' on the target system
Re: OpenSSH Oddity
Brad Hinson wrote: Looking at arch/s390/kernel/traps.c, do_trap(), I bet interruption code 0x4 corresponds to EINTR 'interrupted system call'. Actually, it doesn't ;-) This code is what is called program-interruption condition in the the ESA/390 Principles of Operation, and code 4 corresponds to a Protection Exception. This can be caused by a variety of conditions, but in Linux user space it always means your program tried to write to an read-only memory mapping. In this particular case we have a User PSW of 070dc000 c0006318, that is we wereexecuting the instruction at 0x40006318 (which tends to be within the main executable .text section); this instructions happened to be User Code: 50 00 70 00, i.e. st %r0, 0(%r7). Register %r7 contained 0x40016f3c, and this is confirmed by failing address: 40016000 (note that the hardware reports the failing address only by page). Now it's hard to say where exactly the address 0x40016f3c points to, but it is low enough to have some likelyhood of also residing with the main executable's .text (or .rodata) section, which is generally mapped read-only. So the question is why the program tries to write to a read-only location. I'd recommend to use GDB to track this down further, either on the live process or else on a core dump. Mit freundlichen Gruessen / Best Regards Ulrich Weigand -- Dr. Ulrich Weigand GNU compiler/toolchain for Linux on System z and Cell BE IBM Deutschland Entwicklung GmbH, Schoenaicher Str. 220, 71032 Boeblingen Phone: +49-7031/16-3727 --- Email: [EMAIL PROTECTED] -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Ulrich, Thanks for your insight. I've been trying to use GDB on this, and it hasn't been working for me. What ever I do just seems to change the abend, or move where it occurs. Any specific suggestions on how to go about it would be very helpful. My level of expertise with GDB is very low. :( Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Ulrich Weigand Sent: Sunday, September 24, 2006 5:56 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity -snip- Now it's hard to say where exactly the address 0x40016f3c points to, but it is low enough to have some likelyhood of also residing with the main executable's .text (or .rodata) section, which is generally mapped read-only. So the question is why the program tries to write to a read-only location. I'd recommend to use GDB to track this down further, either on the live process or else on a core dump. Mit freundlichen Gruessen / Best Regards Ulrich Weigand -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
-rw-r--r-- 1 root root27 1994-07-07 11:39 /etc/host.conf -rw-r--r-- 1 root root 664 2006-09-22 19:00 /etc/hosts -rw-r--r-- 1 root root 18503 2006-09-23 14:57 /etc/ld.so.cache -rw-r--r-- 1 root root71 2006-09-22 15:33 /etc/resolv.conf drwxr-xr-x 4 root root 4096 2006-09-22 18:05 /lib/ lrwxrwxrwx 1 root root19 2006-09-22 15:15 /lib/libnss_dns.so.2 - libnss_dns-2.3.4.so* drwxr-xr-x 18 root root 12288 2006-09-23 14:57 /usr/lib/ Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Brad Hinson Sent: Sunday, September 24, 2006 3:44 AM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Looking at arlibnss_dns.so.2ch/s390/kernel/traps.c, do_trap(), I bet interruption code 0x4 corresponds to EINTR 'interrupted system call'. An strace of the sshd process while you attempt to connect may help identify which syscall is failing, if any. I used: # ps aux | grep sshd root 1267 0.0 0.9 5224 2344 ?Ss Sep16 0:00 /usr/sbin/sshd # strace -p 1267 -o /tmp/strace.out Process 1267 attached - interrupt to quit ((connect with client)) ^c Process 1267 detached I wonder if anything is returning -1.. Also, does it make a difference if you run 'ldconfig' on the target system? I tend to blame glibc for syscall errors, but maybe it's not related. On Sat, 2006-09-23 at 17:46 -0400, Post, Mark K wrote: Nope, no differences in sshd_config, since that is delivered as part of the package, and Pat doesn't do PAM, so no PAM configuration files at all. Thanks for the suggestions to check, Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Marcy Cortes Sent: Saturday, September 23, 2006 5:35 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Is you sshd_config the same? How about the /etc/pam.d stuff? Marcy Cortes -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Post, Mark K Sent: Saturday, September 23, 2006 2:21 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] OpenSSH Oddity One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/ Please do not reply off-list
Re: OpenSSH Oddity
Post, Mark K wrote: Ulrich, Thanks for your insight. I've been trying to use GDB on this, and it hasn't been working for me. What ever I do just seems to change the abend, or move where it occurs. Any specific suggestions on how to go about it would be very helpful. My level of expertise with GDB is very low. :( One of the drawbacks to debuggers is that they change the environment, sometimes in ways that matter as to whether the program under test fails. I commonly resort to adding debugging code (which can have the same problem). Last week I had glibc spewing about double-linked-list corruption, and google suggested valgrind as a useful tool in looking for the problem. You can also test with electricfence (shocking idea!); it involves linking with another library, but no code changes. Broadly, the problems that give rise to an 0c4 are those that can cause glibc to complain (and abort), buffer overruns, and if it manages to overwrite code, practically anything including incorrect output. -- Cheers John -- spambait [EMAIL PROTECTED] [EMAIL PROTECTED] Tourist pics http://portgeographe.environmentaldisasters.cds.merseine.nu/ Please do not reply off-list -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Ulrich, All, Using GDB, I figured out that the S0C4 is coming from the path_open routine of elf/dl-load.c in glibc. Is there any way to figure out which assembler instructions belong to wchich lines of C source code? I don't presume to be able to debug glibc, but at least I can try to get the glibc developers to look at the problem. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Ulrich Weigand Sent: Sunday, September 24, 2006 5:56 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Brad Hinson wrote: Looking at arch/s390/kernel/traps.c, do_trap(), I bet interruption code 0x4 corresponds to EINTR 'interrupted system call'. Actually, it doesn't ;-) This code is what is called program-interruption condition in the the ESA/390 Principles of Operation, and code 4 corresponds to a Protection Exception. This can be caused by a variety of conditions, but in Linux user space it always means your program tried to write to an read-only memory mapping. In this particular case we have a User PSW of 070dc000 c0006318, that is we wereexecuting the instruction at 0x40006318 (which tends to be within the main executable .text section); this instructions happened to be User Code: 50 00 70 00, i.e. st %r0, 0(%r7). Register %r7 contained 0x40016f3c, and this is confirmed by failing address: 40016000 (note that the hardware reports the failing address only by page). Now it's hard to say where exactly the address 0x40016f3c points to, but it is low enough to have some likelyhood of also residing with the main executable's .text (or .rodata) section, which is generally mapped read-only. So the question is why the program tries to write to a read-only location. I'd recommend to use GDB to track this down further, either on the live process or else on a core dump. Mit freundlichen Gruessen / Best Regards Ulrich Weigand -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
All, I think I may have found the answer here: http://sources.redhat.com/ml/libc-hacker/2005-02/msg5.html I try to rebuilt glibc and see if it helps. Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Maybe some difference in the sshd/pam configurations? --- LINUX-390@VM.MARIST.EDU [EMAIL PROTECTED] wrote: One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 __ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Is you sshd_config the same? How about the /etc/pam.d stuff? Marcy Cortes This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Post, Mark K Sent: Saturday, September 23, 2006 2:21 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] OpenSSH Oddity One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390 -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390
Re: OpenSSH Oddity
Nope, no differences in sshd_config, since that is delivered as part of the package, and Pat doesn't do PAM, so no PAM configuration files at all. Thanks for the suggestions to check, Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Marcy Cortes Sent: Saturday, September 23, 2006 5:35 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Is you sshd_config the same? How about the /etc/pam.d stuff? Marcy Cortes This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Post, Mark K Sent: Saturday, September 23, 2006 2:21 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: [LINUX-390] OpenSSH Oddity One other item of interest. If I try to connect using keys, and not a password, things work just fine. But, as I said, when I am not using keys, I don't even get prompted for a password, so it's something in the processing that's going wrong before it gets to asking for a password. Mark Post -Original Message- From: Post, Mark K Sent: Saturday, September 23, 2006 5:10 PM To: 'Linux on 390 Port' Subject: RE: OpenSSH Oddity From my note: I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. So, not that I can tell. Mark Post -Original Message- From: Linux on 390 Port [mailto:[EMAIL PROTECTED] On Behalf Of Rick Troth Sent: Saturday, September 23, 2006 3:38 PM To: LINUX-390@VM.MARIST.EDU Subject: Re: OpenSSH Oddity Target system has different run-time libs? -- R; On Sat, 23 Sep 2006, Post, Mark K wrote: I'm having a very strange problem show up with OpenSSH 4.3p1. On the development system where I built it, it works fine. When I ship the binary package to another Linux guest on the same z/VM system, it doesn't work. When I try to ssh into the system, the client gets a Connection closed by 192.168.0.20 message, without even being prompted for a password. The sshd daemon on the other system throws off this error in the kernel ring buffer (but keeps on running): User process fault: interruption code 0x4 failing address: 40016000 CPU:0Not tainted Process sshd (pid: 13181, task: 0152c000, ksp: 0152dd00) User PSW : 070dc000 c0006318 User GPRS: 40017738 00010dd0 7fffcfa8 40016f3c 40016f3c 7fffcf68 40017000 c0006164 c00064b2 7fffcf48 User ACRS: 40010870 User Code: 50 00 70 00 a7 f4 ff ed 58 80 b0 d4 58 90 d0 40 58 20 80 00 I've checked the md5 checksums for all the files in the openssh and openssl packages, as well as all the shared libraries the sshd binary uses. They all match on both systems. Even if I build the binary on the target system, I get the same results. I'm at a loss to explain why it works fine on one system, but not others. Anyone have any ideas where to look further? Thanks, Mark Post -- For LINUX-390 subscribe / signoff / archive access instructions, send email to [EMAIL PROTECTED] with the message: INFO LINUX-390 or visit http://www.marist.edu/htbin/wlvindex?LINUX-390