hi I managed to debug:
#apt-get source rdesktop #cd rdesktop-1.8.3 #debuild -b -uc -us then simply run the ./rdesktop binary in that directory. in order to debug I did #./configure --prefix=/home/global/erik/localinst/rdesktop --with-debug --with-debug-kbd --with-debug-rdp5 --with-debug-clipboard --with-debug-sound --with-debug-channel --with-debug-seamless --with-debug-smartcard --with-debug-credssp # make clean # make then run ./rdesktop in kdbg to find the segmentation fault at this line in ssl.c: algor = X509_PUBKEY_get0_param(NULL, NULL, 0, &algor, key); according to the documentation that is wrong because the return value is 0 or 1 depending of if the function worked or made error. the return value is NOT a pointer! the compiler also issues a warning here: ssl.c: In function ‘rdssl_cert_to_rkey’: ssl.c:154:8: warning: assignment makes pointer from integer without a cast [-Wint-conversion] algor = X509_PUBKEY_get0_param(NULL, NULL, 0, &algor, key); ^ also see that algor is passed as a pointer in order for the function to manipulate it. it is completely wrong to assign algor from the return value! so i replaced the code by this: DEBUG_RDP5(("Now running patched code\n")); if ( ! X509_PUBKEY_get0_param(NULL, NULL, 0, &algor, key)) { error("X509_PUBKEY_get0_param failed\n"); return NULL; } (yes I missed freeing up resources before returning) now I ran rdesktop and it does not crash but also does not work: # ./rdesktop xpcrash > foo 2>&1 # cat foo (snipped info marked as SNIPPED) Autoselected keyboard map de Failed to negotiate protocol, retrying with plain RDP. ERROR: Failed to extract public key from certificate ERROR: send: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt RDP depth: 24, display depth: 24, display bpp: 32, X server BE: 0, host BE: 0 Adding translation, keysym=0xffe2, scancode=0x36, modifiers=0x0 Adding translation, keysym=0xffe1, scancode=0x2a, modifiers=0x0 Adding translation, keysym=0xffea, scancode=0xb8, modifiers=0x0 Adding translation, keysym=0xff7e, SNIPPED LOT OF SCANCODE STUFF scancode=0x11, modifiers=0x2 Adding translation, keysym=0x65, scancode=0x12, modifiers=0x0 Adding translation, keysym=0x45, scancode=0x12, modifiers=0x2 Adding sequence for keysym (0xe8, egrave) -> 0xfe50, 0x65, Adding sequence for keysym (0xc8, Egrave) -> 0xfe50, 0x45, Adding sequence for keysym (0xe9, eacute) -> 0xfe51, 0x65, Adding sequence for keysym (0xc9, Eacute) -> 0xfe51, 0x45, Adding sequence for keysym (0xea, ecircumflex) -> 0xfe52, 0x65, Adding sequence for keysym (0xca, Ecircumflex) -> 0xfe52, 0x45, Adding sequence for keysym (0xeb, ediaeresis) -> 0xfe57, 0x65, SNIPPED MANY ADDING SEQUENCE AND ADDING TRANSLATION STUFF Adding translation, keysym=0xb7, scancode=0x34, modifiers=0x4 Adding translation, keysym=0xf7, scancode=0x34, modifiers=0x6 Adding translation, keysym=0x2d, scancode=0x35, modifiers=0x0 Adding translation, keysym=0x5f, scancode=0x35, modifiers=0x2 Adding translation, keysym=0xfe60, scancode=0x35, modifiers=0x4 Adding translation, keysym=0xfe56, scancode=0x35, modifiers=0x6 server bpp 24 client bpp 32 depth 24 g_num_channels is 4 Requesting channel cliprdr Requesting channel rdpsnd Requesting channel snddbg Requesting channel rdpdr Server RDP version is 4 We're going for the RDP5-style encryption Ignored certs left: 6 Ignored Certificate length is 1046 cert #6 (ignored): Certificate: Data: Version: 3 (0x2) Serial Number: c1:00:SNIPPED CENSORED SERIAL NUMBER Algorithm: md5WithRSAEncryption Issuer: OU=Copyright (c) 1997 Microsoft Corp., OU=Microsoft Corporation, CN=Microsoft Root Authority Validity Not Before: Jan 10 07:00:00 1997 GMT Not After : Dec 31 07:00:00 2020 GMT Subject: OU=Copyright (c) 1997 Microsoft Corp., OU=Microsoft Corporation, CN=Microsoft Root Authority Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:a9:02:bd:c1:70:e6:3b:f2:4e:1b:28:9f:97:78: SNIPPED MODULUS DETAIL ca:bc:f0:08:a3:22:30:b3:06:85:c9:b3:20:77:13: 85:df Exponent: 65537 (0x10001) X509v3 extensions: 2.5.29.1: 0....[.p.ir.#Q~..M....r0p1+0)..U..."Copyright (c) 1997 Microsoft Corp.1.0...U....Microsoft Corporation1!0...U....Microsoft Root Authority......<<...>.c..@ Signature Algorithm: md5WithRSAEncryption 95:e8:0b:c0:8d:f3:97:18:35:ed:b8:01:24:d8:77:11:f3:5c: SNIPPED HEX DATA a2:8c:d3:d5:54:3f:46:cd:1c:55:a6:70:db:12:3a:87:93:75: 9f:a7:d2:a0 Ignored certs left: 5 Ignored Certificate length is 1269 cert #5 (ignored): Certificate: Data: Version: 3 (0x2) Serial Number: eb:aa:SNIPPED SERIAL NUMBER Signature Algorithm: md5WithRSAEncryption Issuer: OU=Copyright (c) 1997 Microsoft Corp., OU=Microsoft Corporation, CN=Microsoft Root Authority Validity Not Before: Feb 25 08:00:00 2002 GMT Not After : Feb 26 08:00:00 2010 GMT Subject: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, OU=Copyright (c) 1999 Microsoft Corp., CN=Microsoft Enforced Licensing Intermediate PCA Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:b4:00:54:6f:9b:51:26:76:b7:25:13:dd:4e:33: SNIPPED MODULUS bd:1c:85:0e:c1:0f:0c:62:da:c7:3e:de:d6:d5:62: a4:e7 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Extended Key Usage: Code Signing, 1.3.6.1.4.1.311.10.6.1, 1.3.6.1.4.1.311.10.6.2 2.5.29.1: 0....[.p.ir.#Q~..M....r0p1+0)..U..."Copyright (c) 1997 Microsoft Corp.1.0...U....Microsoft Corporation1!0...U....Microsoft Root Authority......<<...>.c..@ 1.3.6.1.4.1.311.21.1: ... X509v3 Subject Key Identifier: A7:F6:94:65:92:SNIPPED DATA 1.3.6.1.4.1.311.20.2: . .S.u.b.C.A X509v3 Key Usage: Digital Signature, Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE Signature Algorithm: md5WithRSAEncryption 57:50:d5:08:d3:5f:fd:d2:8e:23:1d:34:de:bd:0b:b2:71:3c: SNIPPED HEX STUFF 11:62:f5:c0:2d:11:47:23:f0:cd:8c:d0:95:3b:3c:02:94:45: 02:b4:83:95 Ignored certs left: 4 Ignored Certificate length is 1422 cert #4 (ignored): Certificate: Data: Version: 3 (0x2) Serial Number: 61:06:d:SNIPPED Signature Algorithm: md5WithRSAEncryption Issuer: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, OU=Copyright (c) 1999 Microsoft Corp., CN=Microsoft Enforced Licensing Intermediate PCA Validity Not Before: May 9 20:17:34 2003 GMT Not After : Feb 26 08:00:00 2010 GMT Subject: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, OU=Copyright (c) 2003 Microsoft Corp., CN=Microsoft Enforced Licensing Registration Authority CA Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:ea:58:89:4d:af:9d:5b:50:f7:6e:de:a4:7f:a9: SNIPPED 9d:57:eb:2b:1d:07:21:d0:65:d9:03:3c:a9:72:6d: a5:ad Exponent: 65537 (0x10001) X509v3 extensions: 1.3.6.1.4.1.311.21.1: ... X509v3 Subject Key Identifier: D5:90:6D:CD:F5:83:SNIPPED 1.3.6.1.4.1.311.20.2: . .S.u.b.C.A X509v3 Key Usage: Digital Signature, Non Repudiation, Certificate Sign, CRL Sign X509v3 Basic Constraints: critical CA:TRUE X509v3 Authority Key Identifier: keyid:A7:F6:94:65:SNIPPED DirName:/OU=Copyright (c) 1997 Microsoft Corp./OU=Microsoft Corporation/CN=Microsoft Root Authority serial:EB:AA:11:SNIPPED X509v3 CRL Distribution Points: Full Name: URI:http://crl.microsoft.com/pki/crl/products/MSEnforcedLicPCA.crl X509v3 Extended Key Usage: Code Signing, 1.3.6.1.4.1.311.10.6.2 Signature Algorithm: md5WithRSAEncryption 26:af:46:5b:51:b1:bb:96:f9:4c:9d:4c:34:2e:4c:e4:29:98: SNIPPED 8c:99:6f:aa Ignored certs left: 3 Ignored Certificate length is 1476 cert #3 (ignored): Certificate: Data: Version: 3 (0x2) Serial Number: 61:07:68:SNIPPED Signature Algorithm: md5WithRSAEncryption Issuer: C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, OU=Copyright (c) 2003 Microsoft Corp., CN=Microsoft Enforced Licensing Registration Authority CA Validity Not Before: Jul 6 21:19:50 2005 GMT Not After : Feb 26 08:00:00 2010 GMT Subject: CN=Microsoft License Server Registration Authority PA Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:ee:bb:eb:b6:60:c2:2f:60:ec:09:89:7d:fa:b2: SNIPPED 16:7d:bb:91:33:a5:c5:b0:1e:3d:55:a3:e5:34:36: da:a9 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: critical CA:TRUE, pathlen:0 X509v3 Subject Key Identifier: DE:F2:79:SNIPPED X509v3 Key Usage: Digital Signature, Non Repudiation, Certificate Sign, CRL Sign 1.3.6.1.4.1.311.21.1: ... X509v3 Authority Key Identifier: keyid:D5:90:6D:CD:F5:83:18:2B:F1:7C:92:F7:62:E5:0E:44:4B:6E:3A:E1 DirName:/C=US/ST=Washington/L=Redmond/O=Microsoft Corporation/OU=Copyright (c) 1999 Microsoft Corp./CN=Microsoft Enforced Licensing Intermediate PCA serial:61:06:SNIPPED X509v3 CRL Distribution Points: Full Name: URI:http://crl.microsoft.com/pki/crl/products/MSEnfLicRegAuthCA.crl Full Name: URI:http://www.microsoft.com/pki/crl/products/MSEnfLicRegAuthCA.crl Authority Information Access: CA Issuers - URI:http://www.microsoft.com/pki/certs/MSEnfLicRegAuthCA.crt Signature Algorithm: md5WithRSAEncryption 8e:d8:de:bb:7a:41:0f:7e:f1:d6:52:2f:ba:cf:ff:28:d9:6e: SNIPPED 53:e4:11:44 CA Certificate length is 1291 Certificate length is 878 Now running patched code <<<<<<<<<< HERE OUR PATCHED CODE!! Re-setting algorithm type to RSA in server certificate Didn't parse X509 correctly <<<< ??? Failed to parse crypt info <<<< ??? Sending CJRQ for channel #1008 Sending CJRQ for channel #1003 Sending CJRQ for channel #1004 Sending CJRQ for channel #1005 Sending CJRQ for channel #1006 Sending CJRQ for channel #1007 Sending RDP5-style Logon packet Called sec_init with packetlen 340 Sending encrypted packet: 0000 00 00 00 00 33 01 00 00 00 00 08 00 00 00 00 00 ....3........... SNIPPED POTENTIALLY PRIVATE STUFF 0120 05 00 02 00 00 00 00 00 00 00 c4 ff ff ff 00 00 ................ 0130 00 00 27 00 00 00 00 00 ..'..... and now rdesktop terminated without displaying anything. maybe this is of help to anyone. cya erik