Re: gpgsm: ksba_cms_parse failed: Broken pipe
Hi, On 2023-12-14 19:43, Werner Koch wrote: On Thu, 14 Dec 2023 16:19, Jakob Bohm said: zcat ${infl} | faketime "${DSTAMP}" gpgsm --verify --validation-model shell --assume-binary --status-fd 3 --output - - 3>${wrkdir}/sigdec.status || : gpgsm: ksba_cms_parse failed: Broken pipe gpgsm expects more data but zcat already sent its EOF. This might be due to a broken ASN.1 structure which announced N bytes but gpgsm was not able to read thes N bytes. CMS chunks up the data so that you can have a very long stream of data. (I have sometimes seen chunks of just one byte). That is the error message you meant, or did you see more? Adding --verbose is often very helpful. Message on status-fd (copied to stderr by cat command) [GNUPG:] ERROR verify.leave 32877 This is the machine interface; which you rightfully parse. $ gpg-error 32877 32877 = (0, 32877) = (GPG_ERR_SOURCE_UNKNOWN, GPG_ERR_EPIPE) = (Unspecified source, Broken pipe) verify.leave helps us to locate the place where the error as emitted: Here is is at the end of gpgsm_verify function but unfortunately we can't get the offset of the input data where it stopped parsing. You may use "--debug hashing" to write a file with the data hashed (i.e. what is verified). That gives an indication where inthe stream the error occured. Anothyer useful tool is the venerable dumpasn1 tool which you could use instead of gpgsm to find error in the input data structure. Which of the inputs was being truncated on the way to the that parsing function? The decompressed $infl . Thanks for this decoding, based on this I did further testing and found the following: 1. The input file was DER encoded, not BER, so no BER chunking, just a single 1.3 GiB OCTET STRING as inner data. (Checked with dumpasn1 and a lot of patience), and was actually OK. 2. The same error messages are produced when the /output/ hits a broken pipe such as piping into head(1) or having a script typo (my case). So based on 2, it would have been much more helpful if the error message had stated if the "parser" hit an I/O error on output or input, such as "output: -: Broken pipe" while the status-fd stream could have said [GNUPG:] ERROR verify.leave 32877 output - E_PIPE Preferably also change GPG_ERR_SOURCE_UNKNOWN to some other constant indicating output failure. It would be much easier if the gpgsm code could be used standalone like the old gpg1 could, with no entanglement to pinentry, dirmgr Yeah But for security reasons we have separated tasks in a way similar to what postfix does. status-fd protocol etc. etc. scripted signing and verification of computer generated data is not suited to a human-centric user Right, it is a machine interface. For gpg we have the extra tool gpgv to make verification against a known set of keys easy. And recently we added --assert-signer which basically does the same. There has never been a demand to have a dedicated verification tool for CMS but I can imagine to add a --assert-signer also to gpgsm. My point was that status-fd isn't an easy interface to script against, but your bug tracker already contains a rejected task to simplify things. Your use of faketime is interesting ;-) I use faketime to reflect the age of the data being processed, as the certificates are rotated as necessary and the overall scripts contain a table mapping origin dates to certificates and cert validity ranges. Similar tricks may be needed to verify/decrypt stored e-mail based on the delivery times found in the newest (topmost) "Received:" header. P.S. Your mail server ellsberg.gnupg.com appears to be a bit aggressive in its greylisting (mails rejected first 3 to 4 times), causing NDR reports to be mailed back to me after 30 minutes. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: gpgsm: ksba_cms_parse failed: Broken pipe
Hi! On Thu, 14 Dec 2023 16:19, Jakob Bohm said: > zcat ${infl} | > faketime "${DSTAMP}" gpgsm --verify --validation-model shell > --assume-binary --status-fd 3 --output - - 3>${wrkdir}/sigdec.status > || : > gpgsm: ksba_cms_parse failed: Broken pipe gpgsm expects more data but zcat already sent its EOF. This might be due to a broken ASN.1 structure which announced N bytes but gpgsm was not able to read thes N bytes. CMS chunks up the data so that you can have a very long stream of data. (I have sometimes seen chunks of just one byte). That is the error message you meant, or did you see more? Adding --verbose is often very helpful. > Message on status-fd (copied to stderr by cat command) > > [GNUPG:] ERROR verify.leave 32877 This is the machine interface; which you rightfully parse. $ gpg-error 32877 32877 = (0, 32877) = (GPG_ERR_SOURCE_UNKNOWN, GPG_ERR_EPIPE) = (Unspecified source, Broken pipe) verify.leave helps us to locate the place where the error as emitted: Here is is at the end of gpgsm_verify function but unfortunately we can't get the offset of the input data where it stopped parsing. You may use "--debug hashing" to write a file with the data hashed (i.e. what is verified). That gives an indication where inthe stream the error occured. Anothyer useful tool is the venerable dumpasn1 tool which you could use instead of gpgsm to find error in the input data structure. > Which of the inputs was being truncated on the way to the that > parsing function? The decompressed $infl . > It would be much easier if the gpgsm code could be used standalone > like the old gpg1 could, with no entanglement to pinentry, dirmgr Yeah But for security reasons we have separated tasks in a way similar to what postfix does. > status-fd protocol etc. etc. scripted signing and verification of > computer generated data is not suited to a human-centric user Right, it is a machine interface. For gpg we have the extra tool gpgv to make verification against a known set of keys easy. And recently we added --assert-signer which basically does the same. There has never been a demand to have a dedicated verification tool for CMS but I can imagine to add a --assert-signer also to gpgsm. Your use of faketime is interesting ;-) Salam-Shalom, Werner -- The pioneers of a warless world are the youth that refuse military service. - A. Einstein openpgp-digital-signature.asc Description: PGP signature ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
gpgsm: ksba_cms_parse failed: Broken pipe
Dear list, I am using gpgsm in a script for its ability to efficiently process CMS format messages larger than available memory. However after a recent script change, it now fails every time on previously accepted data with error messages that are essentialy gpg implementation internals gobbledygook. Versions involved (distribution supplied builds, so not latest upstram source): $ uname -a Linux 5.10.0-0.deb10.21-amd64 #1 SMP Debian 5.10.162-1~deb10u1 (2023-02-14) x86_64 GNU/Linux $ gpgsm --version gpgsm (GnuPG) 2.2.27 libgcrypt 1.8.4 libksba 1.3.5-unknown Copyright (C) 2021 Free Software Foundation, Inc. License GNU GPL-3.0-or-later <https://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Home: /home/me/.gnupg Supported algorithms: Cipher: 3DES, AES128, AES192, AES256, SERPENT128, SERPENT192, SERPENT256, SEED, CAMELLIA128, CAMELLIA192, CAMELLIA256 Pubkey: RSA, ECC Hash: MD5, SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224, WHIRLPOOL Command lines (from a bash script, sorry for long lines): $ # Note: $infl is archived gzipped DER-encoded CMS SignedData of privacy-sensitive data # $DSTAMP is the date/time when the signature is supposed to be valid # $fprint is the fingerprint of the only allowed signer certificate # $wrkdir is a unique temporary directory. # faketime, gpgsm, grep, zcat, cat and false are the usual GNU or distribution tools # shell is GNU bash # non-debug environment is a cron job zcat ${infl} | faketime "${DSTAMP}" gpgsm --verify --validation-model shell --assume-binary --status-fd 3 --output - - 3>${wrkdir}/sigdec.status || : ( ( grep "^\[GNUPG:\] \(GOOD\|EXPKEY\)SIG ${fprint}" ${wrkdir}/sigdec.status && grep "^\[GNUPG:\] VALIDSIG ${fprint}" ${wrkdir}/sigdec.status ) >/dev/null || ( cat ${wrkdir}/sigdec.status >&2 ; false ) ) Message on stderr: gpgsm: ksba_cms_parse failed: Broken pipe Message on status-fd (copied to stderr by cat command) [GNUPG:] ERROR verify.leave 32877 So what do these error messages mean, and why aren't they in they described in the documentation (info and man)? Which of the inputs was being truncated on the way to the that parsing function? P.S. It would be much easier if the gpgsm code could be used standalone like the old gpg1 could, with no entanglement to pinentry, dirmgr status-fd protocol etc. etc. scripted signing and verification of computer generated data is not suited to a human-centric user interface. Enjoy Jakob -- Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10 This public discussion message is non-binding and may contain errors. WiseMo - Remote Service Management for PCs, Phones and Embedded ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users