Re: En-/Decryption speed for large files (GnuPG and Gpg4win)
Can someone please remove my email address from this group! This has nothing to do with me! Sent from Yahoo Mail for iPhone On Tuesday, January 17, 2023, 5:10 AM, Andre Heinecke via Gnupg-users wrote: Hi, On Sunday 15 January 2023 10:52:23 CET Christoph Klassen wrote: > When I was testing the decryption I also tried "gpg --decrypt > test_file.gpg" (without output file) with the 10 GB file and it took 8 > minutes and 47 seconds. I was wondering why it took longer when GnuPG > didn't need to create an output file. Yes that is expected. Gpg encrypt and decrypt with AES should be mostly IO Bound as with AES-NI instructions it is really fast in the CPU. So not writing the output to disk will result in faster operations. And one of the biggest differences you get is when you encrypt / decrypt on a faster disk. Another big difference what you will see in the perfomance of GnuPG is if you use -z 0 which disables compression. Currently GnuPG on the command line disables compression when the input file name already looks compressed depending on the file name. We want to improve that, especially since Kleopatra hands the filename only in a way that is not used in that compression calculation. E.g. Adding Media data formats there might already help in a lot of use cases. For uncompressable output, like random data, this will make the largest difference. You can put "compress-level 0" into your gpg.conf to cause Kleopatra to also use that. That issue is: https://dev.gnupg.org/T6332 If you could do a run of your tests and comment in that issue with the results that would be helpful. It does not surprise me that Kleopatra is much slower. Due to our Architecture Kleopatra passes Data, through GPGME directly to GnuPG. This results in additional overhead but gives us more flexibility what kind of data we encrypt / decrypt. E.g. a mail or something that is not even written on the File system. For some parts we want to change that. Most notably Ingo is currently working on Gpgtar. Gpgtar can nowadays directly encrypt / decrypt so there is no need to pipe the input / output of GnuPG to or from GpgTar. Using GpgTar directly should help a lot when working with larger Archives. https://dev.gnupg.org/ T5478 We also already increased the buffer size in GPGME to reduce the number of callbacks we do internally but there can be more optimization there. Currently our recommendation for Large Data is to use the command line directly, which will always be fastest as there is no overhead. > Did someone of you also try to en-/decrypt larger files? Maybe even > files that are larger than 1 TB? It would be really nice to know how > long GnuPG and Gpg4win are busy with such large files. I think my largest tests were around 40GB. But I don't have the numbers anymore, the testing I did there was mostly because there were reports that Kleopatra crashes on such large files. Maybe you can open a ticket for this with a reference to https:// dev.gnupg.org/T5478 about performance problems when decrypting / encrypting large files (In contrast to archives.) Best Regards, Andre P.S. we are currently also looking at the startup / initial keycache building time of Kleopatra. This might also be intresting for those looking at Kleo performance. https://dev.gnupg.org/T6259 -- GnuPG.com - a brand of g10 Code, the GnuPG experts. g10 Code GmbH, Erkrath/Germany, AG Wuppertal HRB14459 GF Werner Koch, USt-Id DE215605608, www.g10code.com. GnuPG e.V., Rochusstr. 44, D-40479 Düsseldorf. VR 11482 Düsseldorf Vorstand: W.Koch, B.Reiter, A.Heinecke Mail: bo...@gnupg.org Finanzamt D-Altstadt, St-Nr: 103/5923/1779. Tel: +49-211-28010702___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: En-/Decryption speed for large files (GnuPG and Gpg4win)
Thanks a lot for your reply Andre! On 17.01.23 13:08, Andre Heinecke wrote: Another big difference what you will see in the perfomance of GnuPG is if you use -z 0 which disables compression. I tried that with the 10GB file and, indeed, it was much faster. The encryption took only 51 seconds (with compression it was: 6 min. 21 sec.). -- Christoph Klassen | https://intevation.de Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer: Frank Koormann, Bernhard Reiter OpenPGP_signature Description: OpenPGP digital signature ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
switching off compression (was: En-/Decryption speed for large files (GnuPG and Gpg4win))
Am Dienstag 17 Januar 2023 13:08:18 schrieb Andre Heinecke via Gnupg-users: > Another big difference what you will see in the perfomance of GnuPG is if > you use -z 0 which disables compression. According to the GnuPG documentation (2.4.0) https://gnupg.org/documentation/manuals/gnupg/GPG-Configuration-Options.html#index-compress_002dlevel '-z 0' is equivalent to the following long options '--compress-level 0 --bzip2-compress-level 0' yes, both have to be given. > You can put "compress-level 0" into > your gpg.conf to cause Kleopatra to also use that. Would not be enough to disable bip2 encryption (according to the documentation). Looking at https://gnupg.org/documentation/manuals/gnupg/GPG-Esoteric-Options.html#index-compress_002dalgo what shall work with just one option is `compress-algo uncompressed` Best, Bernhard -- https://intevation.de/~bernhard +49 541 33 508 3-3 Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998 Geschäftsführer Frank Koormann, Bernhard Reiter signature.asc Description: This is a digitally signed message part. ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: En-/Decryption speed for large files (GnuPG and Gpg4win)
Hi, On Sunday 15 January 2023 10:52:23 CET Christoph Klassen wrote: > When I was testing the decryption I also tried "gpg --decrypt > test_file.gpg" (without output file) with the 10 GB file and it took 8 > minutes and 47 seconds. I was wondering why it took longer when GnuPG > didn't need to create an output file. Yes that is expected. Gpg encrypt and decrypt with AES should be mostly IO Bound as with AES-NI instructions it is really fast in the CPU. So not writing the output to disk will result in faster operations. And one of the biggest differences you get is when you encrypt / decrypt on a faster disk. Another big difference what you will see in the perfomance of GnuPG is if you use -z 0 which disables compression. Currently GnuPG on the command line disables compression when the input file name already looks compressed depending on the file name. We want to improve that, especially since Kleopatra hands the filename only in a way that is not used in that compression calculation. E.g. Adding Media data formats there might already help in a lot of use cases. For uncompressable output, like random data, this will make the largest difference. You can put "compress-level 0" into your gpg.conf to cause Kleopatra to also use that. That issue is: https://dev.gnupg.org/T6332 If you could do a run of your tests and comment in that issue with the results that would be helpful. It does not surprise me that Kleopatra is much slower. Due to our Architecture Kleopatra passes Data, through GPGME directly to GnuPG. This results in additional overhead but gives us more flexibility what kind of data we encrypt / decrypt. E.g. a mail or something that is not even written on the File system. For some parts we want to change that. Most notably Ingo is currently working on Gpgtar. Gpgtar can nowadays directly encrypt / decrypt so there is no need to pipe the input / output of GnuPG to or from GpgTar. Using GpgTar directly should help a lot when working with larger Archives. https://dev.gnupg.org/ T5478 We also already increased the buffer size in GPGME to reduce the number of callbacks we do internally but there can be more optimization there. Currently our recommendation for Large Data is to use the command line directly, which will always be fastest as there is no overhead. > Did someone of you also try to en-/decrypt larger files? Maybe even > files that are larger than 1 TB? It would be really nice to know how > long GnuPG and Gpg4win are busy with such large files. I think my largest tests were around 40GB. But I don't have the numbers anymore, the testing I did there was mostly because there were reports that Kleopatra crashes on such large files. Maybe you can open a ticket for this with a reference to https:// dev.gnupg.org/T5478 about performance problems when decrypting / encrypting large files (In contrast to archives.) Best Regards, Andre P.S. we are currently also looking at the startup / initial keycache building time of Kleopatra. This might also be intresting for those looking at Kleo performance. https://dev.gnupg.org/T6259 -- GnuPG.com - a brand of g10 Code, the GnuPG experts. g10 Code GmbH, Erkrath/Germany, AG Wuppertal HRB14459 GF Werner Koch, USt-Id DE215605608, www.g10code.com. GnuPG e.V., Rochusstr. 44, D-40479 Düsseldorf. VR 11482 Düsseldorf Vorstand: W.Koch, B.Reiter, A.HeineckeMail: bo...@gnupg.org Finanzamt D-Altstadt, St-Nr: 103/5923/1779. Tel: +49-211-28010702 signature.asc Description: This is a digitally signed message part. ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: How to make remote gpg talk to specific Unix socket (for forwarded local gpg-agent)
On Mon, 16 Jan 2023 07:56, Troy said: > The problem is that there's already a gpg-agent running at the remote > (I think started by systemd or the X server), which I don't want to > disturb. For my ssh connection, I try to create a new Unix domain Don't run the gpg-agent for your account. I put no-autostart into ~/.gnupg/common.conf to avoid that any tools on the remote start the gpg-agent. Of course you need to disable the systemd stuff to autostart gpg-agent - using systemd for autostart is deprecated because it creates races. iirc, the wiki says that you should put StreamLocalBindUnlink yes into the sshd_config. I prefer to manually delete the socket using ssh remote "rm $(gpgconf -L agent-socket)" if the connection does not work. I have this in ~/.ssh/config Host remote RemoteForward /run/user/1042/gnupg/S.gpg-agent /run/user/1000/gnupg/S.gpg-agent.extra (1042 is my uid on remote). Then you just need to "ssh remote" > - gpg (GnuPG) 2.2.27 Well, the single common option no-autostart is only available in stable (since 2.3.8) Salam-Shalom, Werner -- The pioneers of a warless world are the youth that refuse military service. - A. Einstein openpgp-digital-signature.asc Description: PGP signature ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: En-/Decryption speed for large files (GnuPG and Gpg4win)
On Mon, 16 Jan 2023 16:47, Christoph Klassen said: > For some reason in that test gpg didn't output anything or at least > the PowerShell didn't show anything. Powershell and stdout and stderr are a bit problematic. I can't remember the details so I usually stick to cmd.exe or run tools directly via ssh from a Unix shell. Shalom-Salam, Werner -- The pioneers of a warless world are the youth that refuse military service. - A. Einstein openpgp-digital-signature.asc Description: PGP signature ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: Multithreading with GPGME Python Bindings
On Mon, 16 Jan 2023 07:55, Jan Girlich said: > Is the call to "gpgme_check_version" maybe done implicitly by the > Python bindings? Yes. See gpgme/lang/python/src/core.py # check_version also makes sure that several subsystems are properly # initialized, and it must be run at least once before invoking any # other function. We do it here so that the user does not have to do # it unless she really wants to check for a certain version. check_version() > Do I have to call "gpgme-config --version" before to safely do No, this is a helper to compile against gpgme - it returns suggested compiler and linker flags. Make sure that one context object is only used by one thread at a time. Salam-Shalom, Werner -- The pioneers of a warless world are the youth that refuse military service. - A. Einstein openpgp-digital-signature.asc Description: PGP signature ___ Gnupg-users mailing list Gnupg-users@gnupg.org https://lists.gnupg.org/mailman/listinfo/gnupg-users
