Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Here are some news about the analysis of this boot problem. First, starting and stopping test partitions with cryptdisks_start and cryptdisks_stop does work properly after the computer has boot, but does not work at boot time. Here is the example of the command (with decrypt_keyctl modified to display the keys at the end, as suggested in an earlier post). root@marislae:/lib/systemd/system# cryptdisks_start sda3_crypt [] Starting crypto disk...[info] sda3_crypt (starting)... Caching passphrase for /dev/disk/by-uuid/4f04656d-00fc-4fea-8c8a-44a7bdc7b6e5: ** BEGIN DECRYPT_KEYCTL DEBUG OUTPUT id: cryptkey-pw1 | key id: 657815207 key content: testpw key list: 1 key in keyring: 657815207: --alswrv 0 0 user: cryptkey-pw1 END DECRYPT_KEYCTL DEBUG OUTPUT [ ok crypt (started)...done. root@marislae:/lib/systemd/system# cryptdisks_start sda5_crypt [ ok ] Starting crypto disk...sda5_crypt (running)...done. root@marislae:/lib/systemd/system# As you can see, running it on an already started system is fine. On the other hand, the same setting does not work at boot time. Back to the original problem, without the test partitions and using only my regular two work partitions sda5 and sdb1 holding LVM volumes. The modification of the decrypt_keyctl to display the key on boot showed that after the key for the first partition was entered, it was stored in the user keyring. I got something like: id: cryptkey-sda5b1 | key id: 1007139041 key list: 1 key in keyring: 1007139041: --alswrv 0 0 user: cryptkey-sda5b1 Then the system asked for the second key, ignoring the one it already known, and the following is displayed: id: cryptkey-sda5b1 | key id: 422064942 key list: 1 key in keyring: 422064942: --alswrv 0 0 user: cryptkey-sda5b1 So the id was exactly the same, but the key id was a new one. It seems the keyring is cleared sometime between the two runs of the script and in each case only one key was stored in the keyring. I think the culprit is there: why did key 1007139041 disappear from keyring between the two commands? Another important information is that this system uses systemd, and it seems decrypting is handled differently with systemd. If I run journalctl to get information, I see this message: oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Password file path 'sda5b1' is not absolute. Ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Encountered unknown /etc/crypttab option 'keyscript=decrypt_keyctl', ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Volume sda5_crypt already active. oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Password file path 'sda5b1' is not absolute. Ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Encountered unknown /etc/crypttab option 'keyscript=decrypt_keyctl', ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Volume sdb1_crypt already active. However, this message seems to be generated after the partitions have already been activated, and from the message displayed on the console, I know decrypt_keyctl did run, despite these messages. Yet another information is I noticed earlier in journalctl the message oct. 12 12:08:05 marislae systemd[1]: /usr appears to be on its own filesytem and is not already mounted. This is not a supported setup. This problem is explained here: http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/. I don't know if it is related to my problem, I will try to fix this by moving /usr content into the / partition and see if it helps. best regards, Luc -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Hey Luc, Am 12.10.2014 um 12:57 schrieb Luc Maisonobe: Here are some news about the analysis of this boot problem. First, great that you finally managed to use the debugging version that I sent earlier. First, starting and stopping test partitions with cryptdisks_start and cryptdisks_stop does work properly after the computer has boot, but does not work at boot time. Here is the example of the command (with decrypt_keyctl modified to display the keys at the end, as suggested in an earlier post). root@marislae:/lib/systemd/system# cryptdisks_start sda3_crypt [] Starting crypto disk...[info] sda3_crypt (starting)... Caching passphrase for /dev/disk/by-uuid/4f04656d-00fc-4fea-8c8a-44a7bdc7b6e5: ** BEGIN DECRYPT_KEYCTL DEBUG OUTPUT id: cryptkey-pw1 | key id: 657815207 key content: testpw key list: 1 key in keyring: 657815207: --alswrv 0 0 user: cryptkey-pw1 END DECRYPT_KEYCTL DEBUG OUTPUT [ ok crypt (started)...done. root@marislae:/lib/systemd/system# cryptdisks_start sda5_crypt [ ok ] Starting crypto disk...sda5_crypt (running)...done. root@marislae:/lib/systemd/system# As you can see, running it on an already started system is fine. Looks good so far. On the other hand, the same setting does not work at boot time. Back to the original problem, without the test partitions and using only my regular two work partitions sda5 and sdb1 holding LVM volumes. The modification of the decrypt_keyctl to display the key on boot showed that after the key for the first partition was entered, it was stored in the user keyring. I got something like: id: cryptkey-sda5b1 | key id: 1007139041 key list: 1 key in keyring: 1007139041: --alswrv 0 0 user: cryptkey-sda5b1 Then the system asked for the second key, ignoring the one it already known, and the following is displayed: id: cryptkey-sda5b1 | key id: 422064942 key list: 1 key in keyring: 422064942: --alswrv 0 0 user: cryptkey-sda5b1 So the id was exactly the same, but the key id was a new one. It seems the keyring is cleared sometime between the two runs of the script and in each case only one key was stored in the keyring. I think the culprit is there: why did key 1007139041 disappear from keyring between the two commands? Is this output from the boot process or did you try to unlock the encrypted LVM volumes after boot process finished? Another important information is that this system uses systemd, and it seems decrypting is handled differently with systemd. If I run journalctl to get information, I see this message: This information changes everything! I never tried systemd myself so far and I don't know nothing about how the systemd boot process handles dm-crypt devices/cryptdisks/crypttab at all. oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Password file path 'sda5b1' is not absolute. Ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Encountered unknown /etc/crypttab option 'keyscript=decrypt_keyctl', ignoring. This looks suspicious to me. Maybe systemd has its own crypttab implementation which doesn't support all the crypttab options/functions from Debian sysvinit init and initramfs scripts? At least some 'systemd-cryptsetup' process seems to process crypttab entries and detects an unsupported option 'keyscript'. oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Volume sda5_crypt already active. At the same time, the encrypted device is detected as 'already active'. Doesn't make sense to me. But nevertheless, I can imagine that systemd-cryptsetup detects unknown crypttab options and falls back to the default password prompt afterwards. That would explain why you're asked for the password of every encrypted device. oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Password file path 'sda5b1' is not absolute. Ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Encountered unknown /etc/crypttab option 'keyscript=decrypt_keyctl', ignoring. oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Volume sdb1_crypt already active. However, this message seems to be generated after the partitions have already been activated, and from the message displayed on the console, I know decrypt_keyctl did run, despite these messages. Ok, another option would be that the initscript is executed first, and afterwards the systemd-cryptsetup helper binary iterates over crypttab entries again and detects that the devices are already unlocked. Yet another information is I noticed earlier in journalctl the message oct. 12 12:08:05 marislae systemd[1]: /usr appears to be on its own filesytem and is not already mounted. This is not a supported setup. This problem is explained here: http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/. I don't know if it is related to my problem, I will try to fix this by moving /usr content into the / partition and see if it helps. Yeah, I remember I've read somewhere that systemd
Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Le 12/10/2014 14:20, Jonas Meurer a écrit : Hey Luc, Am 12.10.2014 um 12:57 schrieb Luc Maisonobe: Back to the original problem, without the test partitions and using only my regular two work partitions sda5 and sdb1 holding LVM volumes. The modification of the decrypt_keyctl to display the key on boot showed that after the key for the first partition was entered, it was stored in the user keyring. I got something like: id: cryptkey-sda5b1 | key id: 1007139041 key list: 1 key in keyring: 1007139041: --alswrv 0 0 user: cryptkey-sda5b1 Then the system asked for the second key, ignoring the one it already known, and the following is displayed: id: cryptkey-sda5b1 | key id: 422064942 key list: 1 key in keyring: 422064942: --alswrv 0 0 user: cryptkey-sda5b1 So the id was exactly the same, but the key id was a new one. It seems the keyring is cleared sometime between the two runs of the script and in each case only one key was stored in the keyring. I think the culprit is there: why did key 1007139041 disappear from keyring between the two commands? Is this output from the boot process or did you try to unlock the encrypted LVM volumes after boot process finished? It was the output of the boot process. So I don't even know at which state systemd executes the cryptdisks initscript, and how it interacts with the kernel keyrings. I'm eager to say that your problems are related to systemd init system. Honestly, I don't know when I will find time to give systemd a try and further debug this issue :-/ If you can have a running system without systemd, you are lucky. I was really forced to migrate by some packages updates and so far am really not happy with it. It changed a lot of things and it seems not all packages are ready for it. I'll try to debug further and understand what happens to the keyring. best regards, Luc Cheers, jonas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Hi Jonas, Le 09/10/2014 10:39, Jonas Meurer a écrit : Hey Luc, Am 09.10.2014 um 09:54 schrieb luc: Am 03.10.2014 um 21:55 schrieb Jonas Meurer: Did you find time to do the additional testing/debugging yet? I'd like to fix this bug in time for Debian Jessie, provided that it's really a bug in the package and not an issue on your side ;) Not yet, but I intend to do so. The problem occurs on a computer with some critical information on it, and the partitions already use the full disk space. This implies I have to do some careful work of shrinking filesystems, logical volumes and so on before I can set up a test partition aside. I see. But you don't need to resize your filesystems or go through similar hassle. Simply use file containers for testing. The following commands should setup a testing environment (use carefully, even though I tested them): # dd if=/dev/urandom of=/cont1 bs=1M count=3 # dd if=/dev/urandom of=/cont2 bs=1M count=3 # echo testpw | cryptsetup luksFormat /cont1 # echo testpw | cryptsetup luksFormat /cont2 # echo cont1_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl \ /etc/crypttab # echo cont2_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl \ /etc/crypttab # cryptdisks_start cont1_crypt # cryptdisks_start cont2_crypt On the first invocation (for cont1_crypt), I got this dialog: root@marislae:~# cryptdisks_start cont1_crypt [[] Starting crypto disk...[info] cont1_crypt (starting)... Caching passphrase for /cont1: ** keyctl_set_timeout: Permission denied Error setting timeout on key (2524288), removing Caching passphrase for /cont1: ** keyctl_set_timeout: Permission denied Error setting timeout on key (612589418), removing [Here I pressed ctrl-C to stop the attempts] Caching passphrase for /cont1: Erreur de lecture de la phrase secrète. I was running the commands from root. I initially logged in to the computer from SSH to a regular user, than did sudo -i to get root access if this matters. As I suspected this may be a problem, I allowed root direct SSH access and tried again, login directly to root account, this time it worked: root@marislae:~# cryptdisks_start cont1_crypt [] Starting crypto disk...[info] cont1_crypt (starting)... Caching passphrase for /cont1: ** [ ok _crypt (started)...done. root@marislae:~# cryptdisks_start cont2_crypt [] Starting crypto disk...[info] cont2_crypt (starting)... Using cached passphrase for /cont2. [ ok _crypt (started)...done. root@marislae:~# The /dev/mapper/cont1_crypt and /dev/mapper/cont2_crypt did appear correctly. Is there a way I could check the keyring just after boot, before it is cleared? I could probably set an independent init script to run after disks are mounted to dump the list of the keys in the keyring to some file in /tmp so I can retrieve them once the system is up and debug. I thing I could do this using some keyctl command, but don't know which one to use for a given entry in /etc/crypttab. Should it be simply keyctl list pw1 in the case of your example or something else? I saw in the decrypt_keyctl script some cryptkey-$1 identifier (probably used with an _ appended). How could I use this? best regards Luc As already mentioned - up to now I'm unable to reproduce the bug. For me, decrypt_keyctl works in all tested setups. The provided passphrase is stored in kernel keyring with first invokation and used for all the following device unlockings that have the same keyscript/keyname configured. I understand your point. It is difficult to debug here (as mentioned earlier putting some echo commands completely trashed the boot sequence). I'll do my best. I'm sorry that I brought you in troubles here. The echo command was untested and I at least should have written that. It was meant for debugging purposes only but it completely broke the keyscript. I'll try to not make such premature requests again :-/ Cheers, jonas I test the decrypt_keyctl script with the following setup, and it works as expected. Maybe you could try a similar setup: - create two small lvm logical volumes (5MB are more than enough) - luksformat both logical volumes - add them to your crypttab: clv1_crypt /dev/VG/LV1 testkey1 luks,keyscript=decrypt_keyctl clv2_crypt /dev/VG/LV2 testkey1 luks,keyscript=decrypt_keyctl - try unlocking them via cryptdisks_start: # cryptdisks_start clv1_crypt # cryptdisks_start clv2_crypt The second unlocking should use the key cached during first unlocking. It would be awesome if you could test this. I as well tested this setup during boot process, and it works as expected as well. Also tested with UUID instead of source device path in crypttab, same result. I've no glue what's different on your setups, and any help with debugging would be highly appreciated. In case that you still encounter the bug, please paste your full /etc/fstab and /etc/crypttab again. /etc/crypttab: sdb1_crypt
Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Hey Luc, Am 10.10.2014 um 10:29 schrieb Luc Maisonobe: Le 09/10/2014 10:39, Jonas Meurer a écrit : I see. But you don't need to resize your filesystems or go through similar hassle. Simply use file containers for testing. The following commands should setup a testing environment (use carefully, even though I tested them): # dd if=/dev/urandom of=/cont1 bs=1M count=3 # dd if=/dev/urandom of=/cont2 bs=1M count=3 # echo testpw | cryptsetup luksFormat /cont1 # echo testpw | cryptsetup luksFormat /cont2 # echo cont1_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl \ /etc/crypttab # echo cont2_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl \ /etc/crypttab # cryptdisks_start cont1_crypt # cryptdisks_start cont2_crypt On the first invocation (for cont1_crypt), I got this dialog: root@marislae:~# cryptdisks_start cont1_crypt [[] Starting crypto disk...[info] cont1_crypt (starting)... Caching passphrase for /cont1: ** keyctl_set_timeout: Permission denied Error setting timeout on key (2524288), removing Caching passphrase for /cont1: ** keyctl_set_timeout: Permission denied Error setting timeout on key (612589418), removing [Here I pressed ctrl-C to stop the attempts] Caching passphrase for /cont1: Erreur de lecture de la phrase secrète. I was running the commands from root. I initially logged in to the computer from SSH to a regular user, than did sudo -i to get root access if this matters. As I suspected this may be a problem, I allowed root direct SSH access and tried again, login directly to root account, this time it worked: Interesting, never saw these kind of problems before. I'm testing on a Laptop with Ubuntu 14.04 installed and use 'sudo -i' a lot. Indeed I used it for testing the commands above as well. For me it worked. But let's keep that aside. It's another issue and out of scope for this bugreport ;) root@marislae:~# cryptdisks_start cont1_crypt [] Starting crypto disk...[info] cont1_crypt (starting)... Caching passphrase for /cont1: ** [ ok _crypt (started)...done. root@marislae:~# cryptdisks_start cont2_crypt [] Starting crypto disk...[info] cont2_crypt (starting)... Using cached passphrase for /cont2. [ ok _crypt (started)...done. root@marislae:~# The /dev/mapper/cont1_crypt and /dev/mapper/cont2_crypt did appear correctly. Looks like it worked for you as well this time. Do you still discover the bug during the boot process with your older decrypt_keyring setup? Did you try the test setup from above during boot process? It would be helpful do know whether the keyscript works on boot process with the test setup (it does work for me), and what's the exact difference between your older decrypt_keyctl devices and the new test ones. Is there a way I could check the keyring just after boot, before it is cleared? I could probably set an independent init script to run after disks are mounted to dump the list of the keys in the keyring to some file in /tmp so I can retrieve them once the system is up and debug. I thing I could do this using some keyctl command, but don't know which one to use for a given entry in /etc/crypttab. Should it be simply keyctl list pw1 in the case of your example or something else? I saw in the decrypt_keyctl script some cryptkey-$1 identifier (probably used with an _ appended). How could I use this? The script uses the user keyring '@u' for storing the cryptdisks key. You can list the keyring with 'keyring list @u'. In order to get the particular key from the keyring, do 'keyring search @u user IDENTIFIER', where IDENTIFIER' is cryptkey-pw1 in the example. Then you get the key id, which is needed to get the key itself using 'keyctl print KEY-ID' or 'keyctl pipe KEY-ID'. If you want to output all values during keyscript invokation for debugging purposes, the easiest way is to modify the script at /lib/cryptsetup/decrypt_keyctl. Add the following just after the last line to the script: echo -e BEGIN DECRYPT_KEYCTL DEBUG OUTPUT\nid: $ID_ | key id: $KID_\nkey content: $(keyctl pipe $KID_)\nkey list:\n$(keyctl list @u)\nEND DECRYPT_KEYCTL DEBUG OUTPUT\n 2 Be warned, this line will print your key in plaintext to the console. If you want to redirect the output to a logfile, simply replace '2' by e.g. '/root/decrypt_keyctl.log'. In order to not break your boot process again, after modifying first test the decrypt_keyctl keyscript with the test setup from above by using cryptdisks_start/_stop and only do a reboot after you verified that the keyscript still works. Kind regards, jonas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Hi encountered the exact same problem on a laptop with two different encrypted partition (one for a LVM on a SSD and one for a LVM on a HDD). Setting echo CRYPTTAB_TRIED: $CRYPTTAB_TRIED' at line 36 of the script does NOT work, and in fact it completely prevent the computer to boot, as it gets stuck in an infinite loop complaining about wrong passwphrase. The echo is not displayed at all during this loop. I don't know were the echo goes and I don't know what passphrase is read (I wonder the echo may garble the input). I had to reboot the computer on a live system containing cryptsetup to manually reset everything back to work (running all the commands manually: cryptsetup luksOpen for the two encrypted partitions, vgchange -a y to find the LVM components, mounting everything, mounting /dev, /sys and /proc with --bind, chroot, edit the script to remove the echo, update-initramfs -u -k all to make sure the updated script is in the initramfs, and reboot). So, back to square one, how can we identify the reason why the caching script does not work anymore? best regards, Luc -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Bug#758788: [pkg-cryptsetup-devel] Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
Hello Marek, thanks for your bugreport. Am 21.08.2014 um 13:22 schrieb Marek Królikowski: Dear Maintainer, after upgrade to 2:1.6.6-1 I have to enter passphrase for each of crypttab entries. Desired behavior is for cryptsetup to try cached passphrase for crypptab entries with the same identifier (pw1 in this case). Working configurations: - cryptsetup 2:1.6.4-4 - cryptsetup 2:1.6.6-1 with decrypt_keyctl copied from 2:1.6.4-4 The decrypt_keyctl keyscript was changed with the last upload of cryptsetup in order to fix a bug when a wrong passphrase was entered. See the bugreport for more details: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748368 Do you unlock your devices in initramfs or with the initscript? I guess that something is wrong with the environment variable CRYPTTAB_TRIED. This variable should contain the count of tries to unlock one particular dm-crypt device. Can you add some debugging code to your script in order to print the value of CRYPTTAB_TRIED? A simple 'echo CRYPTTAB_TRIED: $CRYPTTAB_TRIED' at line 36 of the script should already do the job. Please report back your observations. Also, can you please report back what happens if you enter a wrong password (both with the new version of decrypt_keyctl and with the old one). Kind regards, jonas -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
