Re: Bug#735927: general: X *always* crashes when ram is full
previously on this list Roger Leigh contributed: With an SSD, you really don't want /tmp or swap on it; Why?, due to limited write cycles? As long as it is a modern SSD (years) or one of the old ones one with a sandforce controller (OpenBSD dev let me know about that) then it has a good 20% extra space above it's listed gigabytes reserved unusable for wear levelling meaning this is a non issue even when full? Unless he's worried about not being able to wipe the swap? -- ___ 'Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface' (Doug McIlroy) In Other Words - Don't design like polkit or systemd ___ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/801542.40858...@smtp127.mail.ir2.yahoo.com
Re: Bug#735927: general: X *always* crashes when ram is full
On Mon, Jan 20, 2014 at 12:29:28PM +, Kevin Chadwick wrote: previously on this list Roger Leigh contributed: With an SSD, you really don't want /tmp or swap on it; Why?, due to limited write cycles? That's one reason, but the one I was thinking of was the shocking performance. I accidentally disabled tmpfs on /tmp on my main system with an SSD rootfs a few months back and was wondering why I was experiencing such slow builds, desktop freezing for seconds at a time, and other intermittent stalls. Turns out it was thrashing the SSD since /tmp was on the rootfs. Enabling tmpfs on /tmp resolved the problems, changing the performance from dire to excellent. (I have the swap on a RAID1 LVM LV on fast HDDs). This is a system with 8 cores @4GHz, 16GiB RAM, over 16GiB swap, so should be pretty performant, yet /tmp on an SSD made it crawl and freeze continually. This is purely down to the slow write speed of the SSD I have (Intel 320 128GB). If you have a faster SSD, maybe it won't be an issue, but given the amount of junk being written to /tmp when running a desktop environment and applications, from dozens of tmp files and sockets to streaming video, it's likely it will make a significant difference to make /tmp a tmpfs or HDD filesystem. Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linuxhttp://people.debian.org/~rleigh/ `. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools `-GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140120143024.gc6...@codelibre.net
Re: Bug#735927: general: X *always* crashes when ram is full
On Mon, 20 Jan 2014 14:30:24 + Roger Leigh wrote: This is a system with 8 cores @4GHz, 16GiB RAM, over 16GiB swap, so should be pretty performant, yet /tmp on an SSD made it crawl and freeze continually. Interesting, have a look if it states the write access time spec in the datasheet (if available) of that SSD? Though when I've looked for write access time on off the shelf SSD drives it usually only mentions throughput or reading. Do you know if it slows the build if you use it for the source code? If your not sure it's a sandforce or has similar wear levelling then obviously don't try it in case you wear it out, though as it's your root I don't suppose you will try it. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/829972.77857...@smtp115.mail.ir2.yahoo.com
Re: Bug#735927: general: X *always* crashes when ram is full
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 01/20/2014 09:56 AM, Kevin Chadwick wrote: On Mon, 20 Jan 2014 14:30:24 + Roger Leigh wrote: This is a system with 8 cores @4GHz, 16GiB RAM, over 16GiB swap, so should be pretty performant, yet /tmp on an SSD made it crawl and freeze continually. Interesting, have a look if it states the write access time spec in the datasheet (if available) of that SSD? Though when I've looked for write access time on off the shelf SSD drives it usually only mentions throughput or reading. He said this was the Intel 320 Series 128GB drive. As far as I know there is no officially 128GB model of that drive, but the 120GB model (that being the advertised capacity after overprovisioning et cetera) was reviewed in depth by The Tech Report back in 2011: http://techreport.com/review/20653/intel-320-series-solid-state-drive The article includes spec-sheet information and detailed benchmark results. Glancing over them again, I'm not surprised by this described result from that drive; there are other SSDs that might do much better. - -- The Wanderer Secrecy is the beginning of tyranny. A government exists to serve its citizens, not to control them. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJS3UTqAAoJEASpNY00KDJrCMUP/1lefX7QpQgPQ8nvK233DPNv etYT9YEMJGfw1Z+HVPro/aFBw1zsk3VXW4Is+xP08W5GPZRpwhpVKPPW/DFOHrJX N37FK//iiqrHLvl2kodBCkgeMbLvNKH8M+727YiqmOqXFQdjOdsCS9wY90nHT7Ia f7D+i0nrxTeSE3kExWeMwIR9hMAVJLeCGftsbpARPDhVcSH1EiDKcUWf1VvfGeoW ca5j4/o1b5m6v/a1oFglMOk5M/OFxIRxQWL3ckFyxfkWgm+pkEQFXkdy2JfAv/Hc YuAK1vXR+d2Q5RdPy/5bj+llpbHy2AA/Tll7aZ1TwNNcf4XQNjxzu10Zp07RYOvY 5pzNwcLApDvKs84OY5ICfigqdoXnzo3WObgkNbcAlvssqzvw2iDhQU9AONY+xooS 0eFD0yXALWlTPLYp1afoeYXP0ruEVu9djtrLlfaJoBJ5QYIB+JReflnxm8D5BL6z 1+uKFr14WQ1+4nWX3vcFsOIwVySeqr9CUX9wpz3TRuJOfzDQwju0srFxcOEhIjxf nVhzhB2qOhFUj+x1IzxCQzYy0jxCqpOzibyMLirLTC9G5vd7bwhV2VlrBA1BpVk7 +C2oZxrjGb8iqxnSPnPbxZTsOSHpZiwd2qVmcdJ9DX/1hp2KmpwZBtavWy8+ntbZ oVDkwQLrNnirDnuB9UJ3 =r/3C -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/52dd44ea.1080...@fastmail.fm
SSDs have extra unused space??? (was: Re: Bug#735927: general: X *always* crashes when ram is full
Hi Kevin, On Montag, 20. Januar 2014, Kevin Chadwick wrote: As long as it is a modern SSD (years) or one of the old ones one with a sandforce controller (OpenBSD dev let me know about that) then it has a good 20% extra space above it's listed gigabytes reserved unusable for wear levelling meaning this is a non issue even when full? wait, what? Do you have any vendor statements to support this 20% extra space? cheers, Holger signature.asc Description: This is a digitally signed message part.
Re: Bug#735927: general: X *always* crashes when ram is full
On Mon, Jan 20, 2014 at 10:46:50AM -0500, The Wanderer wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 01/20/2014 09:56 AM, Kevin Chadwick wrote: On Mon, 20 Jan 2014 14:30:24 + Roger Leigh wrote: This is a system with 8 cores @4GHz, 16GiB RAM, over 16GiB swap, so should be pretty performant, yet /tmp on an SSD made it crawl and freeze continually. Interesting, have a look if it states the write access time spec in the datasheet (if available) of that SSD? Though when I've looked for write access time on off the shelf SSD drives it usually only mentions throughput or reading. He said this was the Intel 320 Series 128GB drive. As far as I know there is no officially 128GB model of that drive, but the 120GB model (that being the advertised capacity after overprovisioning et cetera) was reviewed in depth by The Tech Report back in 2011: http://techreport.com/review/20653/intel-320-series-solid-state-drive The article includes spec-sheet information and detailed benchmark results. Glancing over them again, I'm not surprised by this described result from that drive; there are other SSDs that might do much better. You are correct--it is indeed 120GB, my mistake. I'm not sure if it's just the write speed. It's also reading stuff back from /tmp and the rest of the rootfs so it may just be a pathological workload. That said, I would have classed it as very light--/var, /home and the build chroots etc. are all on a separate RAID/LVM array, so /tmp was the sole part of the rootfs being actively written. It would be interesting to have further data from users who could try running with /tmp on the rootfs SSD and on a tmpfs (with or without the swap on the SSD). Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linuxhttp://people.debian.org/~rleigh/ `. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools `-GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140120194359.gd6...@codelibre.net
Re: Bug#735927: general: X *always* crashes when ram is full
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 01/20/2014 02:43 PM, Roger Leigh wrote: It would be interesting to have further data from users who could try running with /tmp on the rootfs SSD and on a tmpfs (with or without the swap on the SSD). My swap, /tmp, /boot, /usr, and / are all LVM logical volumes, on a volume group backed by two identical 256GB SSDs in RAID-1 via mdraid, and I haven't noticed any issues. All other filesystems (/var, /opt, /home) are on a separate volume group backed by RAID-5 mechanical storage. However, not only does the RAID-1 question introduce another variable and likely improve performance, these drives were specifically chosen because they were top performers in their capacity class at the time - and with SSDs, higher capacity generally performs better anyway. So this may not be a particularly valuable datapoint. - -- The Wanderer Secrecy is the beginning of tyranny. A government exists to serve its citizens, not to control them. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBCAAGBQJS3ayXAAoJEASpNY00KDJrqBcQAJjEE8TA6cMnzANgXcFCQkXp F3V7L0C9XtJkJCnGQUoIgT3DkC0PtnaDqLXatnhz8nFlT1WWDxLvR2WDBU+Rl7V4 moiHT2l+zksZ1nWE2wfW+V9nLBw0vcQif3vSCvZEPtlYnY4lbPWJdBMgvBZHxeBt wFtW8xNwVoSJvxdGlknH0tnJ8CinTps4S+WU7sHslTP56Z6IUZI2/o4qnidlZNGj QE+2CjmuBdKggnFRXJViv8R6NUYdboY3RRkkFfPXk6V6U7ymI4JUCk6juAg9MyJ+ ob2+zWgeHavUA3ixeKA2hhA1HWKAMCoAGo1jdJELLLpNAx8gyW6Vg7gTutzPJXKj 61FqULfj5LOE3rmL49FyPJtQgai7Ug67RF86z+0oDZodLb/375N4daRQ5+BDmGnC sISc5z0ennqb/i8uXQU8Ywmt382Y8/IZ2RubbZVjZ2YLcXST+oIoSs1WxMIBoMRW b9VPthC5yTgn4vclwsjRxWjUl1ajB3kPJodakZAsVc8tjYVOqfQipMQk25Oekmjs hvJyUyUPxTPHXHzmNL386S+mB0eotQM/+U+OwIk54Wmuksu5OqnLL80UXkOaQwZZ kteRrweWBdfjtUjY8S+p9DBzgyBWqD3KEWkYZi48TNcehcV8lncAdaB6pgiuJDVd pmqWoF090M6xtCbxYKtb =tpWt -END PGP SIGNATURE- -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/52ddac97.2050...@fastmail.fm
Re: SSDs have extra unused space??? (was: Re: Bug#735927: general: X *always* crashes when ram is full
On Tue, Jan 21, 2014 at 1:22 AM, Holger Levsen wrote: wait, what? Do you have any vendor statements to support this 20% extra space? Flash is basically probabilistic storage and you need extra space to ensure that the probability your data is stored remains high, and a (possibly trustworthy) controller with algorithms in front to help with that. Check out Bunnie's talk at 30C3 about SD cards where he mentioned a 16GB flash device with the controller only saying it was 128MB in size (IIRC). I assume similar things apply to SSDs/NAND etc. http://www.bunniestudios.com/blog/?p=3554 https://events.ccc.de/congress/2013/Fahrplan/events/5294.html https://media.ccc.de/browse/congress/2013/30C3_-_5294_-_en_-_saal_1_-_201312291400_-_the_exploration_and_exploitation_of_an_sd_memory_card_-_bunnie_-_xobs.html -- bye, pabs http://wiki.debian.org/PaulWise -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAKTje6EBS5bWzB3te1iLHXB33y=5gcjaq6y1ckf5o9gkjta...@mail.gmail.com
Bug#735927: general: X *always* crashes when ram is full
@Holger: I'm sorry, but I'm closing this bug as google-chrome is supported by Google, not by Debian. You _*REALLY*_ should read the whole ticket first. And other tickets in the future too. Please reopen this ticket. @Martijn: Or (recommended) just add a small swap partition (say 128MB) to allow the kernel to put unused memory pages on disk so the X server can use them instead. I'm sorry, i can not, i only have SSD in that machine. If i enable swap in a machine with only 1GB ram it would be in use all day long, every time. I will try your other recommendations through, please be patient and wait for my feedback. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140119163651.6bb3afa5@moli-desktop
Bug#735927: general: X *always* crashes when ram is full
You could change the OOM priority of the X server, this would cause some other process to be killed instead. Nice idea, i searched for this oom thing (btw it means: out of memory) and found OOM Killer. Some pages for future reference: http://linux-mm.org/OOM_Killer http://lwn.net/Articles/317814/ http://www.oracle.com/technetwork/articles/servers-storage-dev/oom-killer-1911807.html Looks like nice priority is important when something has to be killed to free memory (unfortunatelly it's not that important as it does not take the nice LEVEL into the calculation, if it's -1 or -19, it's the same...), so i ran chrome by the following command: nice -19 /opt/google/chrome/google-chrome --password-store=gnome --disk-cache-size=1 --disk-cache-dir=/dev/null Then i ran the following command in a shell under root: while true ; do for i in $(pidof chrome) ; do pid=$(($i+0)) if [[ $pid -ne 0 ]] ; then echo -999 /proc/$i/oom_score_adj ; fi ; done ; sleep 1 ; done and opened some youtube videos, and the X crashed again. (Please check my commands, did i do everything right?) And no messages from oom_killer in /var/log/messages . So looks like it's not an oom killer thing happening. btw i ran the same tests on the same computer but with ubuntu 11.04 kernel 2.6.38 , no crashes!! Same shared video memory, same intel hardware, different kernel (drivers?). -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140119200443.5f8151db@moli-desktop
Bug#735927: general: X *always* crashes when ram is full
this next test was fast, sorry for the flood. You could disable memory overcommit, that would cause things to fall over sooner. Again, nice idea, thanks. For reference: http://serverfault.com/questions/485798/cent-os-how-do-i-turn-off-or-reduce-memory-overcommitment-and-is-it-safe-to-do Did the above, rebooted, started chrome, it ran but did not appear visually on the screen. The process ran and allocated the memory but probably it couldnt allocate enough to run onwards. Now when i tried to run anything, all failed with the message cannot allocate memory, and a few seconds later the X crashed again. No messages in /var/log from oom-killer (i mistyped its name in my previous email) so again it does not look like oom-killer WAS TAKING action. I'm emphasising this because i have an old message from oom-killer when it really worked: /var/log/kern.log.1:Jan 18 22:25:35 desktop kernel: [ 2027.606581] Xorg invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 /var/log/messages.1:Jan 18 22:25:35 desktop kernel: [ 2027.606581] Xorg invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 /var/log/syslog.1:Jan 18 22:25:35 desktop kernel: [ 2027.606581] Xorg invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 but not this time. (my clock is set) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140119204324.0d588343@moli-desktop
Bug#735927: general: X *always* crashes when ram is full
On 2014-01-18 23:56, moli wrote:
ok, another way to reproduce without chrome or flash or youtube:
reboot to a clean system, dont run anything, only an X and a console
# mkdir /tmp/foo
# mount -t tmpfs none /tmp/foo -o size=900m
# dd if=/dev/zero of=/tmp/foo/bar
(a message comes with device full)
(now you cannot give any new command in bash as there is no memory to
start a new shell for you command: error message: bash: fork: cannot
allocate memory)
now wait a few seconds, move your mouse, etc, boom, the x crashes:
/var/log/Xorg.0.log.old
(EE) intel(0): Failed to submit batch buffer, expect rendering
corruption or even a frozen display: No space left on device.
note: this time i've set the bios parameters to the *MAXIMUM* values:
DVMT MODE: FIXED
IGD DVMT MEMORY: 128MB
IGD APERTURE SIZE: 256MB
so there should be plenty of video memory and note that i didnt run
much program that needs video memory.
I don't think its running out of *video memory*. It's just running out
of normal RAM. What else do you expect after putting the system under
extreme memory pressure (either by sucking it into a tmpfs or by running
google-memory-hog)?
The famous last words from your Xorg.log are from
xserver-xorg-video-intel-2.19.0 src/intel_batchbuffer.c (removed
irrelevant parts):
ret = dri_bo_subdata(intel-batch_bo, 0, intel-batch_used*4,
intel-batch_ptr);
if (ret != 0) {
if (ret == -EIO) {
} else {
xf86DrvMsg(scrn-scrnIndex, X_ERROR,
Failed to submit batch buffer,
expect rendering corruption
or even a frozen display: %s.\n,
strerror(-ret));
}
}
To me this looks like the intel driver cannot do a task because it
cannot allocate memory (this does not neccessarily trigger the kernel's
OOM killer) and shortly afterwards X crashes - maybe the intel driver
corrupted something on the way. The ret value is not returned to the
caller (X), so that does not know about the failure.
Look here for instructions for debugging X crashes:
https://wiki.debian.org/XStrikeForce/XserverDebugging
Good luck that anything here works if the system is out of memory.
btw i ran the same tests on the same computer but with ubuntu 11.04
kernel 2.6.38 , no crashes!! Same shared video memory, same intel
hardware, different kernel (drivers?).
Compare the versions of xserver-xorg-core and xserver-xorg-video-intel
between the Ubuntu version and the Debian version you used.
Could you try the vesa (xserver-xorg-video-vesa) driver instead and see
what happens under extreme memory pressure?
Andreas
PS: and you would be perfectly fine if X wouldn't crash but would be
killed by the oomkiller - with the same situation afterwards ?
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/52dc31dd.7020...@debian.org
Bug#735927: general: X *always* crashes when ram is full
On 2014-01-19 21:13, Andreas Beckmann wrote: Could you try the vesa (xserver-xorg-video-vesa) driver instead and see what happens under extreme memory pressure? And since I suspect xserver-xorg-video-intel, you might want try a newer version of that driver. The Xorg maintainers can probably give some hints how to properly backport the jessie version to wheezy (if a plain rebuild in a wheezy environment is not enough). Andreas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/52dc3642.8040...@debian.org
Bug#735927: general: X *always* crashes when ram is full
On Sun, Jan 19, 2014 at 04:36:51PM +0100, moli wrote: @Holger: Or (recommended) just add a small swap partition (say 128MB) to allow the kernel to put unused memory pages on disk so the X server can use them instead. I'm sorry, i can not, i only have SSD in that machine. If i enable swap in a machine with only 1GB ram it would be in use all day long, every time. I will try your other recommendations through, please be patient and wait for my feedback. Mount a tmpfs on /tmp with a strict size limit. You can do this with /etc/default/tmpfs (RAMTMP, TMPSIZE) or just add an fstab entry. The default size is 50% RAM, which is likely the cause of your problems. This will definitely OOM your system when you fill /tmp; it's normally backed by swap so has somewhere to move the data to. The size limit will definitely constrain what you can store in /tmp though, so it will bring its own limitations (but will avoid crashing things). I'm unsure of a better solution I'm afraid. With an SSD, you really don't want /tmp or swap on it; using a tmpfs on /tmp makes a world of difference. But this is certainly better with swap on a spinning disk. Swap on the SSD would be better than /tmp on the SSD if you really need the space--it'll only get used when memory gets tight for the most part so it might be a compromise worth considering. If we don't already, the installer could make some adjustments to RAMTMP/TMPSIZE if the rootfs is on an SSD and/or swap is absent. Regards, Roger -- .''`. Roger Leigh : :' : Debian GNU/Linuxhttp://people.debian.org/~rleigh/ `. `' schroot and sbuild http://alioth.debian.org/projects/buildd-tools `-GPG Public Key F33D 281D 470A B443 6756 147C 07B3 C8BC 4083 E800 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140119225525.gb6...@codelibre.net
Bug#735927: general: X *always* crashes when ram is full
ok, another way to reproduce without chrome or flash or youtube: reboot to a clean system, dont run anything, only an X and a console # mkdir /tmp/foo # mount -t tmpfs none /tmp/foo -o size=900m # dd if=/dev/zero of=/tmp/foo/bar (a message comes with device full) (now you cannot give any new command in bash as there is no memory to start a new shell for you command: error message: bash: fork: cannot allocate memory) now wait a few seconds, move your mouse, etc, boom, the x crashes: /var/log/Xorg.0.log.old (EE) intel(0): Failed to submit batch buffer, expect rendering corruption or even a frozen display: No space left on device. note: this time i've set the bios parameters to the *MAXIMUM* values: DVMT MODE: FIXED IGD DVMT MEMORY: 128MB IGD APERTURE SIZE: 256MB so there should be plenty of video memory and note that i didnt run much program that needs video memory. Now testing this method on my desktop machine which has 12GB(!) ram and kdm+kde did not crash the X. The system suffered, lagged but did not crashed. After filling the tmpfs i opened some new chrome tabs (the only reason i'm using chrome is nothing else i use eats memory like it), waited for ~10 minutes, chrome did not respond but did not crash. Then i allocated some more ram again in the tmpfs, the system console had some messages it is killing the chrome to free ram, some chrome tabs crashed, but the X did not. After this i've logged out and logged in to my girlfriends account, she uses xfce. Did the same as above, X did not crash, so it's not looking like an xfce-bug. Note that my desktop machine has a video card with DEDICATED video ram. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20140118235621.1522456b@moli-desktop
