Re: [gentoo-user] Re: Alternate Incoming Mail Server
On Mon, Apr 6, 2020 at 4:02 PM Grant Taylor wrote: > > On 4/6/20 1:03 PM, Rich Freeman wrote: > > More often than not, yes. The main exception I've seen are sites > > that email you verification codes, such as some sorts of "two-factor" > > implementations (whether these are really two-factor I'll set aside > > for now). Many of these services will retry, but some just give up > > after one attempt. > > I believe that's a perfect example of services that should send email > through a local MTA that manages a queue and retries mail delivery. > There is no need for this type of queuing logic and complexity to be in > the application. Especially if the application is otherwise stateless > and runs for the duration of a single HTTP request. Sure, but: 1. We're talking about people who think email is a great solution to 2FA. and 2. Why use a standard MTA when you can build one yourself? I believe this is a corollary to Zawinski's Law. -- Rich
Re: [gentoo-user] Re: Alternate Incoming Mail Server
On Mon, Apr 6, 2020 at 12:18 PM Ian Zimmerman wrote: > > On 2020-04-06 14:24, Ashley Dixon wrote: > > > Cheers for the help ! To be honest, I don't think I'd want to receive > > e-mail from someone who cannot resist pressing a button :) > > In fact, "MTAs" that don't retry turn out to be spam robots on close > inspection, more often than not. That is the basis for the spam > fighting tactic called "greylisting". So you will not even be original > in ignoring them. > More often than not, yes. The main exception I've seen are sites that email you verification codes, such as some sorts of "two-factor" implementations (whether these are really two-factor I'll set aside for now). Many of these services will retry, but some just give up after one attempt. Solutions like postgrey make it easy to whitelist particular MTAs or destination addresses to avoid this problem. I won't say that greylisting has solved all my spam problems but it definitely cuts down on it. Also, by delaying never-before-seen MTAs it also makes it more likely that RBLs and such will be updated before the email makes it past greylisting, which will cut down on "zero day" spam. -- Rich
Re: [gentoo-user] Idea for Videoconferencing (Video part)
On Fri, Apr 3, 2020 at 12:44 PM Petric Frank wrote: > > i think eye contact is a good thing while working on the screen. > I think most people think that eye contact is a good thing most of the time. If you have any tips for actually doing it I suspect half of us on the list would benefit. :) I guess one thing I like about web conferencing is not using the camera and then nobody thinks about the fact that I never look at their faces... -- Rich
Re: [gentoo-user] zoom?
On Wed, Mar 25, 2020 at 9:06 AM Jorge Almeida wrote: > > On Wed, Mar 25, 2020 at 12:56 PM Alec Ten Harmsel > wrote: > > > > On Wed, Mar 25, 2020, at 08:54, Jorge Almeida wrote: > > > On Wed, Mar 25, 2020 at 12:42 PM Michael wrote: > > > > Have you looked at using WebRTC with Zoom, rather than installing their > > > > code > > > > on your PC? > > > WebRTC is a web standard. You should be able to use Zoom in-browser, > > without downloading the application. > > > OK. But it seems to offer limited functionality: > https://support.zoom.us/hc/en-us/articles/115005666383-Show-a-Join-from-your-browser-Link > > (And will it be safer that the application?) > Web tabs are fairly highly sandboxed in most browsers. Suffice it to say something running in a web tab isn't going to be spying on your process list/etc. An application can basically do absolutely anything you can do from a shell unless you've done something to contain it. Running it in a container would obviously be one way of containing it. Running it under another UID would be another, though users can generally see all the processes in the system and read any file that is world-readable. I'm not sure how the flatpak version of zoom that was mentioned earlier is packaged. I believe flatpak is container-based, but I haven't used it and I can't speak to how well-contained it actually is, either in general or in its implementation of this particular application. In theory they could make it very secure, but that doesn't mean that they did. Oh, and keep in mind that X11 itself isn't the most secure piece of software in existence. In particular any window on your desktop can spy on the keyboard input into any other window on your desktop, unless you're employing protective measures that nobody actually employs outside of maybe pinentry (I haven't checked that one and I forget if it is completely modal - as in you can't type in any other x11 window while it is displayed). -- Rich
Re: [gentoo-user] Re: SDD strategies...
On Wed, Mar 18, 2020 at 9:49 AM antlists wrote: > > On 17/03/2020 14:29, Grant Edwards wrote: > > On 2020-03-17, Neil Bothwick wrote: > > > >> Same here. The main advantage of spinning HDs are that they are cheaper > >> to replace when they fail. I only use them when I need lots of space. > > > > Me too. If I didn't have my desktop set up as a DVR with 5TB of > > recording space, I wouldn't have any spinning drives at all. My > > personal experience so far indicates that SSDs are far more reliable > > and long-lived than spinning HDs. I would guess that about half of my > > spinning HDs fail in under 5 years. But then again, I tend to buy > > pretty cheap models. > > > If you rely on raid, and use spinning rust, DON'T buy cheap drives. I > like Seagate, and bought myself Barracudas. Big mistake. Next time > round, I bought Ironwolves. Hopefully that system will soon be up and > running, and I'll see whether that was a good choice :-) Can you elaborate on what the mistake was? Backblaze hasn't found Seagate to really be any better/worse than anything else. It seems like every vendor has a really bad model every couple of years. Maybe the more expensive drive will last longer, but you're paying a hefty premium. It might be cheaper to just get three drives with 3x redundancy than two super-expensive ones with 2x redundancy. The main issues I've seen with RAID are: 1. Double failures. If your RAID doesn't accommodate double failures (RAID6/etc) then you have to consider the time required to replace a drive and rebuild the array. As arrays get large or if you aren't super-quick with replacements then you have more risk of double failures. Maybe you could mitigate that with drives that are less likely to fail at the same time, but I suspect you're better off having enough redundancy to deal with the problem. 2. Drive fails and the system becomes unstable/etc. This is usually a controller problem, and is probably less likely for better controllers. It could also be a kernel issue if the driver/failesystem/etc doesn't handle the erroneous data. I think the only place you could impact this risk is with the controller, not the drive. If the drive sends garbage over the interface then the controller should not pass along invalid data or allow that to interface with functioning drives. This is one of the reasons that I've been trying to move towards lizardfs or other distributed filesystems. This puts the redundancy at the host level. I can lose all the drives on a host, the host, its controller, its power supply, or whatever, and nothing bad happens. Typically in these systems drives aren't explicitly paired but data is just generally pooled, so if data is lost the entire cluster starts replicating it to return redundancy, and that rebuild gets split across all hosts and starts instantly and not after you add a drive unless you were running near-full. One host replicating one 12TB drive takes a lot longer than 10 hosts replicating 1.2TB each to another host in parallel as long as your network switches can run at full network capacity per host at the same time and you have no bottlenecks. -- Rich
Re: [gentoo-user] SDD strategies...
On Tue, Mar 17, 2020 at 1:59 AM wrote: > > The HD will contain the whole system including the complete root > filesustem. Updateing, installing via Gentoo tools will run using > the HD. If that process has ended, I will rsync the HD based root > fileystem to the SSD. > ... I'll go ahead and write one consolidated reply to a couple of points raised in these two threads to save everybody emails. First, I'll echo what was said about this being probably an overly-complex solution to the problem. I think you'll spend more time dealing with this than with any SSD failure issues. Next, in general you tend to get what you pay for. With SSDs if you're getting that bargain 512GB drive for $15 when all the comparable drives are $70, and it was made by some company you've never heard of, chances are that you're missing something. I'm not saying you need to go buy that 150% more expensive "signature edition" drive or whatever, though it might very well have some feature that justifies its price. Just be wary if things that look too good to be true. If you buy a reputable brand that is marketed for your intended use chances are you're getting something decent, even if you're paying 10% more. If you really know what you're doing you can certainly use research to save money by critically evaluating your options. Finally, ALL DRIVES FAIL. It doesn't matter what the underlying storage technology is. I've seen hard drives fail in less than a year, with the warranty replacement drive failing less than a year after that. I think next warranty replacement (still in the original warranty period) lasted 5+ years of near-continuous use. The typical failure modes of hard drives and solid state storage are different, but they all fail. You can't perfectly predict WHEN they will fail either. Most drives have SMART and sometimes it can detect failure conditions before failure, but not always. What will you do when your brand new drive fails 3 weeks after you buy it? If you don't have an answer that doesn't involve you losing stuff you don't want to lose, or having downtime you don't want to have, then you need to re-evaluate your approach. Backups and RAID are the obvious solutions - with backup generally being the more reliable solution to data loss, and RAID being the more reliable solution to downtime, but with them both having some overlap. Here is what I've done: 1. Preferred solution to SSD failure and associated downtime is RAID+backups. The most important stuff is backed up to the cloud, encrypted. With SSDs I usually do a full backup to hard drives since that is fairly inexpensive given their relative capacities. I'm using ZFS mirroring as my RAID-like solution right now, and I use ZFS-send/receive to do hourly backups that are very low overhead. 2. If I'm too cheap to use RAID on a host then I just do the hourly ZFS remote snapshots - that is a good solution on hosts where downtime doesn't matter, because I can just get a new drive and restore the snapshot and I'm back in business after a day or two, accepting a one hour recovery point objective. You can of course use rsync as well. For rsync-based backups I recommend rsnapshot, in portage. For zfs remote snapshots I have switched to zfs_autobackup: https://github.com/psy0rz/zfs_autobackup/blob/master/README.md -- Rich
Re: [gentoo-user] The Intel sequel of vulnerabilities continues ...
On Tue, Mar 10, 2020 at 8:20 PM Michael wrote: > > What-ever I may buy in the future, I'll make sure it does not contain Intel > inside ... > > https://www.theregister.co.uk/2020/03/10/lvi_intel_cpu_attack/ > It seems like the PRIMARY vulnerability here is with SGX, which I doubt most of us care about too much (does Linux even support it)? Apparently it needs to be enabled in your firmware to even work. However, it seems that SGX just makes it much easier to execute the attack, and the attack is not limited to SGX, so this still is an issue for everybody. The mitigations are indeed crazy. It seems like any instruction that accesses memory potentially has to be fenced, including the RET instruction (which pops the return value off the stack, which is memory). This is still pretty fresh, so it is hard to gauge how practical these concerns are. If you can narrow down the areas of code that are sensitive and be more selective with the mitigation that will of course greatly reduce the performance cost. If you have to basically banish the RET instruction in your compiler that can't be cheap. I could also see optimizations to inline calls become more prevalent. I'm not sure if AMD has issues like this that nobody has found yet, or if they avoided this problem in their designs. Oh, and this seems like potentially a bigger impact than meltdown as this isn't about reading memory - it is about effectively WRITING it. Not so much writing the memory itself from my initial read so much as making the CPU think it read something from memory that wasn't there, which is very similar. Indeed, with the RET instruction you're modifying a value that is going to go into the instruction pointer so you can basically hijack code execution. Of course, this still requires that you have executable code addressable by the process to jump to (which probably means in a page not marked with NX). It will be very interesting to see where this goes. I'm sure lkml is already buzzing... -- Rich
Re: [gentoo-user] Re: New laptop - AMD or Intel?
On Mon, Mar 9, 2020 at 2:28 PM Manuel McLure wrote: > > On Mon, Mar 9, 2020 at 11:09 AM Grant Edwards > wrote: >> >> On 2020-03-09, Mark Knecht wrote: >> >> > Would that be the consensus of the group here? > > My understanding is that AMD is currently leading both in raw performance as > well as bang-for-buck. > That is generally my understanding as well, at least for desktop/server situations when you're buying parts. For a laptop you really have to look at the whole package. Raw CPU perf-per-dollar is often not the biggest concern with a laptop. As long as it meets your performance specs you're mostly going to be worried about form, battery, and so on. So, I really wouldn't obsess over AMD vs Intel. I have been buying all AMD lately for my regular hosts and do my own builds. My only recent laptop is a combo form that runs Intel, because it was the form I wanted (Acer Switch running Win10 and a bunch of Win-specific software). You're probably going to find more Intel options in this space in any case. -- Rich
Re: [gentoo-user] Now it's AMD's turn ...
On Sun, Mar 8, 2020 at 10:23 AM Rudi wrote: > > While I usually side with AMD for their contributions to the Open > Sourced community, I'm going to go out on a limb and say that even > though they're funded by Intel the fact that they've been keeping the > specifics quiet proves that they're trying to help rather than smear > the name of AMD. IMO all responsible disclosure only makes everybody safer, so if Intel wants to fund making my AMD CPUs safer, I'm all for that. If these researchers can find a flaw and report it, somebody else could find it and not report it. > Hopefully this doesn't cause as much of a recoil as the Spectre/Meltdown > mitigations. What % of performance was lost for those? 20? That's the key. While vulnerabilities should be avoided as much as possible, the fact is that almost all software and hardware ends up having them. The real issues are: 1. Does the vendor provide a mitigation in a timely manner? 2. Is the mitigation free (ie software/etc)? 3. Does the mitigation have any kind of long-term negative impact? With meltdown the issue was #3. Right now we don't have any mitigation, though I can't really speak to how fast is fast enough. Now that this is disclosed they should push to get this fixed ASAP. -- Rich
Re: [gentoo-user] Re: New Intel vulnerability?
On Sat, Mar 7, 2020 at 12:21 PM Grant Edwards wrote: > > On 2020-03-07, Rich Freeman wrote: > > > In this case we're talking about a TPM where a threat model > > is an attacker with physical access that is trying to play games with > > the busses/etc, and as such it is important that it initialize using > > code in ROM that is known-good. > > Note that the person behind the attack doesn't need physical > access. If an attacker can shove malicious firmware into something > like a PCI card with DMA bus-master capabilities, then on power-up > that card can carry out the attack. However, getting the firmware > into the PCI card would probably require root privledges, so there > would need to be a pre-existing privledge-elevation vulnerability. That is a really good point. Attackers with root access are still a concern. You could have a zero day in your system, then somebody breaks in, and then they install some rootkit in your video card or something else capable of exploiting this flaw. They leave no other traces on your system. Then the next day you patch your system to close the zero day, but unknown to you there is a rootkit already installed. Even wiping your drives and reinstalling would not remove it in this scenario. And the firmware updates won't cover you in this case. Of course, exploiting this in software probably requires other vulnerabilities in other pieces of hardware. Also, exactly what kind of hardware access is required isn't disclosed anywhere I can see, so exactly what sorts of attacks like this would/wouldn't work is hard to tell. Of course, attacks mediated through hardware in your system like this are a bigger issue than just this particular flaw. Most of those can be mitigated through IOMMU and so on, so that your graphics card can't just go reading encryption keys out arbitrary process memory, and so on. However, the key here is that you install something that lurks until reboot and then exploits the CPU to get its hooks into the system early enough that it has control over the IOMMU and so on, before any OS even initializes. The security modules in the CPU are supposed to secure the IOMMU using ROM-based code and so on before anything important is in RAM, and while I didn't dig into the details that sounds like where the fault lies. -- Rich
Re: [gentoo-user] New Intel vulnerability?
On Sat, Mar 7, 2020 at 11:26 AM Ivan T. Ivanov wrote: > > Quoting Rich Freeman (2020-03-06 23:13:55) > > > > The patched firmware executes before any software you boot, assuming > > your device was patched before the hacker got his hands on it. > > > > Well, they say that vulnerability is inside ROM code [1], which > is executed before any firmware. And because this is ROM it could > not be patched. > The root vulnerability is indeed in ROM. The firmware patches partially mitigate the vulnerability. Without a firmware patch the CPU is vulnerable to both hardware and software attacks. With the firmware patch the CPU is still vulnerable to hardware/physical attacks, but is apparently no longer vulnerable to software attacks. Obviously both attack vectors matter depending on your use case, but software attacks are obviously far more widely applicable. If you don't patch your firmware then even a server which is completely protected physically is still vulnerable. Even if you don't intend to use these features in your CPU, a hacker who manages to get into your server could use the vulnerability to implant a rootkit that is protected by the CPU from detection by the OS. So, even though a firmware update doesn't entirely close the vulnerability, it is still important to deploy. Note that while in this case it is apparently not possible to fix the problem with firmware, there are known CPU hardware problems that can be fixed via software. It really depends on the nature of the problem. In this case we're talking about a TPM where a threat model is an attacker with physical access that is trying to play games with the busses/etc, and as such it is important that it initialize using code in ROM that is known-good. -- Rich
Re: [gentoo-user] New Intel vulnerability?
On Fri, Mar 6, 2020 at 3:55 PM Wols Lists wrote: > > On 06/03/20 19:39, Rich Freeman wrote: > > > > They don't detail the effort required. If the firmware is patched it > > sounds like it still requires tinkering with hardware. > > By then it's TOO LATE. The firmware is signed for security, AND LOADED > AT BOOT. But if the boot process is compromised, the attacker simply > doesn't load the patched firmware. The patched firmware executes before any software you boot, assuming your device was patched before the hacker got his hands on it. Obviously if you lost your laptop last week and the hacker has it in their closet, then they're going to be able to attack the unpatched firmware with a software hack. >From the descriptions I'm seeing it sounds like the attack against the hardware itself requires physical access to some of the busses. A proper TPM should resist this sort of attack, and that is why this is a vulnerability. However, you can't just stick a USB drive into a computer to hack it if the firmware is patched. > > > > Yes, but keep in mind the signing keys have nothing to do with disk > > encryption. It is for remote attestation. Hence my Netflix comment. > > > Signing keys have EVERYTHING to do with whether you can trust the CPU. > If you can't trust the CPU, then it can simply read the disk encryption > credentials without any reference to whether it SHOULD read them. So, I think you're conflating a few different things here. First, trust is a choice you make. Whether you trust the CPU or not makes no difference in whether somebody else can defeat the embedded TPM. Second, the signing keys we're talking about are keys embedded in the CPU used by the CPU to sign stuff. These keys aren't used for encrypting anything, and they aren't used for decrypting anything. They're also not used to verify firmware/etc. It would make no sense to put the firmware signing key in the CPU - that key would be locked up at the firmware vendor's HQ with the corresponding public key embedded in the CPUs. Don't get me wrong - encryption keys stored in the CPU are compromised by this vulnerability. However, this has nothing to do with the signing keys. I suspect that modifying the firmware might also be possible due to this vulnerability, by replacing the public key in the CPU with a new one that you generated. Again, that has nothing to do with the signing keys on the CPU. The signing keys should just be associated with remote attestation. > And it only takes ONE person to crack that master key ONCE, and > EVERYBODY is up shit creek. No - there isn't any master key stored inside an Intel CPU that can be used to compromise the encryption on other hosts. We're talking about the signing keys used for remote attestation. Again, Netflix and Intel and similar companies are going to be ticked if that master key is leaked, but it won't have any impact on disk encryption. Of course, disk encryption is still compromised by the vulnerability. It just has nothing to do with the master signing keys (which are also model-specific and not completely universal), and hacking any individual device, if patched, will probably still require a hardware-level attack against that specific device. > At the end of the day, it's a "tree of trust". And once the root key is > compromised, you can NOT trust ANY key that was secured by said root > key. Yup. Fortunately the disk encryption keys are unique to each device, and stored in the compromised TPM hardware. They aren't really secured by a root key in any meaningful way. -- Rich
Re: [gentoo-user] New Intel vulnerability?
On Fri, Mar 6, 2020 at 2:07 PM Wols Lists wrote: > > On 06/03/20 13:48, Rich Freeman wrote: > > If you fall into this camp you need to still update your firmware to > > address the non-TPM-user and to avoid making it trivial for software > > to steal your keys/etc. However, you need to be aware that you are no > > longer secure against physical theft of your device. Somebody who > > steals your laptop with passwordless encryption might be able to break > > the encryption on your device. > > It's worse that that, he's dead, Jim! > > The summary on LWN is an easy read. Somebody who steals your Intel > laptop WILL be able to break the encryption on your device. > > tl;dr summary - the microcode that *boots* the cpu has been compromised. > So even while it is setting up tpm and all that malarkey, malware can be > stealing keys etc. They don't detail the effort required. If the firmware is patched it sounds like it still requires tinkering with hardware. However, there really isn't nothing you said that doesn't agree with what I said. Whether they "WILL" be able to break the encryption on your device depends a lot on the details and the knowledge of the attacker. Hence the reason I said "might." In any case, might is good enough to not rely on a broken security feature. > Which means that Intel's master signing key will soon be cracked and > compromised. Yes, but keep in mind the signing keys have nothing to do with disk encryption. It is for remote attestation. Hence my Netflix comment. -- Rich
Re: [gentoo-user] New Intel vulnerability?
On Fri, Mar 6, 2020 at 3:50 AM Michael wrote: > > I have lost count with the naming scheme of Intel's embedded spyware to know > if this is yet another vulnerability, or something to convince me to throw > away the last Intel powered box still in my possession (mind you its >10yr > old): > > https://arstechnica.com/information-technology/2020/03/5-years-of-intel-cpus-and-chipsets-have-a-concerning-flaw-thats-unfixable/ The article is actually pretty well-written. I haven't studied the issue in any depth but here are my impressions: 1. You need a firmware update to prevent software vulnerabilities. 2. Even with a firmware update you are vulnerable to somebody with physical access to your device. The whole issue centers around TPM essentially. This potentially impacts you if you don't care about TPM, but it impacts you more if you do care about TPM. If you don't use TPM (probably many on this list), then your main concern should just be with getting your firmware patched (#1 above). Otherwise you could be vulnerable to rootkits that hijack the TPM on your machine and use it to spy on you in hard-to-detect ways. Based on the article a firmware patch should block the ability for software to get into your TPM and mess with it. Then you're basically safe. If you aren't using TPM you're already vulnerable to somebody with physical access to your device, so there is no real change in the threat model for you. Now let's get to those who use TPM or the other impacted trusted services. You use these if: 1. You rely on secure boot (with any OS - Linux does support this though I imagine it is rare for Gentoo users to use it). 2. You rely on TPM-backed full disk encryption. This includes Bitlocker and most commercial solutions. This doesn't include LUKS. If your disk is unreadable if you remove it from the computer, but you don't need any password to boot it, then you're probably using TPM-backed encryption. 3. You are Netflix/etc and are relying on remote attestation or any of the technologies RMS would term "treacherous computing." 4. You are a corporate owner of computers and are relying on the same technologies in #3 but to actually protect your own hardware. Or maybe if you're the only person in the world using Trusted GRUB. If you fall into this camp you need to still update your firmware to address the non-TPM-user and to avoid making it trivial for software to steal your keys/etc. However, you need to be aware that you are no longer secure against physical theft of your device. Somebody who steals your laptop with passwordless encryption might be able to break the encryption on your device. They would need to steal the entire laptop though - if you throw out a hard drive nobody will be able to recover it from the trash. If you're Netflix I'm not sure why you're even bothering with this stuff because all your content is already available in full quality on torrent sites, but I guess you can lose even more sleep over it if you want to. If you're using secure boot then somebody with physical access might be able to change the authorization settings and let another OS boot. If you're a corporation with sensitive data you probably have the biggest impact, because you're distributing laptops to people who lose them and who don't have a ton of security hygiene to begin with. The only people who probably will consider replacing hardware are corporate users. Most on this list are going to be fine with a firmware update as you're probably not using the TPM features. Indeed, even getting those working on Linux is a PITA - I'm not aware of any distro that has TPM-backed encryption out of the box. Windows has this in the pro edition (Bitlocker) and it is probably fairly popular. If you use LUKS-based encryption you are going to be secure with patched firmware as long as nobody installs a keylogger on your device. That will be easier with the vulnerability, though somebody could just hack the keyboard hardware anyway and LUKS wouldn't protect you against that. TPM has pros and cons compared to LUKS in general. If you don't patch your firmware then it is possible a rootkit might get in there and steal your keys at boot time. If somebody has more to add from researching this more I'm all ears. Now I need to check if my windows tablet with Bitlocker is vulnerable. This also shows the downside to TPM encryption - it is convenient but if somebody steals a laptop and just keeps it stored away they could always use a vulnerability like this to break in sometime in the future. It is probably still worth using as a minimum because it does protect against hard drive loss, and it works if your TPM isn't vulnerable. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Mon, Mar 2, 2020 at 12:22 AM William Kenworthy wrote: > > I thought lizardfs was much more community minded > but you are characterising it as similar to moosefs - a taster offering > by a commercial company holding back some of the non-essential but > jucier features for the paid version - is that how you see them? I don't see much of an active community. It seems like most actual development happens outside of the public repo, with big code drops by the private team doing the work (which seems to be associated with a company). A bit like the Android model. I'm sure they'll accept pull requests, but that isn't how most of the work is getting done. It seems like the main difference between them and moosefs is that they're making more stuff FOSS to entice users over. Shadow masters are FOSS as opposed to just having metadata loggers. HA is FOSS in the latest RC. So, it seems like their model is to trickle out the non-free stuff and make it free after a delay. It really seems like Ceph is the best fully open platform out there, but the resource requirements just make it impractical. I have no doubt that it can scale FAR better with its design, but that design basically forces every node to be a bit of a powerhouse, versus Lizardfs where you just have one daemon with all the intelligence and the rest are just dumping files on disks. And you really don't need much CPU/RAM for the master if you're serving large files - the demands would go up with IOPS and number of files, and multimedia is low on both. > By the way, to keep to the rpi subject, I did have a rpi3B with a usb2 > sata drive attached but it was hopeless as a chunkserver impacting the > whole cluster. Having the usb data flow and network data flow through > the same hub just didn't go well Hard drives plus 100Mbps LAN sharing a single USB 2.0 hub is definitely not a recipe for NAS success... When I upgraded to UniFi switches I really only noticed for the first time how many hosts I have that aren't gigabit, and they're mostly Pis at this point. They're nice little project boards but for anything IO-intensive they're almost always the wrong choice. The RockPro64 I'm using has gigabit plus PCIe 3.0 x8 plus USB3 and as far as I can tell they don't have any contention. Maybe they're all on a PCIe bus or something but obviously that can handle quite a bit. Only issue was that the rk3399 PCIe drivers were not the most robust in the kernel, but ayufan and the IRC channel were both helpful and his kernel branch is actively maintained, so I was able to get everything sorted (some delays needed during training to allow boards to initialize and I was having power issues in the beginning). Much of the rk3399 support in the kernel was pushed by Google for Chromebooks and LSI HBAs weren't exactly on their list of things to test with those - not sure if the Chromebooks put much of anything on PCIe. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sun, Mar 1, 2020 at 8:52 PM William Kenworthy wrote: > > For those wanting to run a lot of drives on a single host - that defeats > the main advantage of using a chunkserver based filesystem - > redundancy. Its far more common to have a host fail than a disk drive. > Losing the major part of your storage in one go means the cluster is > effectively dead - hence having a lot of completely separate systems is > much more reliable Of course. You should have multiple hosts before you start putting multiple drives on a single host. However, once you have a few hosts the performance improves by adding more, but you're not really getting THAT much additional redundancy. You would get faster rebuild times by having more hosts since there would be less data to transfer when one fails and more hosts doing the work. So, it is about finding a balance. You probably don't want 30 drives on 2 hosts. However, you probably also don't need 15-30 hosts for that many drives either. I wouldn't be putting 16 drives onto a single host until I had a fair number of hosts. As far as the status of lizardfs goes - as far as I can tell it is mostly developed by a company and they've wavered a bit on support in the last year. I share your observation that they seem to be picking up again. In any case, I'm running the latest stable and it works just fine, but it lacks the high availability features. I can have shadow masters, but they won't automatically fail over, so maintenance on the master is still a pain. Recovery due to failure of the master should be pretty quick though even if manual - just have to run a command on each shadow to determine which has the most recent metadata, then adjust DNS for my master CNAME to point to the new master, and then edit config on the new master to tell it that it is the master and no longer a shadow, and after restarting the daemon the cluster should be online again. The latest release candidate has the high availability features (used to be paid, is now free), however it is still a release candidate and I'm not in that much of a rush. There was a lot of griping on the forums/etc by users who switched to the release candidate and ran into bugs that ate their data. IMO that is why you don't go running release candidates for distributed filesystems with a dozen hard drives on them - if you want to try them out just run them in VMs with a few GB of storage to play with and who cares if your test data is destroyed. It is usually wise to be conservative with your filesystems. Makes no difference to me if they take another year to do the next release - I'd like the HA features but it isn't like the old code goes stale. Actually, the one thing that it would be nice if they fixed is the FUSE client - it seems to leak RAM. Oh, and the docs seem to hint at a windows client somewhere which would be really nice to have, but I can't find any trace of it. I only normally run a single client but it would obviously perform well as a general-purpose fileserver. There has been talk of a substantial rewrite, though I'm not sure if that will actually happen now. If it does I hope they do keep the RAM requirements low on the chunkservers. That was the main thing that turned me off from ceph - it is a great platform in general but needing 1GB RAM per 1TB disk adds up really fast, and it basically precludes ARM SBCs as OSDs as you can't get those with that much RAM for any sane price - even if you were only running one drive per host good luck finding a SBC with 13GB+ of RAM. You can tune ceph to use less RAM but I've heard that bad things happen if you have some hosts shuffle during a rebuild and you don't have gobs of RAM - all the OSDs end up with an impossible backlog and they keep crashing until you run around like Santa Claus filling every stocking with a handful of $60 DIMMs. Right now lizardfs basically uses almost no ram at all on chunkservers, so an ARM SBC could run dozens of drives without an issue. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sun, Mar 1, 2020 at 2:13 AM William Kenworthy wrote: > > Keep in mind that rpi are not the only cheap, capable arm hardware out > there. > I completely agree. Anytime I'm looking at an application I consider the SBCs available as options. Certainly the odroids are highly spoken of. Main advantage of the Pi is its ubiquity - just about anything you could want is already packaged and documented for it. It is also pretty cheap. > backed by an Odroid HC2 moosefs cluster (though I am using an intel > powered Odroid H2 for the master). I considered an HC2 for lizardfs. My problem with it is that it has a single SATA port, which means you're buying a $50 SBC for every hard drive in your cluster. For a single drive per node it is probably your best bet. However, my chunkservers are: ~$65 RockPro64 $20 used LSI HBA $5 wall wart $25 cheap ATX PSU $5 ATX power switch $5 extra SATA cables $5 powered 16x PCIe riser cable (these are a bit hard to find) That is ~$125, and will support 16 hard drives. You're saving money on the 3rd drive per node. If you want some kind of enclosure for the drives you'll pay maybe another $5/drive. The other option that might be worth considering if you don't mind losing some bandwidth to the drives is just using SATA3 and hubs/etc and external drives. I'm shucking external drives anyway. So, any SBC with a SATA3 port would work for that, with nothing else needed. I could see USB3 bandwidth (shared) being a constraint if you're rebuilding, but it would keep up with gigabit ethernet. Oh, and for any kind of NAS/etc solution make sure that whatever you get has gigabit ethernet. The Pi3s at least don't have that - not sure about the Pi4. Wouldn't help in a Pi3 anyway as I think the LAN goes through the internal USB2 bus - the Pi is pretty lousy for IO in general - at least conventional PC IO. That GPIO breakout is of course nice for projects. -- Rich
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sat, Feb 29, 2020 at 9:49 AM Dale wrote: > > I have noticed the OOM killing the wrong thing as well. In a way, how > does it know what it should kill really??? After all, the process using > the most memory may not be the problem but another one, or more, could. > I guess in most cases the one using the most is the bad one but it may > not always be the case. I'm not sure how OOM could determine that tho. > Maybe having some setting like you mentions would help. It's a thought. Oh, plenty of people have given thought to it. The algorithm is actually not as well-documented as I thought it was. Lots of documents, but they're fragmented. Behavior is also configurable. For example, you can just tell Linux to panic on OOM, or just have it kill the process that triggered OOM even if it isn't using much memory. Best docs I could find are at (among other places): https://github.com/torvalds/linux/blob/master/Documentation/filesystems/proc.txt#L1520 Aside from setting limits on services so that they die/restart before overall system memory is threatened, adjusting oom_score_adj lets you tweak overall priorities for any process. By default it mostly comes down to what process is hogging the most RAM, with slight preference for root-owned processes. Really though setting resource limits is your best bet. Then you can set a threshold above normal use, and if something misbehaves it is going to get restarted before most of RAM is taken. User session cgroups can of course be limited as well so that interactive processes can't just go nuts and use all your RAM. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Sat, Feb 29, 2020 at 10:17 AM Daniel Frey wrote: > > Yes, I'm aware linux does VLANs... I set up netifrc to do this (I > already have some "smart" switches set up - not full layer 3.) I thought > about running containers but if I ever have to do something like > emergency maintenance on my server the whole LAN would be down. Seems > like a no-brainer to have a tiny device like an RPi to do this. Yup. It really depends on your requirements. My main LAN uses a Pi as a DHCP+DNS server, for exactly this reason. I don't want to be replacing a hard drive in my server and now my lights/TV/whatever don't work. OpenHab runs on a Pi for this reason as well. On the other hand, for my other VLANs DHCP+DNS is handled by stuff like my UniFi gateway or other embedded solutions. These don't have the same requirements as my main LAN and being mostly self-contained a more consumer-oriented solution is fine. I don't want to be doing security updates on a bazillion Pis either. I use VLAN on Linux more for providing services on the VLANs. Not that I have much of this. Don't think I'm running some kind of datacenter. I just have a typical home LAN, and I'm running AREDN which basically needs two more VLANs of its own (one for the network it serves, and one for backhaul to the internet for tunnels/etc - don't want that stuff getting into my LAN and the IP address space conflicts in any case). I could see adding an IOT VLAN maybe, but the problem is that so much of that stuff needs to interact. If I stuck my TVs/Chromecasts/etc on a separate VLAN, then I couldn't cast to them from my phone or anything else unless it was on that VLAN too. > I'm not so sure I'll try installing Gentoo on it though, it doesn't > really seem suitable for compiling tasks. I'm pretty sure the kit I > ordered has a card with Raspbian on it, I'll check that out first. I run Raspbian on my Pis for this reason. If I had some niche use where Gentoo added value I'd go with it, but otherwise it just seems too painful. As it is I have to compile kernel modules on my RockPro64 boards and that takes forever even without having to build the actual kernel. When I've built kernels on those while troubleshooting issues with PCIe it would literally take an hour or more. If you do want to run Gentoo on a Pi you really should be cross-compiling it. Something like Gentoo Reference Platform on steroids would certainly be nice for ARM. -- Rich
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sat, Feb 29, 2020 at 9:13 AM Dale wrote: > > Runaway processes is one reason I expanded my memory to 32GBs. It gives > me more wiggle room for portage to be on tmpfs. > That is my other issue. 99% of the time the OOM killer is preferred when this happens versus having the system just grind to a complete halt. Either way some service is going to stop working, but at least with the OOM killer it probably will only be one service. OOM doesn't always kill the right thing, but it happens so infrequently I haven't bothered to address this. Setting limits on VM use on each service would of course be a good idea. Also, you can tune OOM priority as well for any process. With systemd these are unit config settings. I haven't looked at openrc recently but certainly you can just edit the init.d script to set these if there isn't just a config option. I've found OOM guessing wrong is more of an issue when you have a lot of medium-sized processes vs one large one. If one process is using 10GB of RAM and goes haywire it is very likely that this is the one OOM-killer will go after. On the other hand if you're building with make -j16 and you hit some really intensive part of a build and you get 16 processes demanding half a GB each then it is more likely that the OOM killer will first target some service that is RAM-hungry but not usually a problem, because it is using more than any one of the gcc processes. I wonder if you can make OOM-killer cgroup-aware. Services are generally in separate cgroups while make would all be in another, so if it looked at total use of the cgroup and not the individual process then it would weigh something that forks heavily a lot higher. -- Rich
Re: [gentoo-user] Re: webrsync {SOLVED}
On Sat, Feb 29, 2020 at 4:33 AM Wols Lists wrote: > > I just have a massive swap space, and /var/tmp/portage is a tmpfs. So > everything gets a fast tmpfs build, and it spills into swap as required > (hopefully almost never). > I can articulate a bunch of reasons that on paper say that this is the best approach. In practice I've found that swap on linux is sub-optimal at best. I only enable swap when I absolutely have to as a result. I'll reduce -j to lower memory demand before adding swap usually. On more RAM-constrained hosts I'll enable swap when building specific packages, or try to avoid those packages entirely. Maybe something has changed in the last few years and swap is actually useful, but I'm skeptical. I always tend to end up with GB of free RAM and a churning hard drive when I enable it. On SSD I'm sure it will perform better, but then I'm running through erase cycles instead. Like I said, on paper adding swap should only make things better. But, that is only true if the kernel makes the correct choices about prioritizing swap vs cache use. Sure, I could set swappiness to zero or whatever, but then that just turns swap into a NOOP best case and it isn't like I have OOM issues, so why bother?. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Fri, Feb 28, 2020 at 8:11 PM Daniel Frey wrote: > > Thanks for the detail, I've just ordered an RPi4B to mess around with. > It would be helpful to move DNS etc off my home server as I'm trying to > separate everything into VLANs. > Keep in mind that Linux supports VLAN tagging, so if you set up your switch to trunk your server you can have containers or even services on multiple VLANs on the same host. I have this configured via systemd-networkd - I'm sure you could do it with various other network managers as well. I just have a bridge for each VLAN and then I can attach container virtual ethernet interfaces to the appropriate VLAN bridge for each container. KVM uses bridges and it should be just as easy to put VMs on the appropriate bridges. If you assign IPs on the host to each VLAN interface then as long as the VLANs don't have conflicting IP addresses you can just attach services to the appropriate VLANs by binding to their addresses. A service that binds to 0.0.0.0 or to multiple addresses would listen on all of them. Now, if your VLANs have conflicting address spaces then I'd probably just stick to containers so that no host actually sees conflicting IPs, otherwise you're probably going to have to go crazy with iproute2 and netfilter to get all the packets going to the right places. And all of that should work from a Pi as well as long as long as you enable CONFIG_VLAN_8021Q. You also need to make sure the tagged VLAN traffic is passed from the switch (which is not what you normally want to do for a non-VLAN-aware host where you would filter out all but one VLAN and remove the tag). I run my DHCP server on a Pi so that it is more independent. -- Rich
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Fri, Feb 28, 2020 at 6:09 AM Wols Lists wrote: > > On 27/02/20 21:49, Rich Freeman wrote: > > A fairly cheap amd64 system can run a ton of services in containers > > though, and it is way simpler to maintain that way. I still get quick > > access to snapshots/etc, but now if I want to run a gentoo container > > it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one > > core, but once a month it needs 4GB of RAM and 100% of 6 cores. As > > long as I'm not doing an emerge -u world on half a dozen containers at > > once it is no big deal at all. > > Do all your containers have the same make options etc? Can't remember > which directory it is, but I had a shared emerge directory where it > stored this stuff and I emerged with -bk options (use binary if it's > there, create binary if it isn't). > They're probably not too far off in general, but not exact. I only run one instance of any particular container, so I haven't tried to do parallel builds. If portage had support for multiple binary packages co-existing with different build options I might. If I ever get really bored for a few weeks I could see playing around with that. It seems like it ought to be possible to content-hash the list of build options and stick that hash in the binary package filename, and then have portage search for suitable packages, using a binary package if one matches, and doing a new build if not. Many of my containers don't even run Gentoo. I have a few running Arch, Ubuntu Server, or Debian. If some service is well-supported in one of those and is poorly supported in Gentoo I will tend to go that route. I'll package it if reasonable but some upstreams are just not very conducive to this. There was a question about ARM-based NAS in this thread which I'll go ahead and tackle to save a reply. I'm actually playing around with lizardfs (I might consider moosefs instead if starting from scratch - or Ceph if I were scaling up but that wouldn't be practical on ARM). I have a mix of chunkservers but my target is to run new ones on ARM. I'm using RockPro64 SBCs with LSI HBAs (this SBC is fairly unique in having PCIe). There is some issue with the lizardfs code that causes performance issues on ARM though I understand they're working on this, so that could change. I'm using it for multimedia and I care more about static space than iops, so it is fine for me. The LSI HBA pulls more power than the SBC does, but overall the setup is very low-power and fairly inexpensive (used HBAs on ebay). I can in theory get up to 16 drives on one SBC this way. The SBC also supports USB3 so that is another option with a hub - in fact I'm mostly shucking USB3 drives anyway. Main issue with ARM SBCs in general is that they don't have much RAM, so IMO that makes Ceph a non-starter. Otherwise that would probably be my preferred option. Bad things can happen on rebuilds if you don't have 1GB/TB as they suggest, and even with the relatively under-utilized servers I have now that would be a LOT of RAM for ARM (really, it would be expensive even on amd64). Lizardfs/moosefs chunkservers barely use any RAM at all. The master server does need more - I have shadow masters running on the SBCs but since I'm using this for multimedia the metadata server only uses about 100MB of RAM and that includes processes, libraries, and random minimal service daemons like sshd. I'm running my master on amd64 though to get optimal performance, shadowed on the chunkservers so that I can failover if needed, though in truth the amd64 box with ECC is the least likely thing to die and runs all the stuff that uses the storage right now anyway. The other suggestion to consider USB3 instead of SATA for storage isn't a bad idea. Though going that route means wall warts and drives as far as the eye can see. Might still be less messy than my setup, which has a couple of cheap ATX PSUs with ATX power switches, 16x PCIe powered risers for the HBAs (they pull too much power for the SBC), and rockwell drive cages to stack the drives in (they're meant for a server chasis but they're reasonably priced and basically give you an open enclosure with a fan). I'd definitely have a lot fewer PCBs showing if I used USB3 instead. I'm not sure how well that would perform though - that HBA has a lot of bandwidth if the node got busy with PCIe v2 x4 connectivity (SAS9200-16E) and with USB3 it would all go through 1-2 ports. Though I doubt I'd ever get THAT many drives on a node and if I needed more space I'd probably expand up to 5 chunkservers before I'm putting more than about 3 drives on each - you get better performance and more fault-tolerance that way. One big reason I went the distributed filesystem approach was that I was getting tired of trying to cram as many drives as I could into a single host and then dealing with some of the inflexibilities of zfs. The inflexibility bit is improving s
Re: [gentoo-user] Rasp-Pi-4 Gentoo servers
On Thu, Feb 27, 2020 at 4:25 PM james wrote: > > Yea, I was not clear. I'd run the mail-server, on a 'cluster' (4 or > more), not an individual pi-board unless it was beef up, processor and > ram wise. Gig E would also be on my list. > Unless you have some niche need I wouldn't generally run servers on Pis. The biggest issue with ARM is that all the cheap platforms are starved for RAM, and RAM is one of the biggest issues when running services. And of course the Pi in particular has IO issues (as do many other cheap SBCs but this is less of an ARM issue). The RAM issue isn't so many an ARM issue as a supply/demand thing - the only people asking for 64GB ARM boards are big companies that are willing to pay a lot for them. I do actually run a few services on Pis - DNS, DHCP, and a VPN gateway. That's about it. These are fairly non-demanding tasks that the hardware doesn't struggle with, and the data is almost entirely static so an occasional backup makes any kind of recovery trivial. The only reason I run these services on Pis is that they are fairly fundamental to having a working network. Most of my services are running in containers on a server, but I don't want to have to think about taking a server down for maintenance and then literally every IOT device in the house won't work. These particular services are also basically dependency-free which means I can just boot them up and they just do their jobs, while they remain a dependency for just about everything else on the network. When you start running DHCP in a container you have more complex dependency issues. A fairly cheap amd64 system can run a ton of services in containers though, and it is way simpler to maintain that way. I still get quick access to snapshots/etc, but now if I want to run a gentoo container it is no big deal if 99% of the time it uses 25MB of RAM and 1% of one core, but once a month it needs 4GB of RAM and 100% of 6 cores. As long as I'm not doing an emerge -u world on half a dozen containers at once it is no big deal at all. Now, if I needed some server in some niche application that needed to be able to operate off of a car battery for a few days, then sure I'd be looking at Pis and so on. -- Rich
Re: [gentoo-user] speedup remote portage
On Thu, Feb 27, 2020 at 2:27 PM james wrote: > > On 2/26/20 7:08 PM, William Kenworthy wrote: > > Hi, > > > > ��� due to space considerations on my laptop I have moved portage > > onto a > > network share (moosfs, mfsmounted) - slower but works fine.� However, > > being a laptop trying to update/install when out and about is both very > > slow and network intensive through a vpn - again, it works but is even > > slower to the point of not always being practical > > > > Is there a way to localise/speedup portage scanning parts of the > > update/install process? > > > A simpler, much less sophisticated update, what I do is > use emerge -f option to 'fetch only' option first. The selectively > update; or you can use a usb-3.1+ device for fast easy upgrades, due to > laptop limitations, but the communications data channel limitations > leave you at the mercy of the available wireless bandwidth characteristics. So, a few thoughts here: 1. Definitely make sure you're splitting out distfiles. Any new install does this already, but mixing up the tree and distfiles is mixing very different types of data with different requirements. 2. Consider whether you REALLY need to share all this stuff. The whole tree isn't THAT big, so it might be easier to just locally sync. If network bandwidth is an issue, consider setting up a local mirror so that you sync (and fetch distfiles) within your network when available, and then just sync directly from the internet when you're remote. 3. If you REALLY need that portage tree to be on the network, consider keeping just the metadata directory local. That can be generated on-demand or synced, but either way it needs to be updated anytime the repo itself is updated. Generally with rsync we distribute both together. I suspect that having local metadata will cut down on your network IO when you're doing dep resolution/etc. 4. Squashfs was mentioned. In theory that is a nice compact solution. In practice you might find it to be a pain to update since it is read-only. But you could update your network tree, then create a squashfs and stick that on your network, and then sync the squashfs file for local mounting. I'm not sure how well moosefs performs for this sort of thing in general. I personally use lizardfs to store large multimedia files and it works fine, but streaming multi-GB multimedia that ends up getting buffered is a very different access pattern than sourcing a bazillion ebuild scripts. I'd be interested in how you're liking moosefs for this sort of thing in general. For accessing tons of little files I think all your latencies matter, but in particular you want low latency to your master server as that is going to end up limiting every read/open you perform. You might be able to play with the cache settings on the client side, though at least in lizardfs I've found some of those to be buggy - they're generally passed as mount options. In the past I've had one main Gentoo box sync its repo from the web, and had other hosts in the LAN either bind-mount this (containers), or sync this over the network. Likewise I generally share my main host distfiles as a mirror which is preferentially used by all my other hosts - you don't get a 100% hit rate and the main host won't get a copy of missed files, but it probably cuts down on 80+% of the network fetching as long as the main host does its fetches before the other hosts do. If the main host lacks a distfile then the MIRRORS setting has a regular Gentoo mirror listed next. Honestly, unless you're running a completely private repo for whatever reason I would not try to go remotely mounting your repo over the internet from your moosefs server. I bet you could fetch anything needed faster from a regular internet mirror in that case, and if portage could use mirrors seamlessly over internet latencies chances are we'd all be doing that already. Pretty much every distro maintains a local repo on every host for this reason. -- Rich
Re: [gentoo-user] webrsync
On Sun, Feb 23, 2020 at 2:11 PM Ian Zimmerman wrote: > > emerge-webrsync has just eaten my /usr/portage :-( ;-( > > What has happened and how can I fix it? > Offhand I'm not sure why it broke, but the gentoo repo is completely disposable. When this sort of thing happens I generally just: cd /usr mv portage portage-old emerge-webrsync or emerge --sync You might consider moving the sync location out of /usr while you're at it. Also, if you haven't already done so get /usr/portage/distfiles outside of the repo directory. For new installs both of these are moved to /var in non-nested directories, but existing installs will continue to use their present locations. -- Rich
Re: [gentoo-user] Re: Compiling first and then installing using -K
On Tue, Feb 18, 2020 at 4:32 PM Nikos Chantziaras wrote: > > On 18/02/2020 21:22, Rich Freeman wrote: > > On Tue, Feb 18, 2020 at 2:06 PM Nikos Chantziaras wrote: > >> > >> It gets worse. The container reconfigured the keyboard shortcuts on the > >> host! After booting a container, alt+Fn or alt+left/right on the host > >> started switching to the linux text-mode console. I pressed alt+f2 to > >> bring up the plasma search, I ended up on TTY2... ha ha. > >> > >> Remember how I said I'll use nspawn from now on? I take that back. Let's > >> just say this thing is not even remotely production ready. > > > > Never had any issues with it, but I've never tried to use my host root > > as the input filesystem. > > I didn't do that either. Unless you mean the throw-away VM experiment, > which is not where this happened. Strange. I use it all the time and have never seen anything like that happen. That said, I can't remember the last time I fired up an X server on the host. -- Rich
Re: [gentoo-user] Re: Compiling first and then installing using -K
On Tue, Feb 18, 2020 at 2:06 PM Nikos Chantziaras wrote: > > It gets worse. The container reconfigured the keyboard shortcuts on the > host! After booting a container, alt+Fn or alt+left/right on the host > started switching to the linux text-mode console. I pressed alt+f2 to > bring up the plasma search, I ended up on TTY2... ha ha. > > Remember how I said I'll use nspawn from now on? I take that back. Let's > just say this thing is not even remotely production ready. Never had any issues with it, but I've never tried to use my host root as the input filesystem. I suspect the issue is that this is giving the container access to the host /dev, /sys and so on, and thus the container isn't ending up being contained. Normally you don't go mounting a host /dev inside a container image before launching it... -- Rich
Re: [gentoo-user] Re: Compiling first and then installing using -K
On Mon, Feb 17, 2020 at 6:31 PM Nikos Chantziaras wrote: > > On 18/02/2020 01:21, Rich Freeman wrote: > > On Mon, Feb 17, 2020 at 6:00 PM Nikos Chantziaras wrote: > >> Hm. I'm too chicken to try it because I'm not sure it does what I think > >> it does, but does the "--ephemeral" option pretty much do *exactly* what > >> Dale was asking about? Can you start your current "/" as a container > >> as-is, emerge packages in it and save them as binaries, then install > >> those from the outside, then shutdown the container and all is forgotten? > > > > Obvious way to test this would be to just set up a VM. It has the > > obvious advantage of always being in-sync with your host config. > > > > I think I might actually try playing around with this. I'm on zfs > > though so I'm not sure how it will perform. > > I just tested it in a throw-away Ubuntu VM running on ext4. It crashed > and burned due to disk space. It tried to duplicate the whole "/" with > zero error checks. So free space reached 0 but it still didn't abort. I > had to abort with ctrl+c. Free space was then 200MB (out of 20GB). I did > "du -sh /*" to find where all the GBs went, but it doesn't find it. > Hmm, if it just resorted to doing a cp it might have tried to copy the copy, or if it was really brain-dead it might not have limited itself to the root filesystem. Granted, the necessary files might not all be on one filesystem to begin with, but it would obviously have to avoid copying /proc and so on. I mean, it might have trouble with: -r 1 root root 128T Feb 11 14:31 /proc/kcore -- Rich
Re: [gentoo-user] Re: Compiling first and then installing using -K
On Mon, Feb 17, 2020 at 6:00 PM Nikos Chantziaras wrote: > > On 17/02/2020 21:46, Rich Freeman wrote: > > > > Well, if you decide to play with it I'll offer up: > > https://rich0gentoo.wordpress.com/2014/07/14/quick-systemd-nspawn-guide/ > > Hm. I'm too chicken to try it because I'm not sure it does what I think > it does, but does the "--ephemeral" option pretty much do *exactly* what > Dale was asking about? Can you start your current "/" as a container > as-is, emerge packages in it and save them as binaries, then install > those from the outside, then shutdown the container and all is forgotten? You know, I think that might actually work. Note that it depends on reflinks or snapshots for efficient operation, and I'm not sure what the full list of supported filesystems are. They do mention btrfs. I'm not sure if zfs is supported (zfs supports snapshots but clones would be needed here and those have some limitations, and zfs does not support reflinks). You'd obviously have to bind-mount your binary package directory - I think you could do that even using the same root as this would enable writes to that one path to escape the mount namespace and get into your host filesystem. Obvious way to test this would be to just set up a VM. It has the obvious advantage of always being in-sync with your host config. I think I might actually try playing around with this. I'm on zfs though so I'm not sure how it will perform. -- Rich
Re: [gentoo-user] Re: Compiling first and then installing using -K
On Mon, Feb 17, 2020 at 2:24 PM Nikos Chantziaras wrote: > > On 17/02/2020 21:05, Rich Freeman wrote: > > I wouldn't use a chroot for anything at this point - anything you can > > do with one you can do just as easily with a container, with more > > separation. They're just as easy to set up as well - I personally use > > nspawn to run my containers but I'm sure lxc is almost as simple and > > of course it doesn't require running systemd. > > nspawn seems very nice indeed. Haven't used it before, and that's simply > because I never heard of it :-) Now that I did, it looks like it's what > I'll be using from now on: > > https://wiki.archlinux.org/index.php/Systemd-nspawn Well, if you decide to play with it I'll offer up: https://rich0gentoo.wordpress.com/2014/07/14/quick-systemd-nspawn-guide/ That, and: ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=guest --directory=/path/to/container/root --network-bridge= KillMode=mixed Type=notify Though, if I didn't already have this recipe handy I'd be using nspawn units I suppose. Oh, this does require a bridge for your networking. If you're using KVM you probably already have one set up - the approach is identical. Rich
Re: [gentoo-user] Re: Compiling first and then installing using -K
On Mon, Feb 17, 2020 at 1:21 PM Nikos Chantziaras wrote: > > probably much slower.) A chroot or container on the other hand is > extremely lightweight. There's no virtualization involved (or very > little of it), so it should be pretty much as fast as a native system. Chroots and containers are exactly as fast as the native system, and don't involve any virtualization. In fact, on linux you can't NOT run a process in a chroot or container. Every process has a root directory and a set of namespaces applied to it, including init, and every new process just inherits the settings of the process that execed it. All linux users are essentially using at least one container. As such, running more than one container doesn't involve any kernel behavior that running a single container doesn't involve. Now, it is true that if you're running multiple containers you're more likely to have multiple copies of glibc and so on in RAM, and thus there is a memory overhead, though that applies system-wide, and not just to the processes running inside the additional containers. Maybe the one bit of overhead is the first time you launch a particular process in a particular container any shared libraries it uses will have to be loaded into RAM, while on the host there is a decent chance that some of them are already in RAM. We're really splitting hairs at this point, however. I wouldn't use a chroot for anything at this point - anything you can do with one you can do just as easily with a container, with more separation. They're just as easy to set up as well - I personally use nspawn to run my containers but I'm sure lxc is almost as simple and of course it doesn't require running systemd. Getting back to the original topic - you can just build binary packages for stuff like qt without using a container, but if you do so you won't be able to build more than one layer of dependencies. It still cuts down on the merge time considerably, but obviously not as much as it does if you build everything ahead of time. -- Rich
Re: [gentoo-user] lxc filling /dev
On Sun, Feb 16, 2020 at 7:57 PM William Kenworthy wrote: > > 2 ~ # lxc-attach -n mail -- bash -c "df -h" > none492K 320K 172K 66% /dev > du and ls -al do not give any clues, the host /dev is normal and all > running lxc instances do it, but at different rates Are you running ls -al from INSIDE the container? If you're running it on the host you won't see anything because it is almost certainly in a separate mount namespace, and so it is invisible from the host. In particular, any files you see in rootdir/dev from the host are NOT visible in the container, and vice-versa. I don't use lxc, but if I had to take a wild guess your /dev isn't being properly initialized inside, and some typical device node is being created as a regular file and stuff like "echo foo > /dev/null" is actually writing to a real file there, filling up the tmpfs. Try: lxc-attach -n mail -- bash -c "ls -l --recursive /dev" Or launch an interactive shell inside the container and just poke around in there. I have no idea what the "lxc way" to launch a shell is, but you can always use: nsenter --target --all /bin/bash (where is the pid on the host of a process inside the container) nsenter is part of util-linux -- Rich
Re: [gentoo-user] Number of open Bugzilla bugs
On Sat, Feb 15, 2020 at 8:39 AM Kai Peter wrote: > > On 2020-02-15 01:46, Rich Freeman wrote: > > On Fri, Feb 14, 2020 at 7:25 PM Marc Joliet wrote: > >> > > > >> like, say, HAL: https://bugs.gentoo.org/401257. > > > > That isn't a HAL bug - it is a bug for a relatively recent version of > > pm-utils. > This is one of the issues with a general discussion of overridden > points: switch to an unimportant detail (of an example). Actually, I only brought it up because it actually is a pretty good illustration with the problem here. It is really easy to run a query and get a bunch of results and extrapolate from there. The problem is that when you actually start looking at the results you're returning they are a lot more nuanced. I am certain that there are plenty of open bugs that are no longer valid or which were resolved upstream and so on. The problem is that it takes a fair bit of effort to actually identify these without just closing out a ton of bugs wholesale which are still valid and relevant. > I couldn't agree > with some changes in the direction Gentoo is going. It looks a bit like > swarm intelligence. That is hardly a change. Gentoo has been basically operating like a swarm intelligence for most of its existence. That was less the case in the drobbins days, but that was a long time ago. Now Gentoo is largely an amalgamation of volunteer developers working on their personal interests with some common rules and a very minimal shared set of values. It has been this way for a good 10-15 years really. It is a model that actually works pretty well for many things. Closing bugs isn't one of them, however, as: 1. Many packages were put there by somebody who was interested in them at the time and is either no longer around or is no longer interested, and these can be targets of bugs. 2. Any dev can work on whatever they want, not necessarily the stuff that most of the bugs are open for. 3. Since anybody could take up an interest in anything at any time, nobody can say with confidence that a particular bug will or won't get fixed in the next week. Maybe nobody today is interested in fixing it, but somebody new could show up tomorrow and work on it. It is a very bottom-up approach. It is basically the opposite of something like Agile where you target some MVP and everybody focuses on a narrow scope to get a bolus of work done. > Nothing personal, just IMHO. Will I get banned from this list now? Nobody gets banned from lists for having an opinion. It is pretty hard to get banned in general but it tends to happen when it starts to turn into harassment/spam or gets personal. Basically see the code of conduct. Having an opinion on something isn't really a problem. Misinformation can become a problem but it has to be pretty bad before anything is done. > > > > If an issue no longer exists then of course the bug should be closed. > Why doesn't this not happen? That goes back to the example above. To know if an issue still exists somebody needs to investigate. The reason that no developer has investigated all those ancient bugs is the same as the reason you personally haven't investigated them. There just isn't any particular individual interested in doing it. If somebody wanted to propose a bug cleanup crew project that went looking to close out old bugs with some defined rules around what they're going to do, I bet there would be support for it, as long as the rules seemed reasonable. I doubt anybody is going to get push-back for closing bugs for stuff that is no longer in the tree, or which doesn't apply to current versions in the tree, and so on. Where you'd hit controversy is stuff that probably is still valid but just has nobody interested in fixing it right now. But, IMO it is a lot of work for little reward. I think some of this is the whole "sort vs search" email workflow argument. Different people have different preferences, but for stuff that probably won't get looked at much it makes more sense to not invest a ton of effort into housekeeping. If you spent a lot of time closing out 30 out of 40 bugs for some unmaintained package, the other 10 will still sit and rot for quite a while. If the bugs were bad enough to be causing serious issues chances are the treecleaners would have already caught it. Things like security bugs aren't treated the same as "package randomly doesn't build in this 0.1% edge case with a simple workaround." -- Rich
Re: [gentoo-user] Number of open Bugzilla bugs
On Fri, Feb 14, 2020 at 7:25 PM Marc Joliet wrote: > > personally, I care about closing bugs that are done with or > can't be acted upon. As do I. > I honestly think it would > be best to close bugs that are just not applicable anymore, e.g., for ebuilds > or versions of packages that have not been in the tree for a looong time, Certainly. > like, say, HAL: https://bugs.gentoo.org/401257. That isn't a HAL bug - it is a bug for a relatively recent version of pm-utils. I'm not sure if the bug is still an issue, but it could very well be. And that is the issue with just closing bugs because they're old. They can still be issues. If an issue no longer exists then of course the bug should be closed. -- Rich
Re: [gentoo-user] Number of open Bugzilla bugs (Was: Question about gentoo-sources kernel release versions)
I just wanted to preface this that my intent has been to be frank with you, and I appreciate that you have done the same. I appreciate that you have a different perspective and I think it is actually a pretty common one. So, take what I say for what it is worth, and I certainly won't be offended if you remain in disagreement, and if you feel that a different distro would better suit your needs I certainly wouldn't find that offensive at all either. Also, bear in mind that I speak for nobody but myself but I do think that some of my observations are fairly accurate in general, and most seem to agree with your own. On Mon, Feb 10, 2020 at 3:48 PM Michael Jones wrote: > > On Sun, Feb 9, 2020 at 8:07 PM Rich Freeman wrote: >> >> You could have jumped through all the required hoops and still had it >> ignored. > > That's pretty horrible, honestly. Why isn't Gentoo doing better than that? Simple, "Gentoo" can't do anything - it has no hands, no feet, no brain. It is a collection of volunteers. They work on what they wish to work on. That means that some things end up being worked on by nobody. That said, plenty do work on proxy maintaining and IMO are doing better in this area than has been at some points in the past. > Yes, yes, Gentoo is run by volunteers, so on and so forth. But it's entirely > possible (and regularly accomplished in other organizations) to emphasise a > culture that actively tries to not ignore low hanging fruit for so long. Most devs work on the things that interest them. If your contribution is in an area that no devs find interesting, then you're offering low-hanging fruit that doesn't taste very good. It might be easy to grasp, but nobody wants to eat it. That is just an analogy. I'm not saying that NOBODY will find the contribution useful. But, it could very well be that no devs interested in proxy-maintaining happen to find it personally useful. >> > I'm not attempting to be contradictory for the sake of being >> > contradictory, but the situation is significantly more complicated >> > than what you said >> >> Not sure how that could be. I literally said "If you do report an >> issue it might or might not get fixed." I'm pretty sure that hits all >> the examples you supplied, being that it was basically a tautology. >> There simply are no guarantees. > > Honestly, this conversation is just making me less interested in contributing > to Gentoo in the future. Not only do 1-liner pull requests that fix broken > packages get rejected / not fixed for a year, but now I'm being replied to > with word-games when I try to discuss the issues that I face as an end-user. I completely get that you find the absence of any guarantee frustrating. I was just telling you the truth. You could do everything right and end up being completely ignored. This seems to match your own observations. That isn't meant as a word-game. Just as frank conversation. > The problem is that you're lying to people if you keep a bug in bugzilla open > for 10+ years. > > You know it won't be worked on, they know it won't be worked on. So just > close the bug. Actually, I don't agree with this. Leaving a bug open makes no statement at all about whether it will be worked on. I guess it is a bit like the debate about whether agnostics are atheists. :) It is entirely possible a 10 year old bug will get worked on - it just takes somebody interested to start looking at it. >> >> > Surely if something hasn't been addressed in 20 years, it won't be? >> >> If nobody can bother to fix 20 year old bugs on some piece of >> software, why would you be running it today, and thus why would you be >> searching for bugs for it? > > > Then why is it still in the Gentoo package repository? Maybe the package works fine for the most part. Just because a package has a bug doesn't mean that it is useless. Why remove a mostly-working piece of software over a minor bug? > > If it's not in the Gentoo package repository, why is there an open bug in > bugzilla about it? You can file a bug to request having something added to the repository, so there are actually many bugs pertaining to software that isn't in the repo. >> The only person getting reminded is the requester. A maintainer that >> is deliberately ignoring bugs will be sending bot mail to /dev/null. >> If requesters start pinging devs in other ways to get their attention >> about such bugs, that seems more likely to just have these devs become >> more aggressive about blocking such attempts from users to >> communicate. That's probably part of why so few devs are on this list >> at all. :) > > Why is that person allowed to be a maintainer for that package then? Sounds > like a pretty co
Re: [gentoo-user] Number of open Bugzilla bugs (Was: Question about gentoo-sources kernel release versions)
On Sun, Feb 9, 2020 at 7:23 PM Michael Jones wrote: > > On Sun, Feb 9, 2020 at 5:43 PM Rich Freeman wrote: >> >> Bugs get closed all the time. Bugs also get opened and and linger all >> the time. I couldn't tell you the ratio, but that is the nature of >> things. >> >> If you don't report an issue, and nobody else is aware of it, I can >> pretty much guarantee that nobody will fix it. If you do report an >> issue it might or might not get fixed. That's the nature of the >> beast. > > Or in my case, I sometimes post 1-line pull requests to the Gentoo > github, which fix packages being unable to compile, which get rejected > because I didn't jump through enough hoops, and the bug remains > unfixed for over a year after I open the PR. I stopped posting PRs > after that, since it's a waste of my time. You could have jumped through all the required hoops and still had it ignored. > I'm not attempting to be contradictory for the sake of being > contradictory, but the situation is significantly more complicated > than what you said Not sure how that could be. I literally said "If you do report an issue it might or might not get fixed." I'm pretty sure that hits all the examples you supplied, being that it was basically a tautology. There simply are no guarantees. > Add to that, Gentoo has *so many bugs* that your bug tracking > software, when told to simply "Give me all of the bugs" refuses to > actually do so. It generally isn't a problem unless you run queries with no filters at all. Sure, if you literally ask it for "all of the bugs" you won't get them. So don't ask it for all of them. I'll note that even if we closed all the bugs you'd still get the same error unless you only asked for OPEN bugs. :) And if what you want is all old bugs closed, you can just filter by date, and then you'll get all the benefits of those bugs being filtered as far as query response limits are concerned. > Why should I continue opening new bugs, (or even better, provide > patches) when I have new problems? Simple. If you provide a patch or bug you're more likely to get a response than if you don't. There are no guarantees either way. > I don't see the problem as Gentoo not knowing that there are issues > that should be tracked. I see it as a problem of Gentoo can't engage > with it's user community in an effective way. And I see having over > 10,000 open bugs as one of the barriers between effective user > engagement and what we have today. I don't see how open bug counts are really the problem here. Let's suppose we put in a bot that closes all bugs 5 minutes after they are opened, unconditionally, and locked them. We'd have nearly zero open bugs at all times that way. Obviously that wouldn't increase user engagement. I don't think your frustration is really that bugs that were opened 15 years ago by somebody else are still open. I think your frustration is that a bug you open tomorrow is reasonably likely to not be fixed. CLOSED-EXPIRED or whatever isn't the status you want - you want RESOLVED-FIXED. And that is completely normal. However, the only thing that is going to lead to this is people actually fixing bugs, not bots playing with statuses. Gentoo will NEVER engage with its user community in a way you consider effective by these sorts of standards. If Gentoo had 50,000 developers that were all super-active we'd STILL have this problem, because such a high level of quality would bring in millions of users, and they'd be filing millions of bugs, and those mere 50,000 developers would still not get everything resolved. Granted, I won't say that this will scale up infinitely but I think that even high-quality or professional distros still end up with more bugs than they can really fix. >> Honestly, I'm not sure how having bots beg bug reporters about letting >> their bugs be closed relentlessly (albeit at a very slow rate) until >> they finally stop responding is going to improve things. Somebody >> reports an issue and is frustrated that nobody does anything about it. > > Is there ever a time cutoff, after which a bug should automatically be > closed, in your opinion? No. > Surely if something hasn't been addressed in 20 years, it won't be? If nobody can bother to fix 20 year old bugs on some piece of software, why would you be running it today, and thus why would you be searching for bugs for it? > Either: > > 1. The bug hasn't been acted on in the previous 5 years on bugzilla, > but maybe it's been fixed and the original reporter / developer > forgot to do anything in bugzilla about it. Or no one realized it was > fixed. This kind of thing happens all the time. Chances are if anybody is maintaining the package then it will eventually get noticed and fixed. If nobody is
Re: [gentoo-user] Re: Question about gentoo-sources kernel release versions
On Sun, Feb 9, 2020 at 4:04 PM Michael Jones wrote: > > On Fri, Feb 7, 2020 at 2:25 PM Rich Freeman wrote: >> >> On Fri, Feb 7, 2020 at 2:34 PM Michael Jones wrote: >> > >> > Honestly I'd rather see the 30 day stabilization policy apply to LTS >> > kernels vs. being stabilized faster. Maybe I'm once bitten twice shy. >> >> One issue here is that the kernel upstream doesn't really consistently >> label kernels for bug criticality (including security bugs), or >> severity of regressions. >> > > But Rich, Linux-4.19.97 was released on Jan 17th, and then > gentoo-sources-4.19.97 was released on Jan 18th, whereas > https://bugs.gentoo.org/706036 was acknowledged to be fixed by > Linux-4.19.99 on Jan 29th, and it's now Feb 9th and no newer > gentoo-sources package has been stabilized. > > So we've got some inconsistency in how the gentoo-sources package is > stabilized. But, Michael, I really have no idea how what you said in any way contradicts what I said... Per the Gentoo kernel page they generally follow the 30 day policy except for security issues. My guess is that somebody thought 4.19.97 contained a security fix, and 4.19.99 did not. The bug you linked seemed to have nothing to do with security. Pretty much EVERY kernel release fixes bugs. I get that you want stuff that personally affects you fixed faster, and stuff that doesn't personally fixed you given careful examination before release. I don't see how you're going to get that without doing your own QA. > I'm not saying the current cadance isn't the right one, and i'm not > saying it is. I'm just saying in this particular situation, it didn't > fit one specific end-users needs, and maybe that's fine, and maybe > it's not. I'm not the one putting in the work, so I'm not going to say > the project *should* take an action. Just point out that there was a > problem from my perspective. Well, obviously I'm sympathetic. If I thought that Gentoo's release cadence fit my needs I'd be running Gentoo kernels. :) This topic has been discussed a few times but not recently that I am aware of. IMO they're doing better now than they were the last time I actually used a Gentoo kernel (being mindful that the "they" likely aren't the same people, and the better/worse part is subjective in any case). >> I mean, you could just close bugs without resolving them after a >> period of time. That would make the open bug counts look better, but >> it wouldn't change the actual quality of the distro, and would just >> tend to hide problems packages that are still in the repo. > > That's 655 bugs with a last modified date of January 1st 2010 or older. > > And 2461 bugs with a last modified date between January 1st 2010 and Jan 1st > 2015. > > Please be aware that at least one person who uses and (minorly) > contributes to Gentoo finds this off putting, bizarre, and incredibly > difficult to interact with. It's intimidating to new users and old > users of Bugzilla alike, and is a constant source of uncertainty and > confusion on where to find something or if the bugzilla platform is > actually even the right place to file an issue. I'm sure lots of people find it bizarre. I'm also sure that lots of people would find doing what you propose bizarre also. It is just a matter of taste. Granted, I think most people have bad taste, and that most people would probably think I have bad taste, so there is that. :) If you want to pretend that there aren't any bugs older than 10 years, then just filter your searches and you won't see them. Just pretend they don't exist. Problem solved. > > From my perspective, Bugzilla is where issues go to linger forever, > because it's rare for me to see a bug opened there get any attention > at all, such as being closed or commented on, much less transition to > the confirmed state. Bugs get closed all the time. Bugs also get opened and and linger all the time. I couldn't tell you the ratio, but that is the nature of things. If you don't report an issue, and nobody else is aware of it, I can pretty much guarantee that nobody will fix it. If you do report an issue it might or might not get fixed. That's the nature of the beast. Honestly, I'm not sure how having bots beg bug reporters about letting their bugs be closed relentlessly (albeit at a very slow rate) until they finally stop responding is going to improve things. Somebody reports an issue and is frustrated that nobody does anything about it. Will reminding them that we didn't do anything about it in 5-10 years improve how they feel about the issue? If they reply that it still is an issue, will it help that we reply again in another 5 years to ask if it is still an issue help? It seems like picking at a scab when the only people paying attention to a bug
Re: [gentoo-user] Re: Question about gentoo-sources kernel release versions
On Fri, Feb 7, 2020 at 2:34 PM Michael Jones wrote: > > Here's an example of how 4.19.97 being stabilized might have exposed users to > functionality breaking bugs: https://bugs.gentoo.org/706036 > > Honestly I'd rather see the 30 day stabilization policy apply to LTS kernels > vs. being stabilized faster. Maybe I'm once bitten twice shy. One issue here is that the kernel upstream doesn't really consistently label kernels for bug criticality (including security bugs), or severity of regressions. So, in general any newer release should only make things better, but if a particular release made things much worse you'd only know from general discussion on high-volume lists. I follow upstream directly and just tend to hold off a day or two before updates, and look for signs of chatter before doing so. That is hardly optimal. A company like RedHat just hires a ton of kernel engineers to basically do their own QA on top of upstream's - they can stay on top of those sorts of problems. I'm sure the Gentoo kernel team does a better job than I personally do, but I doubt they can sink as much time as that. > As an aside: The gentoo bug tracker has way too many open bugs > (Thousands and thousands of them opened over 10 years ago), and the > search interface is... frustrating. Took me over 5 minutes to find > that bug despite being a commenter on it. Does anyone know if there's > any plans for that situation to change in any way? I doubt it, but the search interface is probably more likely to change than the number of open bugs. I mean, you could just close bugs without resolving them after a period of time. That would make the open bug counts look better, but it wouldn't change the actual quality of the distro, and would just tend to hide problems packages that are still in the repo. Obviously fixing bugs would be the ideal path, but we're volunteer driven, so that is up to the volunteers. I mean, we could just remove any package from the repo that has open bugs older than a certain time, but that seems likely to just end up with a repo without any packages in it. Unless the bugs have security implications or are causing impact to updates to other packages there usually isn't any policy requiring that they be fixed. I'm sure lots of people would be up for improving the UI, though that still requires volunteer work. Also, if the fix involves switching away from bugzilla that is a much bigger hurdle, and one challenge is that Gentoo doesn't like to host stuff written in Java/Ruby, which tends to exclude a lot of popular stuff. I don't sysadmin webservers for a living, so I won't comment on whether that policy is a good one or a bad one, but I suspect those behind it know a lot more about maintaining webservers than I do... -- Rich
Re: [gentoo-user] python 2 deprecation
On Sat, Jan 25, 2020 at 12:20 PM james wrote: > > I have removed all syntax/expressions from > /etc/portage/make.conf > > related to python; hoping python-2 would just die a natural, slow and > painless (for me) death, on this ancient gentoo install. > > Any and all comments related to what to check/do are most welcome, > related to removing python-2* Disclaimer: I'm not involved with the python team, so details could change around this... Right now that is your best bet. If you don't care about python, avoid specifying python-related flags in your config files. Then portage will just apply defaults wherever possible. It doesn't always work, but that is mainly when packages can only support one version of python at a time (which isn't too many of them). As long as you aren't going crazy with accepting keywords then the profiles should be updated so that python versions are removed only after all the packages that depend on them have been removed, ideally with newer v3 packages having been stabilized. I suspect this process will drag out for months as everybody involved realizes that migrating can be painful but it has to be done, so expect packages to steadily get masked and to see v2 stuff start disappearing once the packages that depend on them are gone. Users can always do overlays if they want to maintain their own v2 forks, but I suspect this will be a lot of work. Maybe somebody will step up and do it but I'm not seeing many signs of it. Some packages in Gentoo just aren't super well-maintained and so upstream might have v3 packages that aren't in the repo. Bugs and especially pull requests will doubtless be welcomed in these cases. Just about every major distro is pushing to get rid of v2 so upstreams that are dragging their feet are going to have motivation to fix things or they just won't have any users left. I'm sure at some point there will be a news announcement if users actually need to do anything. I doubt the rug will just get pulled out from under you. Even the package masks can be unmasked for 30 days until the packages start getting pulled which gives you a bit of pain-free time to deal with it. -- Rich
Re: [gentoo-user] Portage unable to write to /usr/portage
On Sat, Jan 11, 2020 at 9:37 AM Dale wrote: > > I run emerge as root but the proper permissions, or at least was several > years ago, is portage:portage and rwx access for both. This is my settings. By default portage drops permissions to portage:portage during most phases, including reading the repository (which requires executing ebuilds and eclasses even for what are intended to be read-only functions). If every file in the repository isn't readable by portage, then you will have problems. Portage also drops permissions during syncing, so if you have files that aren't modifiable by portage then that can also cause issues if you sync. In general it is best if everything is 664/775 portage:portage in the repo. It is pretty easy to mess this up if you try to update the repo manually, such as by running git pull as root in a git repo. If you use emerge --sync to update then you won't have this problem. If you've messed up permissions you can go fixing them with chown/chmod, or you can just delete the whole repository directory tree and do an emerge --sync to re-create it. > If I recall correctly, if you > add your user to the portage group, you can run a lot of commands as > user. Unless you have needed files set to be non-readable by everyone you should be able to run read-only portage commands under any user, like emerge --pretend. If you've locked anything down then being in the portage group would obviously help with that, assuming you've done the locking-down correctly. > I think you have to be root to actually install something tho. Obviously. Nothing in portage is suid so unless you've modified your system to have a very non-conventional security model you can't go installing almost anything as non-root. Portage drops permissions by default during most operations, but not during install or running install-related scripts. -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Mon, Jan 6, 2020 at 10:05 AM Dale wrote: > > The drive I have is likely > done on the drive itself, device managed, which is good for me. Really the ideal situation are the Host Aware drives. I have no idea what percentage of the markets they make. They fall back to being device managed if the host doesn't do anything to manage them. Your drive is device managed. See the link I posted earlier to the mfr info. > I've been noticing that too. Only bad thing is, I can't always tell > what is in the enclosure. Sometimes the info is given but sometimes > not. I've also seen a few people complain that what they got was not > the model of drive they thought. Yeah, there are no guarantees as to what you'll get if you go the shucking route. If you absolutely need a certain amount of cache or a red firmware then you're just going to have to pay double to get that guarantee. For my application I'd definitely prefer the red firmware, but it isn't really the end of the world if the drive takes a few seconds to timeout on that one failure every 5 years. I'm not going to pay double just to guarantee a particular model. If it were $20 more that would be another matter, but we're talking $180 vs $350 here. The other gotcha is that if you want to do a warranty replacement at some point you're going to have to put it back in the enclosure to do so. That means hanging onto enclosures, and is of course a bit of a pain besides. Again, with a 50% reduction in cost you'd need to have a lot of drive failures to be worth worrying about, especially since I believe drive warranties are getting shorter anyway. If you're in a typical enterprise situation then you're going to want to just buy the bare drives with the standard warranty/etc. If you buy in bulk chances are you're getting a discount anyway (and then maybe you can get them in caddies or whatever). The Best Buy deals are sporadic anyway and limited in quantity - you could never run a data center that way. That's why they're priced the way they are... -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Mon, Jan 6, 2020 at 9:18 AM Dale wrote: > > Rich Freeman wrote: > > On Mon, Jan 6, 2020 at 8:25 AM Mick wrote: > >> If they are used as normal PC drives for regular writing > >> of data, or with back up commands which use rsync, cp, etc. then the disk > >> will > >> fail much sooner than expected because of repeated multiple areas being > >> deleted, before each smaller write. I recall reading about how short the > >> life > >> of SMR drives was shown to be when used in NAS devices - check google or > >> youtube if you're interested in the specifics. > > Can you give a link - I'm not finding anything, and I'm a bit dubious > > of this claim, because they still are just hard drives. These aren't > > SSDs and hard drives should not have any kind of erasure limit. > > > > Now, an SMR used for random writes is going to be a REALLY busy drive, > > so I could see the drive being subject to a lot more wear and tear. > > I'm just not aware of any kind of serious study. And of course any > > particular model of hard drive can have reliability issues (just look > > up the various reliability studies). > > > > I ran up on this article however, it is a short time frame. Still might > be a interesting read tho. > > https://blogs.dropbox.com/tech/2019/07/smr-what-we-learned-in-our-first-year/ That article makes no mention of reliability issues with SMR. In fact, they mention that they want 40% of their storage to be on SMR by now. Clearly they wouldn't be doing that if the drives failed frequently. Note that they did modify their software to have write patterns suitable for SMR. That is the key here. You absolutely have to engineer your application to be suitable for SMR, or only choose SMR if your application is already suitable. You can't just expect these drives to perform remotely acceptably if you just throw random writes at them. > I'm still a bit curious and somewhat untrusting of those things tho. > Regular hard drives go bad often enough as it is. We don't need some > fancy unknown thing inserted just to add more issues. Sort of reminds > me of the init thingy. Each thing added is another failure point. Obviously they're relatively new, but they seem reliable enough. They're just not suitable for general purpose use. > I'm going to test my ebay skills and see if I can find some non-SMR > drives. It sounds like some require some research to know if they are > or not. :/ That's pretty simple. Find a drive that looks reasonable price/capacity/etc-wise. Then just google the model number to confirm it isn't SMR. If you're in the US though you're probably best off shucking drives from Best Buy these days. A drive that costs $350 as a bare drive will get sold for $180 in a USB enclosure. I think it is just market segmentation. They want to get top dollar from enterprise users, and they aren't going to be shucking drives from Best Buy bought on "limit 1 item per customer" sales. By shucking I'm getting 12TB red drives for less than the cost of a 6TB green drive. Just be aware that if your PSU is old you'll need to tape over some of the SATA power pins. New PSUs - even cheap ones - haven't given me any trouble. I'm sure there are more up-to-date guides as these days the drives are 12TB, but here is the gist of it: https://www.reddit.com/r/DataHoarder/comments/7fx0i0/wd_easystore_8tb_compendium/ If you aren't in the US I have no idea whether equivalent deals are available. That subreddit is a good place to go for info on cheap hard drives though. -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Mon, Jan 6, 2020 at 8:25 AM Mick wrote: > > If they are used as normal PC drives for regular writing > of data, or with back up commands which use rsync, cp, etc. then the disk will > fail much sooner than expected because of repeated multiple areas being > deleted, before each smaller write. I recall reading about how short the life > of SMR drives was shown to be when used in NAS devices - check google or > youtube if you're interested in the specifics. Can you give a link - I'm not finding anything, and I'm a bit dubious of this claim, because they still are just hard drives. These aren't SSDs and hard drives should not have any kind of erasure limit. Now, an SMR used for random writes is going to be a REALLY busy drive, so I could see the drive being subject to a lot more wear and tear. I'm just not aware of any kind of serious study. And of course any particular model of hard drive can have reliability issues (just look up the various reliability studies). > Personally, I would only use such a drive for 'keepers'. Say, films I intend > to write once and watch many times, ripped music albums, family photos, etc. > For OS files and other temporary backups I would use a normal PC drive. Certainly I would never use an SMR for an OS or /home. Backups should be fine, as long as you're using a sequential backup file format. tar/duplicity should be fine. Dar is probably fine but I'd need to check (I think it just writes the index to the end, so the seeking issues are on reading and not writing). Even zip/etc is going to be fine. What is going to be a problem is anything that just replicates the original data as all the separate files/directories that exist on the original drive, like rsync/rsnapshot/etc. Those formats are of course attractive because the backup is just a replica of the original, but they involve random writes. Most formats that just create a bunch of files named archive-001.foo that need a special command to restore are going to be fine. I personally haven't encountered a need to consider an SMR drive as you can shuck those 12TB Easystore drives for something like $180 on sale, at least in the US. Those are just standard drives (often with red firmware). I couldn't even use them for my multimedia as I'm storing that stuff on lizardfs right now and that breaks everything into chunks. Granted, I don't rewrite it often but unless zfs is SMR-aware it is still going to be writing lots of modest-sized files as the original files get chunked up and distributed across the nodes. On the disk lizardfs data just looks like a browser cache, with everything in numbered files about 60MB in size in my case. The files also appear to turn over a bit during rebalancing. -- Rich
Re: [gentoo-user] Re: External hard drive and idle activity
On Sat, Jan 4, 2020 at 10:13 PM Ian Zimmerman wrote: > > On 2020-01-02 14:12, Rich Freeman wrote: > > > > Device Model: ST8000AS0003-2HH188 > > > > > > I recall reading about SMR but can't recall the details of what it is. > > > far as I know, this is just a basic 8TB drive. > > > > This is an SMR drive. You should DEFINITELY read up on what they are. > > How do you know? The identfying string doesn't appear in the kernel > source (I did a case-insensitive recursive grep). > https://www.seagate.com/www-content/product-content/hdd-fam/seagate-archive-hdd/en-us/docs/100827317b.pdf These drives provide the following key features: • 8TB capacity for efficient storage-per-slot. • Affordable efficiencies with 2TB-per-disk Drive Managed SMR-based hard drive technology. ... -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Fri, Jan 3, 2020 at 8:51 AM Spackman, Chris wrote: > > udisksctl power-off --block-device /dev/sdx > > I didn't see that command mentioned in the thread yet. I've been using > it, after umount, for about 8 months for roughly weekly backups and > some misc storage. So far, I've not seen any problems with it. The drive > immediately shuts down, and there haven't been any data or performance > issues. > > But because no one else has mentioned it, I wonder udisksctl is not the > best tool in this case? > I suspect that running this command will either not power off the drive in this case, or if it does it will have the exact same impact as pulling the plug. If the drive is unmounted then the kernel has received guarantees from the drive that all data is written persistently. It is unlikely that a drive will lose the data after this point, but if has some bug then short of a firmware update I doubt there is any reliable workaround short of leaving it on forever. -- Rich
Re: [gentoo-user] Dracut and how to specify names
On Fri, Jan 3, 2020 at 5:57 AM Dale wrote: > > Can you post a ls -al /boot for both kernels and images? That way I can > see how it names them when doing it your way. If I can make sense of > it, I may try doing it that way. Thing is, it'll change eventually > too. lol I use the standard kernel names: config-4.19.92 initramfs-4.19.92.img System.map-4.19.92 vmlinuz-4.19.92 /lib/modules/4.19.92 I create the initramfs using: dracut "" 4.19.92 Dracut is going to need the path to the modules more than anything else, so I suspect it will work if you substitute 4.19.92 with whatever the path of your modules directory is, within /lib/modules. Also, could you actually post the command lines you're using? You posted 4 fairly long emails elaborating on how everything isn't working right, and I don't think you actually posted a single dracut command line. When something isn't working right it is usually best to start with what you're actually doing, along with what is happening and what you expected to happen. You mainly covered the last bit of those three but left out most of the first two. I actually use a script to do my kernel updates - this is intended mainly for bumps and isn't entirely suitable when I need to change things, in which case I usually just build manually following the same steps: #!/bin/bash cd /usr/src/linux || exit git pull || exit rm -rf /var/tmp/linux || exit export KBUILD_OUTPUT=/var/tmp/linux make O=/var/tmp/linux oldconfig || exit nice -n20 make O=/var/tmp/linux -j12 -l20 || exit make O=/var/tmp/linux modules_install || exit make O=/var/tmp/linux install || exit emerge @module-rebuild || exit NEWVER=$(make --no-print-directory kernelversion) || exit dracut "" $NEWVER || exit grub-mkconfig -o /boot/grub/grub.cfg (This does all the building in /var/tmp and leaves me with a clean kernel source directory. That is actually the upstream-recommended way but it does create the issue that if any package that builds kernel modules gets updated it will fail. I usually just delay updating these packages until I do my next kernel update, but I can just run this script again to re-create /var/tmp/linux with the necessary files to build further modules. Note that you need a few GB in /var/tmp for this to work, and this script doesn't clean up - I usually want that directory left for any module updating, and it gets cleared on reboot anyway which usually follows a kernel update. This works great on tmpfs if you have the space. Note also that I'm using upstream stable vanilla sources - I checkout a longterm branch which is what is getting pulled at the start. This should work with gentoo sources as well if you just tweak the start. I like to maintain more control over what kernel I'm following as I tend to use out-of-tree modules like zfs, or experimental ones like btrfs, or newer CPUs like Ryzen - for one reason or another just following random stable releases is problematic.) -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Thu, Jan 2, 2020 at 7:43 PM Wol wrote: > > And yes - the 8TB capacity gave it away - I think the largest "normal" > drives available are 4TB at present ... anything bigger must be shingled. > I have several non-SMR drives in the 8-12TB range, shucked from those WD external drives that occasionally are on sale at Best Buy. While the drives inside are white-label people have managed to socially engineer the support team at WD to reveal that they aren't SMR. But, you definitely need to check. -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Thu, Jan 2, 2020 at 1:41 PM Dale wrote: > > Rich Freeman wrote: > > > > Out of curiosity, what model drive is it? Is it by chance an SMR / > > archive drive? > > Device Model: ST8000AS0003-2HH188 > > I recall reading about SMR but can't recall the details of what it is. > As far as I know, this is just a basic 8TB drive. This is an SMR drive. You should DEFINITELY read up on what they are. For reads they're completely normal. For sequential writes to unused space they're completely normal. For random writes or overwrites they are significantly different from traditional hard drives. They work a bit like an SSD in the sense that blocks are arranged into larger erase regions. Within a region blocks can only be written sequentially. If you want to overwrite one block in the middle of a region, the drive will read the entire region into RAM, then write the entire region sequentially with the overwritten block to a new spot on the disk. This is just like in an SSD where if try to overwrite a block in a region with any unTRIMmed blocks the drive must read the entire region, erase the region, and write the modified region. Except that in an SSD those extra reads/writes operate with SSD access times. With an SMR drive those extra reads/writes operate with hard drive latencies, so they're MUCH more costly. For backup use they're usually fine, IF you're writing in a sequential file format that is appended to. If you're using rsync to do your backups then that isn't what you're doing and you're probably paying a heavy penalty. If you were doing incremental backups using tar/duplicity/whatever then you'd probably be fine. Some filesystems might be optimized for these drives to reduce the amount of overwriting in place. I haven't looked into it. I'd expect a log-based filesystem to work fairly well, though those can have high levels of fragmentation which is better suited for SSD than SMR. These drives all have fairly active firmware that manages this special overwrite process so that they can be used with operating systems that are naive to how they work. I wouldn't be surprised if this is what is causing the drive to be active after you unmount it. In theory it should be harmless to power it off. However, leaving it powered on probably will improve its performance as it can take care of any garbage collection before the next time you use it. If whatever journal it is using to speed things up gets full then you'll feel the full brunt of any write penalties until it is flushed. You might want to seriously consider changing to a backup format that just creates big tail-appended files containing incremental changes. Something like rsync that just outputs bazillions of little files is going to create lots of random writes when things change, vs consolidating all those changes into one file that just grows at the end. Treat them the way you would a tape (which is what tar was designed for). Nothing wrong with SMR drives per se - they can potentially be cheaper especially for backup (using an appropriate file format), and are just as fast for reading so they're also great for infrequently changing bulky data. However, random writes are very costly and you should be aware of that going in... -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Thu, Jan 2, 2020 at 11:23 AM Mick wrote: > > On Thursday, 2 January 2020 14:43:58 GMT Rich Freeman wrote: > > > > Out of curiosity, what model drive is it? Is it by chance an SMR / > > archive drive? > > Good catch! I hadn't thought of this - the Linux kernel will need to have > DM_ZONED enabled I think, for the OS to manager the shingled writes > sequentially, but I don't have this enabled here because AFAIK I have no such > drives in my possession. I haven't looked into the details of how it works. Certainly the kernel should be able to optimize writes to such disks and utilize TRIM if supported by the drive. But it isn't strictly needed as you go on to say. > > Due to the limitations on how those write data out I > > could see them implementing an internal filesystem that journals > > incoming data and then writes it back out after the fact. > > SMR drives which implement a 'device managed' write mechanism, will use their > own firmware to control data storage. The OS would not be aware of anything > being different to a conventional drive. Correct. > > If so then > > that might happen even after the kernel thinks it is unmounted. > > However, such a drive firmware would probably use a journal that > > ensures data is safe even if power is cut mid-operation. The drive > > isn't supposed to report that a write is completed until it is > > durable. > > Which I take it to mean the drive would not be unmounted by the OS until it is > safe to do so and for all intends and purposes it will also be safe to be > powered down thereafter. Yes - even if the drive is doing its own data shuffling after being unmounted, it should still be safe to power off. In any case, I was just speculating as to why it might be doing writes when not mounted. I don't know the internal details of how these drives all work. > I would think this would be within seconds of > successfully unmounting it. Spinning for 30 minutes or more after it is > unmounted sounds excessive to me, if it is only being spun by the firmware for > flushing its journal buffers. Indeed. That really makes me wonder whether it is actually writing anything. It could just be that the drive has some vibration or otherwise is giving the sensation that it is doing something when it isn't. Though, if the thing really does have a large journal inside to improve performance it could actually buffer a pretty large number of writes. SMR drives can have a very large amount of write amplification. If each erase region on the disk contains 10k blocks then every 1MB of data in the journal could potentially lead to 10GB of disk rewrites, assuming that writes are randomly distributed. It also makes sense that when replaying the journal the drive is going to prioritize erase regions with the most updates, which means that by the time you're unmounting the drive you're going to expect the stuff left in the journal to lead to the most write amplification. I would imagine that a drive that works in this manner is going to use a logfs for the journal (if the journal is even in SMR format), and then it would just keep an erase region free at all times. Anytime a partial write happens in a region the data would get written to the free region, then that region would be remapped in place of the old region, and now the old region is unallocated for the next write. This could be interrupted due to power loss at any time without any real loss to data, since data is not overwritten in place. Before the journal record is closed out the old region is still valid, and after the new region is valid. An interrupted write is just repeated on power up since the new region can just be overwritten again safely. In any case, the bottom line is that a drive should be safe to unplug if all filesystems on it are umounted and the umount commands all return. If the disk loses data after that point it is a manufacturing design flaw. -- Rich
Re: [gentoo-user] External hard drive and idle activity
On Thu, Jan 2, 2020 at 5:49 AM Dale wrote: > > lol I didn't think of that and I don't recall anyone else thinking of > it either. That is because syncing before unmounting doesn't do anything. Unless you use --lazy umount blocks until all writes are complete to a device. The instant it returns as far as the kernel is concerned the device should be safe to power off. If you do a sync first then of course the umount will complete more quickly, since all writes should already be flushed. I have no idea what your device is doing after it is unmounted, but it doesn't have anything to do with the linux kernel unless some process is directly accessing the raw device (very unlikely). Maybe the drive firmware is doing some kind of housekeeping, or maybe the drive has some kind of vibration in it that just makes it feel like it is doing something. Or maybe the NSA or Red Army has hacked your firmware and it is doing who knows what (yes, the NSA bit at least is a thing). In any case, chances are the drive manufacturer has accounted for sudden power loss in the design because if they didn't there would be a ton of complaints, since there is nothing you can do about this sort of thing assuming the firmware is up to something. Out of curiosity, what model drive is it? Is it by chance an SMR / archive drive? Due to the limitations on how those write data out I could see them implementing an internal filesystem that journals incoming data and then writes it back out after the fact. If so then that might happen even after the kernel thinks it is unmounted. However, such a drive firmware would probably use a journal that ensures data is safe even if power is cut mid-operation. The drive isn't supposed to report that a write is completed until it is durable. -- Rich
Re: [gentoo-user] is the X11 Security extension required for xauth?
On Mon, Dec 23, 2019 at 1:15 AM Adam Carter wrote: > >> The difference between -X and -Y is in providing a layer of security >> so that remote clients can't play games like keyboard sniffing with >> your local X server. > > > I had conflated those extra checks with .Xauthority. > > How would the keyboard sniffing attack work? Since everything over the > network is made confidential by ssh, i'm guessing the attack would be by a > local user on the ssh client/X server box somehow? Anything that can connect to your X server can sniff your keyboard. Connecting to your X server requires: 1. A network/socket connection (which usually means being on the same machine/LAN, or having access to the other side of the ssh tunnel which probably means being on the remote ssh server). 2. Authenticating with the X server, typically using a cookie from an .Xauthority file, a copy of which is in your home directory when you log in using a display manager, and another copy will be created in your home directory on the remote ssh server when you connect using ssh X11 forwarding. There are non-cookie methods of authenticating with X11 I think but I'm less familiar with them, and they're much less common. So, anything with access to your home directory on your X server (ie your desktop), or on the remote ssh server, which would generally include root on both hosts, can connect to your X server. Any process which does so can sniff your keyboard/mouse input (and I think screen as well) for the most part. X11 does NOT restrict client applications to only receiving keystrokes destined for its own windows. That has been a very longstanding security weakness in X11. In the case of local processes it isn't as much of a hole, because in most cases the only processes that can talk to your X server can also read all your data from your home directory, and attach to your own processes to read their memory/etc. Unless you're using some kind of application containerization layer (not common yet), Linux doesn't provide much isolation between processes running under the same UID if one is malicious. However, when you're connecting to a remote server with X11 forwarding, you now are giving access to your keyboard/mouse/etc to processes on that remote host, which otherwise would NOT be able to access your local host. If you remotely admin servers/etc you probably aren't intending to give the owners of those servers the ability to spy on your desktop. There are a couple of ways to protect yourself against this: 1. ssh -X, if you have the security extension in your X server, will block these sorts of attacks at the X11 level. X11 API calls that try to intercept this sort of data will fail (I don't know the gory details). 2. Some applications are hardened against these attacks by taking exclusive access of input (similar to a modal dialog): a. The X11 implementations of app-crypt/pinentry grab exclusive input access when they run. This is why you should ALWAYS use the X11 version of these programs for ssh/gpg agent passphrase entry, and not just have the agent read input in an xterm/etc. The X11 application will block key sniffing via hostile X clients. b. Some xterm implementations have buried in their menus a secure mode option, which while selected grabs exclusive input access. It is intended to be used anytime you enter a password or other sensitive data, but must be manually turned on/off when needed. You can't just leave it on all the time because then no other applications will be able to respond to keyboard input. These issues date WAY back to the beginning of X11. I remember first reading about them in the 90s in a tip to avoid giving untrusted hosts the ability to run X clients (back in an era where things like X terminals were more common). Apologies if I got any details wrong here, but I'm pretty sure this is the gist of how the issue works. How big an issue really depends on how you use X11 forwarding in ssh. If you're the sort of person who remotely admins client servers via ssh you probably should avoid using ssh -Y though. While I've never heard of this being exploited in recent times, it is definitely a risk. -- Rich
Re: [gentoo-user] is the X11 Security extension required for xauth?
On Mon, Dec 23, 2019 at 4:23 AM n952162 wrote: > > My understanding was that this message was due to the lack of an .Xauthority > file: > > Warning: untrusted X11 forwarding setup failed: xauth key data not generated This is not correct. The problem is that it is trying to set up trusted forwarding, and no security extension is present. While an .Xauthority file isn't created, that is just a symptom of the earlier problem. If you create an .Xauthority file it will do you no good whatsoever if you want to connect with -X. > > If I interpret that correctly, one simply failed, and the other used fake > security. Someone subsequently mentioned that the -Y flag would generate the > .Xauthority file, which I tested and saw that that indeed was true - > although, the thusly generated .Xauthority file did not allow a subsequent > usage with -X. -X will generate a .Xauthority file if it negotiates a connection, as will -Y. The reason -Y is working and not -X is that it doesn't need the missing security extension to work. Thus it sets up the connection and creates the .Xauthority file. However, this is of no use for starting a subsequent connection using -X, because the problem with -X has nothing to do with the .Xauthority file in this case. -- Rich
Re: [gentoo-user] is the X11 Security extension required for xauth?
On Sun, Dec 22, 2019 at 8:27 PM Adam Carter wrote: > > But -X produces > "Warning: untrusted X11 forwarding setup failed: xauth key data not generated" > > but the server has an ~/.Xauthority file, which oddly gets created by an ssh > -Y session -Y still uses an .Xauthority so that isn't a surprise. MIT-MAGIC-COOKIE security is used by both -X and -Y, which is what this file is used for. The difference between -X and -Y is in providing a layer of security so that remote clients can't play games like keyboard sniffing with your local X server. Whether this ought to be a default was apparently a debate over a decade ago, when the USE flag was at least added to make it possible. I haven't used it in a while though so I can't vouch for whether there are any issues with -X when the USE flag is enabled to build the extension. https://bugs.gentoo.org/237778 -- Rich
Re: [gentoo-user] Re: trying to upgrade some old, never upgraded image for an embedded system …
On Thu, Dec 19, 2019 at 6:45 PM Thomas Schweikle wrote: > > So I've tried now to upgrade in various ways: > 1. the one given in https://anongit.gentoo.org/git/repo/sync/gentoo.git > But this fails as soon as I try to emerge git. python-exec is at version > 2.4.6 now. Without any 2.0.1 packed, legal versions left. Same for all other > dependencies. Not really a way to go … Not sure what you mean by this. Not really questioning it. It is just hard to provide responses to one-liners on technical issues. What python-exec is at 2.4.6? The one in the repo as of a few years ago? The one that is current (which will almost certainly not work)? A quick check of the log for that repo says that 2.0.1 was removed in Feb 2017, so as long as you check out a version from before then you should be fine as far as that is concerned. You might want to check out an even older version though depending on how old your system is. You still haven't mentioned that, and it matters. I'm not sure if you read my post but I wasn't suggesting just syncing that repo's current HEAD as if that was going to fix things. You need to check out an old commit from it. 6e534e02ec5a71c676dfbae1e9eb6041ef9316ab was the last commit before 2.0.1 was removed, for example - I don't think the tree that far back has metadata so you might need to build that too. > Since there isn't a stage3 and some portage tree matching, I'd ask: armv7 and > others have gone unsupported? Mentioning the arch in your first email probably wouldn't have hurt either. I'm not sure what the state of armv7 support is but somebody else might be able to comment on that. -- Rich
Re: [gentoo-user] How does the emerge process recognizes, whether a certain package is installed?
On Wed, Dec 18, 2019 at 1:55 PM Neil Bothwick wrote: > > On Wed, 18 Dec 2019 19:24:03 +0100, tu...@posteo.de wrote: > > > What is the mechanism and how does it work, which > > cares, that a certain package of a certain version > > gets installed only once? > > The package database is at /var/db/pkg > It is probably worth noting that there really isn't anything preventing a package from being re-installed over the same version of itself. Portage shouldn't generally try to do this unless there is some reason to rebuild it (slot op dep change/etc). However, you can force it to do so (harmlessly) at any time just by running emerge -1 . In general when a package is installed and another package exists in the same slot (whether the same version or not), first portage installs the new package, and then it removes the old version. However, portage tracks files by hash and modification date so when it goes to remove the old package, any files that were overwritten by the new package will no longer match, and thus will not be removed. Any files installed by the old version which were not overwritten by the new version (which could be the same version) will get removed. This also allows updates to system/toolchain/etc packages in-place without too much disruption to running processes (obviously already open files are unaffected regardless due to unix mechanics, but you could get race conditions with files that aren't actually open if it were done the other way around). -- Rich
Re: [gentoo-user] Re: trying to upgrade some old, never upgraded image for an embedded system …
On Wed, Dec 18, 2019 at 11:03 AM Grant Edwards wrote: > > On 2019-12-18, (Nuno Silva) > wrote: > > > The EAPI problem is in a package that is pulled as a dependency of > > portage. > > > > Unless there's a simple hack to solve this, you will need to use older > > ebuilds or split the update in several steps, using older versions of > > the portage tree. The following notes show a way of achieving this: > > > > https://wiki.gentoo.org/wiki/User:NeddySeagoon/HOWTO_Update_Old_Gentoo > > In my experience of situations like this, it's often a lot less work > to just backup /etc and user home directories and re-install from > scratch. > That wiki article seems a bit dated, though it has the right general concept. IMO it is way simpler than that. You could of course do a reinstall and move your /etc and /home - that will certainly be the cleanest approach. You'll probably clear out a lot of orphans or things that are config-protected that have moved that way (well, less so if you keep /etc whole). I think some of this hinges on just HOW old that system is. What was the date that it was last updated on? Assuming it isn't older than 2015 I think the simplest safe approach is to switch to a git repo, and then update it by date. You can use https://anongit.gentoo.org/git/repo/sync/gentoo.git as it has the metadata cache included, but that didn't really start until Aug 2018. Commits before that date won't include metadata, though you can build that yourself. It also uses CI checks so in theory every merge commit is clean and consistent. You can do date-based checkouts. I'd try jumping one year at a time updating @system or at least portage+toolchain. If one of those updates fails you can do a shorter update interval. You probably don't need to update @world until you get up to the current date. As long as @system is updated it should be able to bootstrap everything else. You can't just jump to the current portage as the current portage ebuild is going to use an EAPI that isn't supported by the version of portage you already have. Portage is usually updated in EAPI conservatively to minimize this issue, but if you want to jump multiple years at a time it won't work. Jumping 6-12mo at a time will minimize this issue. -- Rich
Re: [gentoo-user] what does the "USE flags" section of the packages.gentoo.org/package page mean?
On Mon, Dec 16, 2019 at 2:25 PM n952162 wrote: > > It's strange ... on coming home, I see that my machine here can display > all the usual filetypes and has *no* use flags: > > media-gfx/imagemagick-7.0.8.11 > > I'm still curious what that "USE flags" section in the package document > represents. > Without fetching an out-of-date repo I have no idea what 7.0.8.11 supported as it is no longer in the repo. However, 7.0.9.8 supports: equery uses '=media-gfx/imagemagick-7.0.9.8' [ Legend : U - final flag setting for installation] [: I - package is installed with flag ] [ Colors : set, unset ] * Found these USE flags for media-gfx/imagemagick-7.0.9.8: U I + + X : Add support for X11 + + bzip2 : Use the bzlib compression library + + corefonts : Use media-fonts/corefonts which is required by some commands + + cxx : Build support for C++ (bindings, extra libraries, code generation, ...) - - djvu: Support DjVu, a PDF-like document format esp. suited for scanned documents - - fftw: Use FFTW library for computing Fourier transforms - - fontconfig : Support for configuring and customizing font access via media-libs/fontconfig - - fpx : Enable media-libs/libfpx support - - graphviz: Add support for the Graphviz library - - hdri: Enable High Dynamic Range Images formats - - heif: Enable support for ISO/IEC 23008-12:2017 HEIF/HEIC image format using media-libs/libheif - - jbig: Enable jbig-kit support for tiff, Hylafax, ImageMagick, etc + + jpeg: Add JPEG image support + + jpeg2k : Support for JPEG 2000, a wavelet-based image compression format + + lcms: Add lcms support (color management engine) - - lqr : Enable experimental liquid rescale support using media-libs/liblqr + + lzma: Support for LZMA (de)compression algorithm - - openexr : Support for the OpenEXR graphics file format + + openmp : Build support for the OpenMP (support parallel computing), requires >=sys-devel/gcc-4.2 built with USE="openmp" + + pango : Enable Pango support using x11-libs/pango - - perl: Add optional support/bindings for the Perl language + + png : Add support for libpng (PNG images) - - postscript : Enable support for the PostScript language (often with ghostscript-gpl or libspectre) - - q32 : Set quantum depth value to 32 - - q8 : Set quantum depth value to 8 - - raw : Add support for raw image formats - - static-libs : Build static versions of dynamic libraries as well + + svg : Add support for SVG (Scalable Vector Graphics) - - test: Enable dependencies and/or preparations necessary to run tests (usually controlled by FEATURES=test but can be toggled independently) + + tiff: Add support for the TIFF image format + + truetype: Add support for FreeType and/or FreeType2 fonts - - webp: Add support for the WebP image format - - wmf : Add support for the Windows Metafile vector image format + + xml : Add support for XML files + + zlib: Add support for zlib (de)compression I'm pretty skeptical that the older version supported no USE flags. How are you coming to this conclusion? -- Rich
Re: [gentoo-user] Gentoo's Python policy drives me crazy
On Mon, Dec 16, 2019 at 2:00 PM Helmut Jarausch wrote: > > Today's updating involves some package which causes rebuilding > a package which needs Python2.7 and another one which needs > python_single_target_python3_8 > required by that mysterious @__auto_slot_operator_replace_installed__ > > To emerge the Python2.7 package (Scribus) I do need to set > PYTHON_SINGLE_TARGET="python2_7" > in /etc/portage/make.conf > which makes the whole update fail since the other packages needs > PYTHON_SINGLE_TARGET="python3_8" > > So, what can I brave Gentoo user do? > Set PYTHON_SINGLE_TARGET in package.env and not make.conf. That is definitely a setting that can't be set globally. That said, the way python works on Gentoo certainly gets a lot of people confused. Also, just a heads-up that python 2.7 probably will be leaving the main repository in the next few weeks unless somebody steps up to maintain it. -- Rich
Re: [gentoo-user] PYTHON_TARGETS something must have changed
On Sun, Dec 8, 2019 at 9:38 AM Dale wrote: > > The best I can come up with, start figuring out a way to keep python 2 > around on your own, use a overlay if one is available or start expecting > python 2 to disappear, real soon. It seems the devs want it gone even > before it's end of life. > Nobody really wants it gone per se. It is just that nobody has stepped up to keep it around. Maintaining it is fairly complex, and I suspect the people most interested in the nuts and bolts of python are also the ones who are less interested in 2.7. I think most who want it to stay are more interested in it from the standpoint of keeping other software running, but may not be interested in actually taking care of 2.7 itself. These sorts of situations usually cause controversy. Sometimes somebody cares enough to maintain the software. Sometimes it happens in an overlay (which allows a maintainer to be a non-dev more easily, and it also eliminates most QA requirements so that can ease the burden of maintenance, though with the caveat that those QA standards exist for a reason so there are downsides). In the past when stuff like this has happened the software has generally ended up being taken out of the tree, because the fact is that stuff like this can break pretty quickly if nobody is fixing bugs, and if nobody wants to maintain it then that will be what happens. But, it is entirely possible that somebody will step up to maintain it. Python is a bit messier than some previous cases like this because of the whole way that PYTHON_TARGETS and such work, and the complexity of the dependency graph. And keep in mind that the upstream announced EOL is less than a month off. Not that this means the code instantly stops working, but that is why we're starting to see masks and such and more discussion/planning. -- Rich
Re: [gentoo-user] How to tell what the current AMD microcode level is?
On Sat, Dec 7, 2019 at 7:22 PM Adam Carter wrote: > > On Sun, Dec 8, 2019 at 9:39 AM Daniel Frey wrote: >> >> Does anyone know of a list of microcode versions? > > I dont know, so i just use the ~amd64 linux-firmware version. For my 3900X > its currently; > microcode: CPU0: patch_level=0x08701013 > > The last update came through in October; > firmware-md5s-2019-09-09.txt:fef89be989f6a160b340027a24cd0a16 > /lib/firmware/amd-ucode/microcode_amd_fam17h.bin > firmware-md5s-2019-09-25.txt:fef89be989f6a160b340027a24cd0a16 > /lib/firmware/amd-ucode/microcode_amd_fam17h.bin > firmware-md5s-2019-10-23.txt:a30e5f81d37ed38faf039b75bc376197 > /lib/firmware/amd-ucode/microcode_amd_fam17h.bin > firmware-md5s-2019-11-12.txt:a30e5f81d37ed38faf039b75bc376197 > /lib/firmware/amd-ucode/microcode_amd_fam17h.bin > AMD's documentation is pretty terrible on this front. I don't think they actually release the microcode binaries anywhere officially. It seems like they patch them through windows, and these versions end up floating around (probably via enterprise support contracts), and then somebody snags one and sticks it in the linux-firmware package. Oh, and there is basically zero official info as far as a changelog goes. So, if you want to know if some particular version addresses some particular CVE you're just going to have to trust whatever somebody said on lkml or on some random internet forum. These aren't even linux-specific drivers. They're just microcode blobs. Nobody but AMD can create them or work on them. The least AMD could do is stick them on their website along with official hashes and release notes. I'm sure the linux-firmware maintainers know what they're doing and do the necessary detective work to ensure nothing gets missing, but something like this should really have formal vendor support. -- Rich
Re: [gentoo-user] rebuilt kernel - now doesn't shutdown
On Sat, Dec 7, 2019 at 2:13 PM Rich Freeman wrote: > > On Sat, Dec 7, 2019 at 10:19 AM n952162 wrote: > > > > On 12/07/19 13:17, Mick wrote: > > > > > On Friday, 6 December 2019 11:38:09 GMT n952162 wrote: > > >> I rebuilt my kernel and now have the Network Block Device, but now my > > >> system doesn't power off anymore, using shutdown -h now, and doesn't > > >> reboot with reboot (orshutdown -r now). > > > I'm not sure which kernel you said you're running at present, but this > > > commit > > > was made to handle a similar behaviour: > > > > > > https://lkml.org/lkml/2019/10/6/338 > > > > > >> Anybody have any idea what could have become misconfigured? > > > As per above commit, it could be a matter of a kernel driver bug rather a > > > misconfiguration on your part. > > Oh! That's interesting. I'm using AMDGPU, which might be related. But > > it's happening in a more recent kernel: 4.19.72 rather than the [PATCH > > 4.14 08/68] listed in the bug report (although, I don't know what 08/68 > > refers to) > > > > First, it looks like that was reverted in > 8d13c187c42e110625d60094668a8f778c092879 due to some problem. > Ok, I'm being dense. The revert is supposed to FIX the reboot problem. For 4.19 the problem was introduced in 4.19.78 and fixed in 4.19.81. If you're running 4.19.72 I don't think you should be affected by this. -- Rich
Re: [gentoo-user] rebuilt kernel - now doesn't shutdown
On Sat, Dec 7, 2019 at 10:19 AM n952162 wrote: > > On 12/07/19 13:17, Mick wrote: > > > On Friday, 6 December 2019 11:38:09 GMT n952162 wrote: > >> I rebuilt my kernel and now have the Network Block Device, but now my > >> system doesn't power off anymore, using shutdown -h now, and doesn't > >> reboot with reboot (orshutdown -r now). > > I'm not sure which kernel you said you're running at present, but this > > commit > > was made to handle a similar behaviour: > > > > https://lkml.org/lkml/2019/10/6/338 > > > >> Anybody have any idea what could have become misconfigured? > > As per above commit, it could be a matter of a kernel driver bug rather a > > misconfiguration on your part. > Oh! That's interesting. I'm using AMDGPU, which might be related. But > it's happening in a more recent kernel: 4.19.72 rather than the [PATCH > 4.14 08/68] listed in the bug report (although, I don't know what 08/68 > refers to) > First, it looks like that was reverted in 8d13c187c42e110625d60094668a8f778c092879 due to some problem. Second, 4.14 and 4.19 are different longerm series, so there are releases of 4.14 that are newer than some releases of 4.19. 4.19.72 was released on Sep 10, 2019, so it definitely doesn't contain that patch. Both 4.14 and 4.19 receive regular updates as bugs are found in either series or fixes are backported from other series where they apply. I'm not sure if this issue was properly fixed in a more recent patch, or if that has made its way to stable yet - I suspect not based on the bug below. The 08/68 that you see is a reference to the 68 patches that were proposed for the 4.14.148 release back in October. There have already been 10 stable releases in the 4.14 series since then. The upstream bug is at https://bugzilla.kernel.org/show_bug.cgi?id=205147 -- Rich
Re: [gentoo-user] To all IPv6-slackers among the Gentoo community
On Wed, Nov 27, 2019 at 12:36 PM Dale wrote: > > I've found that asking here is best. If it wasn't for my post here, I > would have stuck with Linksys because it is what I've used in the past. > Thing is, a post here lead me to a better product, even tho it wasn't a > Linksys product. It's one reason I post questions here quite often. I > get more info from here than I could likely ever find elsewhere because > most people here post about their own experience not some theory. You > should know, you post about yours quite often and it's generally a good > idea to give it some weight when deciding something. > Linksys had that one router eons ago that was capable of running linux (might have run it out of the box - I forget). Back in the day there weren't many options and they were one of the better ones. They only got worse, and a lot of much better options have come out since then. LOTS of better options. There are ARM-based PCs designed to run pfsense and so on with multiple NICs. Buffalo makes routers with DD-WRT pre-installed, and while I'd double check in the past they could all be trivially flashed to OpenWRT. I'd also seriously consider Ubiquiti. An ER-X can be found in the $60 range and supports routing at gigabit speeds. It runs linux already out of the box with ssh/etc and a CLI, or a nice web GUI. It looks like it isn't hard to flash OpenWRT on it as well though there seem to be some caveats (disclaimer: I've never tried it). There are a couple of good options. I'd seriously consider using something that does what you want out of the box before going the OpenWRT route. I don't think EdgeOS is actually FOSS, but it is largely built on FOSS, so if it does what you want out of the box and is easy to maintain that is a win, and if at any point it doesn't get support you can then go the OpenWRT route. That said, I've run a router on OpenWRT for ages as well. I think that is a bit more work without much gain, but you can do it. Oh, one thing I would avoid doing is running a bazillion services on your router. Yes, if it is a linux/bsd box you can run whatever you want on it. Yes, a lot of that stuff is already packaged and easy to install. Just consider why you have a firewall in the first place (ie another layer of isolation), and that this is likely a device with minimum CPU/RAM/etc and whether you REALLY want to be hosting all this stuff on a box that is a serious PITA to backup/image or rescue if it doesn't boot up right. Generally I don't host anything on a router that isn't directly related to its mission, so that could include updating a dynamic DNS address, serving DHCP, or maybe serving DNS. I've tried running OpenVPN and such on them and have found performance generally suffers for it. -- Rich
Re: [gentoo-user] To all IPv6-slackers among the Gentoo community
On Tue, Nov 26, 2019 at 8:10 PM Dale wrote: > > I went to Newegg. Hey, I buy stuff there sometimes. Anyway, I've > looked at several routers and none of them mention IPv6 that I can > find. I even skimmed the reviews and can't find a mention of it. Is > there some secret way to know when IPv6 is supported? Is it called > something else maybe? > IMO there are three reasonable approaches you can take towards getting a router you won't curse yourself for buying a year from now: 1. DIY. PC or other general-purpose computing hardware with multiple NICs. There are SBCs that work well for this. You can run pfsense or some other router-oriented distro/software/wrappers. Or you can just roll your own with netfilter and such. Max flexibility, but also max fuss. Unless you use a SBC you'll also be paying a price in electricity. Don't underestimate how much you pay for any x86-based system that runs 24x7 - especially anything old you have lying around. 2. OpenWRT/DD-WRT/etc. Again it is a bit fussy but generally way less so than going pure DIY unless you're running pfsense or some other appliance-oriented distro. If you go this route then definitely check for recommendations on hardware that is known to work WELL. Some stuff technically works but can be very prone to having to play around with JTAG and such if you make the slightest mistake. You'll probably spend an extra $20 on hardware you won't regret buying - do it. 3. Something commercial that isn't terrible. There are various options, but everybody always points to Ubiquiti and I'm mostly happy with them. If you want something that is more gui-based I'd go with their Unifi line. I'd avoid Amplifi as it is more consumer-oriented and you'll end up being frustrated with it. EdgeOS is getting closer to something like OpenWRT - it runs linux and you can get a shell and mess around with the CLI. However, while the EdgeOS routing options are great they aren't so good with WiFi and EdgeOS and Unifi don't interoperate all that well (not impossible, but they don't really talk to each other so you have to maintain two configs). I also really dislike that the EdgeOS management software is only supplied as a docker image, which is a pain if you're not using docker (one of these days I'll have to get it working with my bridge interface as it always tries to create its own and ends up having no physical network access). The Unifi controller software is packaged for a couple of distros which makes it much more flexible to deploy (and you can use it on docker if you wish). Personally I'm running EdgeOS on my router and Unifi on everything else. If I could go back I might have gone with Unifi on the gateway but it does bug me that it is so much more expensive and does the same thing. If I had it then end-to-end VLAN/etc would be much more practical, though I'd need a pile of managed switches to make it work well. I've run all three options at various points. Unless your needs are special I think there is value in just going with #3. It just runs itself for the most part, and if you want multiple access points or anything like that the network basically runs itself. I just plug in new hardware and then on the controller software it shows up, and one click provisions it which configures it to fit in with all my global settings. -- Rich
Re: [gentoo-user] To all IPv6-slackers among the Gentoo community
On Tue, Nov 26, 2019 at 6:41 PM Ralph Seichter wrote: > > My current ISP offers native IPv6 and has been doing so for years. > While choice varies across different countries, IPv6 availability has > increased considerably over the last 10 years, which is why SiXXs.net > has discontinued services[1] mid 2017. Even a small amount of searching > should turn up a decent ISP in most industrialized countries. > Uh, at least in the US most ISPs serve a particular geographic area, and most areas have 1-2 to choose from. Well, unless you want to pay to actually run a dedicated line to your house. So either you deal with the consumer-oriented services available in your area, or you move to an area that has better options. I can't imagine that most people would move for IPv6. I just hope the local telecoms support IPv6 properly before they go so far as to start doing carrier-grade NAT... -- Rich
Re: [gentoo-user] Re: eno1 became back eth0
On Thu, Nov 14, 2019 at 11:21 AM Grant Edwards wrote: > > The way it was explained to me was that the old way fell down in some > situations with multiple interfaces. Interfaces were named in the > order they were disovered by the kernel during startup. For some > sorts of NICs (e.g. PCI) the discovery order is repeatible, so no > problems. > > However, for some sorts of interfaces (e.g. USB attached devices), the > discovery order isn't always repeatable. The new scheme was > implemented to make sure than every time you reboot you get interface > names that corresponded to the same physical RJ45 jacks they did the > last time. Another issue is that network interfaces wasn't something really accommodated in the original unix design, which is why they don't show up in /dev. I think on plan9 this was remedied, but of course nobody uses that. If they did show up as devices then we could use symlinks such as with /dev/disk/by-id and so on to provide more flexible solutions. I don't think it is possible to have the same physical interface have multiple names in the kernel so that you can have both eth0 and the new scheme side-by-side. I'd suggest that they should be named by MAC but of course even this is malleable in some situations, and there is promiscuous mode as well. Persistent device names are tricky in a lot of situations, really. Back when I was running mythtv I had multiple pl2303 devices and keeping those straight required writing udev rules that created symlinks based on the physical host port they were plugged into, which is of course non-ideal in USB land. It is less of an issue for hardware that is sophisticated enough to present its own UUID. -- Rich
Re: [gentoo-user] links that behave differently per calling app?
On Sun, Nov 10, 2019 at 11:37 PM Caveman Al Toraboran wrote: > > i think if we have this, we can solve slotting in a simpler > way. e.g. we install libs in their own non-conflicting > locations, and then install for them such fancy sym links > with access that routes accessing processes to the right > version of the lib. > I think the respondents so far haven't actually understood what you're asking based on your example. What follows is mainly for the purposes of fun discussion and some awareness of what is out there, but IMO none of this is really a great basis for a solution... This is certainly possible for EXECUTABLES and is in fact commonly done. busybox is probably the most prominent example of this, and sysvinit uses it as well (as do many others no doubt). However, you're asking about libraries and probably files in general, such as config files. With a library it might be possible (though probably inadvisable) to build a wrapper library that intercepts calls, figures out which ABI/API to use, and then dispatches them to the appropriate implementations. Since it runs under the calling process it would know what that process is. Or you could use the following method... For generic files such as text files an even more inadvisable approach would be to use a bazillion FIFOs and possibly fanotify to intercept read/writes and return the appropriate data. However, I think FIFOs are limited to sequential access so that is going to limit what it can be used for. I'm not sure at what point fanotify for permissions blocks but you might also be able to swap out the symlink between the call to open and the time when the file is actually opened, but that seems likely to be subject to race conditions if you have 3 processes trying to open the same file at the same time (even if fanotify gives the opportunity to relink the file in time how would it know that the process now has an open descriptor that no longer depends on the link and that it is safe to relink for the next process). -- Rich
Re: [gentoo-user] where is /usr/portage?
On Mon, Nov 11, 2019 at 5:28 AM Mickaël Bucas wrote: > > My two systems are currently using the old locations. > Is there a documentation about the way to migrate to the new locations > without breaking things ? > The profile links comes to mind but other things are probably necessary ! > > Has anyone already done the migration ? > In this case do you have advices or warnings about it ? Moving both paths around have been trivial for a very long time. Both are completely disposable when you think about it so there isn't much you can do wrong. Just set DISTDIR and PORTDIR anywhere you want. You can optionally move the existing files to those locations, but if you don't portage will just re-create everything on the next emerge --sync and re-fetch distfiles as needed. If you're ever concerned that something bad has happened to either directory you can always just wipe it out and let portage repopulate it. I personally keep both in /var/cache and am not a fan of the non-LSB /var/db in general, as I think the repo basically is a cache. However, this was not the majority view. In any case it is just a default so it really isn't a big deal - any user can put both of these directories anywhere they prefer. Ditto for binary packages, logs, etc. The only thing you'd need to be careful about migrating is the installed package metadata, as that can't be easily regenerated. I also keep a /usr/portage symlink because I must be getting old. :) Some tools also have that path hard-coded. The one thing I do strongly recommend is not nesting DISTDIR inside PORTDIR. That really wasn't a great design from day 1 and I don't think anybody disagreed with separating them. I'm not sure how much I like $PORTDIR/metadata being where it is either, but I can at least see the argument of keeping syncs of it and PORTDIR atomic and that is an easy way to accomplish this. -- Rich
Re: [gentoo-user] visualise openrc initscript start order and dependency tree
On Mon, Nov 11, 2019 at 5:38 AM Wols Lists wrote: > > On 09/11/19 19:51, Rich Freeman wrote: > > > > Only if somebody has created a generator for openrc, which I doubt. > > It was obviously a semi-trollish comment. > > > Now that's harsh! Although yes I'm sure he was tweaking tails - hence > the "tongue in cheek" smiley. I didn't intend to suggest that I thought it was mean-spirited. Just stirring the pot. > > Fact is, there are a lot of people out there who hate systemd because > it's been successful, and it's been successful because it sticks to the > nix philosophy of "do one thing, and do it well". > Now, THAT is a semi-trollish comment if I ever saw one. :) That said, you could argue that the individual components of systemd do generally do one thing well. I think the criticism is more in the packaging, and that the components mostly don't interchange with anything non-systemd. Though as we can see from eudev/elogind and so on that isn't strictly the case. I sometimes describe systemd as the anti-busybox. But, I don't want to derail the thread entirely... -- Rich
Re: [gentoo-user] visualise openrc initscript start order and dependency tree
On Sat, Nov 9, 2019 at 7:01 AM J. Roeleveld wrote: > > On 9 November 2019 11:42:38 CET, Neil Bothwick wrote: > >On Fri, 08 Nov 2019 21:03:13 +0100, J. Roeleveld wrote: > > > >> I had a similar issue and ended up checking every init-script, conf.d > >> file and rc.conf entry and making a dependency-tree manually on a big > >> white-board. > >> > >> I haven't found a tool that does this automatically yet. > > > >systemd's systemctl ;-) > > > >I'll get my coat... > > Does this parse openrc scripts correctly? Only if somebody has created a generator for openrc, which I doubt. It was obviously a semi-trollish comment. That said, if somebody is looking to write a utility and wants to see what else is out there, systemd has a couple of useful utilities that work along these lines. For example, a short snippet out of systemctl list-dependencies: ● ├─remote-fs.target ● │ ├─mnt-lizardfs.mount ● │ ├─var-lib-machines.mount ● │ └─nfs-client.target ● │ ├─auth-rpcgss-module.service ● │ ├─rpc-statd-notify.service ● │ └─remote-fs-pre.target (The full output is obviously long on any complex host, but this gives an idea. A target is basically a virtual service - similar to a virtual package like plasma-meta.) There are also some tools in systemd-analyze that give you bootchart-like capabilities based on log scanning. It can figure out the critical path to getting to your target runlevel and what services slowed that down the most (though on a really busy host you need to bear in mind that other stuff was launching in parallel so the times given aren't what you'd get if nothing else was running.) Oh, and while bootchart doesn't really show dependencies (at least I don't think it does), if you haven't checked it out you might still find it useful. I believe that does work with openrc, and it at least helps you see where the system is spending its time at boot. -- Rich
Re: [gentoo-user] can genkernel install files with different names?
On Fri, Oct 18, 2019 at 9:02 AM Caveman Al Toraboran wrote: > > what one doesn't use grub? > None that I'm aware of, but I use grub so I haven't gone looking. Like I said, I used to do it this way and get why, but since doing it the new grub way has made my life easier than fighting it with some side benefits, I'm doing it that way. Now, if you're using a tool that requires manually editing config files every time make install dumps a version number into the various filenames, I completely sympathize with wanting to avoid this. Oh, one thing you might consider is symlinks assuming your bootloader supports them. Then genkernel/make/whatever can create fancy filenames as much as they want, and you can just symlink /boot/vmlinuz to whatever you want to start up and so on. I have no idea what bootloaders support symlinks on what filesystems - I know they tend to be really simple tools so I wouldn't take this for granted. It would be trivial to test - just create a symlink of a kernel in your existing /boot and copy/paste your existing menu option to add a new one that references the symlink instead of the existing kernel, and see if it boots... -- Rich
Re: [gentoo-user] can genkernel install files with different names?
On Fri, Oct 18, 2019 at 6:51 AM Alexander Openkowski wrote: > > I struggle with the naming of genkernel generated kernels for quite a while > now and have written a small wapper script for this purpose... > Somebody else shared the same problem and wrote a fairly complex wrapper, and it is installed on most reader's systems already. It is called grub-mkconfig. :) Hey, I get it. I used to do it exactly they way you do. However, the kernel's make install, and the default behavior of both dracut and genkernel, all use a consistent naming convention that is compatible with grub-mkconfig, and I found that it was way easier to join them then to try to beat them. As a bonus it is easier to keep a library of past kernel versions in my boot menu. Now, what I could use is a script/tool that will clean up those versions using some kind of rotation strategy like: 1. Keep the last 5 versions of the current series. 2. Keep the last version of each of the last two longterm series. 2. Keep one version of every stable series between the current and the last longterm series. And this would apply to everything in /boot except config files, and to modules as well. Config files outside this range would get moved into some archive directory of old configs. -- Rich
Re: [gentoo-user] packages going stable after sources become unavailable
On Sun, Sep 8, 2019 at 11:38 AM Grant Edwards wrote: > > This seems to happen regularly with Imagemagick. Version 7.0.8.60 > just went stable today, yet it can't be built because version 7.0.8.60 > sources can no longer be downloaded. > > Am I doing something wrong? > > Shouldn't there be a requirement that an ebuild actually be > _buildable_ from scratch when it goes stable? > Do you have a working Gentoo distfile mirror in your make.conf? imagemagic has a mirror:// SRC_URI. That means there is no upstream distfile available, and Gentoo is distributing its own file via the mirrors (this is actually frowned upon these days though I'm not sure if it is banned offhand - I just avoid doing it this way). Usually the better practice is to host the file somewhere (such as the Gentoo webserver), and it will be mirrored anyway. These mirror:// files are manually injected into the mirror system by developers, and then they stick around as long as some ebuild references them, and then they just go poof without a trace (which means users with older repos can have issues). There has been talk about more robust solutions for these files (especially patches/etc), but nothing has happened. If your system isn't configured with a working Gentoo distfile mirror then for most packages you're probably downloading everything from upstream, but these mirror:// URIs have no sources other than the mirrors so if you don't have a working mirror config it simply won't fetch at all. In general anything in the tree should be fetchable. If that isn't the case it should be quickly spotted and fixed/removed, and it is always a valid bug, assuming it isn't just a local issue on your system. I wouldn't file a bug for this issue - I suspect QA would have already if it wanted to make it a strict policy. This sort of thing is trivially detected with a tinderbox or repoman if desired. -- Rich
Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system
On Mon, Aug 19, 2019 at 10:42 AM Raffaele Belardi wrote: > > Rich Freeman wrote: > > Next time you do something like this, keep in mind that Gnome and xfce > > can co-exist on the same system, and so can openrc and systemd. > > Good point, I did not know, in particular for the init systems I thought it > was exactly > the opposite. > The only area of incompatibility I'm aware of are the sysvinit-compatibility links. Both sysvinit and systemd provide implementations of common utilities like poweroff, halt, reboot, telinit, and so on. There is also init itself. The versions that come with sysvinit are compatible with both sysvinit and systemd. If you don't have sysvinit then systemd can supply these. Systemd itself doesn't require these utilities but they are useful both for compatibility and convenience. (ie "systemctl poweroff" works fine, as does sending the command via dbus, but scripts or sysadmins might prefer to be able to just run "poweroff"). The versions of these supplied by systemd are not compatible with sysvinit. A USE flag toggles whether systemd installs these utilities. If it does then it blocks sysvinit. So, you just have to switch that USE flag to install the two in parallel. If you don't have systemd install "init" then you do need to have a kernel command line to launch systemd directly as init. Offhand I think that is really the only conflict between the two. Systemd doesn't use anything but those compatibility utils from sysvinit but it doesn't mind them being around, and nothing in sysvinit/openrc should even notice that systemd is installed. As long as you set the USE flag appropriately you can dual-boot between the two very easily. The only gotcha is keeping all your configs up-to-date as openrc and systemd store things in different places. When you install systemd it takes a snapshot of many of your openrc settings but that is a one-time operation. Some of those settings are hard to change if systemd isn't running as PID 1 - I think the wiki has instructions for how to do this. -- Rich
Re: [gentoo-user] switch from gnome/systemd to xfce/openrc borked my system
On Mon, Aug 19, 2019 at 2:29 AM Raffaele Belardi wrote: > > Yesterday I tried to switch my ~amd64 box from Gnome/systemd to Xfce/openrc. > I followed > the wiki [1], [2] to install Xfce from a Gnome terminal: > > - switch profile from 17.1/desktop/gnome/systemd to 17.1/desktop > - emerge xfce4-meta and some xfce4 applications/panels/extras > - unmerged systemd and emerged OpenRC > - emerge -uDvN world to account for the different profile flags Next time you do something like this, keep in mind that Gnome and xfce can co-exist on the same system, and so can openrc and systemd. At this point you're probably just going to want to troubleshoot what you are left with, though you could consider reverting back to your old config and starting over if you have backups/etc. I imagine that not many people move from systemd to openrc, since the latter is basically the default on Gentoo already. If I were going to migrate a working system between the two I would probably do it stepwise: 1. Rebuild the kernel with support for both systemd and openrc. Boot that (under systemd) and confirm it is working. 2. Install xfce and get that working fine (under systemd). That really has no tie-in to the service manager but if you have this working it is one less thing to mess with and it simplifies your system. 3. Install openrc and reboot under systemd just to make sure everything is still working fine. I forget what the defaults are but you might need to tweak your systemd USE flags so that it uses the sysvinit versions of halt/reboot/poweroff/telinit and so on. It works just fine with its own version of these tools or with sysvinit. 4. Make sure you have your openrc configured the way you want it (I don't think it has any issues with using rc-update and so on while systemd is running, but I haven't tried that). 5. Switch your kernel command line to boot with openrc, and make a note of what it said before. If it boots fine you're now running openrc and just have to clean up the stuff you don't want. If it doesn't boot you just have to edit your command line and you're back up and running with systemd until you sort it out. 6. Switch your profile, do the -uDvN to rebuild anything impacted, and depclean the stuff you don't need. Reboot to test. By doing it this way you will be just making one change at a time with a reboot in-between so that you know what broke if something breaks. The way you did it is potentially more time-efficient, but if something breaks you are going to be hunting to figure out what it was. Since all the packages you're changing are capable of co-existing there is no reason to switch cold turkey. Now, on a new install or a host I didn't care so much about uptime for I'd probably do it your way, and just revert to a backup. In a production environment where reboots are a concern I'd be working out the procedure on a test host. Oh, yeah, and step 0 is to make a backup... :) -- Rich
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 7:41 PM Grant Taylor wrote: > > On 8/6/19 10:28 AM, Rich Freeman wrote: > > An initramfs is just a userspace bootloader that runs on top of linux. > > I disagree. > > To me: > > · A boot loader is something that boots / executes a kernel with > various parameters. > · An initramfs / initrd (concept) is the micro installation that runs > under the kernel that was booted (by the boot loader). Bootloaders are just programs that run other programs, ultimately. We can argue all day about what kinds of programs count as each. But, there is no point arguing definitions. > The initramfs / initrd micro installation does the following: > > 1) fulfills prerequisites (e.g. loads requisite drivers, initializes > networking for and discovers storage, decrypts block devices) > 2) mounts root (and some other) file systems > 3) launches additional init scripts. So, an initramfs doesn't actually have to do ANY of those things, though typically it does. > None of those steps include booting / executing an alternate kernel. Nothing prevents an initramfs from booting an alternate kernel actually, though if it does it will need its own root filesystem initialized one way or another (which could be an initramfs). Linux bootstrapping linux is basically the whole premise behind coreboot. > > > and you don't need to have cruft all over your filesystem to do it. > > Actually yes you do need that cruft. Sure, but you don't need it ALL OVER YOUR FILESYSTEM. Some of the proposed solutions in this thread involved sticking stuff in /bin and so on. An initramfs is nicely bundled up in a single file. > If anything, having an initramfs / initrd means that you're going to > have an additional copy of said core system files in a slightly > different format (initramfs / initrd) that's not usable by the main system. Absolutely true, which means it won't interfere with the main system, as opposed to sticking it in /bin or somewhere where it might. > > and they make your bootstrapping process infinitely flexible. > > Nope. I don't agree. > > There have been many things that I've wanted to do in the past 20 years > that initramfs / initrd aren't flexible enough to do. Such as? It is a linux userspace. It can literally do anything. > But adding an init script that calls tools on the root file system is > infinitely more flexible. I'm not restricted to doing things that > dracut (et al.) know how to do. You don't need to use dracut (et al) to have an initramfs. In fact, you could create your super root filesystem that does all the fancy stuff you claim you can't do with an initramfs, then create a cpio archive of that root filesystem, and now you have an initramfs which does the job. > If I can boot the kernel, point it at a root disk, and launch an init > system, I can do anything I want with the system. Sure, but only if the kernel can find that disk without any userspace code. What if that disk is on the other side of an ssh tunnel? > Let me say it this way. If I can put together commands to do something, > I can get thee system to do it on boot. I don't have to wait for dracut > to be updated to support the next wiz-bang feature. If you know the commands to do something, why would you have to wait for somebody else to implement them? > > > If you want root to be a zip file hosted on a webserver somewhere > > that isn't a problem. If you want root to be a rar file on a > > gpg-encrypted attachment to a message stored on some IMAP server, > > you could do that too. To make all that work you can just script it > > in bash using regular userspace tools like curl or fetchmail, without > > any need to go mucking around with lower-level code. Then once your > > root filesystem is mounted all that bootstrapping code just deletes > > itself and the system operates as if it was never there (strictly > > speaking this isn't required but this is usually how it is done). > > You need /something/ to be your starting root file system. For most > servers / workstations / VMs / containers, that's a local directory > structure. Actually, for most of these the initramfs is the starting root filesystem (just about all linux server installs use one). Usually it just mounts a simple filesystem on a local disk as root and then pivots. However, an initramfs frees you up from the limitation that it be something that can be passed directly on the command line. > Sadly, I think people have thrust additional (IMHO) largely unnecessary > complexity into the init process just to be able to support more exotic > installations. Then, they have said "We want to be consistent in how we > boot, so we need to use this more complex thing everywhere." To which, > many greybeards have respon
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 7:39 PM Ian Zimmerman wrote: > > On 2019-08-06 12:28, Rich Freeman wrote: > > > > Arguing against this trivial (and IMHO, elegant) solution is tilting > > > at windmills. Specially if it is for ideological reasons instead of > > > technical ones. > > > Some of the solutions I've seen tossed out in this thread are more > > complex than just building your own initramfs from scratch. > > > > An initramfs is just a userspace bootloader that runs on top of linux. > > Nobody has any problem with conventional bootloaders, and if you want > > to do anything with one of those you have to muck around in low-level > > C or assembly. > > There is a difference, and that difference is the reason I dislike > initramfs, not one of the other possible reasons you hypothesize. The > difference is that real Unix processes (not just kernel threads and not > just PID 1) survive from the initramfs stage into the "real Unix" > stage. It's like being able to trace matter back before the Big Bang. They only persist if you allow them to. Most implementations I've seen don't leave anything behind. Typically an initramfs will unlink everything inside, kill any stray processes (if it even forks anything in the first place), and then will exec the new init from the previous init. That starts up init as PID 1 with nothing else running, and no trace of the initramfs remaining. But, sure, you can stick a fork bomb in an initramfs and have it create 10GB of junk in the ramfs as well so that you boot up a crippled system if you really want to. Unless something has changed the kernel is actually built with an empty initramfs built-in by default which loads no matter what. So, the framework of an initramfs is always there, and the only thing that changes is whether it does anything. -- Rich
Re: [gentoo-user] Re: USE flag 'split-usr' is now global
On Tue, Aug 6, 2019 at 11:54 AM Canek Peláez Valdés wrote: > > Arguing against this trivial (and IMHO, elegant) solution is tilting at > windmills. Specially if it is for ideological reasons instead of technical > ones. > ++ Some of the solutions I've seen tossed out in this thread are more complex than just building your own initramfs from scratch. An initramfs is just a userspace bootloader that runs on top of linux. Nobody has any problem with conventional bootloaders, and if you want to do anything with one of those you have to muck around in low-level C or assembly. There has been talk of gathering up all the stuff you need from /usr to bootstap everything, and then adding some scripting to mount everything. The proposals have been to shove all that in / somewhere (perhaps even in /bin or /sbin). If instead you just stick it all in a cpio archive you basically have an initramfs, and you don't need to have cruft all over your filesystem to do it. There are already configurable and modular solutions like dracut which do a really nice job of building one, and they make your bootstrapping process infinitely flexible. If you want root to be a zip file hosted on a webserver somewhere that isn't a problem. If you want root to be a rar file on a gpg-encrypted attachment to a message stored on some IMAP server, you could do that too. To make all that work you can just script it in bash using regular userspace tools like curl or fetchmail, without any need to go mucking around with lower-level code. Then once your root filesystem is mounted all that bootstrapping code just deletes itself and the system operates as if it was never there (strictly speaking this isn't required but this is usually how it is done). I doubt we'll see a /usr merge anytime soon, or any huge rush to break a separate /usr completely without an initramfs. However, there are already use cases known where a separate /usr can cause problems without either an initramfs or some kind of early-boot shell script (there have already been solutions tossed out for that which are much simpler than the ones in this thread). The biggest example I've heard is bluetooth keyboards - that was what made most of the zealots give up on supporting anything that could possibly be needed for bootstrapping being in /, as bluetooth has a bunch of deps and at that point you might as well shove gnome in /bin. So, for the forseeable future your system probably won't break if you are using a separate /usr, but if it does the policy has been established for a long time that nobody is obligated to fix it (nor are they prohibited from doing so). Really, though, you should take the time to appreciate an initramfs whether you decide to use one or not. They're a really elegant solution to the problem. Sometimes I think that half the objection is due to the fact that they use cpio which is a bit obscure - if they were tarballs people would be tearing them apart and playing with them more. -- Rich
Re: [gentoo-user] ntp-client slows down the boot process
On Fri, Jul 26, 2019 at 11:32 AM YUE Daian wrote: > > I switched to a faster NTP server. It still takes some seconds but > better than before. > > Maybe you are right. Having correct system time is more important than > several seconds... You're never going to make NTP fast unless you're using a VERY low-latency server - like something on your own LAN. That is just the nature of the protocol - it has to do a round trip, and of course to do anything it needs the interface up, DNS, and so on, and all of these will be starting from cold caches. If you have non-local DNS and non-local NTP then that is multiple round-trips to the internet. > > By the way does "rc_parallel" really makes a difference? > I tried it once before but did not really see much difference. I haven't used OpenRC in ages, but I'm guessing that NTP is set as a dependency somewhere in the chain. It does make sense - lots of services do not like abrupt time changes so generic dependencies will probably assume that you want to set your clocks before starting anything. I'm not sure how ntpdate implements time changes. I know that ntpd will slew the clock gradually for small corrections, but it is a daemon so it can easily implement something like that. A one-shot correction will probably be instant, and thus will be more of an impact on other services. You can probably adjust the dependencies to suit your tastes, but of course you'll have to keep in mind that time changes for running services might or might not be a problem. If you're fairly confident in your hardware clock accuracy (assuming you even have one) that isn't a big deal. If you're talking about some system that doesn't keep time when powered off/etc then you probably don't want your database server to spin up thinking it is 1980 or whatever its epoch is. I did a quick check of what is being done with systemd and ntpdate is needed before the time-sync target, and that is needed before starting cron or any timer units (obvious requirement), and it is also required before spinning up libvirt guests, which also makes sense so that those initialize with a clean clock, though if they update themselves maybe that isn't a hard requirement. -- Rich
Re: [gentoo-user] Black Screen of Death
On Fri, Jul 19, 2019 at 7:52 AM Alan Mackenzie wrote: > > On Fri, Jul 19, 2019 at 01:30:48 +0200, gentoo-u...@c-14.de wrote: > > > > Running ASUS Prime x470 Prime with ryzen 2700 and have upgraded kernel > > > to latest stable when running emerge --update... > > I've been having this problem ever since I built my "new" (April 2017) > machine, Ryzen something or other on an ASUS Prime x370pro. My machine > has been hanging, perhaps, about once a week. > What kernel version are you both running? The new integrated vega APUs and the vega GPUs use the amdgpu kernel driver, which is new. It was fairly unstable until recently. If you're on the latest 4.19 that probably isn't the problem (not that it is impossible). 4.19.59 is the current upstream longterm. If you're on something pre-4.18 I'd expect problems. I don't know if all the amdgpu issues are backported to 4.14 - I would probably stick with 4.19 with anything Ryzen/Vega-related. It is possible this isn't the problem, but this sounds like some kind of KMS-related issue. If you can't switch to a virtual console with KMS that is probably a kernel issue. I'm assuming you are using the in-kernel amdgpu drivers. If you're using some kind of proprietary driver that could be a problem as well. -- Rich
Re: [gentoo-user] AMD microcode updates - where are they?!
On Sat, Jul 13, 2019 at 4:16 PM Wols Lists wrote: > > On 13/07/19 20:23, Mick wrote: > > Thanks Corbin, I wonder if despite articles about microcode patch releases > > to > > deal with spectre and what not, there are just no patches made available for > > my aging AMD CPUs. > > Or Spectre and what not are Intel specific ... > > I know a lot of the reports said many of the exploits don't work on AMD. > It's something to do with the way Intel has implemented speculative > execution, and AMD doesn't use that technique. Some spectre-related vulnerabilities apply to AMD, and some do not. Most of the REALLY bad ones do not, but I believe that some of the AMD ones still require microcode updates to be mitigated in the most efficient way. Take a look in /sys/devices/system/cpu/vulnerabilities on your system for the kernel's assessment of what vulnerabilities apply, and how they are being mitigated. What you want to see is every single one either saying "Not affected" or they start with "Mitigation:" If you see one starting with something like Partial Mitigation or Vulnerable you should Google if there is something you can do to improve this. Note that this assumes you have a current kernel. The kernel can only report the vulnerabilities it knows about, so if you're running some kernel from 9 months ago it won't know about everything. For reference, on my Ryzen 5 1600 I get: for x in * ; do echo -n "$x: " ; cat $x ; done l1tf: Not affected mds: Not affected meltdown: Not affected spec_store_bypass: Mitigation: Speculative Store Bypass disabled via prctl and seccomp spectre_v1: Mitigation: __user pointer sanitization spectre_v2: Mitigation: Full AMD retpoline, STIBP: disabled, RSB filling -- Rich
Re: [gentoo-user] Music player being run from an emerge
On Tue, Jul 9, 2019 at 2:10 PM Andrew Lowe wrote: > > * ACCESS DENIED: open_wr: /dev/snd/controlC0 > * ACCESS DENIED: open_wr: /dev/snd/controlC0 This has nothing to do with permissions, and everything to do with the sandbox. Most emerge phases are sandboxed. This uses LD_PRELOAD to override glibc functions to intercept filesystem access, which is why you're also getting messages related to this (just going from memory so forgive me if I have a minor detail off here). If you want to override the sandbox behavior you need to stick a file in /etc/sandbox.d/ with your changes, such as adding SANDBOX_WRITE="/dev/snd" to your config. In general though portage is configured to lock down filesystem writes even as root so that build systems aren't dropping random files in random places, or wiping out your hard drives. We want the build system to stay in /var/tmp and stick its install files in the image directory, and then portage will catalog them and move them to your filesystem so that it can cleanly uninstall things later. You could also disable sandboxing entirely, but I would strongly recommend NOT doing this. -- Rich
Re: [gentoo-user] cannot upgrade to zfs 0.8.x
On Mon, Jul 8, 2019 at 7:46 AM John Covici wrote: > > OK, so I successfully did build 0.8.1, but it does not like my root > file system parameter, dracut chokes and puts me in an emergency shell > and I have to mount it manually. I tried root=rpool and > root=rpool/root which dracut completely said unable to continue and if > I omit the root= altogether like I have done in previous versions, it > says it can't mount the root file system, and throws me into the > emergency shell where I can mount it manually. I do have rpool/root > designated as rootfs. I'm not sure if this is a dracut issue or a zfs issue, but on my zfs 0.7.13 system I have: root=ZFS=fast/root ...where fast/root is the pool/dataset I'm using. You might try adding the ZFS= to your root string. I wasn't aware it would even work otherwise. I have yet to try 0.8 - I'm not in a huge rush though I'm sure the TRIM support will help. I'm content to let everybody else run into issues like this for me... :) -- Rich
Re: [gentoo-user] Human configurable boot loader, OR useful grub2 documentation
On Fri, Jul 5, 2019 at 4:10 AM Mick wrote: > > On Friday, 5 July 2019 08:24:14 BST mad.scientist.at.la...@tutanota.com wrote: > > Thank you! Now I don't have to read all the grub2 manual right away. > > You could create manually a /boot/grub/grub.cfg file, but this is NOT how > GRUB2 was meant to be used. TBH, if you want to do this, then why bother with > GRUB2 in the first place. You could instead install sys-boot/grub-static from > an overlay and use grub legacy by manually configuring its /boot/grub/ > grub.conf file. The only people who would tell you not to use a manual config would also tell you not to use the old version of grub. There is really no reason to use the old version, except for the fact that the documentation for manual config files on the new one is pretty opaque. The newer version is much more versatile in terms of support for newer filesystems, etc. It just has a preferred mode of operation that basically generates config files that are practically a bootloader in themselves. The old version isn't even in the Gentoo repo, which means that if you do run into problems you'll be using backchannel support. Given that somebody just posted a ready-to-use config file I'd start there. All that said, it probably is worth taking the time to see if you can bend to the tool rather than making the tool bend to you. If you use the standard make install kernel filenames, and edit the grub config files in /etc appropriately, chances are it will generate a nice menu that just works. I used to do it the way you're looking to do it, but find that the tools work pretty well these days and it makes it trivial to maintain a library of old kernel versions which has helped out with the occasional regression. While the autogen config files are a bit complex, they are actually editable and readable. If you skip through all the conditional logic you'll get to the guts of the actual menu options. You can always autogen a config and not send the output to the actual config file if you want to see what it wants to do for reference. -- Rich
Re: [gentoo-user] Profile 17.1 fails at the analyse stage
On Thu, Jun 20, 2019 at 9:21 AM Jack wrote: > > The --analyze phase bailed out before even starting. I filed an issue > upstream (mgorny's github repository) and he made a change (I didn't > look at the actual commit) so this situation should now be handled > correctly. I think he did want to accept anything that ended up > pointing to the right place, but was afraid of ending up with an > unpredictable result, so now it will accept either the relative or > absolute form. I don't know when he will release a new version. > IMO that was the right design choice. You just don't want to mess around with these symlinks without care, so it is better to test that everything is as expected. Otherwise you'll break some system that somebody had tweaked 5 years ago and forgotten about. This way the edge cases get reported, and can be taken into account before opening things up more... -- Rich
Re: [gentoo-user] Inter-package dependencies.
On Thu, Jun 20, 2019 at 1:34 AM aleiphoenix . wrote: > > On Thu, Jun 20, 2019 at 1:30 PM aleiphoenix . wrote: > > See some ebuilds like > > x11-drivers/nvidia-drivers/nvidia-drivers-430.14.ebuild > > > > if use kernel_linux && kernel_is ge 5 2; then > > ewarn "Gentoo supports kernels which are supported by > > NVIDIA" > > ewarn "which are limited to the following kernels:" > > ewarn " > ewarn " > ewarn "" > > ewarn "You are free to utilize epatch_user to provide > > whatever" > > ewarn "support you feel is appropriate, but will not > > receive" > > ewarn "support as a result of those changes." > > ewarn "" > > ewarn "Do not file a bug report about this." > > ewarn "" > > fi > > > > You could die there :) > > Sorry about the typo, I mean you could use "die" there. > This will block installing the package if the kernel is invalid, but will not block installing an invalid kernel later. Also, this needs to be done in an appropriate phase. You probably want to use a blocker for something like this, though portage doesn't always handle these well, and kernels are also a bit different as packages go. A blocker is just a dependency with an ! operator. Note that this blocks installing that kernel package, not running that kernel, which is why kernel dependences tend to be expressed differently. Personally, in situations where I'm very dependent on kernel series I just run upstream kernels. Gentoo barely patches its kernels so unless you REALLY need the gentoo patches it is pretty trivial to just sync from the stable repo and pull the appropriate branch for a longterm series. I do this for zfs, and have done it in the past for btrfs, and would do this anytime I needed to pin a particular kernel series. You would lose the benefit of any Gentoo QA, but IMO that benefits you mainly to the degree that you run typical packages, and you're already deviating from the script. If your package is in-tree and depends on kernel version you could file a bug to address this in some way. If it is in an overlay then you're on your own. -- Rich
Re: [gentoo-user] Re: What is the "halt" user for?
On Wed, Jun 19, 2019 at 3:19 PM Ian Zimmerman wrote: > > On 2019-06-19 15:10, Jack wrote: > > > Won't "sudo halt" work? I frequently do "sudo reboor" or just > > "reboot" from a root shell. (I am also systemd free.) > > I would prefer to avoid sudo for security reasons (to get root I > normally login on an otherwise unused virtual console). But yes, I'm > pretty sure that would work. This certainly isn't the only way to accomplish your goal, but this is a pretty typical use of sudo. Note that sudo isn't limited to just giving users all-or-nothing access to run commands as root. You can give a particular user the ability to run a particular command line as root as well. So, you could give a user the ability to run shutdown/etc as root, perhaps with a specific set of parameters, and possibly without entering a password. The user wouldn't necessarily be able to do anything else. So, if that user were compromised it could only be used to shut down the system. That of course can be used as a DOS, but the same issue applies to your proposed solution. These days there are other ways to do the same - I'm sure you can do something like this with polkit if you're using a PID1 that can accept messages over dbus. I'm not sure if POSIX capabilities would be of use here - maybe to power off but I'm not sure they're granular enough to send signals to PID 1 and do an orderly shutdown. > > I'm also just curious what the intended prupose of the "halt" user is. My beard isn't quite that long, but I'm guessing it is more-or-less what you think it is. It just isn't fully implemented on Gentoo. I'm guessing that the default passwd file had it in there for compatibility to reserve the UID/etc. I doubt anything actually relies on these accounts these days. -- Rich
Re: [gentoo-user] how can I everr switch to profile 17.1?
On Tue, Jun 18, 2019 at 4:31 PM John Covici wrote: > > On Tue, 18 Jun 2019 15:27:21 -0400, > Rich Freeman wrote: > > > > On Tue, Jun 18, 2019 at 2:47 PM John Covici wrote: > > > > > > On Tue, 18 Jun 2019 13:35:16 -0400, > > > Rich Freeman wrote: > > > > > > > > On Tue, Jun 18, 2019 at 1:00 PM John Covici > > > > wrote: > > > > > > > > > > emerge --update --newuse --deep --with-bdeps=y @world > > > > > > > > And did you try actually running this? > > > > > > Yep, I did run this and although the packages I mentioned did not > > > compile, all others did work. Here is what I have for the packages > > > mentioned gnome-desktop and openssl. > > > > > > > So, you're saying that when you tried to run that command, portage > > tried to build gnome-desktop and openssl, and they failed? > > > > Can you attach the build logs for these? > > gnome-desktop and openssl do build successfully, but depclean does > complain about them maybe because of the unresolved dependencies to > totem and inn. What unresolved dependencies? Did it install gnome-desktop and openssl, or not? If it was installed, then it shouldn't be unresolved. I'm having difficulty following this. First these two packages are missing. Then they didn't compile. Then they did build successfully, but are "unresolved?" What is the output of: emerge --update --newuse --deep --with-bdeps=y @world What is the output of: emerge --depclean (yes, you pasted it above, but that was possibly before running the update with-bdeps) What is the output of: emerge -1 media-video/totem net-nntp/inn Also, you're obviously using inn from an overlay since it isn't in the tree. Are its dependencies in the tree or your overlay, and are they correct? The output of the above command will probably help. -- Rich
Re: [gentoo-user] how can I everr switch to profile 17.1?
On Tue, Jun 18, 2019 at 2:47 PM John Covici wrote: > > On Tue, 18 Jun 2019 13:35:16 -0400, > Rich Freeman wrote: > > > > On Tue, Jun 18, 2019 at 1:00 PM John Covici wrote: > > > > > > emerge --update --newuse --deep --with-bdeps=y @world > > > > And did you try actually running this? > > Yep, I did run this and although the packages I mentioned did not > compile, all others did work. Here is what I have for the packages > mentioned gnome-desktop and openssl. > So, you're saying that when you tried to run that command, portage tried to build gnome-desktop and openssl, and they failed? Can you attach the build logs for these? -- Rich
Re: [gentoo-user] how can I everr switch to profile 17.1?
On Tue, Jun 18, 2019 at 1:00 PM John Covici wrote: > > emerge --update --newuse --deep --with-bdeps=y @world And did you try actually running this? -- Rich
Re: [gentoo-user] how can I everr switch to profile 17.1?
On Tue, Jun 18, 2019 at 12:07 PM Helmut Jarausch wrote: > > On 06/18/2019 05:43:55 PM, John Covici wrote: > > It would seem impossible for me to switch to profile 17.1. I say this > > because I can never run emerge --depclean . I have a few packages > > which will not compile and one not in the tree anymore. the packages > > which will not compile are mono-4.4.1.0 and totem-3.30 -- I have filed > > bugs against both of them. Inn no longer configures and I have > > discussed this in another thread. So, where can I go with this? I > > hate to just go forward, although these packages look they don't have > > any lib32 dependencies. Wat do you thihnk about this? > > > > I haven't run emerge --depclean. I have re-emerge gcc / glibc and all > packages with > files /usr/lib32 and /lib32 and some more which I don't remember. If you're unable to run --depclean (not necessarily if you just haven't run it recently), you could run into problems, sometime, probably not a convenient time. You don't HAVE to run it to do the migration, but I'd suggest generally cleaning things up first, because it makes it less likely that you'll run into issues, and it might reduce the time required to migrate. If you CAN'T run --depclean I'd definitely stop and fix whatever is wrong, otherwise you could run into issues during the migration, when they might be harder to resolve. > On the other hand I don't see such a big advantage of profile 17.1. > I have several (64 bit) packages (in my local portage tree) which still > install into /usr/lib > although this folder is now the replacement of the old /usr/lib32. The advantages are more around bringing Gentoo into better alignment with all the other distros and making packages more maintainable in the long-term. If packages are sticking 64-bit libraries in /usr/lib then I would file a bug, if there isn't already one open. There are known packages that don't work with 17.1. There isn't really a reason to rush into this, but long-term support for 17.0 will get dropped, so you will have to migrate, eventually. I would try to find a convenient time to do it. As far as --depclean issues go, it would be more helpful if we actually had details around the issues... -- Rich
Re: [gentoo-user] Profile switch to 17.1.
On Tue, Jun 18, 2019 at 11:51 AM Alan Mackenzie wrote: > > At this stage, /lib32 should be a symlink. I think that step 12 means > just the symlink should be removed, NOT all the stuff inside what it > points to. > > So I think what you should do is: > > $ rm /lib32 > > , but definitely NOT a recursive rm on that symlink. This is the correct answer. Large scale breakage should not be an issue if you've followed the other instructions. It is possible that something minor might break, but this is all 32-bit stuff. I ran revdep-rebuild immediately after removing the symlink and it found nothing wrong, but this is probably a step you can do to be extra-sure. -- Rich
Re: [gentoo-user] Profile 17.1 change errors, who's bug is it?
On Fri, Jun 14, 2019 at 5:20 PM Corbin Bird wrote: > > Most of the info requested is attached in a zip file. Looks like it worked fine to me. Assuming the rest of your emerge completed successfully you can just remove the symlinks as instructed in the news item and you're done. -- Rich
Re: [gentoo-user] Profile 17.1 change errors, who's bug is it?
On Fri, Jun 14, 2019 at 4:14 PM Corbin Bird wrote: > > Deleted those libs in /usr/lib32 and recompilied 'binutils-libs'. > Got new libs in /usr/lib AND --> /usr/lib32. Is it actually installing these in lib32, or are you just seeing them via the compatibility symlink? The emerge log would tell you where it is installing them, or qlist will as well (from portage-utils). Did you actually switch your profile as instructed? You might want to attach your portage log and output of emerge --info. -- Rich
Re: [gentoo-user] Re: Heads Up - switching to profile 17.1 is not as easy as it sounds
On Tue, Jun 11, 2019 at 7:00 PM Nikos Chantziaras wrote: > > On 11/06/2019 13:34, Helmut Jarausch wrote: > > I had some trouble switching to the new profile 17.1. > > Following the advice in the news item didn't suffice. > > I'm not sure if switching to 17.1 would get me anything. I assume 17.0 > will still be there for a long time to come? Eventually it will go away, likely without much warning (just as there was little warning with the recent move to get rid of 13.0). However, you're probably fine for many months. There is a small risk that something might break on 17.0 because nobody will be testing with that config. Other than that I don't see much obvious benefit from the change. It is mostly to bring Gentoo more in line with what has emerged as the standard way of doing things. A lot of the old design predates the standards. -- Rich
Re: [gentoo-user] Heads Up - switching to profile 17.1 is not as easy as it sounds
On Tue, Jun 11, 2019 at 8:42 AM Davyd McColl wrote:
>
> On Tue, 11 Jun 2019 at 14:23, Rich Freeman wrote:
>>
>> I think a big part of that is that before I did ANYTHING I took a lot
>> of steps to clean up...
>
> I guess YMMV. I regularly:
> - emerge --sync
> - emerge --update --newuser --deep @world @preserved-rebuild -a
> - emerge --depclean -a
> (by regularly, I mean at least twice a week). If I uninstall anything, I
> clean out
> package.{use|accept_keywords|licence} where appropriate. AFAIK I followed
> the news item pedantically, following it step-by-step until I got to
> re-merging /lib32
> & /usr/lib32, when things came a little unstuck.
>
> Doesn't mean I'm couldn't miss something, just that I'm not leaving this
> machine out-of-date for months at a time or expecting miracles.
It was just intended as general advice for anybody else doing the
upgrade, not as finger-pointing. Despite my care I still ran into
some minor issues.
> I appreciate all the help and experience available from this list
> and would appreciate any input on my updating procedures above,
> in particular, anything which would have made this transition smoother.
You're not really doing anything wrong. I think this is just the
difference between washing your hands before dinner and washing your
hands before going into surgery. And simply doing everything right
doesn't guarantee a lack of issues for something like this.
> Mostly, I find portage to be very capable, though it's taken me quite a while
> to make heads-or-tails of the error output, but I'm getting better at it.
Portage error output is often cryptic, and usually literally following
its advice is the worst thing you can do. It is fine for a lot of
one-offs but when you get 800 lines of error output and a suggestion
to stick something in a config file I'd make sure you understand what
is going on first.
As with most software it is literally trying to solve a problem it
thinks you asked it to solve. Unfortunately, sometimes the fastest
way to get rid of a disease is to drown the patient in disinfectant so
you could call this 3-laws safe. :)
--
Rich
Re: [gentoo-user] Heads Up - switching to profile 17.1 is not as easy as it sounds
On Tue, Jun 11, 2019 at 7:21 AM Davyd McColl wrote: > > On Tue, 11 Jun 2019 at 12:34, Helmut Jarausch wrote: >> >> I had some trouble switching to the new profile 17.1. >> Following the advice in the news item didn't suffice. >> > > first off, `emerge -v1 /lib /lib32` didn't work out because I had an old > library in there I > had to remove with `emerge --depclean` first. I also have an old install of > sickbeard, which > I had to remove from world for the same reason: `emerge -v1 /lib /lib32` > would just complain > about not being able to find an installable source (my words -- can't > remember the original > terms), but it didn't really look like an error -- all green text. I've updated two hosts. One went very smoothly, but it is a fairly dedicated host. One had a few issues, and it has a LOT of history. I found that anything 32-bit tended to cause more trouble, and I had a few orphans as well. It wasn't a huge deal. I think a big part of that is that before I did ANYTHING I took a lot of steps to clean up. I ran depclean and revdep-rebuild as a start. I reviewed all the migration tool output and anything that looked non-essential was depcleaned. When I did the 32-bit rebuild anything that was giving me trouble was traced back to whatever pulled it in and depcleaned (I forget if I did that up-front or if I just deleted the offending library and depcleaned the rev dep later - obviously don't do that for anything you care about). On a more dedicated host/container/etc I suspect you won't have many issues, because you're not going to have a huge pile of legacy stuff lying around with complicated dependency relationships. Some of my rebuild and depclean issues were resolved with --backtrack and --with-bdeps=y. In general a good principle is that anytime you want to change profiles take some time to do some housekeeping. The less junk you have on your system, the less there is that can go wrong. On my one host I also took the opportunity to decide whether I REALLY needed wine. That is a TON of 32-bit stuff you otherwise probably don't need. After removing it you need to clean out package.use because we don't have soft USE dependencies yet. And of course before I did anything I took a zfs snapshot of my root filesystem which only contains the OS for the most part. So, if I ran into serious issues a rollback would probably have been a one-liner (I'm guessing that I'd do that from a rescue disk just to keep daemons with stuff in /var from going nuts). Overall it went better than I was anticipating actually. We haven't had a migration like this one in a while, but I do think that the risk-level of this one was a bit undersold. Restructuring all your libraries is obviously a risky task and while you shouldn't be alarmist it is something that has a lot of potential to go wrong. To be fair, the news item does say that you should do a backup. -- Rich
Re: Aw: Re: Re: [gentoo-user] Updating portage, continued
On Mon, Jun 10, 2019 at 5:39 PM n952162 wrote: > > On 06/06/19 06:00, n952...@web.de wrote: > > In trying to update portage (before I update my system), I have this: > > !!! All ebuilds that could satisfy > ">=dev-python/setuptools-34[python_targets_pypy(-)?,pn_targets_python3_6(-)?,python_targets_python3_7(-)?,-python_single_target_pypy(-),-pyth-),-python_single_target_python3_6(-),-python_single_target_python3_7(-)]" > have been mas > !!! One of the following masked packages is required to complete your request: > - dev-python/setuptools-::gentoo (masked by: EAPI 7) > - dev-python/setuptools-41.0.1::gentoo (masked by: EAPI 7) > - dev-python/setuptools-41.0.0::gentoo (masked by: EAPI 7) > - dev-python/setuptools-40.9.0::gentoo (masked by: EAPI 7) > - dev-python/setuptools-40.8.0::gentoo (masked by: EAPI 7) > - dev-python/setuptools-40.7.3::gentoo (masked by: EAPI 7) > - dev-python/setuptools-40.6.3::gentoo (masked by: backtracking: slot > conflict) > - dev-python/setuptools-36.7.2::gentoo (masked by: backtracking: slot > conflict) > > Looking at https://packages.gentoo.org/packages/dev-python/setuptools shows > that the only two versions stable for am64 are 40.6.3 and 36.7.2. > > What is backtracking and how can I have a slot conflict if it's the > developers who determine what version sits in a slot? Backtracking refers to how the dependency resolver works - it couldn't figure out a way to satisfy your requirements. You have a slot conflict because two different packages that you asked for require two different versions of setuptools to be installed at the same time in the same slot, or at least that is what portage interprets what it is finding. I'd need to see more of the output to get a sense of what is actually going on - posting 10 lines out of what was probably 1000+ lines of output honestly doesn't help anybody to assist you. Yes, the whole output is tedious but probably contains clues. > > One might say, I have a package already dependent on setuptools and it's not > the right one, but how can it be that two different versions want to go into > the same slot? There are many ways this can happen. Maybe package A wants setuptools 40.7.3 or greater, and package B wants setuptools 40.6.3 or lesser, and you asked for both. Often though it is an issue with not backtracking enough - if you're doing a huge update often you need to add --backtrack=100 or rarely even a larger value in order for portage to find a way to meet the requirements. Sometimes you need to include --with-bdeps=y because something portage isn't considering in-scope is pulling in something that conflicts, and it could be resolved by letting portage update more packages. > Backtracking is something to do with dependency checking. I haven't seen any > explanation of what goes on in dependency checking and why backtracking is > necessary. Can someone point to an explanation? Basically your config files, like the world file, and the profile system set, contain a list of stuff you want. Portage wants to give you want you want. Maybe these files specify 200 packages you're interested in directly. Each of these might ask for 5 more, and each of those 5 more, and so on. Portage works backwards through the dependency tree to generate a list of every requirement of every package. These can form circular loops, and the tree can get quite large very quickly. As an optimization I believe portage avoids traversing the entire thing and only goes back so far - usually it can find a solution to your requirements without traversing the entire tree. The --backtrack option controls how far back portage will try going. Increasing the value will slow things down, but can't hurt anything. > > My emerge output goes on to say: > > The current version of portage supports EAPI '6'. You must upgrade to a > newer version of portage before EAPI masked packages can be installed. > (dependency required by "app-portage/gemato-::gentoo" [ebuild]) > (dependency required by > "sys-apps/portage-2.3.66-r1::gentoo[-build,rsync-verify]" [ebuil > (dependency required by "portage" [argument]) > For more information, see the MASKED PACKAGES section in the emerge > man page or refer to the Gentoo Handbook. > > Is this telling me that I have to *first* update my system and *then* update > portage? So, if you blindly follow portage's instructions there is a good chance that you'll make your life using Gentoo a living nightmare, because portage doesn't really know what you want and often will get in over its head. The risk of this happening goes up quickly if you wait a long time between updates, as you seem to have done. You probably don't need those EAPI 7 packages, but if you do then the version of portage you have can't install them. You could upgrade portage to gain access to more packages, but you should probably get the system consistent with whatever repos you have synced now before you go updating even further. I'd step back
Re: [gentoo-user] Re: sysrescuecd gone rogue
On Wed, Jun 5, 2019 at 3:26 PM J. Roeleveld wrote: > > On June 4, 2019 5:59:49 PM UTC, Neil Bothwick wrote: > >On Tue, 4 Jun 2019 16:38:14 +0300, Nikos Chantziaras wrote: > > > >> > Regardless, have you used the Arch based sysrescuecd to install > >> > Gentoo and are there are any gotchas I should be aware of? > >> > >> You can use anything. I did use the Gentoo-based sysrescuecd once to > >> install Gentoo, and I also used an Ubuntu live DVD too. It doesn't > >> matter. If it's Linux and boots, nothing changes. > > > >One thing to watch out for: if you are installing a UEFI system, you > >need a live CD that boots using UEFI and not CSM. > > The gentoo live dvd can boot into UEFI mode. > Personally I just use an Ubuntu image these days - it is trivial to get that working on anything, and it is also pretty easy to get it working with ZFS (easier than with most alternatives at least). I use zfs on most of my root filesystems. In any case, really any solution should work. I'm all for Gentoo-based ones but the Gentoo install process is pretty generic and doesn't really need anything more sophisticated than tar+bzip+curl/wget+chroot, and then whatever you need to partition and format your filesystems. So, I find that my choice of rescue CD tends to be driven more by practical concerns (what boots on the system I'm using, convenient tools, package repository, etc). -- Rich
