Re: [gentoo-user] NAS and replacing with larger drives
Wols Lists wrote: > On 18/12/2022 22:11, Dale wrote: >> Wol wrote: >>> On 18/12/2022 18:59, Dale wrote: Since this is local, I just use rsync to do my backups. I did have to change the options a bit. It seems TrueNAS doesn't like some of the permissions or something. >>> >>> Are you running the rsync daemon on the NAS? I'm probably teaching >>> grandma to suck eggs, but that massively reduces the need for network >>> traffic. >>> >>> Cheers, >>> Wol >>> >>> >> >> >> I mount the NAS on my Gentoo rig. I mount it under /mnt. Then I run >> rsync and copy from the source to the mount point for the NAS. I may >> could go the other way but never thought about doing it that way. Kinda >> sounds backwards to me but I dunno. ;-) >> > Sounds to me like you're doing it all wrong either way ... > > What is *supposed* to happen is that you have the daemon running on > one machine and the client on the other - doesn't matter which. > > Then the client tells the daemon what files are to be copied, THE TWO > COMPARE CHECKSUMS, and only the stuff that fails the checksum is > copied. So if you're doing an incremental backup, network usage and > writes are kept to a minimum. > > I tell people to an in-place backup if they're running on a snapshot > setup, because again it only writes stuff that has actually changed. > > Cheers, > Wol > > Do you have a link to the proper way to do it? I don't copy to a different machine often so my current method may be the problem. Maybe the way you mention will work much better, even a little better would be nice. ;-) Dale :-) :-)
Re: Living in NGL: was: [gentoo-user] NAS and replacing with larger drives
On 19/12/22 21:30, Rich Freeman wrote: On Mon, Dec 19, 2022 at 7:51 AM Wols Lists wrote: On 19/12/2022 12:00, Rich Freeman wrote: On Mon, Dec 19, 2022 at 12:11 AM Dale wrote: If I like these Raspberry things, may make a media box out of one. I'd like to have a remote tho. 😉 So, I've done that. Honestly, these days a Roku is probably the better option, or something like a Google Chromecast or the 47 other variations on this them. Where do you put that 2TB drive on your Roku or Chromecast? I'm thinking of building a media server, not to drive the TV, but to record and store. I thought that was what a media server was! So, he said "media box," which I assumed meant the client that attaches to the TV. There are some canned solutions for media servers - I think the NVidia Shield can run Plex server for example. However, in general server-side I'd go amd64. My current solution is: 1. Moosefs for storage: amd64 container for the master, and ARM SBCs for the chunkservers which host all the USB3 hard drives. With a modest number of them performance is very good, though certainly not as good as Ceph or local storage. (I do have moosefs in my overlay - might try to get that into the main repo when I get a chance.) 2. Plex server in a container on amd64 (looking to migrate this to k8s over the holiday). 3. Rokus or TV apps for the clients. Very similar to what I have (intel/arm for moosefs) - I am effectively using moosefs as a distributed NAS (fuse mount onto whatever system(s) I am using) with built in data protection and redundancy. LVM and similar pooling is discouraged as it defeats some of the built in data protection. To increase storage, just add a disk, format, add to the config and reload - it automatically redistributes the data. Similarly, you can add/remove storage or whole storage systems while live with no risk to your data (within limits!!!) With LVM, if a drive fails, you are SOL and offline until you can recover and restore the data. On a recent holiday, an SD card failed and a moosefs arm SBC in AU went offline - discovered the next morning when doing status checks from a ship in the Mediterranean(!) - it had already backfilled and protection was back at normal, moosefs was just missing 2Tb of storage space. 5 weeks later when I got home, I replaced the SD card, rebooted and readded the system all with no risk to the data. Dale, I was where you are about 10 or so years ago and was forced to move on when that design hit its limits - forget LVM etc, these days there are lots of better ways to do what you want with less risk to your data. Another factor is power - moosefs is currently 1 intel and 7 arm SBC's that use 90-110w (most of which is due to using ancient WD and Seagate hard drives) - where as my intel desktop is 90w when idle, or over 300 w when compiling etc. so its off unless its being used. Power is important to me as its expensive!! BillK
Re: [gentoo-user] Gaming on gentoo
Am Sat, Dec 17, 2022 at 11:54:28AM -0500 schrieb David Rosenbaum: > Dave Out of curiosity: what is the purpose of those mails which simply full-quote another mail, but have no visible reply from you? I’ve been noticing them quite often of late. Am I missing somehting? -- Grüße | Greetings | Salut | Qapla’ Please do not share anything from, with or about me on any social network. Down with the date line! (32nd September Initiative) signature.asc Description: PGP signature
Re: Living in NGL: was: [gentoo-user] NAS and replacing with larger drives
On Mon, Dec 19, 2022 at 11:43 AM Mark Knecht wrote: > > On Mon, Dec 19, 2022 at 6:30 AM Rich Freeman wrote: > > > My current solution is: > > 1. Moosefs for storage: amd64 container for the master, and ARM SBCs > > for the chunkservers which host all the USB3 hard drives. > > I'm trying to understand the form factor of what you are mentioning above. > Presumably the chunkservers aren't sitting on a lab bench with USB > drives hanging off of them. Can you point me toward and example of > what you are using? Well, a few basically are just sitting on a bench, but most are in half-decent cases (I've found that Pi4s really benefit from a decent case as they will thermal throttle otherwise). I then have USB3 hard drives attached. I actually still have one RockPro64 with an LSI HBA on a riser card but I'm going to be moving those drives to USB3-SATA adapters because dealing with the kernel patches needed to fix the broken PCIe driver is too much fuss, and the HBA uses a TON of power which I didn't anticipate when I bought it (ugh, server gear). Really at this point for anything new 2GB Pi4s are my preferred go-to with Argon ONE v2 cases. Then I just get USB3 hard drives on sale at Best Buy for ~$15/TB if possible. USB3 will handle a few hard drives depending on how much throughput they're getting, but this setup is more focused on capacity/cost than performance anyway. The low memory requirement for the chunkservers is a big part of why I went with MooseFS instead of Ceph. The OSDs for Ceph recommend something like 4GB per hard drive which adds up very fast. The USB3 hard drives do end up strewn about a fair bit, but they have their own enclosures anyway. I just label them well. > > I've been considering some of these new mini-computers that have > a couple of 2.5Gb/S Ethernet ports and 3 USB 3 ports but haven't > moved forward because I want it packaged in a single case. Yeah, better ethernet would definitely be on my wish list. I'll definitely take a look at the state of those the next time I add a node. > Where does the master reside? In a container on your desktop > machine or is that another element on your network? In an nspawn container on one of my servers. It is pretty easy to set up or migrate so it can go anywhere, but it does benefit from a bit more CPU/RAM. Running it in a container creates obvious dependency challenges if I want to mount moosefs on the same server - that can be solved with systemd dependencies, but it won't figure that out on its own. > > 2. Plex server in a container on amd64 (looking to migrate this to k8s > > over the holiday). > > Why Kubernetes? I run it 24x7. This is half an exercise to finally learn and grok k8s, and half an effort to just develop better container practices in general. Right now all my containers run in nspawn which is actually a very capable engine, but it does nothing for image management, so my containers are more like pets than cattle. I want to get to a point where everything is defined by a few trivially backed-up config files. One thing I do REALLY prefer with nspawn is its flexibility around networking. An nspawn container can use a virtual interface attached to any bridge, which means you can give them their own IPs, routes, gateways, VLANs, and so on. Docker and k8s are pretty decent about giving containers a way to listen on the network for connection (especially k8s ingress or load balancers), but they do nothing really to manage the outbound traffic, which just uses the host network config. On a multi-homed network or when you want to run services for VLANs and so on it seems like a lot of trouble. Sure, you can go crazy with iptables and iproute2 and so on, but I used to do that with non-containerized services and hated it. With nspawn it is pretty trivial to set that stuff up and give any container whatever set of interfaces you want bridged however you want them. I actually fussed a little with running a k8s node inside an nspawn container so that I could just tie pods to nodes to do exotic networking but clusters inside containers (using microk8s which runs in snapd) was just a bridge too far... -- Rich
Re: [gentoo-user] NAS and replacing with larger drives
On 12/18/22 23:08, Dale wrote:
Mark Knecht wrote:
On Sun, Dec 18, 2022 at 12:20 PM Dale wrote:
> root@truenas[~]# iperf -s
>
> Server listening on TCP port 5001
> TCP window size: 64.0 KByte (default)
>
>
>
> And nothing. Several minutes later, still nothing. And it
continues to sit there. I don't think it is working. :/
>
> Still, odds are, whatever it is, I'm not likely going to be able to
change it. That poor old CPU just may not have the needed
instruction set to be really fast for this. Maybe a different
encryption would be better. I dunno. It's temporary anyway.
>
> And still nothing. It's been sitting there since I read the last
message. Still, nothing.
>
Sadly, you didn't read all of my instructions or apparently read the
man page or help file
All you've done is start the server which listens for a connection.
Now go to your Gentoo Land machine that you want to backup and execute
iperf -c IP.ADDR.OF.SERVER
Wait 10 seconds and hit ctrl-C
Oh, I read that but didn't get that one worked with the other. Ooops.
Thing is, I don't have the second command on my Gentoo install. :/
It sort of grumbles about that. May look into that later. Got other
things in the air right now.
Dale
$ eix iperf
[I] net-misc/iperf
Available versions:
(2) 2.0.14a **2.*l
(3) 3.12
{debug ipv6 sctp threads}
Installed versions: 3.12(3)(10:20:30 AM 10/08/2022)(-sctp)
Homepage: https://github.com/esnet/iperf
Description: A TCP, UDP, and SCTP network bandwidth
measurement tool
Re: Living in NGL: was: [gentoo-user] NAS and replacing with larger drives
Hi Rich On Mon, Dec 19, 2022 at 6:30 AM Rich Freeman wrote: > My current solution is: > 1. Moosefs for storage: amd64 container for the master, and ARM SBCs > for the chunkservers which host all the USB3 hard drives. I'm trying to understand the form factor of what you are mentioning above. Presumably the chunkservers aren't sitting on a lab bench with USB drives hanging off of them. Can you point me toward and example of what you are using? I've been considering some of these new mini-computers that have a couple of 2.5Gb/S Ethernet ports and 3 USB 3 ports but haven't moved forward because I want it packaged in a single case. Where does the master reside? In a container on your desktop machine or is that another element on your network? > 2. Plex server in a container on amd64 (looking to migrate this to k8s > over the holiday). Why Kubernetes? Is the Plex server safer when not being used? How long does it take to spin up an instance and do the TV apps understand this operation? Or would it be up and running all the time? Thanks, Mark
Re: Living in NGL: was: [gentoo-user] NAS and replacing with larger drives
On Mon, Dec 19, 2022 at 7:51 AM Wols Lists wrote: > > On 19/12/2022 12:00, Rich Freeman wrote: > > On Mon, Dec 19, 2022 at 12:11 AM Dale wrote: > >> If I like these Raspberry things, may make a media box out of one. I'd > >> like to have a remote tho. 😉 > > > So, I've done that. Honestly, these days a Roku is probably the > > better option, or something like a Google Chromecast or the 47 other > > variations on this them. > > Where do you put that 2TB drive on your Roku or Chromecast? > > I'm thinking of building a media server, not to drive the TV, but to > record and store. I thought that was what a media server was! So, he said "media box," which I assumed meant the client that attaches to the TV. There are some canned solutions for media servers - I think the NVidia Shield can run Plex server for example. However, in general server-side I'd go amd64. My current solution is: 1. Moosefs for storage: amd64 container for the master, and ARM SBCs for the chunkservers which host all the USB3 hard drives. With a modest number of them performance is very good, though certainly not as good as Ceph or local storage. (I do have moosefs in my overlay - might try to get that into the main repo when I get a chance.) 2. Plex server in a container on amd64 (looking to migrate this to k8s over the holiday). 3. Rokus or TV apps for the clients. -- Rich
Re: Living in NGL: was: [gentoo-user] NAS and replacing with larger drives
On 19/12/2022 12:00, Rich Freeman wrote: On Mon, Dec 19, 2022 at 12:11 AM Dale wrote: If I like these Raspberry things, may make a media box out of one. I'd like to have a remote tho. 😉 So, I've done that. Honestly, these days a Roku is probably the better option, or something like a Google Chromecast or the 47 other variations on this them. Where do you put that 2TB drive on your Roku or Chromecast? I'm thinking of building a media server, not to drive the TV, but to record and store. I thought that was what a media server was! Cheers, Wol
Re: Living in NGL: was: [gentoo-user] NAS and replacing with larger drives
On Mon, Dec 19, 2022 at 12:11 AM Dale wrote: > > If I like these Raspberry things, may make a media box out of one. I'd > like to have a remote tho. ;-) So, I've done that. Honestly, these days a Roku is probably the better option, or something like a Google Chromecast or the 47 other variations on this them. Keep in mind that low-powered devices like ARM SBCs (or Rokus) are very picky about codecs. If you're running something like Plex the software will offload the transcoding to a server when the codec isn't supported by the player, but if you're doing something more DIY then you need to ensure your library conforms to the specs or your player will just die. Even 1080p can be pushing it for software decoding on an ARM SBC, and forget 4k. Heck, realtime software transcoding 4k is pretty hard on most amd64 servers as well - they will do it but it will use a significant amount of CPU just for one stream. These days I'm using Pis just for moosefs storage, and all the compute-heavy stuff is on amd64. For players I just use TV clients or Rokus plus Plex server on amd64. I don't really get any benefit out of completely DIYing the client side and LIRC is a PITA. -- Rich
Re: [gentoo-user] NAS and replacing with larger drives
On 18/12/2022 22:11, Dale wrote: Wol wrote: On 18/12/2022 18:59, Dale wrote: Since this is local, I just use rsync to do my backups. I did have to change the options a bit. It seems TrueNAS doesn't like some of the permissions or something. Are you running the rsync daemon on the NAS? I'm probably teaching grandma to suck eggs, but that massively reduces the need for network traffic. Cheers, Wol I mount the NAS on my Gentoo rig. I mount it under /mnt. Then I run rsync and copy from the source to the mount point for the NAS. I may could go the other way but never thought about doing it that way. Kinda sounds backwards to me but I dunno. ;-) Sounds to me like you're doing it all wrong either way ... What is *supposed* to happen is that you have the daemon running on one machine and the client on the other - doesn't matter which. Then the client tells the daemon what files are to be copied, THE TWO COMPARE CHECKSUMS, and only the stuff that fails the checksum is copied. So if you're doing an incremental backup, network usage and writes are kept to a minimum. I tell people to an in-place backup if they're running on a snapshot setup, because again it only writes stuff that has actually changed. Cheers, Wol
