Re: [RDD] Fwd: Re: FLAC support
On Mon, Jul 1, 2013 at 10:44 AM, Bill Putney bi...@wwpc.com wrote: Andy, I'm curious, what's on those 300,000 tracks? We have a pretty eclectic music mix here. Everything from Roma and Tribal to Hip-hop and Classical. With 25,000 cuts there's still a good percentage of those cuts that have never been played. 12 TB is about 18,460 hours of wav audio. You can play 24 hours a day for about 2 years before you have to replay a piece of music. You'll have about 3 years of music before the existing disk runs out. With a little housekeeping that could last you for another couple of years. We're a freeform station and we've ripped all of our existing collections (CDs in every genre over the span of about 25 years) as well all new incoming CDs. We do curate the collection (with an approximately 50% reject rate), but since we're a freeform station, we keep pretty much anything of significance in any genre. It's less about what gets played and more about what somebody might want to play someday. An easier to implement solution might be to ask to be able to have /var/snd be able to be split across multiple volumes. Maybe by track range or by group. Then you could add another volume of 16TB on an external USB3 drive array as a second volume. It's one more indirection from the track name in the data base or if the volumes were all sub-directories of /var/snd a volume prefix could be added to the track number in the db. I agree that RD support for libraries spread across multiple directories might be nice. Although it would cause issues in terms of managing which directory new music goes to. Most file systems, etc allow some form of aggregation for multiple back-end disks into a single virtual front-end partition, so there are ways to do that without requiring native RD support. But like I said, the issue is more one of metadata and widely usebale format compatibility than one of pure storage space. ___ Rivendell-dev mailing list Rivendell-dev@lists.rivendellaudio.org http://caspian.paravelsystems.com/mailman/listinfo/rivendell-dev
Re: [RDD] Fwd: Re: FLAC support
Andy, I wonder how you'd convert to FLAC and imbed the metadata from they library you have. You'd have to get the data out of the RD database and put it in the FLAC header? That's a big conversion job on it's own. I think RD will let you have chart-chunk metadata in sound files, it just won't do anything with it. I don't know what if any badness would accrue if you did a file by file header update. I don't think RD keeps track of a checksum of the sound files. So long as you don't change the length of the audio payload. The files would probably end up in different inodes when you wrote them back but I don't think RD cares about that either. RD seems to want all the audio files in one flat directory. While you can do symbolic links to other volumes as subdirectories under /var/snd and individual audio files could live on other subdirectories symbolically linked back into /var/snd there isn't a way to divide it out systematically. Right now you have a group named music (for instance) and it has a file number range. When you rip a CD RD tries to use the next available file number in that range. Hypothetically, RD could be changed so that the range could include a volume name prefix. All the audio file references in RD would have to change to reflect that change or at least the database would have to return the full (or a /var/snd relative) path name for each audio cut. You could keep every audio file name unique within RD and have everything do a table look up to find the path for that file name. That would include the subdirectory. I don't know of a linux hack that would allow directory partitioning by leading character string without a delimiter. I suppose no one ever thought there'd be a use for it. Bill On 7/1/13 10:53 AM, Andy Sayler wrote: On Mon, Jul 1, 2013 at 10:44 AM, Bill Putney bi...@wwpc.com mailto:bi...@wwpc.com wrote: Andy, I'm curious, what's on those 300,000 tracks? We have a pretty eclectic music mix here. Everything from Roma and Tribal to Hip-hop and Classical. With 25,000 cuts there's still a good percentage of those cuts that have never been played. 12 TB is about 18,460 hours of wav audio. You can play 24 hours a day for about 2 years before you have to replay a piece of music. You'll have about 3 years of music before the existing disk runs out. With a little housekeeping that could last you for another couple of years. We're a freeform station and we've ripped all of our existing collections (CDs in every genre over the span of about 25 years) as well all new incoming CDs. We do curate the collection (with an approximately 50% reject rate), but since we're a freeform station, we keep pretty much anything of significance in any genre. It's less about what gets played and more about what somebody might want to play someday. An easier to implement solution might be to ask to be able to have /var/snd be able to be split across multiple volumes. Maybe by track range or by group. Then you could add another volume of 16TB on an external USB3 drive array as a second volume. It's one more indirection from the track name in the data base or if the volumes were all sub-directories of /var/snd a volume prefix could be added to the track number in the db. I agree that RD support for libraries spread across multiple directories might be nice. Although it would cause issues in terms of managing which directory new music goes to. Most file systems, etc allow some form of aggregation for multiple back-end disks into a single virtual front-end partition, so there are ways to do that without requiring native RD support. But like I said, the issue is more one of metadata and widely usebale format compatibility than one of pure storage space. ___ Rivendell-dev mailing list Rivendell-dev@lists.rivendellaudio.org http://caspian.paravelsystems.com/mailman/listinfo/rivendell-dev
Re: [RDD] Fwd: Re: FLAC support
On Mon, Jul 1, 2013 at 12:24 PM, Bill Putney bi...@wwpc.com wrote: I wonder how you'd convert to FLAC and imbed the metadata from they library you have. You'd have to get the data out of the RD database and put it in the FLAC header? That's a big conversion job on it's own. I think RD will let you have chart-chunk metadata in sound files, it just won't do anything with it. I don't know what if any badness would accrue if you did a file by file header update. I don't think RD keeps track of a checksum of the sound files. So long as you don't change the length of the audio payload. The files would probably end up in different inodes when you wrote them back but I don't think RD cares about that either. If flac support were ever added, conversion wouldn't be a big deal. Between the Rivendell export tools and/or some raw SQL queries/scripting, and the standard flac tool set, I'm sure we could hack together a conversion tool without too much trouble. RD seems to want all the audio files in one flat directory. While you can do symbolic links to other volumes as subdirectories under /var/snd and individual audio files could live on other subdirectories symbolically linked back into /var/snd there isn't a way to divide it out systematically. Right now you have a group named music (for instance) and it has a file number range. When you rip a CD RD tries to use the next available file number in that range. Hypothetically, RD could be changed so that the range could include a volume name prefix. All the audio file references in RD would have to change to reflect that change or at least the database would have to return the full (or a /var/snd relative) path name for each audio cut. You could keep every audio file name unique within RD and have everything do a table look up to find the path for that file name. That would include the subdirectory. I don't know of a linux hack that would allow directory partitioning by leading character string without a delimiter. I suppose no one ever thought there'd be a use for it. There are a lot of existing systems out there for making multiple back-end stores look like a single location. From RAID0 and JBOD systems to LVM to modern distributed files systems like Ceph, there are a lot of ways to solve the many discreet storage devises to single logical partition problem. If you really wanted to roll your own, you could whip together a FUSE file-system that spread Rivendell files with their number-based names across a multitude of directories or devices while still presenting a flat front-end to Rivendell itself. But there's really no need to write you own code with so many off the shelf solutions available. Compression and multi-device mappings can all be handled at the file-system level. It's the per-file metadata that would really require implicit Rivendell support. ___ Rivendell-dev mailing list Rivendell-dev@lists.rivendellaudio.org http://caspian.paravelsystems.com/mailman/listinfo/rivendell-dev
Re: [RDD] Fwd: Re: FLAC support
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 As many have noted - the best way to go about this is usually at the filesystem block level, or driver level. Ceph represents a relatively new approach - there are plenty of 'odd' filesystems via FUSE, and rolling your own is not a particularly monumental task, and avoids mucky hacks. Until you get above 100 or so terabytes, though, I'd just put this in a single box and use ZFS - it's not that complex. If you want redundancy, pair up two boxes and rsync them. Cheers, James Harrison On 01/07/2013 20:03, Bill Putney wrote: One could imagine a virtual delimiter based on position. If the first two characters of a file name within a specific directory were defined as the subdirectory part and the remainder as the file name within the subdirectory. I.e., 4350978.wav would be for instance ./43/50978.wav really. Like I said, I don't know of a hack like that. Bill On Jul 1, 2013, at 11:30, Cowboy c...@cwf1.com wrote: On Monday 01 July 2013 02:24:55 pm Bill Putney wrote: I don't know of a linux hack that would allow directory partitioning by leading character string without a delimiter. How would you delimit without a delimiter ? -- Cowboy http://cowboy.cwf1.com Whom are you? said he, for he had been to night school. -- George Ade ___ Rivendell-dev mailing list Rivendell-dev@lists.rivendellaudio.org http://caspian.paravelsystems.com/mailman/listinfo/rivendell-dev ___ Rivendell-dev mailing list Rivendell-dev@lists.rivendellaudio.org http://caspian.paravelsystems.com/mailman/listinfo/rivendell-dev -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (MingW32) iEYEARECAAYFAlHR36IACgkQ22kkGnnJQAxJBwCgheIDb3Fgv/YkBFul7ihNQA3q 3ZQAn2ZCFKk6YifB53LGT68j9NbJIhAl =Q44C -END PGP SIGNATURE- ___ Rivendell-dev mailing list Rivendell-dev@lists.rivendellaudio.org http://caspian.paravelsystems.com/mailman/listinfo/rivendell-dev
