Re: [LAD] [A plugin scanner:] Cache text file formats: rdf, ttl, custom xml?
Am 29.10.18 um 15:34 schrieb Robin Gareus: On 10/29/2018 12:40 AM, Hermann Meyer wrote: Downside of cached information is, that it could clash on plugin load when the plugin have changed it's ports (updated). If that happens you have a much bigger issue than stale caches. Released plugins should not change ports or behavior in a way that is not backwards compatible. Doing so would break existing sessions using the plugin. I know, but it happens. So it's better to be prepared for this case. ___ Linux-audio-dev mailing list Linux-audio-dev@lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev
Re: [LAD] [A plugin scanner:] Cache text file formats: rdf, ttl, custom xml?
On 10/29/2018 10:34 AM, Robin Gareus wrote: On 10/29/2018 06:52 AM, Tim wrote: On 10/29/2018 12:40 AM, Hermann Meyer wrote: Downside of cached information is, that it could clash on plugin load when the plugin have changed it's ports (updated). If that happens you have a much bigger issue than stale caches. Released plugins should not change ports or behavior in a way that is not backwards compatible. Doing so would break existing sessions using the plugin. So when you save plugin information on your own, you need to check before you at least load the plugin, if the cached information is still valid. Yes that was of great concern from the start of this effort. Scenario 1: While the the program is not running, the user installs, moves, removes, or upgrades a plugin, then runs the program. I have an idea at program startup to compare the date/time of each given plugin path with the one of the cache file. If different, reload the cache. I suggest to prefer a checksum or hash (at least additionally), or use inotify or some existing watch mechanism. Ardour uses stat()'s mtime. There are various issues with this: e.g. zip files do not include timezone, some tar archives include strange timestamps, installers on other platforms do strange things, and some users toy with clocks on occasion. Edge cases like these have lead to countless headaches in the past. Scenario 2: While the the application is running - perhaps already for hours into a project - the user decides to install, move, remove, or upgrade a plugin. Watch out ! I have an idea to 'watch' all the given plugin paths for changes and take appropriate action: Crash ;-) I don't think the scenario is too drastic though. The plugins already loaded into memory should still hopefully be OK. yep. On Unix systems already loaded .so will be kept in memory. On Windows you cannot write/replace to a file that is currently opened. You can skip and postpone scanning of plugins that are currently in use until the next session load. Yes, that's what I was thinking. It might not be wise to alter my data or info of an already loaded plugin if it is deleted or removed or upgraded, but at the very very least I would like to be able to support loading a 'brand new' plugin if it suddenly appears. Why? I've never seen someone installing/removing software while recording or mixing. It also sounds like a bad idea to me. There's the inconvenience of having to shut down and restart the program. I've had a couple of requests for that. Would be nice since sometimes users just want to experiment with all the various plugins installing and trying each one out. What happens if you run two instances of the same DAW or two different programs that share the plugin cache? Do all programs watch for changes and use lock-files to prevent concurrent scanning? Hm yeah good point. Should be do-able though. One solution is to ask a user to directly or indirectly initiate a scan. Directly by pressing a button or indirectly by opening a session. These actions are rarely performed concurrently. Also IIUC sometimes plugin ports can be created on-the-fly? Ouch. Do you need to know the ports a-priori? The host needs to load the plugin anyway to insert it. So the only interesting part to cache is * Plugin-Name * Plugin-Author * Factory & User Presets * Category/Tags (if any) * Description (if any) I/O and Control Ports are not usually something that's worth caching, except perhaps for "has MIDI I/O" (to detect Instruments for plugins that don't specify a category). What other information are you interested to cache? Is there a benefit to caching port names, default values, for each plugin? Absolutely right. I went a bit overboard with this, storing much more information than I really needed. Sort of to the point of obsessing over it. And to what end? Once instantiated, we re-gather all that information from the plugin anyway. So a lot of it is useless. I should probably stop where I am and just finish up what I've got. About my cache format: Since the highest denominator (most advanced) in all these plugin types would be LV2, I think it would be really cool if the Turtle tools or libraries could report ALL types of plugins and generate new ttl files on the fly from found plugins. RDF is generic, it can describe pretty much anything. I thought there is already a LADPSA ontology. Also LADSPA uses RDF for presets. TTL is just a RDF serialization. There is also some discussion to allow JSON-LD as alternative to turtle for LV2. LV2's ontology provides for a nice description of the basics, it really uses foaf[1] and doap[2,3] for the basics. This leaves some classification (categories, tags) and a presets-name to ID map. I have no opinion about the de/serialization format. turtle is fine with me, as would be JSON-LD. Both can be efficiently parsed (low CPU, low memory, serial I/O) and satisfy
Re: [LAD] [A plugin scanner:] Cache text file formats: rdf, ttl, custom xml?
On 10/29/2018 06:52 AM, Tim wrote: > On 10/29/2018 12:40 AM, Hermann Meyer wrote: >> Downside of cached information is, that it could clash on plugin load >> when the plugin have changed it's ports (updated). If that happens you have a much bigger issue than stale caches. Released plugins should not change ports or behavior in a way that is not backwards compatible. Doing so would break existing sessions using the plugin. >> So when you save plugin information on your own, you need to check >> before you at least load the plugin, if the cached information is >> still valid. > > > Yes that was of great concern from the start of this effort. > > Scenario 1: While the the program is not running, the user > installs, moves, removes, or upgrades a plugin, then runs > the program. > I have an idea at program startup to compare the date/time > of each given plugin path with the one of the cache file. > If different, reload the cache. I suggest to prefer a checksum or hash (at least additionally), or use inotify or some existing watch mechanism. Ardour uses stat()'s mtime. There are various issues with this: e.g. zip files do not include timezone, some tar archives include strange timestamps, installers on other platforms do strange things, and some users toy with clocks on occasion. Edge cases like these have lead to countless headaches in the past. > Scenario 2: While the the application is running - perhaps already > for hours into a project - the user decides to install, move, remove, > or upgrade a plugin. Watch out ! > I have an idea to 'watch' all the given plugin paths for changes > and take appropriate action: Crash ;-) > I don't think the scenario is too drastic though. > The plugins already loaded into memory should still hopefully be OK. yep. On Unix systems already loaded .so will be kept in memory. On Windows you cannot write/replace to a file that is currently opened. You can skip and postpone scanning of plugins that are currently in use until the next session load. > It might not be wise to alter my data or info of an already loaded > plugin if it is deleted or removed or upgraded, but at the very > very least I would like to be able to support loading a > 'brand new' plugin if it suddenly appears. Why? I've never seen someone installing/removing software while recording or mixing. It also sounds like a bad idea to me. What happens if you run two instances of the same DAW or two different programs that share the plugin cache? Do all programs watch for changes and use lock-files to prevent concurrent scanning? One solution is to ask a user to directly or indirectly initiate a scan. Directly by pressing a button or indirectly by opening a session. These actions are rarely performed concurrently. > > Also IIUC sometimes plugin ports can be created on-the-fly? Ouch. > Do you need to know the ports a-priori? The host needs to load the plugin anyway to insert it. So the only interesting part to cache is * Plugin-Name * Plugin-Author * Factory & User Presets * Category/Tags (if any) * Description (if any) I/O and Control Ports are not usually something that's worth caching, except perhaps for "has MIDI I/O" (to detect Instruments for plugins that don't specify a category). What other information are you interested to cache? Is there a benefit to caching port names, default values, for each plugin? > About my cache format: Since the highest denominator (most advanced) > in all these plugin types would be LV2, I think it would be really > cool if the Turtle tools or libraries could report ALL types > of plugins and generate new ttl files on the fly from found plugins. RDF is generic, it can describe pretty much anything. I thought there is already a LADPSA ontology. Also LADSPA uses RDF for presets. TTL is just a RDF serialization. There is also some discussion to allow JSON-LD as alternative to turtle for LV2. LV2's ontology provides for a nice description of the basics, it really uses foaf[1] and doap[2,3] for the basics. This leaves some classification (categories, tags) and a presets-name to ID map. I have no opinion about the de/serialization format. turtle is fine with me, as would be JSON-LD. Both can be efficiently parsed (low CPU, low memory, serial I/O) and satisfy requirements (semantic description, ordered data structure). > Then we could all simply use our same LV2 discovery codes to examine > all plugin types found. I'm not sure if that's very likely, nor practical. While serd/sord are generic to deal with turtle/RDF, liblilv is rather specific for LV2. The mechanism to scan/discover plugins and presets is rather different for each standard. Realistically, these days the only interesting plugin formats are LV2, VST and AudioUnit. VST does not even specify a system-wide install location for discovery. The AU format is abstracted by Apple, you can't directly interact with it but have to use Apple's libs. This leaves LV2. On a related