Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > >> Running this kind of stuff in a loop would likely only hit SQLite's > >> buffer - which is not representative either. > >> > > Not sure about that. If I were to speculate: SQLite doesn't seem to > do > > any real query caching, and the buffer seems very limited, leading to > a > > weird spiking performance profile: > > If I get that chart right, then your LMS is CPU bound. Which means it's > not waiting for I/O. > Yes, it's 100% CPU bound. There is zero I/O during the run (the DB files are already cached in RAM), which is is one of the main reasons I prefer this benchmark to rebuild library / scan for files. The I/O of reading the files introduces a lot of overhead unrelated to the DB performance, and increases variability (e.g. due to file system caching). I've posted the (quite promising) results in the ' other thread ' (https://forums.slimdevices.com/showthread.php?109460). SW: 'Web UI for LMS' (http://forums.slimdevices.com/showthread.php?98186-Announce-Alternative-Web-Interface-(beta)) | 'Playlist Editor / Generator' (http://forums.slimdevices.com/showthread.php?108199-Announce-LMS-Playlist-Editor) | 'Music Classification' (http://forums.slimdevices.com/showthread.php?108278-Announce-Essentia-Integration-music-classification-(moods-genres-)) | 'Similar Music' (http://forums.slimdevices.com/showthread.php?108495-Announce-LMSmusly-play-similar-music) | 'LMSlib2go' (https://www.nexus0.net/pub/sw/lmslib2go/) HowTos: 'build a self-contained LMS' (http://forums.slimdevices.com/showthread.php?99648-Howto-build-a-self-contained-LMS) | 'Ogg Opus' (http://forums.slimdevices.com/showthread.php?107011-Howto-play-Ogg-Opus-files) | 'Bluetooth/ALSA' (http://forums.slimdevices.com/showthread.php?107230-Howto-Bluetooth-streaming-to-from-LMS-(ALSA-only-no-PulseAudio)) Roland0's Profile: http://forums.slimdevices.com/member.php?userid=56808 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Respect what you are doing , perl seems to vary between 5.8 to 5.24 in the user base . 5.26 ;-) Ill test when its aviable no need to rush i have modest collection of 50k tracks and a reasonably fast machine anyway. It should be available now. Please keep a copy of the old package: I don't have a system running this Perl version, therefore built using https://perlbrew.pl for the first time. If that is working fine, that'll save me some headaches in the future. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > > Dang so Linux mint 18 has such old Perl version :) > > Guess why I have 5.10 binaries? Because my system is still CentOS 6 > based. That's an old Perl :-). > > -- > > Michael Respect what you are doing , perl seems to vary between 5.8 to 5.24 in the user base . Ill test when its aviable no need to rush i have modest collection of 50k tracks and a reasonably fast machine anyway. Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3 sub. Bedroom/Office: Boom Kitchen: Touch + powered Fostex PM0.4 Misc use: Radio (with battery) iPad1 with iPengHD & SqueezePad (spares Touch, SB3, reciever ,controller ) server HP proliant micro server N36L with ClearOS Linux http://people.xiph.org/~xiphmont/demo/neil-young.html Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Dang so Linux mint 18 has such old Perl version :) Guess why I have 5.10 binaries? Because my system is still CentOS 6 based. That's an old Perl :-). -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > > I did not get the latest version for some reason using the amd64 > deb... > > > > Perl Version: 5.22.1 - x86_64-linux-gnu-thread-multi > > The reason is explained in the announcement :-). I'll add that to the > todo list. > -- > > Michael Dang so Linux mint 18 has such old Perl version :) Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3 sub. Bedroom/Office: Boom Kitchen: Touch + powered Fostex PM0.4 Misc use: Radio (with battery) iPad1 with iPengHD & SqueezePad (spares Touch, SB3, reciever ,controller ) server HP proliant micro server N36L with ClearOS Linux http://people.xiph.org/~xiphmont/demo/neil-young.html Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Running this kind of stuff in a loop would likely only hit SQLite's buffer - which is not representative either. Not sure about that. If I were to speculate: SQLite doesn't seem to do any real query caching, and the buffer seems very limited, leading to a weird spiking performance profile: If I get that chart right, then your LMS is CPU bound. Which means it's not waiting for I/O. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
I did not get the latest version for some reason using the amd64 deb... Perl Version: 5.22.1 - x86_64-linux-gnu-thread-multi The reason is explained in the announcement :-). I'll add that to the todo list. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
I did not get the latest version for some reason using the amd64 deb... Logitech Media Server Version: 7.9.2 - 1536946607 @ Fri Sep 14 20:16:26 CEST 2018 Hostname: SqueezeVM Server IP Address: 192.168.1.82 Server HTTP Port Number: 9000 Operating system: Debian - EN - utf8 Platform Architecture: x86_64-linux Perl Version: 5.22.1 - x86_64-linux-gnu-thread-multi Audio::Scan: 0.95 IO::Socket::SSL: 2.024 Database Version: DBD::SQLite 1.34_01 (sqlite 3.7.7.1) Total Players Recognized: 5 Main hifi: Touch + CIA PS +MeridianG68J MeridianHD621 MeridianG98DH 2 x MeridianDSP5200 MeridianDSP5200HC 2 xMeridianDSP3100 +Rel Stadium 3 sub. Bedroom/Office: Boom Kitchen: Touch + powered Fostex PM0.4 Misc use: Radio (with battery) iPad1 with iPengHD & SqueezePad (spares Touch, SB3, reciever ,controller ) server HP proliant micro server N36L with ClearOS Linux http://people.xiph.org/~xiphmont/demo/neil-young.html Mnyb's Profile: http://forums.slimdevices.com/member.php?userid=4143 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Tbh given how many apps go for the complexity of full blown mysql/postgres when they are only storing a few configuration items I'm amazed lms works so well with sqlite underneath. -Transcoded from Matt's brain by Tapatalk- -- Hardware: 3x Touch, 1x Radio, 2x Receivers, 1 HP Microserver NAS with Debian+LMS 7.9.0 Music: ~1300 CDs, as 450 GB of 16/44k FLACs. No less than 3x 24/44k albums.. drmatt's Profile: http://forums.slimdevices.com/member.php?userid=59498 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote:
> Running this kind of stuff in a loop would likely only hit SQLite's
> buffer - which is not representative either.
>
Not sure about that. If I were to speculate: SQLite doesn't seem to do
any real query caching, and the buffer seems very limited, leading to a
weird spiking performance profile:
25622
(bar chart = seconds one execution of the benchmark takes)
No I/O, Database Memory Config: high (LMS consumes about 400MB RAM),
nothing else running on the RPi3
>
> In general I've had little reason for complaints
> with my setup in day-to-day use. But I know that eg. some of Erland's
> plugins are extremely heavy on database stuff. If you're running one of
>
> those, that might be interesting, too.
>
I've added some trackstat:
Code:
BCMDS='{"id":1,"method":"slim.request","params":["-", ["playlists", 0, 999]]}
{"id":1,"method":"slim.request","params":["-", ["titles", 0,
99,"tags:acCdefgGiJklnpPDqrROstTuwy"]]}
{"id":1,"method":"slim.request","params":["-", ["albums", 0,
999,"genre_id:771"]]}
{"id":1,"method":"slim.request","params":["-", ["artists", 0,
999,"genre_id:771"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:topratedgenres",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:topratedartists",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:notcompletelyratedalbums",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:leastplayedalbums",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:mostplayednotrecent",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:mostplayednotrecentartists",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["aa:bb:cc:dd:ee:ff",
["trackstat", "statisticsjive", "statistics:partlyplayedartists",
"_start:1","_itemsPerResponse:999"]]}
{"id":1,"method":"slim.request","params":["-", ["search", 0,
999,"term:Front"]]}
{"id":1,"method":"slim.request","params":["-", ["playlists", "tracks",
0, 999,"playlist_id:475485", "tags:acCdefgGiJklnpPDqrROstTuwy"]]}'
Results are compareable (one benchmark run takes 42s/26s) , only the
spike occurs every third run.
Increasing Database Memory Config to max shaves off ~1sec, but doesn't
do anything else (there is no significant increase of memory
consumption, either)
I'll repeat these with a newly-built library.db and with the the old
SQLite version.
+---+
|Filename: atop.png |
|Download: http://forums.slimdevices.com/attachment.php?attachmentid=25622|
+---+
SW: 'Web UI for LMS'
(http://forums.slimdevices.com/showthread.php?98186-Announce-Alternative-Web-Interface-(beta))
| 'Playlist Editor / Generator'
(http://forums.slimdevices.com/showthread.php?108199-Announce-LMS-Playlist-Editor)
| 'Music Classification'
(http://forums.slimdevices.com/showthread.php?108278-Announce-Essentia-Integration-music-classification-(moods-genres-))
| 'Similar Music'
(http://forums.slimdevices.com/showthread.php?108495-Announce-LMSmusly-play-similar-music)
| 'LMSlib2go' (https://www.nexus0.net/pub/sw/lmslib2go/)
HowTos: 'build a self-contained LMS'
(http://forums.slimdevices.com/showthread.php?99648-Howto-build-a-self-contained-LMS)
| 'Ogg Opus'
(http://forums.slimdevices.com/showthread.php?107011-Howto-play-Ogg-Opus-files)
| 'Bluetooth/ALSA'
(http://forums.slimdevices.com/showthread.php?107230-Howto-Bluetooth-streaming-to-from-LMS-(ALSA-only-no-PulseAudio))
Roland0's Profile: http://forums.slimdevices.com/member.php?userid=56808
View this thread: http://forums.slimdevices.com/showthread.php?t=109325
___
discuss mailing list
discuss@lists.slimdevices.com
http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
FWIW: I just pushed support for DBD::SQLite 1.58 on select platforms to 7.9. See https://forums.slimdevices.com/showthread.php?109460 for details. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
I've just updated my LMS to use the latest DBD::sqlite (1.58) and could run some benchmarks, then downgrade to 1.34 and re-run them. The question is how to benchmark (re-scanning the library is simple, but not really representative of normal use) - how did you measure the 20% performance gain? I only tested the scan. It's very heavy on database stuff, with a good mix of read and write. When scanning the collection is 20% faster then something has been improved. Either optimized query paths, processing of data, or I/O or whatever. Running this in a loop Code: while true; do ./bench-lms.sh; done puts 100% load on the server and should be fully DB-bound. Agreed, benchmarking is difficult to do correctly. Running this kind of stuff in a loop would likely only hit SQLite's buffer - which is not representative either. In general I've had little reason for complaints with my setup in day-to-day use. But I know that eg. some of Erland's plugins are extremely heavy on database stuff. If you're running one of those, that might be interesting, too. One other reason why I was looking into updating SQLite is that it supports new features in the fulltext indexing I'd like to leverage. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote:
> > I've been running LMS with 'updated modules / libraries '
> >
> (http://forums.slimdevices.com/showthread.php?107040-Howto-update-perl-modules-bundled-with-LMS)
> > (DBD::SQLite 1.54 (sqlite 3.13.0)) since 03/2017 on a RPi3 without
> any
> > issues.
>
> Great! Any improvements you've experienced?
>
Unfortunately, this was a new setup on new hardware, so I don't have
anything to compare it to.
>
> That's an odd one. In my previous tests this was constant. Every test I
>
> did was a massive slow down in FTS. But I always did delete the
> library.db to start from scratch for every test. Now I repeated the test
>
> for the same collection, without deleting that file, and performance was
>
> as before (plus speed up in other aspects). As if re-using a file was
> much faster than growing it from zero.
>
> I'm now running my own system at home on an oldish Intel Atom CPU with
> the latest DBD::SQLite. In this "real life" scenario with about 20k
> tracks, several plugins which use the database in some aspect
> extensively (Music And Artist!), it's about 20% faster over all, with
> FTS being pretty much the same as before.
>
I've just updated my LMS to use the latest DBD::sqlite (1.58) and could
run some benchmarks, then downgrade to 1.34 and re-run them.
The question is how to benchmark (re-scanning the library is simple, but
not really representative of normal use) - how did you measure the 20%
performance gain?
Currently, I'm using this script to simulate regular use:
Code:
#!/bin/bash
# adjust genre_id, playlist_id, search term and URL
BCMDS='{"id":1,"method":"slim.request","params":["-", ["playlists", 0, 999]]}
{"id":1,"method":"slim.request","params":["-", ["titles", 0,
99,"tags:acCdefgGiJklnpPDqrROstTuwy"]]}
{"id":1,"method":"slim.request","params":["-", ["albums", 0,
999,"genre_id:771"]]}
{"id":1,"method":"slim.request","params":["-", ["artists", 0,
999,"genre_id:771"]]}
{"id":1,"method":"slim.request","params":["-", ["search", 0,
999,"term:Front"]]}
{"id":1,"method":"slim.request","params":["-", ["playlists", "tracks",
0, 999,"playlist_id:475485", "tags:acCdefgGiJklnpPDqrROstTuwy"]]}'
LCNT=1
TS=$(date +%s.%N)
VERBOSE=1 # 0 to get CSV values
while IFS= read -r line ; do
[ $VERBOSE -ne 0 ] && echo "running query $LCNT"
RES=$(curl -Ss -H "Content-Type: application/json" -X POST -d "${line}"
-o - "http://raspi3-64:9000/jsonrpc.js";)
[ $? -ne 0 ] && echo "curl error:$? " && break
echo "$RES"|grep -q '"result":{"count":0}' && echo "No results for
$line"
LCNT=$((LCNT+1))
done <<< "$BCMDS"
TE=$(date +%s.%N)
RT=$(bc <<< "$TE-$TS")
if [ $VERBOSE -ne 0 ]; then
echo "runtime: $RT"
else
echo "$(date +'%H:%M:%S');$RT"
fi
Running this in a loop
Code:
while true; do ./bench-lms.sh; done
puts 100% load on the server and should be fully DB-bound.
SW: 'Web UI for LMS'
(http://forums.slimdevices.com/showthread.php?98186-Announce-Alternative-Web-Interface-(beta))
| 'Playlist Editor / Generator'
(http://forums.slimdevices.com/showthread.php?108199-Announce-LMS-Playlist-Editor)
| 'Music Classification'
(http://forums.slimdevices.com/showthread.php?108278-Announce-Essentia-Integration-music-classification-(moods-genres-))
| 'Similar Music'
(http://forums.slimdevices.com/showthread.php?108495-Announce-LMSmusly-play-similar-music)
| 'LMSlib2go' (https://www.nexus0.net/pub/sw/lmslib2go/)
HowTos: 'build a self-contained LMS'
(http://forums.slimdevices.com/showthread.php?99648-Howto-build-a-self-contained-LMS)
| 'Ogg Opus'
(http://forums.slimdevices.com/showthread.php?107011-Howto-play-Ogg-Opus-files)
| 'Bluetooth/ALSA'
(http://forums.slimdevices.com/showthread.php?107230-Howto-Bluetooth-streaming-to-from-LMS-(ALSA-only-no-PulseAudio))
Roland0's Profile: http://forums.slimdevices.com/member.php?userid=56808
View this thread: http://forums.slimdevices.com/showthread.php?t=109325
___
discuss mailing list
discuss@lists.slimdevices.com
http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
DJanGo wrote: > Hi Michael, To answer my own "question" - maybe... eg running a clean scan from scratch -> no library.db (with the std. "old" Sqlite Version) on 7.92 (latest) needs 38min. running a clean scan over the gui on the same hardware/software needs 22 mins. The is some very little minor difference if using DROP TABLE IF EXISTS fulltext and DROP TABLE IF EXISTS fulltext_terms; or not. DJanGo's Profile: http://forums.slimdevices.com/member.php?userid=1516 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > That's an odd one. In my previous tests this was constant. Every test I > did was a massive slow down in FTS. But I always did delete the > library.db to start from scratch for every test. Now I repeated the test > > for the same collection, without deleting that file, and performance was > > as before (plus speed up in other aspects). As if re-using a file was > much faster than growing it from zero.-- > > Michael Hi Michael, is that because of /usr/share/squeezeboxserver/lib/DBIx/Migration.pm that calls /usr/share/squeezeboxserver/SQL/SQLite/schema_20_up.sql Code: DROP TABLE IF EXISTS fulltext; and DROP TABLE IF EXISTS fulltext_terms; DJanGo's Profile: http://forums.slimdevices.com/member.php?userid=1516 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
I've been running LMS with 'updated modules / libraries ' (http://forums.slimdevices.com/showthread.php?107040-Howto-update-perl-modules-bundled-with-LMS) (DBD::SQLite 1.54 (sqlite 3.13.0)) since 03/2017 on a RPi3 without any issues. Great! Any improvements you've experienced? I can confirm that fulltext indexing is painfully slow, however, since I haven't tried it with any other version of SQLite, I have no comparison. That's an odd one. In my previous tests this was constant. Every test I did was a massive slow down in FTS. But I always did delete the library.db to start from scratch for every test. Now I repeated the test for the same collection, without deleting that file, and performance was as before (plus speed up in other aspects). As if re-using a file was much faster than growing it from zero. I'm now running my own system at home on an oldish Intel Atom CPU with the latest DBD::SQLite. In this "real life" scenario with about 20k tracks, several plugins which use the database in some aspect extensively (Music And Artist!), it's about 20% faster over all, with FTS being pretty much the same as before. I'll try to update my pCP based system, too, to get another datapoint. And I'll run a bunch more tests with my testing environment and the 100k library, using the tweaked DBD::SQLite update (stock vs. some stuff disabled, plus ICU support added). And then I'd have to figure out how to compile this for Windows... -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > > Running from the source would allow you to replace DBD::SQLite (inside > the CPAN folder). Please share your experience. I'm still considering > updating... Just have to figure out what was causing the slow-down in > the fulltext indexing. > I've been running LMS with 'updated modules / libraries ' (http://forums.slimdevices.com/showthread.php?107040-Howto-update-perl-modules-bundled-with-LMS) (DBD::SQLite 1.54 (sqlite 3.13.0)) since 03/2017 on a RPi3 without any issues. I can confirm that fulltext indexing is painfully slow, however, since I haven't tried it with any other version of SQLite, I have no comparison. SW: 'Web UI for LMS' (http://forums.slimdevices.com/showthread.php?98186-Announce-Alternative-Web-Interface-(beta)) | 'Playlist Editor / Generator' (http://forums.slimdevices.com/showthread.php?108199-Announce-LMS-Playlist-Editor) | 'Music Classification' (http://forums.slimdevices.com/showthread.php?108278-Announce-Essentia-Integration-music-classification-(moods-genres-)) | 'Similar Music' (http://forums.slimdevices.com/showthread.php?108495-Announce-LMSmusly-play-similar-music) | 'LMSlib2go' (https://www.nexus0.net/pub/sw/lmslib2go/) HowTos: 'build a self-contained LMS' (http://forums.slimdevices.com/showthread.php?99648-Howto-build-a-self-contained-LMS) | 'Ogg Opus' (http://forums.slimdevices.com/showthread.php?107011-Howto-play-Ogg-Opus-files) | 'Bluetooth/ALSA' (http://forums.slimdevices.com/showthread.php?107230-Howto-Bluetooth-streaming-to-from-LMS-(ALSA-only-no-PulseAudio)) Roland0's Profile: http://forums.slimdevices.com/member.php?userid=56808 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
If you want to run LMS from the source, grab a copy from https://github.com/Logitech/slimserver. Then run "perl slimserver.pl". Running from the source would allow you to replace DBD::SQLite (inside the CPAN folder). Please share your experience. I'm still considering updating... Just have to figure out what was causing the slow-down in the fulltext indexing. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > > You're running LMS from a different computer than where the files are > saved? This kind of I/O might be the bottleneck during the scan. But I > could imagine that when TrackStat is doing its job, it's LMS server side > > only, no access to the files. I might be wrong. But please double check, > > because your machine seems more than powerful enough to process 30k > tracks. Check the resource monitor while TrackStat is running. Monitor > disk, network, cpu and memory usage. Correct yes, music is on a powerful NAS box, my setup has always been like this, so I think its due to the laptop now running from USB2 ouch! mherger wrote: > > > It has 8x (i7) CPU cores and 16Gb RAM so is strong[/color] > > Did you check LMS' memory settings for the database? > Yep its set to Maximum 2Gb RAM Cheers www.spicefly.com - ** Spicefly SugarCube ** - A hassle free acoustic journey through your music library using MusicIP. Plus the finest MusicIP installation guides, enhanced MIP Interface and SpyGlass MIP the Windows Automated MusicIP Headless Installer. cparker's Profile: http://forums.slimdevices.com/member.php?userid=2083 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
My library is 30k tracks (on a NAS) and the bottleneck is I/O for sure, the SATA died and wiped out the disks, so I rebuilt it with Win7 booting from a USB drive. You're running LMS from a different computer than where the files are saved? This kind of I/O might be the bottleneck during the scan. But I could imagine that when TrackStat is doing its job, it's LMS server side only, no access to the files. I might be wrong. But please double check, because your machine seems more than powerful enough to process 30k tracks. Check the resource monitor while TrackStat is running. Monitor disk, network, cpu and memory usage. It has 8x (i7) CPU cores and 16Gb RAM so is strong Did you check LMS' memory settings for the database? -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
mherger wrote: > > Thanks for the research. Long story, short.. my interest is due to a > > low powered wintel system grinding to a stop when doing a full rescan > > due to Trackstat. So I was looking at the db and noticed the old > > version and hence the interest. Is there anyway of > > decompressing/patching a wintel system to play around with this > further? > > You can run LMS from the source if you're willing to install Activestate > > Perl 5.14 (which I could provide you a copy if needed). > > I guess you've already upped the memory parameter for the database? It > can make a huge difference (depending on library size, storage type > etc.). > > How large is your library? > What is the bottleneck? Memory? CPU? Disk I/O? If you run out of memory, > > then any other optimization might be irrelevant. > Do you know what kind of query it is that plugin is chewing so badly on? > > -- > > Michael hi Michael That would be great if you could supply me the source. My library is 30k tracks (on a NAS) and the bottleneck is I/O for sure, the SATA died and wiped out the disks, so I rebuilt it with Win7 booting from a USB drive. It has 8x (i7) CPU cores and 16Gb RAM so is strong enough. I can't change the OS due to other software, it works fine day to day but definitely chewing on the db rebuild when Trackstat does a massive update query after a full rescan. So I would like to see if I can play around with the database side of things and happy to share anything useful I find. Thanks in advance www.spicefly.com - ** Spicefly SugarCube ** - A hassle free acoustic journey through your music library using MusicIP. Plus the finest MusicIP installation guides, enhanced MIP Interface and SpyGlass MIP the Windows Automated MusicIP Headless Installer. cparker's Profile: http://forums.slimdevices.com/member.php?userid=2083 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Thanks for the research. Long story, short.. my interest is due to a low powered wintel system grinding to a stop when doing a full rescan due to Trackstat. So I was looking at the db and noticed the old version and hence the interest. Is there anyway of decompressing/patching a wintel system to play around with this further? You can run LMS from the source if you're willing to install Activestate Perl 5.14 (which I could provide you a copy if needed). I guess you've already upped the memory parameter for the database? It can make a huge difference (depending on library size, storage type etc.). How large is your library? What is the bottleneck? Memory? CPU? Disk I/O? If you run out of memory, then any other optimization might be irrelevant. Do you know what kind of query it is that plugin is chewing so badly on? -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Thanks for the research. Long story, short.. my interest is due to a low powered wintel system grinding to a stop when doing a full rescan due to Trackstat. So I was looking at the db and noticed the old version and hence the interest. Is there anyway of decompressing/patching a wintel system to play around with this further? Cheers www.spicefly.com - ** Spicefly SugarCube ** - A hassle free acoustic journey through your music library using MusicIP. Plus the finest MusicIP installation guides, enhanced MIP Interface and SpyGlass MIP the Windows Automated MusicIP Headless Installer. cparker's Profile: http://forums.slimdevices.com/member.php?userid=2083 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Ok, I thought I'd give it the quickest try I could, running a test on my dev machine (2017 MBP, SSD, 16GB RAM). Here are the results: DBD::SQLite 1.58 (sqlite 3.22.0) Discovering files/directories: /Users/mh/Documents/SqueezeZeugs/flac10 (107993 of 107993) Complete 00:00:19 Scanning new music files: /Users/mh/Documents/SqueezeZeugs/flac10 (99663 of 99663) Complete 00:06:01 Building full text index (7 of 7) Complete 00:05:30 Create library views (2 of 2) Complete 00:00:00 Database Optimize (2 of 2) Complete 00:00:08 The server has finished scanning your media library. Total Time: 00:11:58 (Thursday, August 9, 2018 / 2:57 PM) Discovering files/directories: /Users/mh/Documents/SqueezeZeugs/flac10 (107993 of 107993) Complete 00:00:17 Scanning new music files: /Users/mh/Documents/SqueezeZeugs/flac10 (99663 of 99663) Complete 00:06:11 Building full text index (7 of 7) Complete 00:05:35 Create library views (2 of 2) Complete 00:00:00 Database Optimize (2 of 2) Complete 00:00:08 The server has finished scanning your media library. Total Time: 00:12:11 (Thursday, August 9, 2018 / 3:34 PM) DBD::SQLite 1.34_01 (sqlite 3.7.7.1) Discovering files/directories: /Users/mh/Documents/SqueezeZeugs/flac10 (107993 of 107993) Complete 00:00:20 Scanning new music files: /Users/mh/Documents/SqueezeZeugs/flac10 (99663 of 99663) Complete 00:08:43 Building full text index (7 of 7) Complete 00:00:21 Create library views (2 of 2) Complete 00:00:00 Database Optimize (2 of 2) Complete 00:00:15 The server has finished scanning your media library. Total Time: 00:09:39 (Thursday, August 9, 2018 / 3:08 PM) Discovering files/directories: /Users/mh/Documents/SqueezeZeugs/flac10 (107993 of 107993) Complete 00:00:19 Scanning new music files: /Users/mh/Documents/SqueezeZeugs/flac10 (99663 of 99663) Complete 00:08:55 Building full text index (7 of 7) Complete 00:00:30 Create library views (2 of 2) Complete 00:00:01 Database Optimize (2 of 2) Complete 00:00:17 The server has finished scanning your media library. Total Time: 00:10:02 (Thursday, August 9, 2018 / 3:20 PM) There clearly is an improvement in some parts of the newer code, but unfortunately a massive loss in another part. Now you could say I don't use FTS. But I do. That might show you that a decision pro or con can't easily be taken. Things are a bit more complicated than they might seem. Michael http://www.herger.net/slim-plugins - Spotty, MusicArtistInfo mherger's Profile: http://forums.slimdevices.com/member.php?userid=50 View this thread: http://forums.slimdevices.com/showthread.php?t=109325 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
Re: [slim] Why is LMS using such an old SQLite version?
Is there any reason why LMS is using such an old SQLite version, 3.7.7.1 which dates back to 2011. There are many potential reasons. Call it laziness, or "never touch a running system", or lack of priorities, premature optimization, lack of time... Updating the library for all platforms is virtually impossible. Dealing with the potential issues of running different versions on different platforms might lead to some unwanted decisions (eg. rollback, drop support for some platforms etc.) and certainly would consume considerable amounts of time. Which probably would better be spent elsewhere. Maybe there just isn't any compelling reason to update. Looking through the SQLite release log there seem to be pretty impressive performance gains made over the years so just wondering why this hasn't been rolled into LMS. (DBD-SQLite-1.58 released March 2018) Did you carefully read where those improvements would apply? Does it apply to LMS? Did you confirm it's any faster at all? And where do you see the database as the bottleneck, btw? I'd be happy to see a thorough comparison of current LMS and LMS on latest SQLite, scanning libraries of various sizes (5k, 30k, 100k) on various platforms (Pi, Mac, Windows, desktop Linux). Or at least one of them showing that "impressive performance gain". What would you measure? Scanning time? Browsing performance? Time to playback? PS Can we have the old forum software back.. thanks :p Can it get any older?... it's vbulletin 4, announced in 2009. Its successor v5 was announced in 2012. We never updated beyond security updates. -- Michael ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/mailman/listinfo/discuss
