Hey again, I found a quick way to test the plugins and discovered it's the chroma plugin filling the ram. It's not essential for imports so I've disabled it, however I'm sure someone should probably check on it. I'll post on the linuxserver | beets page and see if they have any ideas.
On Friday, 24 August 2018 08:50:18 UTC+10, Enrico wrote: > > Just a quick update, seems like you were right about the plugins, in the > image below (bottom part show RAM usage, top is CPU) the left of the image > is an import without any plugins, the right has the following plugins > active: > discogs, > fetchart, > lastgenre, > chroma, > scrub, > duplicates, > web, > > The ram is slowly filling as expected (with plugins active), currently at > 5GB and counting, usually stalls at ~28GB. I will isolate which of the > above plugins is the offender and report back later today. > [image: Screen Shot 2018-08-24 at 8.43.10 am.png] > > > > > > > > > > > > > > On Thursday, 23 August 2018 16:18:53 UTC+10, Enrico wrote: >> >> Thanks for the reply, I'll try this out in the next few hours. I don't >> want to crash docker just now as there are other using it. >> >> On Thursday, 23 August 2018 12:29:18 UTC+10, Adrian Sampson wrote: >>> >>> Hi! A good place to start might be by pruning your plugin list. I’d bet >>> that one plugin is to blame, and trying imports without them could help >>> identify which one it is. >>> >>> Adrian >>> >>> >>> On Aug 22, 2018, at 9:04 PM, Enrico <[email protected]> wrote: >>> >>> Hey, I'm running beets in a docker container (linuxserver/beets) and can >>> only perform small imports, anything over ~100 albums at a time will stall >>> the import. Occasionally the stall will shut down all docker containers >>> leading me to think beets may be writing to RAM somewhere. Any path not >>> mapped by docker will write to RAM so I've been trying to find where the >>> leak in beets is. Imports with quiet mode set to 'off' typically do fine >>> until they encounter a large album, however when quiet mode is 'on' the >>> crash is quite quick. I've attached a graph of beet's activity during a >>> typical crash, the target folder contains 133 albums, the crash occurred >>> about 25mins into the import on about the 35th album. The server has 32GB >>> of RAM, however at the time of the crash beets appears to be using around >>> 27GB. >>> >>> Beets' active folder mappings are: >>> /music >>> /downloads >>> /config >>> >>> My config file is pasted below: >>> >>> # Beets configuration >>> -------------------------------------------------------- >>> >>> directory: /music >>> library: /config/musiclibrary.blb >>> >>> import: >>> write: yes >>> move: yes >>> copy: no >>> delete: no >>> autotag: yes >>> timid: no >>> resume: no >>> incremental: no >>> none_rec_action: ask >>> log: /config/beet.log >>> >>> quiet: no # enable with command line option >>> quiet_fallback: skip >>> default_action: apply >>> >>> singletons: no >>> languages: [en] >>> detail: no >>> flat: no >>> >>> # use the release-date of the original (first) release of an album? >>> original_date: no >>> >>> # on multi-disk releases, assign track numbers for the whole album. >>> # If "per disk", make sure tracknames do not collide ("paths" setting). >>> per_disc_numbering: yes >>> >>> # files matching these patterns are deleted from source after import >>> clutter: ["Thumbs.DB", ".DS_Store", "*.m3u", ".pls", "*.jpg", "*.m3u", >>> "*.log", "*.cue"] >>> >>> # files/directories matching one of these patterns are ignored during >>> import >>> ignore: [".*", "*~", "System Volume Information"] >>> >>> >>> # Paths >>> ---------------------------------------------------------------------- >>> >>> # Paths and filenames for music files >>> # relative to music directory >>> paths: >>> default: $albumartist/$album%aunique{} [$format]/$track - $title >>> singleton: Non-Album [$format]/$artist - $title >>> comp: Compilations/$album%aunique{} [$format]/$track - $title >>> albumtype_soundtrack: Soundtracks/$album%aunique{} [$format]/$track >>> $title >>> >>> # replace special characters in generated filenames >>> replace: >>> '[\\/]': _ >>> '^\.': _ >>> '[\x00-\x1f]': _ >>> '[<>:"\?\*\|]': _ >>> '\.$': _ >>> '\s+$': '' >>> >>> path_sep_replace: _ >>> >>> # filename for the album art >>> art_filename: cover # results in "cover.jpg" >>> >>> max_filename_length: 0 # unlimited >>> >>> >>> # General >>> -------------------------------------------------------------------- >>> >>> # use mutliple threads during import >>> threaded: yes >>> timeout: 5.0 >>> verbose: no >>> >>> >>> # User Interface >>> ------------------------------------------------------------- >>> >>> color: yes >>> list_format_item: %upper{$artist} - $album - $track. $title >>> list_format_album: %upper{$albumartist} - $album >>> time_format: '%Y-%m-%d %H:%M:%S' >>> terminal_encoding: utf8 >>> >>> ui: >>> terminal_width: 80 >>> length_diff_thresh: 10.0 >>> >>> >>> # Auto Tagger >>> ---------------------------------------------------------------- >>> >>> match: >>> strong_rec_thresh: 0.17 # match 90% or better for auto import >>> medium_rec_thresh: 0.25 >>> rec_gap_thresh: 0.25 >>> max_rec: >>> missing_tracks: medium >>> unmatched_tracks: medium >>> distance_weights: >>> source: 2.0 >>> artist: 3.0 >>> album: 3.0 >>> media: 1.0 >>> mediums: 1.0 >>> year: 1.0 >>> country: 0.5 >>> label: 0.5 >>> catalognum: 0.5 >>> albumdisambig: 0.5 >>> album_id: 5.0 >>> tracks: 2.0 >>> missing_tracks: 0.9 >>> unmatched_tracks: 0.6 >>> track_title: 3.0 >>> track_artist: 2.0 >>> track_index: 1.0 >>> track_length: 2.0 >>> track_id: 5.0 >>> preferred: >>> countries: [] >>> media: [] >>> original_year: no >>> ignored: [] >>> track_length_grace: 10 >>> track_length_max: 30 >>> >>> >>> # Plugins >>> -------------------------------------------------------------------- >>> >>> plugins: [ >>> discogs, >>> #lyrics, >>> #echonest_tempo, >>> fetchart, >>> #acousticbrainz, >>> #embedart, >>> lastgenre, >>> chroma, >>> scrub, >>> #mbsync, >>> #mpdupdate, >>> #mpdstats, >>> #missing, >>> duplicates, >>> web, >>> #fromfilename >>> ] >>> #pluginpath: [] >>> >>> >>> # Plugins Config >>> ------------------------------------------------------------- >>> >>> #lyrics: >>> #auto: yes >>> #fallback: '' >>> >>> #echonest_tempo: >>> # auto: yes >>> >>> #discogs: >>> #source_weight: 0.0 >>> >>> lastgenre: >>> # whitelist: ~/.config/beets/genres.txt >>> #canoncical: ~/.config/beets/genres-tree.yaml >>> #fallback: '' >>> auto: yes >>> source: album >>> >>> fetchart: >>> auto: yes >>> #maxwidth: 600 >>> #cautious: no >>> cover_names: cover folder back >>> >>> #embedart: >>> # auto: yes >>> # maxwidth: 600 >>> >>> replaygain: >>> auto: yes >>> overwrite: yes >>> albumgain: yes >>> >>> # required for mpdstats >>> #mpd: >>> #host: localhost >>> #port: 6600 >>> #password: # none >>> >>> #musicbrainz: >>> #host: 192.168.1.200:5000 >>> #ratelimit: 100 >>> >>> #mpdstats: >>> # rating: False >>> # two ratings are calculated: >>> # "rolling" based on recent development >>> # "stable" based on all-time development >>> # rating mix 0.0 is all "rolling", 1.0 is all "stable" >>> # rating_mix: 0.75 >>> >>> #missing: >>> # format: $albumartist - $album - $track - $title >>> # count: no >>> # total: no >>> >>> duplicates: >>> checksum: no # expensive >>> tiebreak: { items: [bitrate] } >>> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "beets" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/d/optout. >>> <Screen Shot 2018-08-23 at 10.45.31 am.png> >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "beets" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
