Re: [slim] Looking at improving multi-player synchronization
Generally it works quite well, and the interface to sync/unsync is good. However, sync has a number of issues which prevent it from being as 1st-class as the rest of the system - so, thanks very much for working to improve it! A few comments about things that could be better: 1) gapless + sync should work together, and IMHO this needs to be engineered into any sync design from the start. This is my biggest disappointment with the current system, and is the one thing that could lead me to switch to The Competition 2) the music sometimes gets stuck between tracks (never happens when not sync'd). Presumably this relates to buffering and the players being in different states. 3) the ability to sync mid track is required for various other problems: - when a player gets behind (eg from a buffer underrun), it's behind until the next track. What _should_ happen in this circumstance is that the affected player starts playing again, at the same point it would be at if it hadn't dropped out - fast forwarding/rewinding when players are sync'd is not usable - syncing a player to a player which is already playing something should just join in rather than forcing the other player to start the current track again. This sounds petty but it's not when tracks are long and fast-forwarding doesn't work I'm more than happy to assist with testing any new software/firmware to get the above issues sorted out. -- bukharin bukharin's Profile: http://forums.slimdevices.com/member.php?userid=10859 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
I have 2 SB3s in different (yet quite close) rooms, both running wirelessly (from 6.5.1 SS on Linux FC5). When synced for music (usual stuff - album tracks 3-5 minutes each), I haven't noticed a problem. I frequently use AlienBBC to Listen Again to (UK) BBC Radio 4 shows though, and I do sometimes notice the sync drifting very noticeably into the programmes. I guess this probably relates to the duration of the item (up to 75 minutes) -- SadGamerGeek SadGamerGeek's Profile: http://forums.slimdevices.com/member.php?userid=6145 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
bukharin;197654 Wrote: - fast forwarding/rewinding when players are sync'd is not usable - syncing a player to a player which is already playing something should just join in rather than forcing the other player to start the current track again. This sounds petty but it's not when tracks are long and fast-forwarding doesn't work Yep, this is a massive pain for me. I might be 30 minutes into a Listen Again programme via AlienBBC, and want to carry on listening in another room, but there is basically no way to do this without listening to he same 30 minutes again I'm also happy to help test any firmware/code changes. -- SadGamerGeek SadGamerGeek's Profile: http://forums.slimdevices.com/member.php?userid=6145 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
I'm synching two players, and I'd like to adjust the time delay between them to minimize echo. Is there some way like a plugin or hidden player setting to do this? I'm currently synching an SB3 and a computer running SoftSqueeze 3.3. All files are lossless (flac), and I have high-quality digital converters in each room (UA 2192 and RME HDSP). SoftSqueeze has a configurable buffer size for the audio output, but it needs to be quite large (1.5s), so there's a lot of delay. Btw, audio travels a little over a foot every ms (1.12'/ms), so a 90' separation will be about 80ms, the time where delays start to become perceived as distinct echos (as opposed to doubling or thickening). Delays under about 80ms are micenuts as someone mentioned above, and delays under about 20ms are knatscocks. -- pbjbryan pbjbryan's Profile: http://forums.slimdevices.com/member.php?userid=3034 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
The problem with a set delay is that there's going to be a variance in the amount of network delay each time the players sync up at the start of each track. This is even worse for wireless networks. -- andyg andyg's Profile: http://forums.slimdevices.com/member.php?userid=3292 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
For what it's worth, here is my experience with synchronization. In general, things actually work very very well. I do have two problems, however: 1) The current sync implementation breaks gapless playback and inserts a very audible gap between consecutive tracks. I understand (and agree) that the best time to resync players is between tracks, but there has got to be a way to insert only as much gap as is required to get the players back in sync. In many cases, this gap would be so small as to be unnoticeable. 2) Syncing between SB3 and Transporter doesn't work with native Ogg playback, so I have to transcode to FLAC. Not a big deal in itself (and an easy enough workaround), but this may indicate a problem in how the basic sync process currently works, because the sync logic in this case doesn't maintain a lack of delay between players, it merely maintains a constant and unvarying delay. -- CatBus CatBus's Profile: http://forums.slimdevices.com/member.php?userid=7461 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
I usually sync between 8 and 10 players, all wired, and do not have any problems with drifting except on Internet Radio streams. I do have a problem where the playlist will get stuck between songs when players are synced. It never happens with a single independent player. -- upstatemike upstatemike's Profile: http://forums.slimdevices.com/member.php?userid=6409 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
CatBus;197588 Wrote: 2) Syncing between SB3 and Transporter doesn't work with native Ogg playback, so I have to transcode to FLAC. Not a big deal in itself (and an easy enough workaround), but this may indicate a problem in how the basic sync process currently works, because the sync logic in this case doesn't maintain a lack of delay between players, it merely maintains a constant and unvarying delay. #2 is caused by the fact that there's a startup delay on the Ogg decoder, to avoid skipping. And since Transporter has a faster CPU, it starts before the SB. I think Richard is planning on fixing this so the decoder runs even while paused. -- andyg andyg's Profile: http://forums.slimdevices.com/member.php?userid=3292 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
My work on improving the synchronization system is progressing. I'd like some feedback from people who use synchronization regularly as to what actual problems they have. Also, people who have tried to use it and found it inadequate for their purposes your feedback would be good too. I'd like to break this down a bit. It would be really helpful if you could include as much detail as possible from the issues below: - Multiple players not starting up a track in sync. - Players drifting out of sync during play. - Playing tracks from local server - Playing remote streams (Internet radio, AlienBBC, etc.) - Mix of player types: SliMP3/SB/SB2/SB3/Transporter/SoftSqueeze Alan. -- awy awy's Profile: http://forums.slimdevices.com/member.php?userid=7480 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list [EMAIL PROTECTED] http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
I'm glad you're working on this, because sync was a major attraction in deciding to go with SD, but it has always been a roll of the dice for me. I have two wired SB2s and a wireless SB3, although the sync problems show up even if I connect the SB3 via ethernet. Sometimes the sync is fine, other times it's dreadful. Most often, sync problems are a matter of getting out of sync at the beginning of a track. I also have problems with drifting out of sync while playing a track, but those seem much less common. I haven't confirmed it yet, but I suspect that the sync problems are worse when I am playing random tracks (such as with Erland's plugins). My thinking (possibly incorrect) is that slimserver is trying to pick a new random track to add to the end of the playlist at the same time that it's trying to start the next track on the playlist simultaneously on 2-3 different SBs, but it just doesn't cope sometimes. But I haven't done controlled testing of that theory, nor am I sufficiently fluent in Perl to know if slimserver is really trying to do all of that at once. This is all with my local music library. I don't try to sync internet radio because it *always* drifts out of sync before very long. -- aubuti aubuti's Profile: http://forums.slimdevices.com/member.php?userid=2074 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list [EMAIL PROTECTED] http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
I have two SB2s permanently synced. One in the living room and one in the kitchen. I have others but all used independently. The two synced SB2s are wired and play mainly flac and occasional MP3 albums. The server is Windows XP box with 2.5GHZ processor and 500MB memory and doing not much else. And I'm using Slimserver 6.2.2 (well it works ok for me so not worth the aggro of upgrading). I find that sync failures are extremely rare, amounting to weeks between failures. The system is in use daily. If I start one of the two while the other is running both fall back to the beginning of the playing track. Drift between the two is only noticeable if playing Internet Radio or DAB radio using DABBAR and for both these it will only become apparent they are not in sync after 45 minutes or so. Only issues are that gapless playback (live stuff) is no longer gapless due to re-sync at each track. And originally I tried syncing an SB1 with an SB2 and found that usually within three or four tracks the syncing would break down. Ken -- Ken Ken's Profile: http://forums.slimdevices.com/member.php?userid=957 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list [EMAIL PROTECTED] http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
So which of the above tends to be the real cause of the problem? I guess that (3) could be tackled by forcing a re-sync whenever an under-run is detected. Does (2) happen? If so, what are the clock tolerances on the different devices? -- awy awy's Profile: http://forums.slimdevices.com/member.php?userid=7480 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
Hmm, 10-20ms, perhaps it is doable. I realise that both getting the initial sync precise and then maintaining it would best be done with firmware support. I suspect that firmware change may be needed in the end but here is an outline of how it could possibly be done without it. On my LAN at home I typically get 2ms round trip time (RTT) for wired connections and 4ms for wireless. There are a couple of switches (100baseT) between the server and the wired SlimP3, plus a WLAN .11g base-station and ethernet-bridge on the path to the wirelessly-connected one. I have measured this with both UDP echo and (by modifying the server software) using normal stream data packets. There is a quite a large variation, with many packets above the median, but at the median (or perhaps a lower percentile) the figures are pretty stable. I realise that there are several reasons for the variation in RTT (and I understand at-least some of them). Nonetheless, it is probably possible to get a good measure of the -true- RTT by taking the median value from the last 20-or-so packets. Given the (lack of) accuracy of these measurements, it is also reasonable to assume that the one-way delay is half the RTT. As each ACK packet is received on the server, we know the following: - the time ACK was received by the server and, so long as the RTT is considered valid ( median), we can deduce the time-stamp (T) of the following data; - the number of bytes in the player's buffer (F); - we can get the number of bytes into the song stream (Sb = $client-streamBytes()); - from which we can deduce the play point at T; Pb = Sb F; We can collect this data for each player. The next problem is how to compare the (T, Pb) tuples for multiple players. If the stream rate is known and constant then this is relatively straightforward. If it is not known or is variable then one could either measure it over time (using either rate-of-consumption data from the player or by inspecting the stream using knowledge of the stream protocol) and assume that the average rate (perhaps the -recent- average rate) is a sufficiently-accurate measure for our purposes, or we could keep a copy of all the data in the player's buffer and calculate the run-time of the current buffer (again, by knowledge of the stream protocol). There are probably a variety of tricks that could be employed to improve the efficiency of these measures and associated calculations. *Question:* is the average bit-rate of variable-bit-rate streams likely to be good enough in practice? With the data above we can calculate the relative play-point delay for each player, at least to the point where it is handed off to the decoder. The next problem is how to get the players which are behind to -catch up-. I work on the assumption that it is better to do this than to cause players which are ahead to pause. I do not know much about the construction of the stream formats (MP3, FLAC, etc.) but I presume that one cannot simply steal a few bytes out of the stream and have it still work. I guess that there must be some kind of framing. Is it possible to detect the frame boundaries just by looking at the byte stream starting at an arbitrary point (I would have expected the protocols to have been defined this way to enable a receiver to resynchronize)? Having found the frame boundaries, can one simple cut out a frame and the stream will still play? What time period do frames represent (typical, range)? Assuming that there are good answers to all the above, then one could steal one or more frames from the next packet(s) sent to the player, so that the relative delay is removed. One problem is that this takes buffer-size-play-time seconds to take effect. So perhaps the pause idea has some merit. By sending a pause command followed by a resume (play) command at a known interval, one could probably make adjustments to around the 1ms level. It has the advantage of being effective more quickly, before running through the data in the player's buffer. The trouble would be if the resume command happened to get caught in a network packet delay then all the other players would have to be paused by the same period to get things back in sync. This could quickly get quite silly. In the case of network streams, such as Internet radio, one might find oneself either exceeding the buffering/delay that can be tolerated by the stream, causing a stream reset, or just get more and more behind real time. Any comments on the above or other suggestions would be appreciated. Alan. -- awy awy's Profile: http://forums.slimdevices.com/member.php?userid=7480 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
A couple of simple firmware enhancements could possibly be helpful. - Have each ACK include the player's 1kHz clock tick value - Have a -start-playing (unpause) at clock-tick value- command - Have a -pause for N clock-ticks- command Using the mechanism described in the previous message to determine the typical round-trip-time, this would probably be sufficiently accurate to get good synchronized player starts and could also compensate for known differences in decoder delays. The third option would allow for accurate implementation of the pause method for resynchronization described above. -- awy awy's Profile: http://forums.slimdevices.com/member.php?userid=7480 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
Given the the speed of sound in air is 345 m/s, even a 10m separation of Squeezeboxes (one upstairs, one downstairs) would result in the distant sound being delayed ~29ms to someone at the local player. -- amcluesent amcluesent's Profile: http://forums.slimdevices.com/member.php?userid=10286 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
Each time you send a 'stat' request to the player ($client-requestStatus), you do get back the player's tick value in ms since last reboot (jiffies). See Slim::Networking::Slimproto::_stat_handler. stat is called once per second while the player is playing. -- andyg andyg's Profile: http://forums.slimdevices.com/member.php?userid=3292 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
awy wrote: Hello, I'm giving some serious thought about how to improve the multi-player synchronization problem. In particular, the issue that players can get out of synchronization with each other. Good, there's room for improvement. To make sure that I am starting in the right place, let me briefly assert my understanding of how the current synchronization mechanism works. Players are synchronized at the start of a track by filling each player's buffer above a low-water mark while in 'buffering' mode (not playing), and then putting all synchronized players into play mode as near simultaneously as possible. Synchronization only occurs at the start of a track, including when the track is a remote stream such as an Internet radio station. So the first question is why do players get out of sync? Some possibilities: 1.They start out of sync because the command to start playing is not received sufficiently simultaneously by all players. I know I see this fairly regularly. I use my living room and kitchen players synced. When the 'living' player is playing and I switch on the kitchen one, they don't always start synchronized properly, even though the song restarts (annoying enough by itself). Both players are SB3's and they're both connected to the same wireless access point. Since Slimserver uses TCP I'm assuming they don't use broadcast to turn on the players at exactly the same time, but two commands must go out over TCP. This is never exactly at the same time and apparently sometimes the difference is too great for a decent sync. My idea would be to use a UDP broadcast command to trigger playback. If my understanding is correct, all players should receive that at exactly the same time - or not at all if there's a network error ;). Regards, Peter ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
I guess that I have been concentrating on the SliMP3 protocol (as I only have SliMP3s, no SBs). Reading Slim::Networking::Slimproto I see that the stat response seems to include loads of good stuff. How accurate is the 'elapsed_seconds' measure (presumably only integral seconds) and is the 'bytes_received' 64-bit counter since the start of this stream or some other epoc? -- awy awy's Profile: http://forums.slimdevices.com/member.php?userid=7480 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
Re: [slim] Looking at improving multi-player synchronization
Yes you've touched on all the right issues. The most delay the human ear will tolerate is something like 10-20ms I believe. What needs to happen in order to have better syncing is to use a synced clock with millisecond-resolution on all players and the server, using NTP, SNTP or something similar. Then the server will tell all players to start at time X, where X is some millisecond in the future greater than the largest latency to any one player. This is a firmware issue, and so I'm not sure there's much you can do with just the SlimServer code. An even harder problem is dealing with players that slowly drift out of sync on long radio streams for example. Also see 'bug 259' (http://bugs.slimdevices.com/show_bug.cgi?id=259). -- andyg andyg's Profile: http://forums.slimdevices.com/member.php?userid=3292 View this thread: http://forums.slimdevices.com/showthread.php?t=34133 ___ discuss mailing list discuss@lists.slimdevices.com http://lists.slimdevices.com/lists/listinfo/discuss
