----- Original Message ----- > From: "Yonit Halperin" <yhalp...@redhat.com> > To: spice-de...@freedesktop.org > Sent: Tuesday, February 26, 2013 1:03:46 PM > Subject: [Spice-devel] [PATCH spice-server 00/28] adaptive video streaming > > Hi, > > The following patch series introduces adaptive video streaming to > spice. > > Until now, the mjpeg quality was constant (70), and the frame rate > was modified > according to the rate of frame drops in the server side (a drop > occurs when a new frame reaches the server while > an older frame is still queued in the pipe). In the client side, the > video playback is synchronized according > to the audio playback (each audio and video frame holds an mm-time > field). The jitter-buffer size in the client > was constant as well - 100 ms. When video frames arrive late to the > client (i.e., when the audio playback is ahead of them), > they are dropped. > > The adaptive video streaming is implemented by the following > heuristic: > Given a bit rate, we calculate the best combination of mjpeg quality > and frame rate (henceforth, the stream parameters) for this > bit rate. In order to decide this combination, we evaluate the > encoding size for different jpeg > qualities by applying them on successive frames. > Every new stream is assigned with an initial bit rate. The bit rate > is re-estimated and > modified during the stream life time. The bit-rate is modified based > on: > 1) periodic reports from the client: > The client reports includes information about drops and the > playback latency. > In response to drops, or too short playback latency, we decrease > the bit rate. > In response to reports that suggest that the client playback is > stable with the > current configuration, we try to increase the bit-rate. > 2) server drops: the bit-rate is decreased when server drops occur. > > Each time the bit rate changes, the stream parameters are > re-evaluated. > In addition, we monitor the frames' encoding size, and when there is > a change > that may allow improving the stream parameters, or alternatively, > requires decreasing the > quality, we again re-evaluate them.
What kind of metrics do we expose - how can I get the bitrate back on the client and/or server? > > Other changes: > -------------- > > Besides the client reports, I also added to the protocol a message > that controls the > audio playback latency, for allowing better synchronization of the > audio and video playback buffering. > > The roundtrip time is used for estimating the required playback > delay. In order to get a more accurate estimation > of the roundtrip time I also added an option to measure it > periodically instead of just on startup, and > take the minimum measurement as estimation. > > Results > ------- > I compared the video quality of the current spice master, and of the > new spice, under different network setups. > Spice master was a bit modified for making the comparison more fair: > I increased the audio jitter buffer to 200ms (instead of 100), > and also included the patch "red_worker: stream agent - fix > miscounting of frames". > The network setup was emulated using tc. > > You can find the tests details and the results in a following email. > > For 5Mpbs and 60ms roundtrip (Test1), in spice-master, more than 70% > of the frames that are sent to the client are being dropped, and the > video > is unwatchable. With new spice, while the average frame rate is about > the same, only about 2% of the frames are being dropped by the > client. > For 2.5Mbps and 60ms (Test2), as expected, things gets worse for > spice-master, and the drops rate reaches 90%. For the new spice, it > is less then 20%, and > the video is watchable. > > I also tested a setup of 10Mbps with high latency (170ms, Test3). The > latency affects the initial bit rate estimation in spice (probably > due to the tcp acks overhead). > Thus, the stream is started with a bit-rate estimation of less then > 1.25Mbps. The adaptive video heuristic gradually converges to a > higher bit rate (the column "end-bit-rate"), and > the next video stream will be started with the improved bit rate > estimation. > In Test5 I tested a real environment with a network setup similar to > Test3. However, the test are not comparable because in Test5 setup > (different server and guest), > the basic frame rate (i.e., from the guest to the server) is much > smaller (still need to investigate why). > > In Test4 (20Mbps; <1 ms roundtrip), I evaluated and unlimited setup, > i.e., a setup which will allow the best frame rate and jpeg-quality > for the stream. > With new spice, the capacity of the channel is exploited efficiently. > With spice-master, the condition for dropping frames according to > the defined fps is too strict, > and the observed frame rate is smaller then the maximum possible. > > Video streaming short-term TODO: > ---------------- > - Implement playback-latency adjustments for spice-gtk gstreamer > front-end. > - Add vp8 encoding > - Solve some problems we have with video identification. > - Try to achieve faster convergence to the "right" bit-rate when we > start with a wrong estimation. > > long-term TODO: > --------------- > video pass-through > > Regards, > Yonit. > > Yonit Halperin (28): > red_worker: stream agent - fix miscounting of frames > server/red_worker: streams: moving mjpeg_encoder from Stream to > StreamAgent > mjpeg_encoder: configure mjpeg quality and frame rate according to > a > given bit rate > mjpeg_encoder: re-configure stream parameters when the frame's > encoding size changes > mjpeg_encoder: adjust the stream bit rate based on periodic client > feedback > mjpeg_encoder: modify stream bit rate based on server side pipe > congestion > mjpeg_encoder: update the client with estimations for the required > playback latency > mjpeg_encoder: move the control over frame drops to mjpeg_encoder > mjpeg_encoder: keep the average observed fps similar to the defined > fps > mjpeg_encoder: add stream warmup time, in which we avoid server and > client drops > server: spice_timer_queue > server/red_worker: assign timer callbacks to worker_core, using > spice_timer_queue > red_channel: monitor connection latency using MSG_PING > red_worker: stream - update periodically the input frame rate > server/red_worker: enable latency monitoring in the display channel > red_worker: start using mjpeg_encoder rate control capabilities > red_worker: support SPICE_MSGC_DISPLAY_STREAM_REPORT > red_worker: notify mjpeg_encoder on server frame drops > red_worker: ignoring video frame drops that are not due to pipe > congestion > dispatcher.h: fix - s/#define MAIN_DISPATCHER_H/#define > DISPATCHER_H > snd_worker: support sending SPICE_MSG_PLAYBACK_LATENCY > reds: support mm_time latency adjustments > red_worker: video streams - adjust client playback latency > server/red_worker.c: use the bit rate of old streams as a start > point > for new streams > server/red_worker: add an option to supply the bandwidth via env > var > collect and print video stream statistics > red_worker: increase the interval limit for stream frames > red_worker: assign mm_time to vga frames > > server/Makefile.am | 2 + > server/dispatcher.h | 6 +- > server/inputs_channel.c | 1 + > server/main_channel.c | 7 +- > server/main_channel.h | 1 + > server/main_dispatcher.c | 32 ++ > server/main_dispatcher.h | 2 + > server/mjpeg_encoder.c | 981 > ++++++++++++++++++++++++++++++++++++++++++++- > server/mjpeg_encoder.h | 70 +++- > server/red_channel.c | 228 +++++++++++ > server/red_channel.h | 18 + > server/red_dispatcher.c | 1 + > server/red_worker.c | 514 +++++++++++++++++++----- > server/reds-private.h | 2 + > server/reds.c | 28 +- > server/reds.h | 2 + > server/smartcard.c | 1 + > server/snd_worker.c | 45 +++ > server/snd_worker.h | 2 + > server/spice_timer_queue.c | 268 +++++++++++++ > server/spice_timer_queue.h | 43 ++ > server/spicevmc.c | 1 + > spice-common | 2 +- > 23 files changed, 2148 insertions(+), 109 deletions(-) > create mode 100644 server/spice_timer_queue.c > create mode 100644 server/spice_timer_queue.h > > -- > 1.8.1 > > _______________________________________________ > Spice-devel mailing list > Spice-devel@lists.freedesktop.org > http://lists.freedesktop.org/mailman/listinfo/spice-devel > _______________________________________________ Spice-devel mailing list Spice-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/spice-devel
