I am trying to implement the MSE[1] API for a project. This is a JavaScript API that is being backed by C++ code. The basic idea is to create a buffer that is appended opaque (from JS perspective) data that should be decoded and rendered in a <video> element. I would like to use ffmpeg (specifically libavformat) to handle demuxing of the content. I intend to use other methods to handle decoding frames.
[1]: https://w3c.github.io/media-source/ The end goal being to use ffmpeg to give us a bunch of demuxed frames. It is fine if these are out of order, we can handle reordering frames. We need to pass media segments in arbitrary order and turn that into demuxed frames that define buffered ranges of playable regions. When we try to play into a region that is not buffered, we will pause the video, waiting for more data. I tried using a custom AVIOContext with a custom read callback to read data from the appended buffers. This is able to demux frames so long as the segments appear in order. If I try to give a segment from the past, there are errors. First, the "current segment" doesn't get updated when adding segments to the index; this has a simple fix. The other problem is that we have no way to see the new frames or to even know if the frames exist. We could use the AVStream::index_entries to find frames we have already demuxed and to use seeks to get the frames that we have not received yet; however, since we don't want to store the original data for long, seeking will be hard to implement. This also seems like using internals and I would prefer a more helpful API. First, if there is a better way to do this, I would love to hear it. If not, we would like a way to have ffmpeg demux frames in file order, rather than in DTS order. The goal being to just use the MP4 parser to pull out frames, not to play the resulting frames directly. I would suggest adding a flag that would allow av_read_frame to output frames in the order they appear in the files. This would probably disable seeking since there is no way to predict where to seek to. Or maybe a different function that would act like av_read_frame which would return in file order. I would be happy to implement this (since I will probably need to do this in a fork anyway). I am wondering if there is a better way to do it now and whether a feature like this would be accepted. Thanks. _______________________________________________ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".