Re: [FFmpeg-user] problems consuming live single file hls
Since hls is a streaming format i would actually expect this to work. >>> It does work for "multi file" which is what hls is meant to be. >> Byte-range is part of the standard (see >> https://tools.ietf.org/html/draft-pantos-http-live-streaming-08#section-3.4.1), >> so single file is as much "meant to be" as multi-file afaict. > But the playlist does not change or does it? Ah, I think now I understand where we missed each other: Yes it does! That is one of the central points in HLS: Video is produced in segments (either in separate files or by appending to one file, but in either case playable individually). After a segment is produced it is "published" via the playlist by appending some lines to it. To do this atomic the file is usually copied, appended to, and then "swapped" back in via rename. The single file hls implementation in ffmpeg somehow fails to properly order these things, i.e. segments become visible in the playlist before they are fully written, breaking the standard hls consumer pattern. > >>> If you use single file, I don't see how the behaviour can be >>> different, you can reproduce it using mpegts on your local >>> file system. >> What do you mean? The desire is for new entries in the playlist to >> become visible only after the data is visible to open/read. You are >> saying this is impossible? I cant imagine that to be honest. Would you >> accept a patch if i can get this to work? > Patches are always preferred over bug reports (but bug reports > are very welcome), and there is no doubt that I may completely > misunderstand the issue. I think to some extent that happened (see above where i try to reconcile), but that my be entirely my fault. I will see what I can do. Best Daniel Oberhoff pEpkey.asc Description: application/pgp-keys ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] problems consuming live single file hls
2019-01-12 11:44 GMT+01:00, Daniel Oberhoff : >>> Since hls is a streaming format i would actually expect this to work. >> It does work for "multi file" which is what hls is meant to be. > > Byte-range is part of the standard (see > https://tools.ietf.org/html/draft-pantos-http-live-streaming-08#section-3.4.1), > so single file is as much "meant to be" as multi-file afaict. But the playlist does not change or does it? >> If you use single file, I don't see how the behaviour can be >> different, you can reproduce it using mpegts on your local >> file system. > > What do you mean? The desire is for new entries in the playlist to > become visible only after the data is visible to open/read. You are > saying this is impossible? I cant imagine that to be honest. Would you > accept a patch if i can get this to work? Patches are always preferred over bug reports (but bug reports are very welcome), and there is no doubt that I may completely misunderstand the issue. Carl Eugen ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] problems consuming live single file hls
(sorry, i keep miss-sending this to your personal address) >> Since hls is a streaming format i would actually expect this to work. > It does work for "multi file" which is what hls is meant to be. Byte-range is part of the standard (see https://tools.ietf.org/html/draft-pantos-http-live-streaming-08#section-3.4.1), so single file is as much "meant to be" as multi-file afaict. > If you use single file, I don't see how the behaviour can be > different, you can reproduce it using mpegts on your local > file system. What do you mean? The desire is for new entries in the playlist to become visible only after the data is visible to open/read. You are saying this is impossible? I cant imagine that to be honest. Would you accept a patch if i can get this to work? Best Daniel pEpkey.asc Description: application/pgp-keys pEpkey.asc Description: application/pgp-keys ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] problems consuming live single file hls
Am 10.01.19 um 18:04 schrieb Carl Eugen Hoyos: > 2019-01-10 14:59 GMT+01:00, Daniel Oberhoff : > >> I previously reported on problems with consuming a live single >> file hls stream. It was quite embedded in the application so i >> failed to make a good reproducible case. Now i reduced it. > Please test current FFmpeg git head. same behavior (i attach logs again) > Are you missing the "-re" option? > Iiuc, you trying to read as-fast-as-possible an input > that is just created: How is this supposed to work, > what behaviour do you expect? I dont think -re is applicable. Actually it may be the consumer is slower than the producer, which is ok. Or it may be slower for some time and later catch up. So, yes, i want to consume as fast as possible. Since hls is a streaming format i would actually expect this to work. It does for multi file. The rationale is simple: the producer produces a m3u8 plalist, appending to it new segments as they are produced. i.e. after they have been fully written. Then the consumer sees them (by polling the m3u8) and consumes them. This is how i would expect it to work over http also, i.e. the consumer polls the playlist and consumes any new segments. We simply shortcut here and do it locally via the file system, which should be ok. Am I missing something? As I said, for multi-file hls we dont have these problems, and i am not sure why single file should be different. The only difference really is that instead of whole files the data is in a byte range of a file. Best Daniel ffmpeg version N-92950-gc65c4ce773 Copyright (c) 2000-2019 the FFmpeg developers built with Apple LLVM version 10.0.0 (clang-1000.11.45.5) configuration: libavutil 56. 25.100 / 56. 25.100 libavcodec 58. 43.100 / 58. 43.100 libavformat58. 25.100 / 58. 25.100 libavdevice58. 6.101 / 58. 6.101 libavfilter 7. 47.100 / 7. 47.100 libswscale 5. 4.100 / 5. 4.100 libswresample 3. 4.100 / 3. 4.100 [hls,applehttp @ 0x7fb60e00] Opening 'test_single_file_mp4.m4s' for reading Last message repeated 1 times Input #0, hls,applehttp, from 'test_single_file_mp4.m3u8': Duration: N/A, start: 28.00, bitrate: 51 kb/s Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 51 kb/s, 30 fps, 30 tbr, 15360 tbn, 30720 tbc Metadata: variant_bitrate : 0 compatible_brands: iso6mp41 major_brand : iso5 minor_version : 512 encoder : Lavf58.25.100 Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native)) Press [q] to stop, [?] for help Output #0, null, to '/dev/null': Metadata: encoder : Lavf58.25.100 Stream #0:0: Video: wrapped_avframe, yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc Metadata: variant_bitrate : 0 compatible_brands: iso6mp41 major_brand : iso5 minor_version : 512 encoder : Lavc58.43.100 wrapped_avframe [hls,applehttp @ 0x7fb60e00] Opening 'test_single_file_mp4.m4s' for reading Last message repeated 3 times [h264 @ 0x7fb60100d600] Invalid NAL unit 0, skipping. frame= 156 fps= 79 q=-0.0 size=N/A time=00:00:05.20 bitrate=N/A speed=2.64x [h264 @ 0x7fb60100dc00] Invalid NAL unit size (-503519357 > 3150). [h264 @ 0x7fb60100dc00] Error splitting the input into NAL units. [h264 @ 0x7fb601810800] Invalid NAL unit size (-1920240576 > 3132). [h264 @ 0x7fb601810800] Error splitting the input into NAL units. [h264 @ 0x7fb601002800] Invalid NAL unit size (-805746105 > 3132). [h264 @ 0x7fb601002800] Error splitting the input into NAL units. [h264 @ 0x7fb60100d000] Invalid NAL unit size (271284657 > 3114). [h264 @ 0x7fb60100d000] Error splitting the input into NAL units. [h264 @ 0x7fb60100d600] error while decoding MB 34 8, bytestream -6 [h264 @ 0x7fb60100d600] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented. [h264 @ 0x7fb60100d600] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-de...@ffmpeg.org) [h264 @ 0x7fb60100d600] concealing 2975 DC, 2975 AC, 2975 MV errors in I frame Error while decoding stream #0:0: Invalid data found when processing input [h264 @ 0x7fb60100d600] Invalid NAL unit size (453635919 > 3121). [h264 @ 0x7fb60100d600] Error splitting the input into NAL units. Error while decoding stream #0:0: Invalid data found when processing input [h264 @ 0x7fb60100dc00] Invalid NAL unit size (-1955315659 > 3109). [h264 @ 0x7fb60100dc00] Error splitting the input into NAL units. Error while decoding stream #0:0: Invalid data found when processing input [h264 @ 0x7fb601810800] Invalid NAL unit size (-653815108 > 3125). [h264 @ 0x7fb601810800] Error splitting the
Re: [FFmpeg-user] problems consuming live single file hls
2019-01-10 14:59 GMT+01:00, Daniel Oberhoff : > I previously reported on problems with consuming a live single > file hls stream. It was quite embedded in the application so i > failed to make a good reproducible case. Now i reduced it. Please test current FFmpeg git head. Are you missing the "-re" option? Iiuc, you trying to read as-fast-as-possible an input that is just created: How is this supposed to work, what behaviour do you expect? Carl Eugen ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-user] problems consuming live single file hls
Hello, I previously reported on problems with consuming a live single file hls stream. It was quite embedded in the application so i failed to make a good reproducible case. Now i reduced it. So i have a producer and a consumer. The producer just generates test video like this: ffmpeg -f lavfi -i testsrc=duration=100:size=1280x720:rate=30 -g 1 -hls_flags single_file -hls_list_size 0 -hls_segment_type fmp4 -f hls test_single_file_mp4.m3u8 the consumer (which i start a few seconds in) just consumes the video and discards the frames: ffmpeg -i test_single_file_mp4.m3u8 -f null /dev/null the consumer after a few seconds reports invalid nal unit sizes. I attach producer and consumer logs. Before i consumed the stream with my own c++ based program and saw that i had to basically ignore the last segment in the file, because it often was not fully written after it had been "published" via the m3u8. I did not have this problem when using multi-file hls. Any Ideas? Thanks for help! We really need single file hls in our product (https://game-on-technologies.com/) since our file systems start having problems with all the small files. Best Daniel Oberhoff ffmpeg version 4.0.2 Copyright (c) 2000-2018 the FFmpeg developers built with Apple LLVM version 10.0.0 (clang-1000.11.45.2) configuration: --prefix=/usr/local/Cellar/ffmpeg/4.0.2_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gpl --enable-libmp3lame --enable-libopus --enable-libsnappy --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-opencl --enable-videotoolbox libavutil 56. 14.100 / 56. 14.100 libavcodec 58. 18.100 / 58. 18.100 libavformat58. 12.100 / 58. 12.100 libavdevice58. 3.100 / 58. 3.100 libavfilter 7. 16.100 / 7. 16.100 libavresample 4. 0. 0 / 4. 0. 0 libswscale 5. 1.100 / 5. 1.100 libswresample 3. 1.100 / 3. 1.100 libpostproc55. 1.100 / 55. 1.100 [hls,applehttp @ 0x7f8ebd800400] Opening '../tests/test_single_file_mp4.m4s' for reading Last message repeated 1 times Input #0, hls,applehttp, from '../tests/test_single_file_mp4.m3u8': Duration: N/A, start: 36.00, bitrate: 66 kb/s Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Video: h264 (High 4:4:4 Intra) (avc1 / 0x31637661), yuv444p, 1280x720 [SAR 1:1 DAR 16:9], 66 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc Metadata: variant_bitrate : 0 compatible_brands: iso6mp41 major_brand : iso5 minor_version : 512 encoder : Lavf58.12.100 Stream mapping: Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native)) Press [q] to stop, [?] for help Output #0, null, to '/dev/null': Metadata: encoder : Lavf58.12.100 Stream #0:0: Video: wrapped_avframe, yuv444p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 30 fps, 30 tbn, 30 tbc Metadata: variant_bitrate : 0 compatible_brands: iso6mp41 major_brand : iso5 minor_version : 512 encoder : Lavc58.18.100 wrapped_avframe [hls,applehttp @ 0x7f8ebd800400] Opening '../tests/test_single_file_mp4.m4s' for reading Last message repeated 3 times [h264 @ 0x7f8ebd822000] Invalid NAL unit 0, skipping. [h264 @ 0x7f8ebd822600] Invalid NAL unit size (-1719266706 > 5149). [h264 @ 0x7f8ebd822600] Error splitting the input into NAL units. frame= 135 fps= 68 q=-0.0 size=N/A time=00:00:04.50 bitrate=N/A speed=2.27x [h264 @ 0x7f8ebd817e00] Invalid NAL unit size (-2095900588 > 5112). [h264 @ 0x7f8ebd817e00] Error splitting the input into NAL units. [h264 @ 0x7f8ebd820400] Invalid NAL unit size (-1875114319 > 5184). [h264 @ 0x7f8ebd820400] Error splitting the input into NAL units. [h264 @ 0x7f8ebd821a00] Invalid NAL unit size (-795588519 > 5120). [h264 @ 0x7f8ebd821a00] Error splitting the input into NAL units. [h264 @ 0x7f8ebd822000] error while decoding MB 6 39, bytestream -18 [h264 @ 0x7f8ebd822000] data partitioning is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented. [h264 @ 0x7f8ebd822000] If you want to help, upload a sample of this file to ftp://upload.ffmpeg.org/incoming/ and contact the ffmpeg-devel mailing list. (ffmpeg-de...@ffmpeg.org) [h264 @ 0x7f8ebd822000] concealing 523 DC, 523 AC, 523 MV errors in I frame [h264 @ 0x7f8ebd822000] Invalid NAL unit size (-746374325 > 5151). [h264 @ 0x7f8ebd822000] Error splitting the input into NAL units. Error while decoding stream #0:0: Invalid data found when processing input Last message repeated 1 times [h264 @ 0x7f8ebd822600] Invalid NAL unit size (1739322824 > 5142). [h264 @ 0x7f8ebd822600] Error splitting the input into NAL units. Error while decoding stream #0:0: Invalid
