Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
On 13.10.2015 08:42, Tomas Härdin wrote: On Mon, 2015-10-05 at 14:25 +0200, Tobias Rapp wrote: On 05.10.2015 09:10, tim nicholson wrote: On 04/10/15 13:07, Tomas Härdin wrote: On Mon, 2015-09-28 at 15:11 +0200, Tobias Rapp wrote: [...] For me the most important thing is that anything dealing with MXF should never write invalid files. +1 for sure. To overstate your point: So it would be OK to skip most input frames and replace them with black frames as long as the output file is valid MXF? I think there are different requirements for determining what makes an "error" depending on the use-case. If I try to recover video frames from an broken hard disk, for example, I probably won't mind some lost frames. If I try to encode a video file from a presumably healthy input file I likely care about lost frames. I'm not sure whether the current code is broken per se, but it does look suspicious. Perhaps someone else has a better idea? Truncate/pad mis sized frames to allow muxing to continue, and issue a warning (as at present)? It is currently quite difficult to ensure that all frames have been transcoded if there is only a warning in the log message because AFAIK the only generic way to separate a info log message from a warning is to parse terminal color code sequences. The other solution would be to maintain a RegEx black-list of log output messages in the parent process. Wouldn't it be helpful to introduce some flag option like "-flags +xerror" on avformat level to toggle between "recover as much as possible" mode and "encode all or fail" mode? Possibly. Feels like it'd be tricky to write tests for. But I do see a point in having a mode where lavf is not allowed to do anything strange, as it often tends to want to do. But this really touches on some rather nasty bits of how lavf is (not) structured, and I think it might be easier if you just keep some local hack for your workflow rather than try to bring some sanity to the situation.. I agree that it would cause too much impact to add some "lavf mode switch" that covers the different use-cases. Will work-around it by some log message pattern matching in my application. Thanks for taking the time to discuss the issue. I think I have a better view on the situation now. Regards, Tobias ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
On Mon, 2015-10-05 at 14:25 +0200, Tobias Rapp wrote: > On 05.10.2015 09:10, tim nicholson wrote: > > On 04/10/15 13:07, Tomas Härdin wrote: > >> On Mon, 2015-09-28 at 15:11 +0200, Tobias Rapp wrote: > >>> [...] > >> > >> For me the most important thing is that anything dealing with MXF should > >> never write invalid files. > > > > +1 for sure. > > To overstate your point: So it would be OK to skip most input frames and > replace them with black frames as long as the output file is valid MXF? > > I think there are different requirements for determining what makes an > "error" depending on the use-case. If I try to recover video frames from > an broken hard disk, for example, I probably won't mind some lost > frames. If I try to encode a video file from a presumably healthy input > file I likely care about lost frames. > > >> I'm not sure whether the current code is > >> broken per se, but it does look suspicious. Perhaps someone else has a > >> better idea? > >> > > > > Truncate/pad mis sized frames to allow muxing to continue, and issue a > > warning (as at present)? > > It is currently quite difficult to ensure that all frames have been > transcoded if there is only a warning in the log message because AFAIK > the only generic way to separate a info log message from a warning is to > parse terminal color code sequences. The other solution would be to > maintain a RegEx black-list of log output messages in the parent process. > > Wouldn't it be helpful to introduce some flag option like "-flags > +xerror" on avformat level to toggle between "recover as much as > possible" mode and "encode all or fail" mode? Possibly. Feels like it'd be tricky to write tests for. But I do see a point in having a mode where lavf is not allowed to do anything strange, as it often tends to want to do. But this really touches on some rather nasty bits of how lavf is (not) structured, and I think it might be easier if you just keep some local hack for your workflow rather than try to bring some sanity to the situation.. /Tomas ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
On 05.10.2015 09:10, tim nicholson wrote: On 04/10/15 13:07, Tomas Härdin wrote: On Mon, 2015-09-28 at 15:11 +0200, Tobias Rapp wrote: [...] For me the most important thing is that anything dealing with MXF should never write invalid files. +1 for sure. To overstate your point: So it would be OK to skip most input frames and replace them with black frames as long as the output file is valid MXF? I think there are different requirements for determining what makes an "error" depending on the use-case. If I try to recover video frames from an broken hard disk, for example, I probably won't mind some lost frames. If I try to encode a video file from a presumably healthy input file I likely care about lost frames. I'm not sure whether the current code is broken per se, but it does look suspicious. Perhaps someone else has a better idea? Truncate/pad mis sized frames to allow muxing to continue, and issue a warning (as at present)? It is currently quite difficult to ensure that all frames have been transcoded if there is only a warning in the log message because AFAIK the only generic way to separate a info log message from a warning is to parse terminal color code sequences. The other solution would be to maintain a RegEx black-list of log output messages in the parent process. Wouldn't it be helpful to introduce some flag option like "-flags +xerror" on avformat level to toggle between "recover as much as possible" mode and "encode all or fail" mode? Regards, Tobias ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/10/15 13:07, Tomas Härdin wrote: > On Mon, 2015-09-28 at 15:11 +0200, Tobias Rapp wrote: >> On 19.09.2015 22:49, Tomas Härdin wrote: >>> On Wed, 2015-09-16 at 14:33 +0200, Tobias Rapp wrote: Hi, attached patch fixes ticket #4759 but I guess it is a bit too hasty to always abort transcoding if a single frame cannot be written. I gue ss it would be better to check for some "exit_on_error" like flag set but couldn't find out how to achieve that. Any comments would be appreciated. Regards, Tobias >>> >>> From 7d6f8de2a411817c970a19d8766e69b6eb604132 Mon Sep 17 00:00:00 2001 From: Tobias Rapp Date: Mon, 14 Sep 2015 12:06:22 +0200 Subject: [PATCH] avformat/mxfenc: stop encoding if unfilled video p acket occurs Fixes ticket #4759. --- libavformat/mxfenc.c | 14 +- 1 file changed, 9 insertions(+), 5 deletions(-) >> [...] >>> >>> Is this really better than not writing anything? >>> >>> /Tomas >> >> (Sorry for the long delay in answering, I was caught by a flu last we ek.) >> >> I want to transcode thousands of files and want to get some indicatio n >> from ffmpeg if the transcoding has been successful (all frames have b een >> transcoded) or not. Currently I am using the process exit code to ver ify >> that. >> >> There are two different user stories when using ffmpeg: >> a) "process the input data and exit with error as early as possible i n >> case of errors/problems" >> b) "process the input data and avoid exit with error as long as possi ble >> in case of error/problems" >> >> If I understand correctly I can basically switch between (a) and (b) >> mode by passing the "-xerror" option to ffmpeg or not. So my plan is to >> transcode all my files with "-xerror" and assume that >99% of the fil es >> will pass. The small amount of failing ones can then be analyzed for >> problems manually and possibly have a re-run without "-xerror". >> >> Unfortunately the "-xerror" option affects only ffmpeg but not the >> libraries, so I cannot use it do decide if "mxf_write_packet" should >> return with an error when the video packet size doesn't match the CBR >> constraints. > > For me the most important thing is that anything dealing with MXF shou ld > never write invalid files. +1 for sure. > I'm not sure whether the current code is > broken per se, but it does look suspicious. Perhaps someone else has a > better idea? > Truncate/pad mis sized frames to allow muxing to continue, and issue a warning (as at present)? > /Tomas > > > > ___ > ffmpeg-devel mailing list > ffmpeg-devel@ffmpeg.org > http://ffmpeg.org/mailman/listinfo/ffmpeg-devel > - -- Tim. Key Fingerprint 38CF DB09 3ED0 F607 8B67 6CED 0C0B FC44 8B0B FC83 -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iQEcBAEBAgAGBQJWEiJtAAoJEAwL/ESLC/yDGQAIAJmqWRatBPd/2SXuzyFGWqB2 sg/FBpbzRqFGxr8dOfqfSkwLWu+EUL66UZKu5liVKkeksiUgx4sPtj6DFGFX7ozV g2DbJxvsFG18ES9C8OHc3qRDsgF4Z+F4GuScWMPrVcyvimSdHeajVkS8slrZlg2Y tMGQSin5jLwzv1pGIhCOdQLEndrr+PzwajPI837UBW2e7znWDb1uNHBy1yiPzcf+ 0pyZluyhkqW6IBzgO34Dc39DfQdGrtDWlzyUacT7nQz/4W5q2KAAhUhBTNtqTabK KCtSUfpqWgu6P2xxNvi87F9acqYprS80RC/ovrdgsXiXTNGSbYVscY748xg8mgA= =Clgr -END PGP SIGNATURE- ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
On Mon, 2015-09-28 at 15:11 +0200, Tobias Rapp wrote: > On 19.09.2015 22:49, Tomas Härdin wrote: > > On Wed, 2015-09-16 at 14:33 +0200, Tobias Rapp wrote: > >> Hi, > >> > >> attached patch fixes ticket #4759 but I guess it is a bit too hasty to > >> always abort transcoding if a single frame cannot be written. I guess it > >> would be better to check for some "exit_on_error" like flag set but > >> couldn't find out how to achieve that. > >> > >> Any comments would be appreciated. > >> > >> Regards, > >> Tobias > > > > > >> From 7d6f8de2a411817c970a19d8766e69b6eb604132 Mon Sep 17 00:00:00 2001 > >> From: Tobias Rapp > >> Date: Mon, 14 Sep 2015 12:06:22 +0200 > >> Subject: [PATCH] avformat/mxfenc: stop encoding if unfilled video packet > >> occurs > >> > >> Fixes ticket #4759. > >> --- > >> libavformat/mxfenc.c | 14 +- > >> 1 file changed, 9 insertions(+), 5 deletions(-) > >> > [...] > > > > Is this really better than not writing anything? > > > > /Tomas > > (Sorry for the long delay in answering, I was caught by a flu last week.) > > I want to transcode thousands of files and want to get some indication > from ffmpeg if the transcoding has been successful (all frames have been > transcoded) or not. Currently I am using the process exit code to verify > that. > > There are two different user stories when using ffmpeg: > a) "process the input data and exit with error as early as possible in > case of errors/problems" > b) "process the input data and avoid exit with error as long as possible > in case of error/problems" > > If I understand correctly I can basically switch between (a) and (b) > mode by passing the "-xerror" option to ffmpeg or not. So my plan is to > transcode all my files with "-xerror" and assume that >99% of the files > will pass. The small amount of failing ones can then be analyzed for > problems manually and possibly have a re-run without "-xerror". > > Unfortunately the "-xerror" option affects only ffmpeg but not the > libraries, so I cannot use it do decide if "mxf_write_packet" should > return with an error when the video packet size doesn't match the CBR > constraints. For me the most important thing is that anything dealing with MXF should never write invalid files. I'm not sure whether the current code is broken per se, but it does look suspicious. Perhaps someone else has a better idea? /Tomas signature.asc Description: This is a digitally signed message part ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
On 19.09.2015 22:49, Tomas Härdin wrote: On Wed, 2015-09-16 at 14:33 +0200, Tobias Rapp wrote: Hi, attached patch fixes ticket #4759 but I guess it is a bit too hasty to always abort transcoding if a single frame cannot be written. I guess it would be better to check for some "exit_on_error" like flag set but couldn't find out how to achieve that. Any comments would be appreciated. Regards, Tobias From 7d6f8de2a411817c970a19d8766e69b6eb604132 Mon Sep 17 00:00:00 2001 From: Tobias Rapp Date: Mon, 14 Sep 2015 12:06:22 +0200 Subject: [PATCH] avformat/mxfenc: stop encoding if unfilled video packet occurs Fixes ticket #4759. --- libavformat/mxfenc.c | 14 +- 1 file changed, 9 insertions(+), 5 deletions(-) [...] Is this really better than not writing anything? /Tomas (Sorry for the long delay in answering, I was caught by a flu last week.) I want to transcode thousands of files and want to get some indication from ffmpeg if the transcoding has been successful (all frames have been transcoded) or not. Currently I am using the process exit code to verify that. There are two different user stories when using ffmpeg: a) "process the input data and exit with error as early as possible in case of errors/problems" b) "process the input data and avoid exit with error as long as possible in case of error/problems" If I understand correctly I can basically switch between (a) and (b) mode by passing the "-xerror" option to ffmpeg or not. So my plan is to transcode all my files with "-xerror" and assume that >99% of the files will pass. The small amount of failing ones can then be analyzed for problems manually and possibly have a re-run without "-xerror". Unfortunately the "-xerror" option affects only ffmpeg but not the libraries, so I cannot use it do decide if "mxf_write_packet" should return with an error when the video packet size doesn't match the CBR constraints. Regards, Tobias ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
On Wed, 2015-09-16 at 14:33 +0200, Tobias Rapp wrote:
> Hi,
>
> attached patch fixes ticket #4759 but I guess it is a bit too hasty to
> always abort transcoding if a single frame cannot be written. I guess it
> would be better to check for some "exit_on_error" like flag set but
> couldn't find out how to achieve that.
>
> Any comments would be appreciated.
>
> Regards,
> Tobias
> From 7d6f8de2a411817c970a19d8766e69b6eb604132 Mon Sep 17 00:00:00 2001
> From: Tobias Rapp
> Date: Mon, 14 Sep 2015 12:06:22 +0200
> Subject: [PATCH] avformat/mxfenc: stop encoding if unfilled video packet
> occurs
>
> Fixes ticket #4759.
> ---
> libavformat/mxfenc.c | 14 +-
> 1 file changed, 9 insertions(+), 5 deletions(-)
>
> diff --git a/libavformat/mxfenc.c b/libavformat/mxfenc.c
> index 84ce979..4eac812 100644
> --- a/libavformat/mxfenc.c
> +++ b/libavformat/mxfenc.c
> @@ -2262,7 +2262,7 @@ static void mxf_write_system_item(AVFormatContext *s)
> mxf_write_umid(s, 1);
> }
>
> -static void mxf_write_d10_video_packet(AVFormatContext *s, AVStream *st,
> AVPacket *pkt)
> +static int mxf_write_d10_video_packet(AVFormatContext *s, AVStream *st,
> AVPacket *pkt)
> {
> MXFContext *mxf = s->priv_data;
> AVIOContext *pb = s->pb;
> @@ -2286,9 +2286,12 @@ static void mxf_write_d10_video_packet(AVFormatContext
> *s, AVStream *st, AVPacke
> ffio_fill(s->pb, 0, pad);
> av_assert1(!(avio_tell(s->pb) & (KAG_SIZE-1)));
> } else {
> -av_log(s, AV_LOG_WARNING, "cannot fill d-10 video packet\n");
> +av_log(s, AV_LOG_ERROR, "cannot fill d-10 video packet\n");
> ffio_fill(s->pb, 0, pad);
> +return AVERROR(EIO);
> }
> +
> +return 0;
> }
Is this really better than not writing anything?
/Tomas
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
[FFmpeg-devel] [RFC] avformat/mxfenc: stop encoding if unfilled video packet
Hi,
attached patch fixes ticket #4759 but I guess it is a bit too hasty to
always abort transcoding if a single frame cannot be written. I guess it
would be better to check for some "exit_on_error" like flag set but
couldn't find out how to achieve that.
Any comments would be appreciated.
Regards,
Tobias
>From 7d6f8de2a411817c970a19d8766e69b6eb604132 Mon Sep 17 00:00:00 2001
From: Tobias Rapp
Date: Mon, 14 Sep 2015 12:06:22 +0200
Subject: [PATCH] avformat/mxfenc: stop encoding if unfilled video packet
occurs
Fixes ticket #4759.
---
libavformat/mxfenc.c | 14 +-
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/libavformat/mxfenc.c b/libavformat/mxfenc.c
index 84ce979..4eac812 100644
--- a/libavformat/mxfenc.c
+++ b/libavformat/mxfenc.c
@@ -2262,7 +2262,7 @@ static void mxf_write_system_item(AVFormatContext *s)
mxf_write_umid(s, 1);
}
-static void mxf_write_d10_video_packet(AVFormatContext *s, AVStream *st, AVPacket *pkt)
+static int mxf_write_d10_video_packet(AVFormatContext *s, AVStream *st, AVPacket *pkt)
{
MXFContext *mxf = s->priv_data;
AVIOContext *pb = s->pb;
@@ -2286,9 +2286,12 @@ static void mxf_write_d10_video_packet(AVFormatContext *s, AVStream *st, AVPacke
ffio_fill(s->pb, 0, pad);
av_assert1(!(avio_tell(s->pb) & (KAG_SIZE-1)));
} else {
-av_log(s, AV_LOG_WARNING, "cannot fill d-10 video packet\n");
+av_log(s, AV_LOG_ERROR, "cannot fill d-10 video packet\n");
ffio_fill(s->pb, 0, pad);
+return AVERROR(EIO);
}
+
+return 0;
}
static void mxf_write_d10_audio_packet(AVFormatContext *s, AVStream *st, AVPacket *pkt)
@@ -2459,9 +2462,10 @@ static int mxf_write_packet(AVFormatContext *s, AVPacket *pkt)
mxf_write_klv_fill(s);
avio_write(pb, sc->track_essence_element_key, 16); // write key
if (s->oformat == &ff_mxf_d10_muxer) {
-if (st->codec->codec_type == AVMEDIA_TYPE_VIDEO)
-mxf_write_d10_video_packet(s, st, pkt);
-else
+if (st->codec->codec_type == AVMEDIA_TYPE_VIDEO) {
+if ((err = mxf_write_d10_video_packet(s, st, pkt)) < 0)
+return err;
+} else
mxf_write_d10_audio_packet(s, st, pkt);
} else {
klv_encode_ber4_length(pb, pkt->size); // write length
--
1.9.1
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
