Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On Fri, Oct 12, 2018 at 04:23:43PM +0200, Jerome Martinez wrote:
> On 12/10/2018 15:30, Jan Ekström wrote:
> >And while it seems like we're focusing on the arbitrary limit in
> >av_malloc (which is an issue of its own), I do note that for this
> >specific case this "worst case packed frame size" allocation heuristic
> >might have to be modified? Or at least explained if possible, so that
> >things can be improved in a proper way.
>
>
> Thank you for the feedback.
> I was focused on ff_alloc_packet2() which accepts 64-bit integers, and I did
> not check what is behind (limited to INT_MAX too).
> I understand now that this is a global limit, not tied to FFV1 encoder (but
> when this limit is removed from mem.h, there will be a review of ode which
> test this limit before calling ff_alloc_packet2()...).
>
> As you said in the other answer, it was unrealistic to hit such limit in
> 2005, but now... 2 GB by packet is still huge (the input frame is "only" 128
> MB), but the multiplier used in FFV1 encoder makes the limit hit for "basic"
> 4K frames. 8K frames are approaching...
> A bit complex for me to remove this 2 GB limit for sure everywhere :(, so I
> try to find another solution.
you can increase the 2gb limit to whatever you like with
av_max_alloc() / -max_alloc
That will likely result in bugs appearing which you will need to fix
also there may be other places hardcoding some limits that would need to be
reviewed and updated
If people feel the 2gb limit is an issue then we could attempt to disable it in
the fuzzer and see what happens. (after some local testing first ...)
>
>
> >
> >For historical context, this limit has been inside ffv1enc as follows,
> >number 1 being the current version:
> >1. avctx->width*avctx->height*37LL*4 ("added support for
> >AV_PIX_FMT_GBRP16" - ce2217b25eccda9f5c14022bd69792e71b417b73)
> >2. avctx->width*avctx->height*35LL*4 ("avcodec/ffv1enc: fix size used
> >for ff_alloc_packet2()" - 904a2864bdafe19d18db95ca54dfb36d72957a16)
> >3. avctx->width*avctx->height*((8*2+1+1)*4)/8 ("ffv1enc: switch to
> >encode2()." - 278d88689ba89c78f6c2667cf98025835567d78d)
> >4. ??? Limit was probably somewhere elsewhere or didn't exist before this
>
> I see it but still don't catch the reason (and I am very curious to find an
> input frame able to e.g. make an output frame 2x bigger than the input
> frame).
> Eager to have some background about it (Michael?).
the limit is a result of the v3 version. Its a upper bound of the worst
case.
With sufficient work you can probably find a much more complex expression
that would bound the size for large resolutions to a smaller value.
You will have to consider the range coder states and how they would
evolve during encoding in the worst case. Their evolution is bound to
tend towards less bad in a large frame so there the expression for the
max size will become more complex than a w*h*constant
and you better arent scared of "math proofs" because this pretty much
needs one. That is to decrease the max size you need to properly proof
that that a new bound is large enough.
The current bound should be reasonable easy to proof but the more tight
the bound is the more work it will be
also you can allocate a smaller buffer and reallocate before it becomes too
small. Multithreaded encoding will make this more complex though.
These problems are avoided in v4, this is what should be used really
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
"Nothing to hide" only works if the folks in power share the values of
you and everyone you know entirely and always will -- Tom Scott
signature.asc
Description: PGP signature
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
2018-10-12 11:59 GMT+02:00, Jerome Martinez : > On 12/10/2018 11:15, Carl Eugen Hoyos wrote: >> 2018-10-11 14:01 GMT+02:00, Jerome Martinez : >> >>> - why is having a number > INT_MAX an issue? modern machines >>> are 64-bit and have 8+ GB of RAM >> So where is the issue? > > The issue is that vanilla FFmpeg shows a warning in the scenario I > indicate (a 128 MB frame, far less than 2 GB), without obvious reason of > this warning (the multiplier used and the underlying limitation to 2 GB, > lower than the available RAM, are hard coded without comments about the > reason). At a very general level, I believe a warning that very large resolutions can have a negative effect seems not absurd. I believe it was already argued that pathological input frames can lead to multiplied output size. >> If above is true for you, can can simply remove the check locally, no? > > My goal is that **others** don't have this warning when they use a > vanilla version of FFmpeg. You seem to be 100% sure that the warning is bad, I am not convinced. > Before sending a patch for removing the limitation I don't understand > (or lowering a number I don't understand), I ask if there is a reason > for it (I guess that if this code is here, there is a reason and I > imagined that FFmpeg developers could kindly explain the reason), isn't > it the right thing to do? Do you prefer that I send directly a patch? > >> (I hope you agree that what you write is not generally true.) > > I don't catch what you want to indicate: do you mean that the only way > in FFmpeg to be compatible with all machines is to set an hard coded > limit to 2 GB? > I don't catch the goal, as you can have machines with 1 GB so this test > would be useless on them (it will not prevent other parts of the code to > have to do checks about RAM allocations) as well for machines having lot > of RAM (they can handle big frames). I wanted to explain that not every system FFmpeg runs on is a 64 bit system, I know very well that FFmpeg will have issues on memory- limited systems, iirc, it hits dnxhd on real-world systems. (The default user memory limits hit you on AIX afaiu although system memory is >100G.) > As I understand, the following test: > if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0) > return ret; > > permits to check that FFmpeg has enough RAM for storing the max encoded > frame size, whatever is the size of the RAM (1 GB or 8 GB), so I don't > understand the reason maxsize is limited to 2 GB before the call to > ff_alloc_packet2. > I kindly request more details about how hard coding 2 GB in the code > helps, for both machines having 1 GB & machines having 8 GB. Looks like > I am personally not smart enough for understanding that alone. I believe there are at least two reasons: One is that on 32bit systems, you cannot allocate larger blocks, the second reason is that dos is considered a security issue and allocating more that 2G for one frame seems like a dos approach. One solution may be to have a configure option that raises the allocation limit on 64 bit systems or to raise it for every 64bit system. Carl Eugen ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On 10/12/18, Jerome Martinez wrote:
> On 12/10/2018 15:30, Jan Ekstroem wrote:
>> And while it seems like we're focusing on the arbitrary limit in
>> av_malloc (which is an issue of its own), I do note that for this
>> specific case this "worst case packed frame size" allocation heuristic
>> might have to be modified? Or at least explained if possible, so that
>> things can be improved in a proper way.
>
>
> Thank you for the feedback.
> I was focused on ff_alloc_packet2() which accepts 64-bit integers, and I
> did not check what is behind (limited to INT_MAX too).
> I understand now that this is a global limit, not tied to FFV1 encoder
> (but when this limit is removed from mem.h, there will be a review of
> ode which test this limit before calling ff_alloc_packet2()...).
>
> As you said in the other answer, it was unrealistic to hit such limit in
> 2005, but now... 2 GB by packet is still huge (the input frame is "only"
> 128 MB), but the multiplier used in FFV1 encoder makes the limit hit for
> "basic" 4K frames. 8K frames are approaching...
> A bit complex for me to remove this 2 GB limit for sure everywhere :(,
> so I try to find another solution.
>
>
>>
>> For historical context, this limit has been inside ffv1enc as follows,
>> number 1 being the current version:
>> 1. avctx->width*avctx->height*37LL*4 ("added support for
>> AV_PIX_FMT_GBRP16" - ce2217b25eccda9f5c14022bd69792e71b417b73)
>> 2. avctx->width*avctx->height*35LL*4 ("avcodec/ffv1enc: fix size used
>> for ff_alloc_packet2()" - 904a2864bdafe19d18db95ca54dfb36d72957a16)
>> 3. avctx->width*avctx->height*((8*2+1+1)*4)/8 ("ffv1enc: switch to
>> encode2()." - 278d88689ba89c78f6c2667cf98025835567d78d)
>> 4. ??? Limit was probably somewhere elsewhere or didn't exist before this
>
> I see it but still don't catch the reason (and I am very curious to find
> an input frame able to e.g. make an output frame 2x bigger than the
> input frame).
> Eager to have some background about it (Michael?).
>
> I guess there is a good reason for the "37LL*4" which is huge (note: I
> am afraid I did not update this limit when I implemented
> AV_PIX_FMT_GBRAP16, which is 4/3x bigger), so now trying to find a way
> to warn the user but not with 1 message per frame.
>
> If I can not (easily) remove this limitation, would limiting the message
> to 1 iteration for a whole stream be an acceptable patch?
> If so I'll send such patch.
I'm looking in adoption of v4 (and improve it) version sooner than never.
Ignoring issue all together may cause crash for specially crafted
input files, even
when such scenario is highly unlikely.
The limit of 2GB or whatever small value should ultimately be removed
at some point.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On 12/10/2018 15:30, Jan Ekström wrote:
And while it seems like we're focusing on the arbitrary limit in
av_malloc (which is an issue of its own), I do note that for this
specific case this "worst case packed frame size" allocation heuristic
might have to be modified? Or at least explained if possible, so that
things can be improved in a proper way.
Thank you for the feedback.
I was focused on ff_alloc_packet2() which accepts 64-bit integers, and I
did not check what is behind (limited to INT_MAX too).
I understand now that this is a global limit, not tied to FFV1 encoder
(but when this limit is removed from mem.h, there will be a review of
ode which test this limit before calling ff_alloc_packet2()...).
As you said in the other answer, it was unrealistic to hit such limit in
2005, but now... 2 GB by packet is still huge (the input frame is "only"
128 MB), but the multiplier used in FFV1 encoder makes the limit hit for
"basic" 4K frames. 8K frames are approaching...
A bit complex for me to remove this 2 GB limit for sure everywhere :(,
so I try to find another solution.
For historical context, this limit has been inside ffv1enc as follows,
number 1 being the current version:
1. avctx->width*avctx->height*37LL*4 ("added support for
AV_PIX_FMT_GBRP16" - ce2217b25eccda9f5c14022bd69792e71b417b73)
2. avctx->width*avctx->height*35LL*4 ("avcodec/ffv1enc: fix size used
for ff_alloc_packet2()" - 904a2864bdafe19d18db95ca54dfb36d72957a16)
3. avctx->width*avctx->height*((8*2+1+1)*4)/8 ("ffv1enc: switch to
encode2()." - 278d88689ba89c78f6c2667cf98025835567d78d)
4. ??? Limit was probably somewhere elsewhere or didn't exist before this
I see it but still don't catch the reason (and I am very curious to find
an input frame able to e.g. make an output frame 2x bigger than the
input frame).
Eager to have some background about it (Michael?).
I guess there is a good reason for the "37LL*4" which is huge (note: I
am afraid I did not update this limit when I implemented
AV_PIX_FMT_GBRAP16, which is 4/3x bigger), so now trying to find a way
to warn the user but not with 1 message per frame.
If I can not (easily) remove this limitation, would limiting the message
to 1 iteration for a whole stream be an acceptable patch?
If so I'll send such patch.
Jérôme
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On Thu, Oct 11, 2018 at 4:04 PM Jerome Martinez wrote:
>
> In this case, the ratio (148 times the size of input) is proportional to
> the input frame size, and the test is on a fixed value (MAX_INT).
> I get now the reason to reserve more than the size of the input frame.
> I don't get the relationship between your explanation and these hard
> coded values (148 times the size of input and MAX_INT limit).
>
> Why 148x (37x4) and not 147 or 149?
> Why a warning and maxsize forced to MAX_INT if maxsize > MAX_INT? Is it
> an internal constraint of FFmpeg which would be not able to store
> encoded frames bigger than MAX_INT?
>
> Can any of that be changed e.g. warning only once for avoiding "spam" of
> such warning which hides other warnings during encoding?
>
> Jérôme
>
And while it seems like we're focusing on the arbitrary limit in
av_malloc (which is an issue of its own), I do note that for this
specific case this "worst case packed frame size" allocation heuristic
might have to be modified? Or at least explained if possible, so that
things can be improved in a proper way.
For historical context, this limit has been inside ffv1enc as follows,
number 1 being the current version:
1. avctx->width*avctx->height*37LL*4 ("added support for
AV_PIX_FMT_GBRP16" - ce2217b25eccda9f5c14022bd69792e71b417b73)
2. avctx->width*avctx->height*35LL*4 ("avcodec/ffv1enc: fix size used
for ff_alloc_packet2()" - 904a2864bdafe19d18db95ca54dfb36d72957a16)
3. avctx->width*avctx->height*((8*2+1+1)*4)/8 ("ffv1enc: switch to
encode2()." - 278d88689ba89c78f6c2667cf98025835567d78d)
4. ??? Limit was probably somewhere elsewhere or didn't exist before this
Best regards,
Jan
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On 12/10/2018 11:15, Carl Eugen Hoyos wrote: 2018-10-11 14:01 GMT+02:00, Jerome Martinez : - why is having a number > INT_MAX an issue? modern machines are 64-bit and have 8+ GB of RAM So where is the issue? The issue is that vanilla FFmpeg shows a warning in the scenario I indicate (a 128 MB frame, far less than 2 GB), without obvious reason of this warning (the multiplier used and the underlying limitation to 2 GB, lower than the available RAM, are hard coded without comments about the reason). If above is true for you, can can simply remove the check locally, no? My goal is that **others** don't have this warning when they use a vanilla version of FFmpeg. Before sending a patch for removing the limitation I don't understand (or lowering a number I don't understand), I ask if there is a reason for it (I guess that if this code is here, there is a reason and I imagined that FFmpeg developers could kindly explain the reason), isn't it the right thing to do? Do you prefer that I send directly a patch? (I hope you agree that what you write is not generally true.) I don't catch what you want to indicate: do you mean that the only way in FFmpeg to be compatible with all machines is to set an hard coded limit to 2 GB? I don't catch the goal, as you can have machines with 1 GB so this test would be useless on them (it will not prevent other parts of the code to have to do checks about RAM allocations) as well for machines having lot of RAM (they can handle big frames). As I understand, the following test: if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0) return ret; permits to check that FFmpeg has enough RAM for storing the max encoded frame size, whatever is the size of the RAM (1 GB or 8 GB), so I don't understand the reason maxsize is limited to 2 GB before the call to ff_alloc_packet2. I kindly request more details about how hard coding 2 GB in the code helps, for both machines having 1 GB & machines having 8 GB. Looks like I am personally not smart enough for understanding that alone. Jérôme ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
2018-10-11 14:01 GMT+02:00, Jerome Martinez : > - why is having a number > INT_MAX an issue? modern machines > are 64-bit and have 8+ GB of RAM So where is the issue? If above is true for you, can can simply remove the check locally, no? (I hope you agree that what you write is not generally true.) Carl Eugen ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On 10/11/18, Jerome Martinez wrote:
> On 11/10/2018 14:08, Paul B Mahol wrote:
>> On 10/11/18, Jerome Martinez wrote:
>>> Hi,
>>>
>>> Testing FFmpeg FFV1 encoder on big frames (more than 4K: 4300x3956), I
>>> have a warning for each frame encoded (so a lot of warnings!):
>>> [ffv1 @ 024a6bcfe880] Cannot allocate worst case packet size, the
>>> encoding could fail
>>>
>>> Checking avcodec/ffv1enc.c, it is due to the following lines:
>>> int64_t maxsize = AV_INPUT_BUFFER_MIN_SIZE
>>> + avctx->width*avctx->height*37LL*4;
>>> [...]
>>> if (maxsize > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32) {
>>> av_log(avctx, AV_LOG_WARNING, "Cannot allocate worst case
>>> packet size, the encoding could fail\n");
>>> maxsize = INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32;
>>> }
>>>
>>> The value of maxsize is 2517614784, more than INT_MAX, sure, but I don't
>>> understand:
>>> - why does FFmpeg need to allocate 148 times the size of a frame?
>>> - why is having a number > INT_MAX an issue? modern machines are 64-bit
>>> and have 8+ GB of RAM, and in practice I currently saw no encoding
>>> failure on thousands of frames.
>>>
>>> Additionally I didn't get why maxsize is reduced ("only" need of 12
>>> times the size of the frame) in case of FFV1 version > 3 (experimental
>>> right now):
>>> if (f->version > 3)
>>> maxsize = AV_INPUT_BUFFER_MIN_SIZE +
>>> avctx->width*avctx->height*3LL*4;
>>>
>>> maxsize is used for calling ff_alloc_packet2(), which accepts 64-bit
>>> numbers.
>>> if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0)
>>> return ret;
>>>
>>> Is it possible to reduce this "37x4" multiplier without risk and/or
>>> remove maxsize reduction in case of sizeof(size_t) == 8 and/or show the
>>> warning only once?
>>>
>>> Issue can be reproduced with:
>>> ./ffmpeg.exe -f lavfi -i testsrc2=size=4300x3956 -t 0.040 -pix_fmt
>>> rgba64 rgba64.dpx
>>> ./ffmpeg.exe -i rgba64.dpx -c:v ffv1 rgba64.mkv
>> You are not using latest version of FFv1.
>
>
> As far as I know, version 3 is the latest (stable) version of FFV1, and
> I use this one.
> version > 3 are experimental.
> I am not sure that advising to use an experimental version for "fixing"
> a warning is good.
>
> Additionally, in theory a 16kx16k content would still produce this
> warning with FFV1 (experimental) v4 encoding, even if there is 16
> Terabytes of RAM (more than enough for having the MAX_INT = 2 GiB raw
> frame in RAM), so I understand that the warning would still be there (in
> theory) with FFV1 (experimental) v4.
>
>
>> In latest version, encoder will not produce packets which are bigger than
>> input ones -- uncompressed raw. And instead if that happens it will encode
>> as raw video.
>>
>> Theoretically output packets for old versions of FFv1 could be several
>> times bigger than raw input frame.
>
>
> In this case, the ratio (148 times the size of input) is proportional to
> the input frame size, and the test is on a fixed value (MAX_INT).
> I get now the reason to reserve more than the size of the input frame.
> I don't get the relationship between your explanation and these hard
> coded values (148 times the size of input and MAX_INT limit).
>
> Why 148x (37x4) and not 147 or 149?
> Why a warning and maxsize forced to MAX_INT if maxsize > MAX_INT? Is it
> an internal constraint of FFmpeg which would be not able to store
> encoded frames bigger than MAX_INT?
>
> Can any of that be changed e.g. warning only once for avoiding "spam" of
> such warning which hides other warnings during encoding?
Real issue is in enforced limit of av_malloc size.
For answer to that question and others ask author who introduced such
limitation.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On 11/10/2018 14:08, Paul B Mahol wrote:
On 10/11/18, Jerome Martinez wrote:
Hi,
Testing FFmpeg FFV1 encoder on big frames (more than 4K: 4300x3956), I
have a warning for each frame encoded (so a lot of warnings!):
[ffv1 @ 024a6bcfe880] Cannot allocate worst case packet size, the
encoding could fail
Checking avcodec/ffv1enc.c, it is due to the following lines:
int64_t maxsize = AV_INPUT_BUFFER_MIN_SIZE
+ avctx->width*avctx->height*37LL*4;
[...]
if (maxsize > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32) {
av_log(avctx, AV_LOG_WARNING, "Cannot allocate worst case
packet size, the encoding could fail\n");
maxsize = INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32;
}
The value of maxsize is 2517614784, more than INT_MAX, sure, but I don't
understand:
- why does FFmpeg need to allocate 148 times the size of a frame?
- why is having a number > INT_MAX an issue? modern machines are 64-bit
and have 8+ GB of RAM, and in practice I currently saw no encoding
failure on thousands of frames.
Additionally I didn't get why maxsize is reduced ("only" need of 12
times the size of the frame) in case of FFV1 version > 3 (experimental
right now):
if (f->version > 3)
maxsize = AV_INPUT_BUFFER_MIN_SIZE +
avctx->width*avctx->height*3LL*4;
maxsize is used for calling ff_alloc_packet2(), which accepts 64-bit
numbers.
if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0)
return ret;
Is it possible to reduce this "37x4" multiplier without risk and/or
remove maxsize reduction in case of sizeof(size_t) == 8 and/or show the
warning only once?
Issue can be reproduced with:
./ffmpeg.exe -f lavfi -i testsrc2=size=4300x3956 -t 0.040 -pix_fmt
rgba64 rgba64.dpx
./ffmpeg.exe -i rgba64.dpx -c:v ffv1 rgba64.mkv
You are not using latest version of FFv1.
As far as I know, version 3 is the latest (stable) version of FFV1, and
I use this one.
version > 3 are experimental.
I am not sure that advising to use an experimental version for "fixing"
a warning is good.
Additionally, in theory a 16kx16k content would still produce this
warning with FFV1 (experimental) v4 encoding, even if there is 16
Terabytes of RAM (more than enough for having the MAX_INT = 2 GiB raw
frame in RAM), so I understand that the warning would still be there (in
theory) with FFV1 (experimental) v4.
In latest version, encoder will not produce packets which are bigger than
input ones -- uncompressed raw. And instead if that happens it will encode
as raw video.
Theoretically output packets for old versions of FFv1 could be several
times bigger than raw input frame.
In this case, the ratio (148 times the size of input) is proportional to
the input frame size, and the test is on a fixed value (MAX_INT).
I get now the reason to reserve more than the size of the input frame.
I don't get the relationship between your explanation and these hard
coded values (148 times the size of input and MAX_INT limit).
Why 148x (37x4) and not 147 or 149?
Why a warning and maxsize forced to MAX_INT if maxsize > MAX_INT? Is it
an internal constraint of FFmpeg which would be not able to store
encoded frames bigger than MAX_INT?
Can any of that be changed e.g. warning only once for avoiding "spam" of
such warning which hides other warnings during encoding?
Jérôme
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
Re: [FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
On 10/11/18, Jerome Martinez wrote:
> Hi,
>
> Testing FFmpeg FFV1 encoder on big frames (more than 4K: 4300x3956), I
> have a warning for each frame encoded (so a lot of warnings!):
> [ffv1 @ 024a6bcfe880] Cannot allocate worst case packet size, the
> encoding could fail
>
> Checking avcodec/ffv1enc.c, it is due to the following lines:
> int64_t maxsize = AV_INPUT_BUFFER_MIN_SIZE
>+ avctx->width*avctx->height*37LL*4;
> [...]
> if (maxsize > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32) {
> av_log(avctx, AV_LOG_WARNING, "Cannot allocate worst case
> packet size, the encoding could fail\n");
> maxsize = INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32;
> }
>
> The value of maxsize is 2517614784, more than INT_MAX, sure, but I don't
> understand:
> - why does FFmpeg need to allocate 148 times the size of a frame?
> - why is having a number > INT_MAX an issue? modern machines are 64-bit
> and have 8+ GB of RAM, and in practice I currently saw no encoding
> failure on thousands of frames.
>
> Additionally I didn't get why maxsize is reduced ("only" need of 12
> times the size of the frame) in case of FFV1 version > 3 (experimental
> right now):
> if (f->version > 3)
> maxsize = AV_INPUT_BUFFER_MIN_SIZE +
> avctx->width*avctx->height*3LL*4;
>
> maxsize is used for calling ff_alloc_packet2(), which accepts 64-bit
> numbers.
> if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0)
> return ret;
>
> Is it possible to reduce this "37x4" multiplier without risk and/or
> remove maxsize reduction in case of sizeof(size_t) == 8 and/or show the
> warning only once?
>
> Issue can be reproduced with:
> ./ffmpeg.exe -f lavfi -i testsrc2=size=4300x3956 -t 0.040 -pix_fmt
> rgba64 rgba64.dpx
> ./ffmpeg.exe -i rgba64.dpx -c:v ffv1 rgba64.mkv
You are not using latest version of FFv1.
In latest version, encoder will not produce packets which are bigger than
input ones -- uncompressed raw. And instead if that happens it will encode
as raw video.
Theoretically output packets for old versions of FFv1 could be several
times bigger than raw input frame.
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
[FFmpeg-devel] ffv1enc: question about "Cannot allocate worst case packet size, the encoding could fail"
Hi,
Testing FFmpeg FFV1 encoder on big frames (more than 4K: 4300x3956), I
have a warning for each frame encoded (so a lot of warnings!):
[ffv1 @ 024a6bcfe880] Cannot allocate worst case packet size, the
encoding could fail
Checking avcodec/ffv1enc.c, it is due to the following lines:
int64_t maxsize = AV_INPUT_BUFFER_MIN_SIZE
+ avctx->width*avctx->height*37LL*4;
[...]
if (maxsize > INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32) {
av_log(avctx, AV_LOG_WARNING, "Cannot allocate worst case
packet size, the encoding could fail\n");
maxsize = INT_MAX - AV_INPUT_BUFFER_PADDING_SIZE - 32;
}
The value of maxsize is 2517614784, more than INT_MAX, sure, but I don't
understand:
- why does FFmpeg need to allocate 148 times the size of a frame?
- why is having a number > INT_MAX an issue? modern machines are 64-bit
and have 8+ GB of RAM, and in practice I currently saw no encoding
failure on thousands of frames.
Additionally I didn't get why maxsize is reduced ("only" need of 12
times the size of the frame) in case of FFV1 version > 3 (experimental
right now):
if (f->version > 3)
maxsize = AV_INPUT_BUFFER_MIN_SIZE +
avctx->width*avctx->height*3LL*4;
maxsize is used for calling ff_alloc_packet2(), which accepts 64-bit
numbers.
if ((ret = ff_alloc_packet2(avctx, pkt, maxsize, 0)) < 0)
return ret;
Is it possible to reduce this "37x4" multiplier without risk and/or
remove maxsize reduction in case of sizeof(size_t) == 8 and/or show the
warning only once?
Issue can be reproduced with:
./ffmpeg.exe -f lavfi -i testsrc2=size=4300x3956 -t 0.040 -pix_fmt
rgba64 rgba64.dpx
./ffmpeg.exe -i rgba64.dpx -c:v ffv1 rgba64.mkv
Jérôme
___
ffmpeg-devel mailing list
ffmpeg-devel@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
