Also remove the redundant casts which were hiding the error here.
---
On 14/09/16 12:33, Diego Biurrun wrote:
> On Tue, Sep 13, 2016 at 08:45:55PM +0100, Mark Thompson wrote:
>> --- a/libavutil/hwcontext_vdpau.c
>> +++ b/libavutil/hwcontext_vdpau.c
>> @@ -304,7 +304,7 @@ static int vdpau_transfer_data_from(AVHWFramesContext
>> *ctx, AVFrame *dst,
>>
>> for (i = 0; i< FF_ARRAY_ELEMS(data) && dst->data[i]; i++) {
>> data[i] = dst->data[i];
>> - if (dst->linesize[i] < 0 || (uint64_t)dst->linesize > UINT32_MAX) {
>> + if (dst->linesize[i] < 0 || (uint64_t)dst->linesize[i] >
>> UINT32_MAX) {
>> av_log(ctx, AV_LOG_ERROR,
>> "The linesize %d cannot be represented as uint32\n",
>> dst->linesize[i]);
>
> There is a second instance of this error in the file.
Urgh, yes.
> I wonder if the cast is not the root of the problem. Why is it necessary
> in the first place?
Right, the cast isn't needed.
If sizeof(linesize) <= sizeof(uint32_t), then the linesize is promoted to the
unsigned type of uint32_t and result of the comparison is correctly always false
since UINT32_MAX is the largest value of that type. (This is probably all
current cases.)
If sizeof(linesize) > sizeof(uint32_t), then UINT32_MAX is promoted to the
longer signed type and the comparison does the right thing. (This is only used
now if int is 64-bit (i.e. pretty much nowhere), but it will get used on 64-bit
machines once linesize is ptrdiff_t everywhere.)
- Mark
libavutil/hwcontext_vdpau.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/libavutil/hwcontext_vdpau.c b/libavutil/hwcontext_vdpau.c
index 9722c10..ebd4825 100644
--- a/libavutil/hwcontext_vdpau.c
+++ b/libavutil/hwcontext_vdpau.c
@@ -304,7 +304,7 @@ static int vdpau_transfer_data_from(AVHWFramesContext *ctx,
AVFrame *dst,
for (i = 0; i< FF_ARRAY_ELEMS(data) && dst->data[i]; i++) {
data[i] = dst->data[i];
- if (dst->linesize[i] < 0 || (uint64_t)dst->linesize > UINT32_MAX) {
+ if (dst->linesize[i] < 0 || dst->linesize[i] > UINT32_MAX) {
av_log(ctx, AV_LOG_ERROR,
"The linesize %d cannot be represented as uint32\n",
dst->linesize[i]);
@@ -355,7 +355,7 @@ static int vdpau_transfer_data_to(AVHWFramesContext *ctx,
AVFrame *dst,
for (i = 0; i< FF_ARRAY_ELEMS(data) && src->data[i]; i++) {
data[i] = src->data[i];
- if (src->linesize[i] < 0 || (uint64_t)src->linesize > UINT32_MAX) {
+ if (src->linesize[i] < 0 || src->linesize[i] > UINT32_MAX) {
av_log(ctx, AV_LOG_ERROR,
"The linesize %d cannot be represented as uint32\n",
src->linesize[i]);
--
2.9.3
_______________________________________________
libav-devel mailing list
libav-devel@libav.org
https://lists.libav.org/mailman/listinfo/libav-devel
