Re: [FFmpeg-user] Disparity in VMAF scores
I'm not familiar with that tool, but one thing I have noticed with some 3rd-party VMAF tools is the scores they generate aren't always accurate if the material in question is not 1080p for the standard model, or 4k in the case of the 4k model. So if your source is smaller than 1080p I wouldn't trust the score. The official vmaf.exe has you tell it what the resolution of the source is and the calculations get adjusted accordingly. You can end up with a wildly different score if the resolution isn't taken into account during the score calculation, and some 3rd-party tools seem to get this part wrong somehow and you end up with a score such as you're seeing, somewhere in the 70s, when with the official tool and the proper resolution set it would probably give you something in the 90s. I noticed this when I was testing one day with some 4k stuff, some 1080p stuff, and some 480p stuff with a DVD source. The DVD stuff was returning scores in the 70s even when encoded with x264 and CRF 16 or 17, which seemed very strange to me. So instead I used ffmpeg to dump to yuv and then used vmaf.exe on those files to double check and the scores came back in the upper mid 90s, which made much more sense. Since VMAF determines your viewing distance with a multiple of the file's vertical resolution as one of the factors it is necessary that the calculations be fed the proper information with regard to the file's resolution or the score will be incorrect. I'm not saying your 3rd-party tool in particular is making this mistake for sure, but it is possible and is worth double checking with the official tool just to be sure you're getting sane scores. I've seen some online VMAF guides written by random people that state something like "You should expect lower scores for lower resolution files such as DVD." The actual case is whatever tool they're using to get their scores isn't handling resolution properly and that's why their lower resolution files are resulting in lower scores. On Mon, May 9, 2022 at 10:35 AM Kamaldeep Tumkur wrote: > On Mon, 9 May 2022 at 21:49, Clayton Macleod > wrote: > > > Using the official VMAF.exe is a bit more of a pain since you have to > > convert to yuv first, but being able to examine the results file is > > sometimes helpful. 75 sounds like something may still be off if the files > > don’t appear drastically different to your eye. > > > > -- > > Clayton Macleod > > If no one comes from the future to stop you from doing it, then how bad > of > > a decision can it really be? > > > > > On May 9, 2022, at 1:14 AM, Kamaldeep Tumkur < > kamaldeep.tum...@gmail.com> > > wrote: > > > > > > So my question is, does libvmaf implementation do any kind of > detection > > of > > > audio in the distorted asset before computing a perceptual video > quality > > > score. > > > > > > > > >> On Mon, 9 May 2022 at 13:37, Kamaldeep Tumkur < > > kamaldeep.tum...@gmail.com> > > >> wrote: > > >> > > >> @Clayton, thanks for your response. > > >> > > >> It turned out that the external encoder was adding an audio track to > the > > >> encode. This was throwing the libvmaf scores off. On removing the > audio > > >> track and checking the encode, the score jumped from 25 to 77.64. > > >> > > >> Now there is basis to compare the ffmpeg and the external encode. > > >> > > >> Regards > > >> Kamaldeep > > >> > > >> > > >> > > >> On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur < > > kamaldeep.tum...@gmail.com> > > >> wrote: > > >> > > >>> Hello, > > >>> > > >>> I'm new to the computation of VMAF scores using libvmaf in ffmpeg. > > While > > >>> trying out generating scores through libvmaf, I have an issue that I > > cannot > > >>> seem to find an explanation for. > > >>> > > >>> I encoded the 'crowd_run' HQ MP4 with two different encoders. The > > >>> resulting files were compared with the same source (original). > > >>> > > >>> VMAF score of ffmpeg encoded clip: > > >>> > > >>> [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 > > >>> > > >>> > > >>> Command used: > > >>> > > >>> ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 > > >>> -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy > > -f > > >>> mpegts pass2.ts > > >>> > > >>> > > >>> VMAF score for externally encoded clip: > > >>> > > >>> [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 > > >>> > > >>> > > >>> > > >>> I just don't know how to explain this disparity in generated scores > > when > > >>> framerates and resolution of the source were retained. The second > clip > > >>> doesn't seem to show such a degradation expected through the score. > > >>> Attaching both the encoded clips here for analysis. > > >>> > > >>> > > >>> Could anyone point me to why the external encoder generates an output > > >>> with low VMAF score. Anything in the frame structure? > > >>> > > >>> > > >>> ffmpeg encode: > > >>> > > >>> > > >>> > > > https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing > > >>> > > >>> >
Re: [FFmpeg-user] Disparity in VMAF scores
On Mon, 9 May 2022 at 21:49, Clayton Macleod wrote: > Using the official VMAF.exe is a bit more of a pain since you have to > convert to yuv first, but being able to examine the results file is > sometimes helpful. 75 sounds like something may still be off if the files > don’t appear drastically different to your eye. > > -- > Clayton Macleod > If no one comes from the future to stop you from doing it, then how bad of > a decision can it really be? > > > On May 9, 2022, at 1:14 AM, Kamaldeep Tumkur > wrote: > > > > So my question is, does libvmaf implementation do any kind of detection > of > > audio in the distorted asset before computing a perceptual video quality > > score. > > > > > >> On Mon, 9 May 2022 at 13:37, Kamaldeep Tumkur < > kamaldeep.tum...@gmail.com> > >> wrote: > >> > >> @Clayton, thanks for your response. > >> > >> It turned out that the external encoder was adding an audio track to the > >> encode. This was throwing the libvmaf scores off. On removing the audio > >> track and checking the encode, the score jumped from 25 to 77.64. > >> > >> Now there is basis to compare the ffmpeg and the external encode. > >> > >> Regards > >> Kamaldeep > >> > >> > >> > >> On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur < > kamaldeep.tum...@gmail.com> > >> wrote: > >> > >>> Hello, > >>> > >>> I'm new to the computation of VMAF scores using libvmaf in ffmpeg. > While > >>> trying out generating scores through libvmaf, I have an issue that I > cannot > >>> seem to find an explanation for. > >>> > >>> I encoded the 'crowd_run' HQ MP4 with two different encoders. The > >>> resulting files were compared with the same source (original). > >>> > >>> VMAF score of ffmpeg encoded clip: > >>> > >>> [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 > >>> > >>> > >>> Command used: > >>> > >>> ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 > >>> -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy > -f > >>> mpegts pass2.ts > >>> > >>> > >>> VMAF score for externally encoded clip: > >>> > >>> [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 > >>> > >>> > >>> > >>> I just don't know how to explain this disparity in generated scores > when > >>> framerates and resolution of the source were retained. The second clip > >>> doesn't seem to show such a degradation expected through the score. > >>> Attaching both the encoded clips here for analysis. > >>> > >>> > >>> Could anyone point me to why the external encoder generates an output > >>> with low VMAF score. Anything in the frame structure? > >>> > >>> > >>> ffmpeg encode: > >>> > >>> > >>> > https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing > >>> > >>> > >>> external encode: > >>> > >>> > >>> > https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing > >>> > >>> > >>> > >>> Thank you. > >>> > > I've been using the crowd_run y4m source at: https://media.xiph.org/video/derf/y4m/crowd_run_1080p50.y4m Yes, I'd prefer a better score too. However, I've only been able to generate an encode with 77.936 at most today, with the external encoder and a qvbr setting. ffmpeg-based encodes are at 90.12, which is quite in the zone. Yes, also need to check the standalone vmaf exec, but what I've learned is using ffmpeg filters to scale up the HLS renditions to the source resolution and frame-rate along with the libvmaf filter is easier. Although, please note that the distorted asset with score of 77 had the same resolution and frame rate as the source, so no additional filtering was performed. ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] Disparity in VMAF scores
Using the official VMAF.exe is a bit more of a pain since you have to convert to yuv first, but being able to examine the results file is sometimes helpful. 75 sounds like something may still be off if the files don’t appear drastically different to your eye. -- Clayton Macleod If no one comes from the future to stop you from doing it, then how bad of a decision can it really be? > On May 9, 2022, at 1:14 AM, Kamaldeep Tumkur > wrote: > > So my question is, does libvmaf implementation do any kind of detection of > audio in the distorted asset before computing a perceptual video quality > score. > > >> On Mon, 9 May 2022 at 13:37, Kamaldeep Tumkur >> wrote: >> >> @Clayton, thanks for your response. >> >> It turned out that the external encoder was adding an audio track to the >> encode. This was throwing the libvmaf scores off. On removing the audio >> track and checking the encode, the score jumped from 25 to 77.64. >> >> Now there is basis to compare the ffmpeg and the external encode. >> >> Regards >> Kamaldeep >> >> >> >> On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur >> wrote: >> >>> Hello, >>> >>> I'm new to the computation of VMAF scores using libvmaf in ffmpeg. While >>> trying out generating scores through libvmaf, I have an issue that I cannot >>> seem to find an explanation for. >>> >>> I encoded the 'crowd_run' HQ MP4 with two different encoders. The >>> resulting files were compared with the same source (original). >>> >>> VMAF score of ffmpeg encoded clip: >>> >>> [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 >>> >>> >>> Command used: >>> >>> ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 >>> -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy -f >>> mpegts pass2.ts >>> >>> >>> VMAF score for externally encoded clip: >>> >>> [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 >>> >>> >>> >>> I just don't know how to explain this disparity in generated scores when >>> framerates and resolution of the source were retained. The second clip >>> doesn't seem to show such a degradation expected through the score. >>> Attaching both the encoded clips here for analysis. >>> >>> >>> Could anyone point me to why the external encoder generates an output >>> with low VMAF score. Anything in the frame structure? >>> >>> >>> ffmpeg encode: >>> >>> >>> https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing >>> >>> >>> external encode: >>> >>> >>> https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing >>> >>> >>> >>> Thank you. >>> >> > ___ > ffmpeg-user mailing list > ffmpeg-user@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-user > > To unsubscribe, visit link above, or email > ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe". ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] Disparity in VMAF scores
So my question is, does libvmaf implementation do any kind of detection of audio in the distorted asset before computing a perceptual video quality score. On Mon, 9 May 2022 at 13:37, Kamaldeep Tumkur wrote: > @Clayton, thanks for your response. > > It turned out that the external encoder was adding an audio track to the > encode. This was throwing the libvmaf scores off. On removing the audio > track and checking the encode, the score jumped from 25 to 77.64. > > Now there is basis to compare the ffmpeg and the external encode. > > Regards > Kamaldeep > > > > On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur > wrote: > >> Hello, >> >> I'm new to the computation of VMAF scores using libvmaf in ffmpeg. While >> trying out generating scores through libvmaf, I have an issue that I cannot >> seem to find an explanation for. >> >> I encoded the 'crowd_run' HQ MP4 with two different encoders. The >> resulting files were compared with the same source (original). >> >> VMAF score of ffmpeg encoded clip: >> >> [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 >> >> >> Command used: >> >> ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 >> -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy -f >> mpegts pass2.ts >> >> >> VMAF score for externally encoded clip: >> >> [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 >> >> >> >> I just don't know how to explain this disparity in generated scores when >> framerates and resolution of the source were retained. The second clip >> doesn't seem to show such a degradation expected through the score. >> Attaching both the encoded clips here for analysis. >> >> >> Could anyone point me to why the external encoder generates an output >> with low VMAF score. Anything in the frame structure? >> >> >> ffmpeg encode: >> >> >> https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing >> >> >> external encode: >> >> >> https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing >> >> >> >> Thank you. >> > ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] Disparity in VMAF scores
@Clayton, thanks for your response. It turned out that the external encoder was adding an audio track to the encode. This was throwing the libvmaf scores off. On removing the audio track and checking the encode, the score jumped from 25 to 77.64. Now there is basis to compare the ffmpeg and the external encode. Regards Kamaldeep On Sun, 8 May 2022 at 15:37, Kamaldeep Tumkur wrote: > Hello, > > I'm new to the computation of VMAF scores using libvmaf in ffmpeg. While > trying out generating scores through libvmaf, I have an issue that I cannot > seem to find an explanation for. > > I encoded the 'crowd_run' HQ MP4 with two different encoders. The > resulting files were compared with the same source (original). > > VMAF score of ffmpeg encoded clip: > > [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 > > > Command used: > > ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 > -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy -f > mpegts pass2.ts > > > VMAF score for externally encoded clip: > > [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 > > > > I just don't know how to explain this disparity in generated scores when > framerates and resolution of the source were retained. The second clip > doesn't seem to show such a degradation expected through the score. > Attaching both the encoded clips here for analysis. > > > Could anyone point me to why the external encoder generates an output with > low VMAF score. Anything in the frame structure? > > > ffmpeg encode: > > > https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing > > > external encode: > > > https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing > > > > Thank you. > ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-user] Disparity in VMAF scores
I haven't looked at your files as I don't have the source file you're using. I presume this is some public domain test file, is that right? Anyway, if you use the official vmaf.exe you can output the comparison results to a file to examine what the score was for each frame. Scrolling quickly through those results I suspect you will find that at some point the frames will no longer be in sync with each other and you will begin to see a bunch of frames that have a score of zero. At least that's what I saw in a project of my own when concatenating several files and using ffmpeg to do the mkv muxing. When I started using mkvmerge to do the muxing my issue went away. I also had to eliminate all streams except the video stream to make sure things worked properly. The presence of audio or subtitles would sometimes screw things up. I'm not sure why that is. Something similar may or may not be happening with your .ts files, but I don't know. You can check using ffprobe to see whether or not the total frame counts match, also. Naturally, if the total number of frames differs then something is happening. On Sun, May 8, 2022 at 3:07 AM Kamaldeep Tumkur wrote: > Hello, > > I'm new to the computation of VMAF scores using libvmaf in ffmpeg. While > trying out generating scores through libvmaf, I have an issue that I cannot > seem to find an explanation for. > > I encoded the 'crowd_run' HQ MP4 with two different encoders. The resulting > files were compared with the same source (original). > > VMAF score of ffmpeg encoded clip: > > [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 > > > Command used: > > ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 > -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy -f > mpegts pass2.ts > > > VMAF score for externally encoded clip: > > [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 > > > > I just don't know how to explain this disparity in generated scores when > framerates and resolution of the source were retained. The second clip > doesn't seem to show such a degradation expected through the score. > Attaching both the encoded clips here for analysis. > > > Could anyone point me to why the external encoder generates an output with > low VMAF score. Anything in the frame structure? > > > ffmpeg encode: > > > https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing > > > external encode: > > > https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing > > > > Thank you. > ___ > ffmpeg-user mailing list > ffmpeg-user@ffmpeg.org > https://ffmpeg.org/mailman/listinfo/ffmpeg-user > > To unsubscribe, visit link above, or email > ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe". > -- Clayton Macleod If no one comes from the future to stop you from doing it, then how bad of a decision can it really be? ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
[FFmpeg-user] Disparity in VMAF scores
Hello, I'm new to the computation of VMAF scores using libvmaf in ffmpeg. While trying out generating scores through libvmaf, I have an issue that I cannot seem to find an explanation for. I encoded the 'crowd_run' HQ MP4 with two different encoders. The resulting files were compared with the same source (original). VMAF score of ffmpeg encoded clip: [libvmaf @ 0x7fc5d8f2cb00] VMAF score: 90.644028 Command used: ffmpeg -i crowd_run_1080p50.mp4 -c:v libx264 -g 100 -keyint_min 100 -sc_threshold 0 -b:v 12000k -maxrate 15000k -bufsize 15000k -c:a copy -f mpegts pass2.ts VMAF score for externally encoded clip: [libvmaf @ 0x7fc53d729500] VMAF score: 25.168431 I just don't know how to explain this disparity in generated scores when framerates and resolution of the source were retained. The second clip doesn't seem to show such a degradation expected through the score. Attaching both the encoded clips here for analysis. Could anyone point me to why the external encoder generates an output with low VMAF score. Anything in the frame structure? ffmpeg encode: https://drive.google.com/file/d/15-9YyNhYWJoTMxdmESkc7b0mPKKUMqG7/view?usp=sharing external encode: https://drive.google.com/file/d/1Mt2jP51KZ4vTG7SYPnJkCG1uMWqOadid/view?usp=sharing Thank you. ___ ffmpeg-user mailing list ffmpeg-user@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-user To unsubscribe, visit link above, or email ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
