Re: [FFmpeg-devel] [PATCH v9 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats
Am 14.01.22 um 18:57 schrieb Romain Beauxis: > Le ven. 14 janv. 2022 à 11:47, Thilo Borgmann a > écrit : >> >> Am 14.01.22 um 13:57 schrieb Marvin Scholz: >>> >>> >>> On 6 Jan 2022, at 15:24, Romain Beauxis wrote: >>> * Implement support for AudioConverter * Switch to AudioConverter's API to convert unsupported PCM formats (non-interleaved, non-packed) to supported formats * Minimize data copy. This fixes: https://trac.ffmpeg.org/ticket/9502 API ref: https://developer.apple.com/documentation/audiotoolbox/audio_converter_services Signed-off-by: Romain Beauxis --- This is the first patch of a series of 3 that fix, cleanup and enhance the avfoundation implementation for libavdevice. These patches come from an actual user-facing application relying on libavdevice’s implementation of avfoundation audio input. Without them, Avfoundation is practically unusable as it will: * Refuse to process certain specific audio input format that are actually returned by the OS for some users (packed PCM audio) * Drop audio frames, resulting in corrupted audio input. This might have been unnoticed with video frames but this makes avfoundation essentially unusable for audio. The patches are now being included in our production build so they are tested and usable in production. >>> >>> Hi, >>> >>> the patches are still corrupt and do not apply. >>> As stated earlier, please either use git send-email or attach the patch >>> to the mail instead of putting its contents in it, as apparently Mail.app >>> messes them up. >> >> Still the same for me. Do you use git send-email or git format-patch? > > Thanks for checking on this y'all and sorry about these complications. > > I used git format-patches. I might try git send-email or the github PR > bridge, that seems like a neat trick. github PR sounds like something you don't want. I use format-patch as well and attach the file (not sending via mail.app though). > > I'm working on a new revision of the patches, I discovered more issues > with audio conversion, possibly linked to bugs with the AudioConverter > API. > > I also discovered an API to do the conversion internally without > having to deal with manually reconverting. Hopefully, this also fixes > the issues we uncovered. > > Will post once we have done more testing. All in all, macos sound APIs > are pretty confusing and buggy around the edges it seems. Cool. -Thilo ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v9 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats
Le ven. 14 janv. 2022 à 11:47, Thilo Borgmann a écrit : > > Am 14.01.22 um 13:57 schrieb Marvin Scholz: > > > > > > On 6 Jan 2022, at 15:24, Romain Beauxis wrote: > > > >> * Implement support for AudioConverter > >> * Switch to AudioConverter's API to convert unsupported PCM > >> formats (non-interleaved, non-packed) to supported formats > >> * Minimize data copy. > >> > >> This fixes: https://trac.ffmpeg.org/ticket/9502 > >> > >> API ref: > >> https://developer.apple.com/documentation/audiotoolbox/audio_converter_services > >> > >> Signed-off-by: Romain Beauxis > >> --- > >> This is the first patch of a series of 3 that fix, cleanup and enhance the > >> avfoundation implementation for libavdevice. > >> > >> These patches come from an actual user-facing application relying on > >> libavdevice’s implementation of avfoundation audio input. Without them, > >> Avfoundation is practically unusable as it will: > >> * Refuse to process certain specific audio input format that are actually > >> returned by the OS for some users (packed PCM audio) > >> * Drop audio frames, resulting in corrupted audio input. This might have > >> been > >> unnoticed with video frames but this makes avfoundation essentially > >> unusable > >> for audio. > >> > >> The patches are now being included in our production build so they are > >> tested > >> and usable in production. > >> > > > > Hi, > > > > the patches are still corrupt and do not apply. > > As stated earlier, please either use git send-email or attach the patch > > to the mail instead of putting its contents in it, as apparently Mail.app > > messes them up. > > Still the same for me. Do you use git send-email or git format-patch? Thanks for checking on this y'all and sorry about these complications. I used git format-patches. I might try git send-email or the github PR bridge, that seems like a neat trick. I'm working on a new revision of the patches, I discovered more issues with audio conversion, possibly linked to bugs with the AudioConverter API. I also discovered an API to do the conversion internally without having to deal with manually reconverting. Hopefully, this also fixes the issues we uncovered. Will post once we have done more testing. All in all, macos sound APIs are pretty confusing and buggy around the edges it seems. Thanks again! -- Romain ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v9 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats
Am 14.01.22 um 13:57 schrieb Marvin Scholz: > > > On 6 Jan 2022, at 15:24, Romain Beauxis wrote: > >> * Implement support for AudioConverter >> * Switch to AudioConverter's API to convert unsupported PCM >> formats (non-interleaved, non-packed) to supported formats >> * Minimize data copy. >> >> This fixes: https://trac.ffmpeg.org/ticket/9502 >> >> API ref: >> https://developer.apple.com/documentation/audiotoolbox/audio_converter_services >> >> Signed-off-by: Romain Beauxis >> --- >> This is the first patch of a series of 3 that fix, cleanup and enhance the >> avfoundation implementation for libavdevice. >> >> These patches come from an actual user-facing application relying on >> libavdevice’s implementation of avfoundation audio input. Without them, >> Avfoundation is practically unusable as it will: >> * Refuse to process certain specific audio input format that are actually >> returned by the OS for some users (packed PCM audio) >> * Drop audio frames, resulting in corrupted audio input. This might have been >> unnoticed with video frames but this makes avfoundation essentially unusable >> for audio. >> >> The patches are now being included in our production build so they are tested >> and usable in production. >> > > Hi, > > the patches are still corrupt and do not apply. > As stated earlier, please either use git send-email or attach the patch > to the mail instead of putting its contents in it, as apparently Mail.app > messes them up. Still the same for me. Do you use git send-email or git format-patch? -Thilo ___ ffmpeg-devel mailing list ffmpeg-devel@ffmpeg.org https://ffmpeg.org/mailman/listinfo/ffmpeg-devel To unsubscribe, visit link above, or email ffmpeg-devel-requ...@ffmpeg.org with subject "unsubscribe".
Re: [FFmpeg-devel] [PATCH v9 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats
On 6 Jan 2022, at 15:24, Romain Beauxis wrote:
* Implement support for AudioConverter
* Switch to AudioConverter's API to convert unsupported PCM
formats (non-interleaved, non-packed) to supported formats
* Minimize data copy.
This fixes: https://trac.ffmpeg.org/ticket/9502
API ref:
https://developer.apple.com/documentation/audiotoolbox/audio_converter_services
Signed-off-by: Romain Beauxis
---
This is the first patch of a series of 3 that fix, cleanup and enhance
the
avfoundation implementation for libavdevice.
These patches come from an actual user-facing application relying on
libavdevice’s implementation of avfoundation audio input. Without
them,
Avfoundation is practically unusable as it will:
* Refuse to process certain specific audio input format that are
actually
returned by the OS for some users (packed PCM audio)
* Drop audio frames, resulting in corrupted audio input. This might
have been
unnoticed with video frames but this makes avfoundation essentially
unusable
for audio.
The patches are now being included in our production build so they are
tested
and usable in production.
Hi,
the patches are still corrupt and do not apply.
As stated earlier, please either use git send-email or attach the patch
to the mail instead of putting its contents in it, as apparently
Mail.app
messes them up.
Changelog for this patch:
* v2: None
* v3: None
* v4: None
* v5: Fix indentation/wrapping
* v6: None
* v7: Removed use of kAudioConverterPropertyCalculateOutputBufferSize
to calculate output buffer size. The calculation is trivial and this
call was
randomly failing for no reason
* v8: None
* v9: None
libavdevice/avfoundation.m | 255 +
1 file changed, 145 insertions(+), 110 deletions(-)
diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m
index 0cd6e646d5..738cd93375 100644
--- a/libavdevice/avfoundation.m
+++ b/libavdevice/avfoundation.m
@@ -111,16 +111,11 @@
int num_video_devices;
-int audio_channels;
-int audio_bits_per_sample;
-int audio_float;
-int audio_be;
-int audio_signed_integer;
-int audio_packed;
-int audio_non_interleaved;
-
-int32_t *audio_buffer;
-int audio_buffer_size;
+UInt32audio_buffers;
+UInt32audio_channels;
+UInt32input_bytes_per_sample;
+UInt32output_bytes_per_sample;
+AudioConverterRef audio_converter;
enum AVPixelFormat pixel_format;
@@ -299,7 +294,10 @@ static void destroy_context(AVFContext* ctx)
ctx->avf_delegate= NULL;
ctx->avf_audio_delegate = NULL;
-av_freep(>audio_buffer);
+if (ctx->audio_converter) {
+ AudioConverterDispose(ctx->audio_converter);
+ ctx->audio_converter = NULL;
+}
pthread_mutex_destroy(>frame_lock);
@@ -673,6 +671,10 @@ static int get_audio_config(AVFormatContext *s)
AVFContext *ctx = (AVFContext*)s->priv_data;
CMFormatDescriptionRef format_desc;
AVStream* stream = avformat_new_stream(s, NULL);
+AudioStreamBasicDescription output_format = {0};
+int audio_bits_per_sample, audio_float, audio_be;
+int audio_signed_integer, audio_packed, audio_non_interleaved;
+int must_convert = 0;
if (!stream) {
return 1;
@@ -690,60 +692,97 @@ static int get_audio_config(AVFormatContext *s)
avpriv_set_pts_info(stream, 64, 1, avf_time_base);
format_desc =
CMSampleBufferGetFormatDescription(ctx->current_audio_frame);
-const AudioStreamBasicDescription *basic_desc =
CMAudioFormatDescriptionGetStreamBasicDescription(format_desc);
+const AudioStreamBasicDescription *input_format =
CMAudioFormatDescriptionGetStreamBasicDescription(format_desc);
-if (!basic_desc) {
+if (!input_format) {
unlock_frames(ctx);
av_log(s, AV_LOG_ERROR, "audio format not available\n");
return 1;
}
+if (input_format->mFormatID != kAudioFormatLinearPCM) {
+unlock_frames(ctx);
+av_log(s, AV_LOG_ERROR, "only PCM audio format are supported
at the moment\n");
+return 1;
+}
+
stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
-stream->codecpar->sample_rate= basic_desc->mSampleRate;
-stream->codecpar->channels = basic_desc->mChannelsPerFrame;
+stream->codecpar->sample_rate= input_format->mSampleRate;
+stream->codecpar->channels =
input_format->mChannelsPerFrame;
stream->codecpar->channel_layout =
av_get_default_channel_layout(stream->codecpar->channels);
-ctx->audio_channels= basic_desc->mChannelsPerFrame;
-ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel;
-ctx->audio_float = basic_desc->mFormatFlags &
kAudioFormatFlagIsFloat;
-ctx->audio_be = basic_desc->mFormatFlags &
kAudioFormatFlagIsBigEndian;
-ctx->audio_signed_integer
[FFmpeg-devel] [PATCH v9 1/3] libavdevice/avfoundation.m: use AudioConvert, extend supported formats
* Implement support for AudioConverter
* Switch to AudioConverter's API to convert unsupported PCM
formats (non-interleaved, non-packed) to supported formats
* Minimize data copy.
This fixes: https://trac.ffmpeg.org/ticket/9502
API ref:
https://developer.apple.com/documentation/audiotoolbox/audio_converter_services
Signed-off-by: Romain Beauxis
---
This is the first patch of a series of 3 that fix, cleanup and enhance the
avfoundation implementation for libavdevice.
These patches come from an actual user-facing application relying on
libavdevice’s implementation of avfoundation audio input. Without them,
Avfoundation is practically unusable as it will:
* Refuse to process certain specific audio input format that are actually
returned by the OS for some users (packed PCM audio)
* Drop audio frames, resulting in corrupted audio input. This might have been
unnoticed with video frames but this makes avfoundation essentially unusable
for audio.
The patches are now being included in our production build so they are tested
and usable in production.
Changelog for this patch:
* v2: None
* v3: None
* v4: None
* v5: Fix indentation/wrapping
* v6: None
* v7: Removed use of kAudioConverterPropertyCalculateOutputBufferSize
to calculate output buffer size. The calculation is trivial and this call was
randomly failing for no reason
* v8: None
* v9: None
libavdevice/avfoundation.m | 255 +
1 file changed, 145 insertions(+), 110 deletions(-)
diff --git a/libavdevice/avfoundation.m b/libavdevice/avfoundation.m
index 0cd6e646d5..738cd93375 100644
--- a/libavdevice/avfoundation.m
+++ b/libavdevice/avfoundation.m
@@ -111,16 +111,11 @@
int num_video_devices;
-int audio_channels;
-int audio_bits_per_sample;
-int audio_float;
-int audio_be;
-int audio_signed_integer;
-int audio_packed;
-int audio_non_interleaved;
-
-int32_t *audio_buffer;
-int audio_buffer_size;
+UInt32audio_buffers;
+UInt32audio_channels;
+UInt32input_bytes_per_sample;
+UInt32output_bytes_per_sample;
+AudioConverterRef audio_converter;
enum AVPixelFormat pixel_format;
@@ -299,7 +294,10 @@ static void destroy_context(AVFContext* ctx)
ctx->avf_delegate= NULL;
ctx->avf_audio_delegate = NULL;
-av_freep(>audio_buffer);
+if (ctx->audio_converter) {
+ AudioConverterDispose(ctx->audio_converter);
+ ctx->audio_converter = NULL;
+}
pthread_mutex_destroy(>frame_lock);
@@ -673,6 +671,10 @@ static int get_audio_config(AVFormatContext *s)
AVFContext *ctx = (AVFContext*)s->priv_data;
CMFormatDescriptionRef format_desc;
AVStream* stream = avformat_new_stream(s, NULL);
+AudioStreamBasicDescription output_format = {0};
+int audio_bits_per_sample, audio_float, audio_be;
+int audio_signed_integer, audio_packed, audio_non_interleaved;
+int must_convert = 0;
if (!stream) {
return 1;
@@ -690,60 +692,97 @@ static int get_audio_config(AVFormatContext *s)
avpriv_set_pts_info(stream, 64, 1, avf_time_base);
format_desc = CMSampleBufferGetFormatDescription(ctx->current_audio_frame);
-const AudioStreamBasicDescription *basic_desc =
CMAudioFormatDescriptionGetStreamBasicDescription(format_desc);
+const AudioStreamBasicDescription *input_format =
CMAudioFormatDescriptionGetStreamBasicDescription(format_desc);
-if (!basic_desc) {
+if (!input_format) {
unlock_frames(ctx);
av_log(s, AV_LOG_ERROR, "audio format not available\n");
return 1;
}
+if (input_format->mFormatID != kAudioFormatLinearPCM) {
+unlock_frames(ctx);
+av_log(s, AV_LOG_ERROR, "only PCM audio format are supported at the
moment\n");
+return 1;
+}
+
stream->codecpar->codec_type = AVMEDIA_TYPE_AUDIO;
-stream->codecpar->sample_rate= basic_desc->mSampleRate;
-stream->codecpar->channels = basic_desc->mChannelsPerFrame;
+stream->codecpar->sample_rate= input_format->mSampleRate;
+stream->codecpar->channels = input_format->mChannelsPerFrame;
stream->codecpar->channel_layout =
av_get_default_channel_layout(stream->codecpar->channels);
-ctx->audio_channels= basic_desc->mChannelsPerFrame;
-ctx->audio_bits_per_sample = basic_desc->mBitsPerChannel;
-ctx->audio_float = basic_desc->mFormatFlags &
kAudioFormatFlagIsFloat;
-ctx->audio_be = basic_desc->mFormatFlags &
kAudioFormatFlagIsBigEndian;
-ctx->audio_signed_integer = basic_desc->mFormatFlags &
kAudioFormatFlagIsSignedInteger;
-ctx->audio_packed = basic_desc->mFormatFlags &
kAudioFormatFlagIsPacked;
-ctx->audio_non_interleaved = basic_desc->mFormatFlags &
kAudioFormatFlagIsNonInterleaved;
-
-if (basic_desc->mFormatID ==
