Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[Moritz Barsnick]

On Thu, Jul 06, 2017 at 03:44:45 -0700, DopeLabs wrote:
> Option 2 in your question looks fine to me... 
> 
> I have used ffmpeg to transcode audio and video streams before to achieve 
> adaptive streaming.
> 
> Here is an example of an audio transcode... input is 256k mp3, output is 
> 128k, 64k, and 24k
> 
> $ ffmpeg -i http://stream.dubstep.fm -c:a mp3 -b:a 128k -f null /dev/null 
> -c:a mp3 -b:a 64k -f null /dev/null -c:a mp3 -b:a 24k -ar 22050 -f null 
> /dev/null

Shalin did confirm in the original posting that this does work and is
the preferred method. The complaint was that it doesn't optimally make
use of parallelism to load all CPUs to the max (and thereby possibly
reduce processing time even more).

Moritz
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[DopeLabs]

Option 2 in your question looks fine to me... 

I have used ffmpeg to transcode audio and video streams before to achieve 
adaptive streaming.

Here is an example of an audio transcode... input is 256k mp3, output is 128k, 
64k, and 24k

$ ffmpeg -i http://stream.dubstep.fm -c:a mp3 -b:a 128k -f null /dev/null -c:a 
mp3 -b:a 64k -f null /dev/null -c:a mp3 -b:a 24k -ar 22050 -f null /dev/null


ffmpeg version N-86413-g50be8f2142 Copyright (c) 2000-2017 the FFmpeg developers
  built with Apple LLVM version 8.1.0 (clang-802.0.42)
  configuration: --enable-libvidstab --enable-shared --enable-pthreads 
--cc=/usr/bin/clang --enable-vda --enable-videotoolbox --arch=x86_64 
--enable-yasm --enable-gpl --enable-postproc --enable-libx264 --enable-libxvid 
--enable-filter=coreimage --enable-filter=coreimagesrc --enable-filter=frei0r 
--enable-filter=frei0r_src --enable-swscale --enable-avfilter 
--enable-avresample --enable-libmp3lame --enable-libvorbis --enable-libopus 
--enable-libtheora --enable-libopenjpeg --enable-libmodplug --enable-libvpx 
--enable-libsoxr --enable-libspeex --enable-libass --enable-libbluray 
--enable-lzma --enable-gnutls --enable-fontconfig --enable-libfreetype 
--enable-libfribidi --enable-libx265 --enable-version3 --enable-libwavpack 
--enable-libssh --enable-librtmp --enable-libfdk-aac --enable-nonfree 
--enable-libzmq --enable-opencl --enable-ffplay --enable-libcaca 
--enable-libvidstab
  libavutil  55. 63.100 / 55. 63.100
  libavcodec 57. 96.101 / 57. 96.101
  libavformat57. 72.101 / 57. 72.101
  libavdevice57.  7.100 / 57.  7.100
  libavfilter 6. 91.100 /  6. 91.100
  libavresample   3.  6.  0 /  3.  6.  0
  libswscale  4.  7.101 /  4.  7.101
  libswresample   2.  8.100 /  2.  8.100
  libpostproc54.  6.100 / 54.  6.100
[mp3 @ 0x7fa3b200] Skipping 88 bytes of junk at 0.
Input #0, mp3, from 'http://stream.dubstep.fm':
  Metadata:
icy-genre   : Dubstep, Dub, Electronic, Garage
icy-name: Dubstep.fm
icy-notice1 : This stream requires http://www.winamp.com;>Winamp
icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724
icy-pub : 1
icy-url : https://www.dubstep.fm
StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded 
From Chicago
StreamUrl   : https://www.dubstep.fm
  Duration: N/A, start: 0.00, bitrate: 256 kb/s
Stream #0:0: Audio: mp3, 44100 Hz, stereo, s16p, 256 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (mp3 (native) -> mp3 (libmp3lame))
  Stream #0:0 -> #1:0 (mp3 (native) -> mp3 (libmp3lame))
  Stream #0:0 -> #2:0 (mp3 (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, null, to '/dev/null':
  Metadata:
icy-genre   : Dubstep, Dub, Electronic, Garage
icy-name: Dubstep.fm
icy-notice1 : This stream requires http://www.winamp.com;>Winamp
icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724
icy-pub : 1
icy-url : https://www.dubstep.fm
StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded 
From Chicago
StreamUrl   : https://www.dubstep.fm
encoder : Lavf57.72.101
Stream #0:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p, 128 kb/s
Metadata:
  encoder : Lavc57.96.101 libmp3lame
Output #1, null, to '/dev/null':
  Metadata:
icy-genre   : Dubstep, Dub, Electronic, Garage
icy-name: Dubstep.fm
icy-notice1 : This stream requires http://www.winamp.com;>Winamp
icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724
icy-pub : 1
icy-url : https://www.dubstep.fm
StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded 
From Chicago
StreamUrl   : https://www.dubstep.fm
encoder : Lavf57.72.101
Stream #1:0: Audio: mp3 (libmp3lame), 44100 Hz, stereo, s16p, 64 kb/s
Metadata:
  encoder : Lavc57.96.101 libmp3lame
Output #2, null, to '/dev/null':
  Metadata:
icy-genre   : Dubstep, Dub, Electronic, Garage
icy-name: Dubstep.fm
icy-notice1 : This stream requires http://www.winamp.com;>Winamp
icy-notice2 : SHOUTcast DNAS/posix(linux x64) v2.5.1.724
icy-pub : 1
icy-url : https://www.dubstep.fm
StreamTitle : ARCHIVE - 2011-01-09 - The Dented McStank With Phaded 
From Chicago
StreamUrl   : https://www.dubstep.fm
encoder : Lavf57.72.101
Stream #2:0: Audio: mp3 (libmp3lame), 22050 Hz, stereo, s16p, 24 kb/s
Metadata:
  encoder : Lavc57.96.101 libmp3lame
size=N/A time=00:00:13.14 bitrate=N/A speed=2.69x
video:0kB audio:347kB subtitle:0kB other streams:0kB global headers:0kB muxing 
overhead: unknown





> On Jul 5, 2017, at 2:23 39PM, sh


> alin  wrote:
> 
> Reuben Martin wrote
>> On Thursday, June 29, 2017 6:56:51 PM CDT shalin wrote:
>>> Cley Faye wrote
>>> 
 More to the point, as far as I know, 

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[shalin]

Reuben Martin wrote
> On Thursday, June 29, 2017 6:56:51 PM CDT shalin wrote:
>> Cley Faye wrote
>> 
>> > More to the point, as far as I know, ffmpeg will not do parallel
>> treatment
>> > on either input or output, as you deduced. It certainly is doable, but
>> > don't seem trivial to implement in the ffmpeg CLI tool itself since
>> that
>> > tool must handle various cases, and introducing (more) special cases
>> would
>> > add complexity for not much benefits.
>> > 
>> > You should look in using the various ffmpeg libraries directly; with
>> that
>> > you could probably write it to decode once, encode multiple time. But
>> > unless you have shockingly similar settings and unlimited processing
>> > power,
>> > you'll most likely stall some of your encodings while other catch up
>> with
>> > them. I don't know how well ffmpeg libraries handle multithreading (I
>> > suppose fairly well since most of it's operations are based on context
>> > structures) but you'll have to do a fair amount of synchronization by
>> > hand.
>> 
>> Thank you for your answer to my original question. So from this, I am
>> assuming that ffmpeg in its current form doesn't support output level
>> parallelism. Actually I have 'shockingly' similar transcodings to perform
>> on
>> a given input stream (I need to crop different parts of the input video
>> and
>> encode them at same bitrate and also cropped parts are same in size for
>> each stream). With that each encode should roughly run at the same rate
>> and
>> I believe multithreading should be really helpful.
> 
> All this is possible but you need to do a lot of reading. Starting with
> input 
> mapping. http://www.ffmpeg.org/ffmpeg-all.html#Advanced-options
> 
> It couldn’t hurt to read up on how anonymous pipes work in a posix shell,
> as 
> well as other shell related concepts.

There's no doubt that I need a lot of reading but I wish you at least read
my question properly. It has nothing to do with pipes or shell. It's about
running multiple encodes in parallel within single ffmpeg process (option 2
in my question). Simply providing the link doesn't help as it doesn't solve
the issue at hand.




--
View this message in context: 
http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-parallel-doesn-t-work-tp4680420p4680482.html
Sent from the FFmpeg-users mailing list archive at Nabble.com.
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[Reuben Martin]

On Thursday, June 29, 2017 6:56:51 PM CDT shalin wrote:
> Cley Faye wrote
> 
> > More to the point, as far as I know, ffmpeg will not do parallel treatment
> > on either input or output, as you deduced. It certainly is doable, but
> > don't seem trivial to implement in the ffmpeg CLI tool itself since that
> > tool must handle various cases, and introducing (more) special cases would
> > add complexity for not much benefits.
> > 
> > You should look in using the various ffmpeg libraries directly; with that
> > you could probably write it to decode once, encode multiple time. But
> > unless you have shockingly similar settings and unlimited processing
> > power,
> > you'll most likely stall some of your encodings while other catch up with
> > them. I don't know how well ffmpeg libraries handle multithreading (I
> > suppose fairly well since most of it's operations are based on context
> > structures) but you'll have to do a fair amount of synchronization by
> > hand.
> 
> Thank you for your answer to my original question. So from this, I am
> assuming that ffmpeg in its current form doesn't support output level
> parallelism. Actually I have 'shockingly' similar transcodings to perform on
> a given input stream (I need to crop different parts of the input video and
> encode them at same bitrate and also cropped parts are same in size for
> each stream). With that each encode should roughly run at the same rate and
> I believe multithreading should be really helpful.

All this is possible but you need to do a lot of reading. Starting with input 
mapping. http://www.ffmpeg.org/ffmpeg-all.html#Advanced-options

It couldn’t hurt to read up on how anonymous pipes work in a posix shell, as 
well as other shell related concepts.

-Reuben


___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[shalin]

Cley Faye wrote
> More to the point, as far as I know, ffmpeg will not do parallel treatment
> on either input or output, as you deduced. It certainly is doable, but
> don't seem trivial to implement in the ffmpeg CLI tool itself since that
> tool must handle various cases, and introducing (more) special cases would
> add complexity for not much benefits.
> 
> You should look in using the various ffmpeg libraries directly; with that
> you could probably write it to decode once, encode multiple time. But
> unless you have shockingly similar settings and unlimited processing
> power,
> you'll most likely stall some of your encodings while other catch up with
> them. I don't know how well ffmpeg libraries handle multithreading (I
> suppose fairly well since most of it's operations are based on context
> structures) but you'll have to do a fair amount of synchronization by
> hand.

Thank you for your answer to my original question. So from this, I am
assuming that ffmpeg in its current form doesn't support output level
parallelism. Actually I have 'shockingly' similar transcodings to perform on
a given input stream (I need to crop different parts of the input video and
encode them at same bitrate and also cropped parts are same in size for each
stream). With that each encode should roughly run at the same rate and I
believe multithreading should be really helpful.

I will probably check with ffmpeg developers to know their thoughts on this
one. Let me know if anyone has tried some thing like this.

Thanks.



--
View this message in context: 
http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-parallel-doesn-t-work-tp4680420p4680436.html
Sent from the FFmpeg-users mailing list archive at Nabble.com.
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[Hans Carlson]

On Wed, 28 Jun 2017, shalin wrote:


Nicolas George wrote

I think you need to take a little step back and learn what Unix pipes 
mean. They are meant to connect the output of a process to the input of 
the next one, but in your examples your commands neither consume their 
input nor produce output.


Piping only shows that I am trying to run all three ffmpeg transcode 
instances in parallel using single command line. If it makes any 
clearer, you can read them as three separate commands issued at the same 
time. With that, I hope we can focus on the concern raised in the 
original question.


As Nicolas already said, you need to take a closer look at "piping" vs 
"background processing" in the UNIX shell.


What you are doing is PIPING:

   |  | 

This takes the output of process 1 and uses it as the input to process 2, 
then the output of process 2 and uses it as the input to process 3.


It sounds like what you want is BACKGROUND PROCESSING:

   &  & process 3 &

This puts all 3 processes in the background and runs them concurrently.

You might want to check out some basics of the UNIX shell, such as this 
link:


  https://www.washington.edu/computing/unix/startdoc/shell.html
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[Cley Faye]

2017-06-28 22:56 GMT+02:00 shalin :

> Piping only shows that I am trying to run all three ffmpeg transcode
> instances in parallel using single command line. If it makes any clearer,
> you can read them as three separate commands issued at the same time. With
> that, I hope we can focus on the concern raised in the original question.
>

​Then you don't want pipes, you just want three parallel processes, which
you can do using &.

More to the point, as far as I know, ffmpeg will not do parallel treatment
on either input or output, as you deduced. It certainly is doable, but
don't seem trivial to implement in the ffmpeg CLI tool itself since that
tool must handle various cases, and introducing (more) special cases would
add complexity for not much benefits.

You should look in using the various ffmpeg libraries directly; with that
you could probably write it to decode once, encode multiple time. But
unless you have shockingly similar settings and unlimited processing power,
you'll most likely stall some of your encodings while other catch up with
them. I don't know how well ffmpeg libraries handle multithreading (I
suppose fairly well since most of it's operations are based on context
structures) but you'll have to do a fair amount of synchronization by hand.
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[shalin]

Nicolas George wrote
> Hi.
> 
> Le decadi 10 messidor, an CCXXV, Shalin Mehta a écrit :
>> Option1: Run 3 instances of ffmpeg using pipe.
>> ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4
>> -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v
>> 12M
>> -f null -
> 
> I think you need to take a little step back and learn what Unix pipes
> mean. They are meant to connect the output of a process to the input of
> the next one, but in your examples your commands neither consume their
> input nor produce output.
> 
> -- 
>   Nicolas George
> 
> signature.asc (849 bytes)
> http://www.ffmpeg-archive.org/attachment/4680421/0/signature.asc;

Piping only shows that I am trying to run all three ffmpeg transcode
instances in parallel using single command line. If it makes any clearer,
you can read them as three separate commands issued at the same time. With
that, I hope we can focus on the concern raised in the original question.




--
View this message in context: 
http://www.ffmpeg-archive.org/Using-FFMPEG-to-encode-multiple-outputs-in-parallel-doesn-t-work-tp4680420p4680422.html
Sent from the FFmpeg-users mailing list archive at Nabble.com.
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] Using FFMPEG to encode multiple outputs in 'parallel' (doesn't work?)

[Nicolas George]

Hi.

Le decadi 10 messidor, an CCXXV, Shalin Mehta a écrit :
> Option1: Run 3 instances of ffmpeg using pipe.
> ffmpeg -i input.mp4 -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4
> -c:v libx264 -b:v 12M -f null - | ffmpeg -i input.mp4 -c:v libx264 -b:v 12M
> -f null -

I think you need to take a little step back and learn what Unix pipes
mean. They are meant to connect the output of a process to the input of
the next one, but in your examples your commands neither consume their
input nor produce output.

Regards,

-- 
  Nicolas George


signature.asc
Description: Digital signature
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
http://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".