Re: Using subprocess to capture a progress line

[Chris Warrick]

On 10 November 2015 at 23:47, Tim Johnson  wrote:
> Using python 2.7.6 on ubuntu 14.04
> The application in question is run with bash and gnome-terminal :
>
> I've written a command-line "wrapper" for youtube-dl, executing
> youtube-dl as a subprocess.
>
> --
> youtube-dl reports download progress on one line. I.E. the line is
> overwritten numerous times with no carriage return until the
> downloading is finished.
> --
>
> The following code runs the youtube-dl command and reports each line
> as output by youtube-dl
> ###
> p = subprocess.Popen(list(args), stderr=subprocess.STDOUT,
>  stdout=subprocess.PIPE)
> while True:
> line = p.stdout.readline()
> if not line:
> break
> tmp = line.strip()
> print tmp
> ###
>
> However this method not does show the download progress _until_ the
> download is complete.

There is no \n character at the end — which means that
p.stdout.readline() cannot return. In fact, if you printed repr() of
the line you read, you would get this:

b'\r[download]  54.9% of 2.73MiB at 26.73KiB/s ETA 00:47\r[download]
55.0% of 2.73MiB at 79.33KiB/s ETA 00:15\r…snip…\r[download] 100% of
2.73MiB in 00:01\n'

The download line is implemented using \r, which is the carriage
return character (return to the first character), and then by
overwriting characters that were already printed.

The solution? There are numerous. I’ll help you by obscuring the worst one.

(1) [recommended] figure out how to make youtube_dl work as a library,
read its main file to figure out the problem. Don’t mess with
subprocess.
(2) [don’t do it] do you need to intercept the lines? If you don’t set
stderr= and stdout=, things will print just fine.
(3) [DON’T DO IT] .ernq() punenpgre ol punenpgre naq znxr n zrff.

PS. Thank you for setting a sensible Reply-To header on your messages.
Which is something the list should be doing.

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Chris Warrick]

On 11 November 2015 at 17:16, Tim Johnson  wrote:
>> (2) [don’t do it] do you need to intercept the lines? If you don’t set
>> stderr= and stdout=, things will print just fine.
>   Got to try that before using the module, just for edification.

At which point your initial code sample will become:
###
p = subprocess.Popen(list(args))
###

(is list(args) really necessary? Wouldn’t plain Popen(args) just work?)

-- 
Chris Warrick 
PGP: 5EAAEA16
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Tim Johnson]

* Chris Warrick  [15 00:55]:
> On 10 November 2015 at 23:47, Tim Johnson  wrote:
> > Using python 2.7.6 on ubuntu 14.04
<..> 
> There is no \n character at the end — which means that
> p.stdout.readline() cannot return. In fact, if you printed repr() of
> the line you read, you would get this:
> 
> b'\r[download]  54.9% of 2.73MiB at 26.73KiB/s ETA 00:47\r[download]
> 55.0% of 2.73MiB at 79.33KiB/s ETA 00:15\r…snip…\r[download] 100% of
> 2.73MiB in 00:01\n'
> 
> The download line is implemented using \r, which is the carriage
> return character (return to the first character), and then by
> overwriting characters that were already printed.
> 
> The solution? There are numerous. I’ll help you by obscuring the worst one.
> 
> (1) [recommended] figure out how to make youtube_dl work as a library,
> read its main file to figure out the problem. Don’t mess with
> subprocess.
  Was my first goal, had some problems, but I have solved them in
  part by finding the good documentation of the developers.

  I.E., the subprocess method _is_ going away and I will be using
  the youtube_dl module.

> (2) [don’t do it] do you need to intercept the lines? If you don’t set
> stderr= and stdout=, things will print just fine.
  Got to try that before using the module, just for edification.

> (3) [DON’T DO IT] .ernq() punenpgre ol punenpgre naq znxr n zrff.
> 
> PS. Thank you for setting a sensible Reply-To header on your messages.
> Which is something the list should be doing.
  LOL! Glad to help :)
  Thanks for the reply and the further education. 
  Cheers
-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Tim Johnson]

* Chris Warrick  [15 07:54]:
> On 11 November 2015 at 17:16, Tim Johnson  wrote:
> >> (2) [don’t do it] do you need to intercept the lines? If you don’t set
> >> stderr= and stdout=, things will print just fine.
> >   Got to try that before using the module, just for edification.
> 
> At which point your initial code sample will become:
> ###
> p = subprocess.Popen(list(args))
> ###
> 
  Yeah, 'list is redundant.
  Progress is now showing, but I forgot to say that I've lost the
  original intent, and that was to examine each line so that I could
  pull out the title.

  No matter. I'm on the way to make the youtube_dl module working.
  cheers

-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Chris Angelico]

On Wed, Nov 11, 2015 at 9:47 AM, Tim Johnson  wrote:
> I've written a command-line "wrapper" for youtube-dl, executing
> youtube-dl as a subprocess.
>
> --
> youtube-dl reports download progress on one line. I.E. the line is
> overwritten numerous times with no carriage return until the
> downloading is finished.
> --
>

Sounds to me like a possible buffering problem. But since youtube-dl
is implemented in Python, you might find it easier to "pip install
youtube_dl" and work with the methods directly:

>>> import youtube_dl
>>> youtube_dl.YoutubeDL().download(["m39ydsOPSww"])

Should be possible to manipulate around the outside of that.

ChrisA
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Tim Johnson]

* Chris Angelico  [151110 14:35]:
> On Wed, Nov 11, 2015 at 9:47 AM, Tim Johnson  wrote:
> > I've written a command-line "wrapper" for youtube-dl, executing
> > youtube-dl as a subprocess.
> >
> > --
> > youtube-dl reports download progress on one line. I.E. the line is
> > overwritten numerous times with no carriage return until the
> > downloading is finished.
> > --
> >
> 
> Sounds to me like a possible buffering problem. But since youtube-dl
> is implemented in Python, you might find it easier to "pip install
> youtube_dl" and work with the methods directly:
> 
> >>> import youtube_dl
> >>> youtube_dl.YoutubeDL().download(["m39ydsOPSww"])

  Frankly, I'd prefer - in the long run - to use youtube_dl (the
  module).

  But, when I do as you have suggested (and have tried previously) I
  get the following:

  youtube_dl.utils.DownloadError: ERROR: no suitable InfoExtractor

  I've briefly researched the error and so far haven't come up with an
  solution - guessing that other setup code is needed ...

  As for the buffering issue, at this point, I'll try  using a
  lower-level function like sys.stdout.write when the line 
  tests for '[download]' and 'ETA'.

  :) more elegant solutions welcome. I do intend to investigate
  using youtube_dl, once subprocess has been dealt with.

  thanks for the quick reply

-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Using subprocess to capture a progress line

[Tim Johnson]

Using python 2.7.6 on ubuntu 14.04 
The application in question is run with bash and gnome-terminal :

I've written a command-line "wrapper" for youtube-dl, executing
youtube-dl as a subprocess.

--
youtube-dl reports download progress on one line. I.E. the line is
overwritten numerous times with no carriage return until the
downloading is finished.
--

The following code runs the youtube-dl command and reports each line
as output by youtube-dl
###
p = subprocess.Popen(list(args), stderr=subprocess.STDOUT,
 stdout=subprocess.PIPE)
while True:
line = p.stdout.readline()
if not line:
break
tmp = line.strip()
print tmp
###

However this method not does show the download progress _until_ the
download is complete. 

To clarify : follows is output from my app running youtube-dl.
I've annotated the line in question with '###'

[youtube] ZIgnHPqp0Dk: Downloading webpage
[youtube] ZIgnHPqp0Dk: Downloading video info webpage
[youtube] ZIgnHPqp0Dk: Extracting video information
[youtube] ZIgnHPqp0Dk: Downloading js player en_US-vfljDEtYP
[youtube] ZIgnHPqp0Dk: Downloading DASH manifest
[download] Destination: Someday Soon - Judy Collins 1969.avi.m4a
### the line below is not seen until download is finished.
[download] 100% of 7.58MiB in 00:12.85KiB/s ETA 00:00
[ffmpeg] Correcting container in "Someday Soon - Judy Collins 1969.avi.m4a"
[ffmpeg] Destination: Someday Soon - Judy Collins 1969.avi.mp3
Deleting original file Someday Soon - Judy Collins 1969.avi.m4a (pass -k to 
keep)

Is there a way to code subprocess so the the progress is being
shown? In my case lines being output aren't being shown unless a
newline is sent, or so I understand it.

FYI : the need for this function in this case is trivial, but the
solution will be enlightening for me and have other uses, I'll bet.

thanks

-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Tim Johnson]

* Tim Johnson  [151110 14:55]:
> * Chris Angelico  [151110 14:35]:
> > On Wed, Nov 11, 2015 at 9:47 AM, Tim Johnson  wrote:
> > > I've written a command-line "wrapper" for youtube-dl, executing
> > > youtube-dl as a subprocess.
> > >
> > > --
> > > youtube-dl reports download progress on one line. I.E. the line is
> > > overwritten numerous times with no carriage return until the
> > > downloading is finished.
> > > --
> > >
> > 
> > Sounds to me like a possible buffering problem. But since youtube-dl
> > is implemented in Python, you might find it easier to "pip install
> > youtube_dl" and work with the methods directly:
> 
>   As for the buffering issue, at this point, I'll try  using a
>   lower-level function like sys.stdout.write when the line 
>   tests for '[download]' and 'ETA'.
That method doesn't work ... so far. Oh well.
-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Using subprocess to capture a progress line

[Tim Johnson]

* Tim Johnson  [151110 14:55]:
> * Chris Angelico  [151110 14:35]:
> > On Wed, Nov 11, 2015 at 9:47 AM, Tim Johnson  wrote:
> > > I've written a command-line "wrapper" for youtube-dl, executing
> > is implemented in Python, you might find it easier to "pip install
> > youtube_dl" and work with the methods directly:
<<...> > 
> > >>> import youtube_dl
> > >>> youtube_dl.YoutubeDL().download(["m39ydsOPSww"])
> 
>   Frankly, I'd prefer - in the long run - to use youtube_dl (the
>   module).
> 
>   But, when I do as you have suggested (and have tried previously) I
>   get the following:
> 
>   youtube_dl.utils.DownloadError: ERROR: no suitable InfoExtractor
> 
It looks as if the developers are constantly upgrading. I just
upgraded youtube_dl

sudo pip install --upgrade youtube_dl

And ran the code which is provided under the topic 
"EMBEDDING YOUTUBE-DL"
at 
https://github.com/rg3/youtube-dl/blob/master/README.md

At I got a successful download, AFAICS
-- 
Tim 
http://www.akwebsoft.com, http://www.tj49.com
-- 
https://mail.python.org/mailman/listinfo/python-list