Re: how to save video on web page

[David Wright]

On Wed 08 Apr 2020 at 13:25:55 (-), Curt wrote:
> On 2020-04-07, Ihor Antonov  wrote:
> >> 
> >> Finally, when all else fails, and if you've read this far,
> >> you can just capture the screen contents with ffmpeg's
> >> x11grab and record it to an mpg file. The disadvantages are
> >> that you capture extraneous screen decorations, and you've got
> >> to dedicate the whole screen to watching the video, remembering
> >> to increase your blanking timeout too. If you can only record
> >> audio through the microphone, you get more extraneous rubbish
> >> there too.
> >> 
> >
> > That is one comprehensive write up!
> > Thanks David, today I learned something new thanks to you.
> >
> 
> Yet in all that detailed thoroughness I believe he neglected to mention
> one of the simpler techniques, which is to view the relevant page source
> in your browser and search for the link (mp4 is a good fishing string)
> to the video (far from infallible, of course, but has worked for me on
> occasion).

I thought that was a something everybody did, and not just for videos.
I probably download more PDFs that way myself. I thought people might
not have thought of the techniques I outlined. Is everyone here doing
them already as a matter of routine?

Anyway, for links, I use

function http-lines {
[ -n "$1" ] && printf '%s\n' "Usage:$FUNCNAME < html-source
breaks lines before any occurrence of "http" (whatever the context)
after first replacing newlines by blanks." >&2 && return 1
while read line; do
printf '%s' "$line"
done | sed -e 's,\(http[s]\?://\),\n\1,g' | less -S
}

to make finding links a bit easier. Sometimes you have to copy and
paste bits to assemble the link, eg where the filename is given
relative to some reference elsewhere in the web page.

I also have http-watches, a variant that looks specifically for
youtu* references, which can be plugged straight into youtube-dl
(which I have wrapped, see below).

And I think I've posted these here before:

function wg-in-quotes {
[ -z "$1" ] && printf '%s\n' "Usage:$FUNCNAME 'URL' (must be in 
single quotes!)
strips the google prefix (up to url=) and suffix (from =) from the 
argument,
translates various common %nn sequences in the URL, and retrieves it." 
>&2 && return 1
wget $(sed -e 
's/http.*url=//;s/=.*$//;s/%2F/\//g;s/%3A/:/g;s/%3D/=/g;s/%3F/?/g;s/%26/\&/g;s/%25/%/g'
 <<<"$1") # % characters must be edited last
}

function gy-in-quotes {
[ -z "$1" ] && printf '%s\n' "Usage:$FUNCNAME 'URL' (must be in 
single quotes!)
strips the google prefix (up to url=) and suffix (from =) from the 
argument,
translates various common %nn sequences in the URL, and retrieves it." 
>&2 && return 1
geto $(sed -e 
's/http.*url=//;s/=.*$//;s/%2F/\//g;s/%3A/:/g;s/%3D/=/g;s/%3F/?/g;s/%26/\&/g;s/%25/%/g'
 <<<"$1") # % characters must be edited last
}

which make it easier to copy and paste google's Link Location address
directly into the commandline. gy and geto[ther] are just my wrappers
round youtube-dl that download to a specific directory and maintain a
history file to avoid accidentally repeating the same download.

Cheers,
David.

Re: how to save video on web page

[Curt]

On 2020-04-07, Ihor Antonov  wrote:
>> 
>> Finally, when all else fails, and if you've read this far,
>> you can just capture the screen contents with ffmpeg's
>> x11grab and record it to an mpg file. The disadvantages are
>> that you capture extraneous screen decorations, and you've got
>> to dedicate the whole screen to watching the video, remembering
>> to increase your blanking timeout too. If you can only record
>> audio through the microphone, you get more extraneous rubbish
>> there too.
>> 
>
> That is one comprehensive write up!
> Thanks David, today I learned something new thanks to you.
>

Yet in all that detailed thoroughness I believe he neglected to mention
one of the simpler techniques, which is to view the relevant page source
in your browser and search for the link (mp4 is a good fishing string)
to the video (far from infallible, of course, but has worked for me on
occasion).

Long Wind's hot page source contains such an external link, in fact (but
it has 301 Moved Permanently, and then when Following: it's a 404 Not
Found).

Re: how to save video on web page

[Carl Fink]

On 4/7/20 7:39 PM, Siard wrote:

Long Wind wrote:

i've just installed Video DownloadHelper for jessie's firefox
but can't install an external app required by it

no luck
i give up

I have FF 75 downloaded from www.mozilla.org/en-US/firefox/new/
Unpack and run, no installation needed.
Menu > Add-ons > Extensions
Here's a screenshot: https://ibb.co/ZfC15rp
I wonder what that 'external app' could be, I am not aware of any.


VDH can't download videos from several sites as one file, because they
aren't. Segments are streamed separately. There's a companion app that
joins the segments after they are all downloaded separately.

--
Carl Fink   nitpick...@nitpicking.com

Read my blog at blog.nitpicking.com.  Reviews!  Observations!

Re: how to save video on web page

[Siard]

Long Wind wrote:
> i've just installed Video DownloadHelper for jessie's firefox
> but can't install an external app required by it
>
> no luck
> i give up

I have FF 75 downloaded from www.mozilla.org/en-US/firefox/new/
Unpack and run, no installation needed.
Menu > Add-ons > Extensions
Here's a screenshot: https://ibb.co/ZfC15rp
I wonder what that 'external app' could be, I am not aware of any.

Re: how to save video on web page

[Greg Wooledge]

On Tue, Apr 07, 2020 at 09:33:57PM +, Long Wind wrote:
>  jessie's firefox

Are you even serious.

Re: how to save video on web page

[Ihor Antonov]

On Tuesday, April 7, 2020 12:27:20 PM PDT David Wright wrote:
> On Tue 07 Apr 2020 at 19:11:57 (+0300), Anastasios Lisgaras wrote:
> > Youtube-dl  is indeed a
> > powerful and very good software for this job with many features and
> > options, but can you download videos *from anywhere ?*
> > 
> > What I want to say is that there are many web pages which greatly
> > hinder (prohibit) this possibility.
> > In this case, what can we do? Can we always find the hidden link
> > (source) of the video? If so, how?
> > If the page requires you to be logged in, what can we do?
> 
> I'm not sure what the implications are of having to login to a site.
> But in general you need different tools for different web sites.
> The BBC iplayer and youtube-dl are two such tools, and sometimes
> a download link is even available, which either the browser or
> wget can use (the latter preserving the metadata).
> 
> Where videos exist in their entirety, some sites still play them
> by downloading to a temporary file (and you can see the
> download in the progress bar, ahead of what's actually playing.
> A technique there is to examine /proc/N/fd where N is the
> process number of the browser tab. (The process name used to
> be xul-runner, Web Content etc, and looks as if it's currently
> /usr/lib/firefox-esr/firefox-esr -contentproc.)
> If you find an fd number F that's pointing to a file (deleted) in
> /tmp, then try copying that /proc/N/fd/F (following links). Do it
> when the download progress bar has reached the end, but the file
> is still playing. (Sometimes everything disappears as soon as the
> end is reached.)
> 
> Another technique is where the source is streaming (and might be
> open-ended). Here, the video can end up as fragments in your
> browser cache. How you handle them depends on whether they are
> audiovisual or in two seperate streams, and whether they are
> timestamped. Some are, some aren't. The former are relatively
> easy to reassemble with ffprobe to read the timings and ffmpeg
> to concatenate the pieces (and merge audio/video if necessary).
> 
> Where there's no internal timestamping, you can sometimes rely
> on the filesystem's own to figure out the correct ordering.
> But I prefer to run a script that watches files in the cache
> as they are closed (with inotifywait), and immediately copies
> them out (if the filetype is of interest) with a sequence
> number and the file type in the filename. The relevant segments
> can then be concatenated quite easily. A timeformat of
> %Y%m%d-%H%M%S works well as a more meaningful sequence number,
> particularly if you append %N to include nanoseconds for the
> necessary time resolution.
> 
> Be aware that the fragments in your cache might not all be
> identified by the file program's defaults. For example, I use
> 0 string  G@  TS transport stream
> in ~/.magic to pickup files that file might otherwise label
> as 'data'.
> 
> Sometimes, even then, you have to use a little ingenuity for
> the quiet life: eg there's a UK railway site that has three
> webcams (two stations and the yard) which run simultaneously
> on the same web page. Fortunately, each webcam runs with a
> different frame speed, so it's quick and easy to distinguish
> their files and divide them up.
> 
> Finally, when all else fails, and if you've read this far,
> you can just capture the screen contents with ffmpeg's
> x11grab and record it to an mpg file. The disadvantages are
> that you capture extraneous screen decorations, and you've got
> to dedicate the whole screen to watching the video, remembering
> to increase your blanking timeout too. If you can only record
> audio through the microphone, you get more extraneous rubbish
> there too.
> 

That is one comprehensive write up!
Thanks David, today I learned something new thanks to you.

-- 
Ihor Antonov
https://useplaintext.email

Re: how to save video on web page

[Long Wind]

 i've just installed Video DownloadHelper for jessie's firefoxbut can't install 
an external app required by it
no lucki give up

On Wednesday, April 8, 2020, 5:13:21 AM GMT+8, Long Wind 
 wrote:  
 
  
Thank Siard!but Video DownloadHelper seems blocked in China, i can't install it.
i haven't been able to follow instruction by Ihor Antonovi can't find .m3u8 file

i'm not aware of debian policy on porn, i'm sorry if it offendsit's not 
honorable to view porn?
maybe it's a waste of time, i have such video clips in hard disks
Thanks to all who reply!





On Wednesday, April 8, 2020, 1:10:44 AM GMT+8, Siard 
 wrote:  
 
 Anastasios Lisgaras wrote:
> Video Download Helper it's a good tool but it never won me over and
> it was never so effective (on all websites) so as to amaze me.
> So much so that I no longer use it at all. Is it better/"stronger"
> in the new versions?

All I can say is that it /can/ download the video the OP wanted.
It can also download videos from sites that youtube-dl reports as
'unsupported'.  For me it has become a useful tool.

Re: how to save video on web page

[Greg Wooledge]

On Tue, Apr 07, 2020 at 09:13:21PM +, Long Wind wrote:
> i'm not aware of debian policy on porn, i'm sorry if it offendsit's not 
> honorable to view porn?

It's polite, when posting a link to a site that may contain porn, to
indicate that the site is "NSFW" (not safe for work), so that people who
are in a position where viewing porn could be a problem will know not
to open that link.

Re: how to save video on web page

[David Wright]

On Tue 07 Apr 2020 at 19:11:57 (+0300), Anastasios Lisgaras wrote:

> Youtube-dl  is indeed a
> powerful and very good software for this job with many features and
> options, but can you download videos *from anywhere ?*
> 
> What I want to say is that there are many web pages which greatly
> hinder (prohibit) this possibility.
> In this case, what can we do? Can we always find the hidden link
> (source) of the video? If so, how?
> If the page requires you to be logged in, what can we do?

I'm not sure what the implications are of having to login to a site.
But in general you need different tools for different web sites.
The BBC iplayer and youtube-dl are two such tools, and sometimes
a download link is even available, which either the browser or
wget can use (the latter preserving the metadata).

Where videos exist in their entirety, some sites still play them
by downloading to a temporary file (and you can see the
download in the progress bar, ahead of what's actually playing.
A technique there is to examine /proc/N/fd where N is the
process number of the browser tab. (The process name used to
be xul-runner, Web Content etc, and looks as if it's currently
/usr/lib/firefox-esr/firefox-esr -contentproc.)
If you find an fd number F that's pointing to a file (deleted) in
/tmp, then try copying that /proc/N/fd/F (following links). Do it
when the download progress bar has reached the end, but the file
is still playing. (Sometimes everything disappears as soon as the
end is reached.)

Another technique is where the source is streaming (and might be
open-ended). Here, the video can end up as fragments in your
browser cache. How you handle them depends on whether they are
audiovisual or in two seperate streams, and whether they are
timestamped. Some are, some aren't. The former are relatively
easy to reassemble with ffprobe to read the timings and ffmpeg
to concatenate the pieces (and merge audio/video if necessary).

Where there's no internal timestamping, you can sometimes rely
on the filesystem's own to figure out the correct ordering.
But I prefer to run a script that watches files in the cache
as they are closed (with inotifywait), and immediately copies
them out (if the filetype is of interest) with a sequence
number and the file type in the filename. The relevant segments
can then be concatenated quite easily. A timeformat of
%Y%m%d-%H%M%S works well as a more meaningful sequence number,
particularly if you append %N to include nanoseconds for the
necessary time resolution.

Be aware that the fragments in your cache might not all be
identified by the file program's defaults. For example, I use
0   string  G@  TS transport stream
in ~/.magic to pickup files that file might otherwise label
as 'data'.

Sometimes, even then, you have to use a little ingenuity for
the quiet life: eg there's a UK railway site that has three
webcams (two stations and the yard) which run simultaneously
on the same web page. Fortunately, each webcam runs with a
different frame speed, so it's quick and easy to distinguish
their files and divide them up.

Finally, when all else fails, and if you've read this far,
you can just capture the screen contents with ffmpeg's
x11grab and record it to an mpg file. The disadvantages are
that you capture extraneous screen decorations, and you've got
to dedicate the whole screen to watching the video, remembering
to increase your blanking timeout too. If you can only record
audio through the microphone, you get more extraneous rubbish
there too.

Cheers,
David.

Re: how to save video on web page

[Siard]

Anastasios Lisgaras wrote:
> Video Download Helper it's a good tool but it never won me over and
> it was never so effective (on all websites) so as to amaze me.
> So much so that I no longer use it at all. Is it better/"stronger"
> in the new versions?

All I can say is that it /can/ download the video the OP wanted.
It can also download videos from sites that youtube-dl reports as
'unsupported'.  For me it has become a useful tool.

Re: how to save video on web page

[Anastasios Lisgaras]

On 4/7/20 1:49 AM, Ihor Antonov wrote:
Having said that you can try downloading it yourself - all you need is 
to find a link to ".m3u8" file in network tab of developer mode when you 
start the video.


youtube-dl https://bla-bla/playlist.m3u8

--

Ihor Antonov

https://useplaintext.email



Youtube-dl  is indeed a powerful 
and very good software for this job with many features and options, but 
can you download videos *from anywhere ?*


What I want to say is that there are many web pages which greatly hinder 
(prohibit) this possibility.
In this case, what can we do? Can we always find the hidden link 
(source) of the video? If so, how?

If the page requires you to be logged in, what can we do?


On 4/7/20 2:39 PM, Siard wrote:

You can install the Video Download Helper browser extension.
It exists for Firefox and Vivaldi, don't know about other browsers.


Video Download Helper it's a good tool but it never won me over and it 
was never so effective (on all websites) so as to amaze me.
So much so that I no longer use it at all. Is it better/"stronger" in 
the new versions?

Re: how to save video on web page

[Carl Fink]

On 4/7/20 7:39 AM, Siard wrote:

Long Wind wrote:

i have stretch and want to save video on web page:
https://cl7v.com/html/14071/
which tool shall i install?

You can install the Video Download Helper browser extension.
It exists for Firefox and Vivaldi, don't know about other browsers.


Also Chrome that I know of, but Google requires that version to
not work for YouTube downloads.

--
Carl Fink   nitpick...@nitpicking.com

Read my blog at blog.nitpicking.com.  Reviews!  Observations!

Re: how to save video on web page

[Siard]

Long Wind wrote:
> i have stretch and want to save video on web page:
> https://cl7v.com/html/14071/
> which tool shall i install?

You can install the Video Download Helper browser extension.
It exists for Firefox and Vivaldi, don't know about other browsers.

Re: how to save video on web page

[Eike Lantzsch]

On Tuesday, 7 April 2020 06:37:20 -04 Michael Lange wrote:
> On Tue, 7 Apr 2020 12:28:39 +0200
>
> Michael Lange  wrote:
> > > Use upstream.
> >
> > or just use backports.
>
> oops, sorry, that had to be "use deb-multimedia", that's where the current
> version here actually comes from.
>
> Regards
>
> Michael
While, in general, I would be extremely cautious with using packages of deb-
multimedia, in the case of youtube-dl it does not pull-in other packages of
deb-multimedia.
-- up to now. That may change in the future. So be on the watch.

Have a nice day
Eike ZP6CGE

Re: how to save video on web page

[Michael Lange]

On Tue, 7 Apr 2020 12:28:39 +0200
Michael Lange  wrote:


> > Use upstream.
> > 
> 
> or just use backports.

oops, sorry, that had to be "use deb-multimedia", that's where the current
version here actually comes from.

Regards

Michael


.-.. .. ...- .   .-.. --- -. --.   .- -. -..   .--. .-. --- ... .--. . .-.

We do not colonize.  We conquer.  We rule.  There is no other way for us.
-- Rojan, "By Any Other Name", stardate 4657.5

Re: how to save video on web page

[Michael Lange]

Hi,

On Mon, 6 Apr 2020 18:20:20 -0400
Greg Wooledge  wrote:

> On Mon, Apr 06, 2020 at 02:51:06PM -0700, Ihor Antonov wrote:
> > On Monday, April 6, 2020 2:37:20 PM PDT Long Wind wrote:
> > > i have stretch and want to save video on web
> > > page:https://cl7v.com/html/14071/which tool shall i install? Thanks!
> > 
> > I usually use youtube-dl
> > 
> > https://packages.debian.org/search?
> > suite=all=all=any=names=youtube-dl[1] 
> > 
> > Official upstream site:
> > https://youtube-dl.org/[2] 
> 
> It's worth pointing out that the Debian packages of youtube-dl have a
> problem: they're stable.  Youtube is not.  You need a continually
> mutating package to retrieve content from Youtube, and Debian's stable
> releases can't provide that.
> 
> Use upstream.
> 

or just use backports.

Regards
Michael


.-.. .. ...- .   .-.. --- -. --.   .- -. -..   .--. .-. --- ... .--. . .-.

... The prejudices people feel about each other disappear when they get
to know each other.
-- Kirk, "Elaan of Troyius", stardate 4372.5

Re: how to save video on web page

[Ihor Antonov]

On Monday, April 6, 2020 3:28:26 PM PDT Long Wind wrote:
>  Thank Ihor Antonov!i've just tried youtube-dl
> zhou@debian:~$  youtube-dl https://cl7v.com/html/14071/
> [generic] 14071: Requesting header
> WARNING: Falling back on generic information extractor.
> [generic] 14071: Downloading webpage
> [generic] 14071: Extracting information
> ERROR: Unsupported URL: https://cl7v.com/html/14071/


> i've tried wget:wget
> http://ppp.downloadxx.com/assets/52c97a151032aafc25c39304ae62d3ee.mp4"it
> doesn't work either
> 
> On Tuesday, April 7, 2020, 5:58:11 AM GMT+8, Ihor Antonov
>  wrote:
> 
>  #yiv6712804207 p, #yiv6712804207 li {white-space:pre-wrap;}
> 
> On Monday, April 6, 2020 2:37:20 PM PDT Long Wind wrote:
> > i have stretch and want to save video on web
> 

Well Long Wind, I was going to get the exact command for you but I have opened 
that 
video link and it appeared to be some sort of porn. 

I am not sure what Debian policy have to say about this but I personally do not 
support 
this. You should at least warn people about such things. A lot of people read 
this mailing list 
at work or at home and nobody want to have porn coming.

Having said that you can try downloading it yourself - all you need is to find 
a link to 
".m3u8" file in network tab of developer mode when you start the video.

youtube-dl https://bla-bla/playlist.m3u8



-- 
Ihor Antonov
https://useplaintext.email

Re: how to save video on web page

[Greg Wooledge]

On Mon, Apr 06, 2020 at 02:51:06PM -0700, Ihor Antonov wrote:
> On Monday, April 6, 2020 2:37:20 PM PDT Long Wind wrote:
> > i have stretch and want to save video on web
> > page:https://cl7v.com/html/14071/which tool shall i install? Thanks!
> 
> I usually use youtube-dl
> 
> https://packages.debian.org/search?
> suite=all=all=any=names=youtube-dl[1] 
> 
> Official upstream site:
> https://youtube-dl.org/[2] 

It's worth pointing out that the Debian packages of youtube-dl have a
problem: they're stable.  Youtube is not.  You need a continually
mutating package to retrieve content from Youtube, and Debian's stable
releases can't provide that.

Use upstream.

Re: how to save video on web page

[Ihor Antonov]

On Monday, April 6, 2020 2:37:20 PM PDT Long Wind wrote:
> i have stretch and want to save video on web
> page:https://cl7v.com/html/14071/which tool shall i install? Thanks!

I usually use youtube-dl

https://packages.debian.org/search?
suite=all=all=any=names=youtube-dl[1] 

Official upstream site:
https://youtube-dl.org/[2] 



-- 
Ihor Antonov
https://useplaintext.email


[1] https://packages.debian.org/search?
suite=all=all=any=names=youtube-dl
[2] https://youtube-dl.org/