Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-18 Thread Reino Wijnsma 
On 2022-04-17T00:45:14+0200, Bo Berglund  wrote:
> One can see an item 'exp=1650232874', where the number seems to be seconds 
> since
> 1970. That is the base of timekeeping on Linux, right?

That's right. See https://en.wikipedia.org/wiki/Unix_time.

$ date -d @1650232874 -u '+%FT%T%:z'
2022-04-17T22:01:14+00:00

$ xidel -se '1650232874 * duration("PT1S") + dateTime("1970-01-01T00:00:00Z")'
2022-04-17T22:01:14Z

On 2022-04-18T07:28:23+0200, Bo Berglund  wrote:
> 2) Whenever it fails and I read the email *I manually start FireFox and visit 
> the
> page with F12 so I can find the new m3u8 URL*. It is then saved to a file 
> used to
> read the URL from the download script.
>
> [...]
>
> But it would be much better if the Firefox procedure could be replaced by an
> automatic extractor like I have for other sites than this:
>
> http://www.freeintertv.com/view/id-2308
>
> So far no luck in using curl or xidel or the like to find the currently active
> m3u8 URL for this site.

Did you totally miss my latest e-mail? Why do I even bother.

On 2022-04-18T12:53:19+0200, stefano roveda  wrote:
> Dear reino, i found the thread very interesting i don see any reason fir
> your comment in that case
> Anyway thank you for taking care.

I'm sorry, but I don't understand what you're trying to say.
What comment in what case? Taking care of what?

-- 
Reino

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-18 Thread Rob Hallam 
On Mon, 18 Apr 2022 at 14:01, Bo Berglund  wrote:
> On Mon, 18 Apr 2022 12:53:19 +0200, stefano roveda  wrote:
>
> >Dear reino, i found the thread very interesting i don see any reason fir
your comment in that case
[ed: quote fixed to original]
> >Anyway thank you for taking care.
>
> I can supply the exact ffmpeg command I use in the script for the download if
> that would make it better suited here...

Including the exact ffmpeg command and complete uncut output is almost
always a good idea.

However, that doesn't necessarily make the question any better suited
for the ffmpeg-user mailing list. Your question boils down to: "how do
I find a changing URL?", the usage of ffmpeg is incidental, I hope you
would agree.

Another example would be something like, "What is the best keyboard to
type out ffmpeg commands?" Yes, this is contrived and ridiculous for
the purposes of illustration. [1]

To address stefano roveda's point- discussions can be interesting to
some, even if they are off-topic. But Reino is right to point out more
productive places to have such discussions. I did too! The point is to
better serve all parties- a place that specialises in extracting URLs
for videos so that they can be downloaded -- like yt-dlp or youtube-dl
! -- is more likely to have experience with that sort of thing and so
serve Bo Berglund better than here. Discussing in that place or with
those folks would also spare the list replies like "my version of
xidel does not have that option" or "the URL worked yesterday but not
today"; which are in my view definitely off-topic, even if the
original query was on-topic.

I don't speak for the ffmpeg or the ffmpeg-user mailing list in any
official capacity. That said, list etiquette is everyone's business :)

Cheers,
Rob



[1]: Stack Overflow had this sort of issue back in the day- people
would append "for programmers" to an off-topic question to make it
appear ostensibly on-topic. [2] This got to the point where someone
asked a question along the lines of 'What do I need to do in order to
be a programmer out at sea?' [3], and hilarity/chaos ensued.

[2]: https://meta.stackexchange.com/a/14486/302572

[3]: https://web.archive.org/web/20100426125115/http://www.mattmcdole.com/boat/
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-18 Thread Bo Berglund 
On Mon, 18 Apr 2022 12:53:19 +0200, stefano roveda  wrote:

>Dear reino, i found the thread very interesting.
>Anyway thank you for taking care.

I can supply the exact ffmpeg command I use in the script for the download if
that would make it better suited here...

#!/bin/bash
URLFILE="/path/to/urlfile"
read -r VIDEOURL M3U8URL < $URLFILE #URLFILE referer and m3u8 URL
MODE="-referer \"${VIDEOURL}\""
CAPTURETIME="$1"
TARGETFILE="$2"
CMD="ffmpeg -hide_banner ${MODE} -i \"${M3U8URL}\" -vf scale=w=-4:h=480 -c:v
libx264 -preset fast -crf 26 -c:a copy -t ${CAPTURETIME} ${TARGETFILE}"
eval "$CMD" #Run the download


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-18 Thread stefano roveda 
Dear reino, i found the thread very interesting i don see any reason fir
your comment in that case
Anyway thank you for taking care.

Il giorno sab 9 apr 2022 alle 13:14 Reino Wijnsma  ha
scritto:

> Hello Bo,
>
> On 2022-04-07T18:48:06+0200, Bo Berglund  wrote:
> > I want to get hold of the m3u8 URL on this site
>
> You do realize this is the FFmpeg mailinglist, right? This question has
> nothing to do with FFmpeg.
> If you want more answers, then this is the wrong place to ask.
>
> > Another example page on the same site going through the same procedure:
> > http://www.freeintertv.com/view/id-2565
>
> By looking at the network-traffic you can see (or at least I can) this
> POST request:
>
> $ curl --data-raw "chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%
> 2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11"
> \
> "http://www.freeintertv.com/myAjax/get_item_m3u8/;
>
> This can be replicated/automated with the HTML/XML/JSON parser Xidel,
> which I've showed you earlier.
>
> The value to be used for "chname":
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e
> 'extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)'
> msnbc_live
>
> "chname" base64 encoded:
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>   string-to-base64Binary(
> extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>   )
> '
> bXNuYmNfbGl2ZQ==
>
> The string to POST:
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>   {
> "chname":string-to-base64Binary(
>   extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
> ),
> "ch":"
> http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
> "html5":"11"
>   }
> '
> {
>   "chname": "bXNuYmNfbGl2ZQ==",
>   "ch": "http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online
> ",
>   "html5": "11"
> }
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>   request-combine(
> (),
> {
>   "chname":string-to-base64Binary(
> extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>   ),
>   "ch":"
> http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>   "html5":"11"
> }
>   )/substring(url,2)
> '
> chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com
> %2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11
>
> And the actual POST-request:
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>   x:request({
> "post":request-combine(
>   (),
>   {
> "chname":string-to-base64Binary(
>   extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
> ),
> "ch":"
> http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
> "html5":"11"
>   }
> )/substring(url,2),
> "url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>   })/raw
> '
> playlist[0]['file']='
> http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8';
> get_item.showPlayer();
>
> Extract the m3u8-url:
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>   x:request({
> "post":request-combine(
>   (),
>   {
> "chname":string-to-base64Binary(
>   extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
> ),
> "ch":"
> http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
> "html5":"11"
>   }
> )/substring(url,2),
> "url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>   })/extract(raw,"http.+m3u8")
> '
>
> http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8
>
> > http://www.freeintertv.com/view/id-2308
>
> This one doesn't work in my browser, but appears to work the same way:
>
> $ xidel -s "http://www.freeintertv.com/view/id-2308; -e
> 'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"
> http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online
> ","html5":"11"})/substring(url,2),"url":"
> http://www.freeintertv.com/myAjax/get_item_m3u8/
> "})/extract(raw,"http.+m3u8")'
> http://livedoc.cgtn.com/500d/prog_index.m3u8
>
> --
> Reino
>
> ___
> ffmpeg-user mailing list
> ffmpeg-user@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
>
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-17 Thread Bo Berglund 
On Sun, 17 Apr 2022 21:35:23 -0300, Micael Silva  wrote:

>> I tried to manually enter a new bigger value for the exp= item to see if it
>> could be extended to 48 hours but it did not work...
>> It returned an error when used with ffmpeg.
>
>
>I don’t know how much you know about tokenized URLs but you couldn’t create
>a valid token just increasing the expire time because it need to match the
>HMAC hash param. That hash is calculated using all parameters plus a secret
>only known by the broadcaster.

It was a test for checking only. I did not really believe it would work given
the way the URL looks.

What I have now is the following:

1) A cron job which tries to use the current URL to download 20 s of video every
hour and send me an email if it fails.

2) Whenever it fails and I read the email I manually start FireFox and visit the
page with F12 so I can find the new m3u8 URL. It is then saved to a file used to
read the URL from the download script.

3) If it has not failed I do step #2 anyway in the evening so it will be OK
during nightly downloads (as seen from here).

But it would be much better if the Firefox procedure could be replaced by an
automatic extractor like I have for other sites than this:

http://www.freeintertv.com/view/id-2308

So far no luck in using curl or xidel or the like to find the currently active
m3u8 URL for this site.


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-17 Thread Micael Silva 
Em sáb., 16 de abr. de 2022 às 19:45, Bo Berglund 
escreveu:

> On Sat, 16 Apr 2022 15:04:02 +0200, Reino Wijnsma 
> wrote:
>
> >...you can see that the m3u8-url for the HLS manifest is only valid for 2
> minutes (1650113048 - 1650112928 = 120).
> >The m3u8-url for the individual videostreams on the other hand is valid
> for 24 hours (1650113048 - 1650199328 = 86400 / 60 / 60 = 24).
>
> I just found this out with the help of my cron job trying to deduce the
> lifetime
> of the m3u8 retrieved via Firefox interaction. Turns out I had to wait
> almost 24
> hours since I created the URL in Firefox before the alarm triggered.
>
> As you say it seems to live for 24 hours.
> In this URL:
>
> https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1650232874~acl=%2f*~data=hdntl~hmac=2f327aa18ebec99f42654f1f6018ef74e98f12ba085743535ca3d6fb6677c7aa/VIDEO_1_4596000.m3u8
>
> One can see an item 'exp=1650232874', where the number seems to be seconds
> since
> 1970. That is the base of timekeeping on Linux, right?
>
> date +%s   gives the seconds since the 1970.
>
> And when I extract an M3U8 URL in Firefox at a certain time and subtract
> the
> value given by the cmd 'date +%s' it returns 86400 +/- change, so the URL
> lifetime is in fact 24 hours.
>
> I tried to manually enter a new bigger value for the exp= item to see if it
> could be extended to 48 hours but it did not work...
> It returned an error when used with ffmpeg.


I don’t know how much you know about tokenized URLs but you couldn’t create
a valid token just increasing the expire time because it need to match the
HMAC hash param. That hash is calculated using all parameters plus a secret
only known by the broadcaster.

> --

—
Micael Silva
@micaelsilva 
(19) 99738-9172
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-16 Thread Bo Berglund 
On Sat, 16 Apr 2022 15:04:02 +0200, Reino Wijnsma  wrote:

>...you can see that the m3u8-url for the HLS manifest is only valid for 2 
>minutes (1650113048 - 1650112928 = 120).
>The m3u8-url for the individual videostreams on the other hand is valid for 24 
>hours (1650113048 - 1650199328 = 86400 / 60 / 60 = 24).

I just found this out with the help of my cron job trying to deduce the lifetime
of the m3u8 retrieved via Firefox interaction. Turns out I had to wait almost 24
hours since I created the URL in Firefox before the alarm triggered.

As you say it seems to live for 24 hours.
In this URL:
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1650232874~acl=%2f*~data=hdntl~hmac=2f327aa18ebec99f42654f1f6018ef74e98f12ba085743535ca3d6fb6677c7aa/VIDEO_1_4596000.m3u8

One can see an item 'exp=1650232874', where the number seems to be seconds since
1970. That is the base of timekeeping on Linux, right?

date +%s   gives the seconds since the 1970.

And when I extract an M3U8 URL in Firefox at a certain time and subtract the
value given by the cmd 'date +%s' it returns 86400 +/- change, so the URL
lifetime is in fact 24 hours.

I tried to manually enter a new bigger value for the exp= item to see if it
could be extended to 48 hours but it did not work...
It returned an error when used with ffmpeg.


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-16 Thread Reino Wijnsma 
On 2022-04-13T01:42:39+0200, Reino Wijnsma  wrote:
> ...and I don't understand how something like 
> "http://188.17.143.39/hls/CH_R04_DOMASHNY/variant.m3u8?version=2; would 
> translate to...
>
> On 2022-04-12T17:54:32+0200, Bo Berglund  wrote:
>> https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1649865178~acl=%2f*~data=hdntl~hmac=5480897632ddf98d23bf870098333af5055752c54d28764b8c894711086d1266/VIDEO_1_4596000.m3u8

Well,...

On 2022-04-13T23:49:26+0200, Reino Wijnsma  wrote:
> Alternatively I could have a look on a much newer system of one of my family 
> members

...I've found out that 
"http://188.17.143.39/hls/CH_R04_DOMASHNY/variant.m3u8?version=2; (from the 
iframe's HTML-source) has nothing to do with the akamaized-url. Very strange.

In the network-traffic I've found 2 different JSON api endpoints that house the 
bare m3u8-url:

$ xidel -s 
"https://api-leap.nbcsports.com/feeds/assets/2007524?application=NBCNews=nbc-player=desktop;
 \
  -e '$json/(videoSources)()/cdnSources/(primary)()/sourceUrl'
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json; \
  -e '$json/(videoSources)()/sourceUrl' \
  -e '$json/(videoSources)()/cdnSources/(primary)()/url'
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8

I have no idea where the identifier 2007524 is coming from though! It's not in 
the HTML-source of "http://www.freeintertv.com/view/id-2308; or its iframe. I 
couldn't find it anywhere else. It must be the program-id of the NBC livestream.

This HLS manifest doesn't work just yet:

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json; \
  -f '{"url":$json/(videoSources)()/sourceUrl,"error-handling":"xxx=accept"}' \
  -e '$headers[1]'
HTTP/1.1 403 Forbidden

The m3u8-url needs to be sent to "https://tokens.playmakerservices.com; in a 
POST-request:

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json; -e 
'$json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl}'
{
  "requestorId": "nbcnews",
  "pid": "2007524",
  "application": "NBCSports",
  "version": "v1",
  "platform": "desktop",
  "token": "",
  "resourceId": "",
  "inPath": "false",
  "authenticationType": "unauth",
  "cdn": "akamai",
  "url": 
"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8;
}

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json; -e 
'serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"})'
{"requestorId":"nbcnews","pid":"2007524","application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":"https:\/\/nbcnews3.akamaized.net\/hls\/live\/723429\/NBCNewsNowa418c506\/clear\/master.m3u8"}

$ xidel -s "https://stream.nbcnews.com/data/live_sources_2007524.json; -e 
'x:request({"post":serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"}),"url":"https://tokens.playmakerservices.com"})/json'
{
  "akamai": [
{
  "token": 
"hdnts=st=1650111897~exp=1650112017~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=bcab5244534ea41b8b5c585c67805765f582cc5386ab89665833c9e8651c16d0",
  "tokenizedUrl": 
"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8?hdnts=st=1650111897~exp=1650112017~acl=/hls/live/723429/NBCNewsNowa418c506/clear/*~hmac=bcab5244534ea41b8b5c585c67805765f582cc5386ab89665833c9e8651c16d0;,
  "sourceUrl": 
"https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/master.m3u8;
}
  ]
}

$ ffmpeg -hide_banner -i "$(xidel -s 
"https://stream.nbcnews.com/data/live_sources_2007524.json; -e 
'x:request({"post":serialize($json/(videoSources)()/{"requestorId":"nbcnews","pid":pid,"application":"NBCSports","version":"v1","platform":"desktop","token":"","resourceId":"","inPath":"false","authenticationType":"unauth","cdn":"akamai","url":sourceUrl},{"method":"json"}),"url":"https://tokens.playmakerservices.com"})/json//tokenizedUrl')"
[...]
Input #0, hls, from

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-14 Thread Bo Berglund 
On Thu, 14 Apr 2022 10:26:56 +0200, Bo Berglund  wrote:

>Surprise!
>All of these small downloads completed and if I test now it still works!
>So I have expanded that test with a new set of at jobs for a couple of hours.
>
>It seems like I have found a "golden" URL by chance!
>
>Previous testing showed that even a 2+ hour old URL from FireFox did not work
>for the actual show download.
>
>I hope this URL will stay operational for at least a day or so.
>
>Could it be that it stays working as long as someone (me) keeps using it at
>regular intervals?
>Like some server side timeout?
>If so I could just set my system up to download a bogus snippet every hour or 
>so
>to keep it living...
>

Well, I installed a cron job to test the URL every hour at xx:32 and send me an
email if it failed.
It did so at 00:32 but I was asleep then so I could not fix it for the
recordings due...

The URL only lived about 24 hours this time.

So I still need an extraction script to call whenever it has failed.
Or every time a recording is about to start.


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-14 Thread Bo Berglund 
On Thu, 14 Apr 2022 10:26:56 +0200, Bo Berglund  wrote:

>Surprise!
>All of these small downloads completed and if I test now it still works!
>So I have expanded that test with a new set of at jobs for a couple of hours.
>
>It seems like I have found a "golden" URL by chance!
>
>Previous testing showed that even a 2+ hour old URL from FireFox did not work
>for the actual show download.
>
>I hope this URL will stay operational for at least a day or so.
>
>Could it be that it stays working as long as someone (me) keeps using it at
>regular intervals?
>Like some server side timeout?
>If so I could just set my system up to download a bogus snippet every hour or 
>so
>to keep it living...

So it is still working!
Same m3u8 URL extracted with Firefox yesterday is still working!

Now I have set up an hourly cron job to download a 20 s video using this static
m3u8 URL and if the ffmpeg call fails it will send me an alert email.
This way I will get a heads-up if the URL needs to be fixed.
But so far not needed, still running fine.

I just wonder why it changed this way suddenly?


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-14 Thread Bo Berglund 
On Thu, 14 Apr 2022 00:15:16 +0200, Reino Wijnsma  wrote:

>On 2022-04-13T23:49:26+0200, Reino Wijnsma  wrote:
>> Or minified:
>>
>> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
>> 'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'
>Alternatively, instead of using x:request() in-query, you could use the 
>command-line option --post / -d, which I guess more resembles curl:
>
>$ xidel -s -d 
>'{request-combine((),{"chname":string-to-base64Binary(doc("http://www.freeintertv.com/view/id-2565;)//meta[@property="og:image"]/extract(@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2)}'
> "http://www.freeintertv.com/myAjax/get_item_m3u8/; -e 
>'extract($raw,"http.+m3u8")'
>
>$ xidel -s \
>  -d '{
>request-combine(
>  (),
>  {
>"chname":string-to-base64Binary(
>  
> doc("http://www.freeintertv.com/view/id-2565;)//meta[@property="og:image"]/
>  extract(@content,"([a-z_]+)\.jpg",1)
>),
>
> "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>"html5":"11"
>  }
>)/substring(url,2)
>  }' \
>  "http://www.freeintertv.com/myAjax/get_item_m3u8/; \
>  -e 'extract($raw,"http.+m3u8")'

This also works to get the URL for video source 2565.


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-14 Thread Bo Berglund 
On Wed, 13 Apr 2022 23:49:26 +0200, Reino Wijnsma  wrote:

>> On Wed, 13 Apr 2022 01:42:39 +0200, Reino Wijnsma  wrote:
>>> That command was for Xidel 0.9.8, which you're not using anymore if I 
>>> understand correctly.
>> What would it be for the most recent xidel?
>
>The final query in my initial post:
>
>On 2022-04-09T13:14:18+0200, Reino Wijnsma  wrote:
>> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>>   x:request({
>> "post":request-combine(
>>   (),
>>   {
>> "chname":string-to-base64Binary(
>>   extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>> ),
>> 
>> "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>> "html5":"11"
>>   }
>> )/substring(url,2),
>> "url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>>   })/extract(raw,"http.+m3u8")
>> '
>> http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8
>
>Or minified:
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
>'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'

This command executed OK and produced a valid m3u8 URL for the 2565 video.
But for that I already had a working extraction command using curl.

Back to video source Id=2308

Yesterday evening I grabbed a new m3u8 URL using FireFox and then I set up a
number of at jobs to use that in order to download small 20 s snippets every 10
minutes for 3 hours overnight.
This was done to check for how long such an M3U8 would be valid. I expected 1-2
hours.

Surprise!
All of these small downloads completed and if I test now it still works!
So I have expanded that test with a new set of at jobs for a couple of hours.

It seems like I have found a "golden" URL by chance!

Previous testing showed that even a 2+ hour old URL from FireFox did not work
for the actual show download.

I hope this URL will stay operational for at least a day or so.

Could it be that it stays working as long as someone (me) keeps using it at
regular intervals?
Like some server side timeout?
If so I could just set my system up to download a bogus snippet every hour or so
to keep it living...


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-13 Thread Reino Wijnsma 
On 2022-04-13T23:49:26+0200, Reino Wijnsma  wrote:
> Or minified:
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
> 'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'
Alternatively, instead of using x:request() in-query, you could use the 
command-line option --post / -d, which I guess more resembles curl:

$ xidel -s -d 
'{request-combine((),{"chname":string-to-base64Binary(doc("http://www.freeintertv.com/view/id-2565;)//meta[@property="og:image"]/extract(@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2)}'
 "http://www.freeintertv.com/myAjax/get_item_m3u8/; -e 
'extract($raw,"http.+m3u8")'

$ xidel -s \
  -d '{
request-combine(
  (),
  {
"chname":string-to-base64Binary(
  
doc("http://www.freeintertv.com/view/id-2565;)//meta[@property="og:image"]/
  extract(@content,"([a-z_]+)\.jpg",1)
),

"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
"html5":"11"
  }
)/substring(url,2)
  }' \
  "http://www.freeintertv.com/myAjax/get_item_m3u8/; \
  -e 'extract($raw,"http.+m3u8")'

-- 
Reino

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-13 Thread Reino Wijnsma 
On 2022-04-13T21:34:32+0200, Bo Berglund  wrote:
> For me it works on both Firefox and Chrome...
> What browser are you using?

I'm using "New Moon 
", a derivative 
of Pale Moon which in turn is based on Firefox.
Why? Believe it or not, but at the moment I'm still using an AMD Athlon XP 
3200+ and it's one of the very few browsers that still works on my cpu. It does 
mean however that websites that utilize the latest Javascript for instance 
won't work for me.
I do know I'm in desperate need of a new system and that'll probably be an AMD 
"Rembrandt" desktop APU or otherwise an AMD ZEN 4 cpu. Both scheduled for 
release later this year. The transition will be a shock I'm sure.

If you can share any useful hits you see in the network-traffic when you visit 
http://www.freeintertv.com/view/id-2308, then I could have a look. 
Alternatively I could have a look on a much newer system of one of my family 
members, which I will visit in a couple of days.

> On Wed, 13 Apr 2022 01:42:39 +0200, Reino Wijnsma  wrote:
>> That command was for Xidel 0.9.8, which you're not using anymore if I 
>> understand correctly.
> What would it be for the most recent xidel?

The final query in my initial post:

On 2022-04-09T13:14:18+0200, Reino Wijnsma  wrote:
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>   x:request({
> "post":request-combine(
>   (),
>   {
> "chname":string-to-base64Binary(
>   extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
> ),
> 
> "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
> "html5":"11"
>   }
> )/substring(url,2),
> "url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>   })/extract(raw,"http.+m3u8")
> '
> http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8

Or minified:

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'

-- 
Reino

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-13 Thread Bo Berglund 
On Wed, 13 Apr 2022 01:42:39 +0200, Reino Wijnsma  wrote:

>As the videostream doesn't work here, you're on your own on this one.

For me it works on both Firefox and Chrome...
What browser are you using?

>On 2022-04-12T17:54:32+0200, Bo Berglund  wrote:
>> Anyway, the command above used with the latest xidel yields an erroneous URL 
>> for
>> the 2565 video:
>> http://livedoc.cgtn.com/500d/prog_index.m3u8
>
>That command was for Xidel 0.9.8, which you're not using anymore if I 
>understand correctly.

What would it be for the most recent xidel?


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-12 Thread Reino Wijnsma 
On 2022-04-12T14:35:39+0200, Bo Berglund  wrote:
> I don't know from where this URL in the command came from (you say you can see
> it...):
> http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online
>
> But it is for the 2565 video...
> Maybe another is needed for 2308?

Apparently. Like I said, the videostream doesn't work in my browser, so I 
couldn't tell whether it was the same videostream.
Then I guess each channel / videostream need it's own approach in trying to 
recover the video-url.

I can only see...

$ xidel -s "http://www.freeintertv.com/view/id-2308; -e '(//iframe)[1]' 
--output-node-format=html
http://www.freeintertv.com/modules/View/screen.php?nametv=NBC+%28USA%29idtv=2308TVSESS=cko6m4au4uea7vh5aqmoi0ec35;
 scrolling="no" frameborder="0">

$ xidel -s "http://www.freeintertv.com/view/id-2308; -f '(//iframe)[1]' -e 
'(//script)[position() = (1,2)]' --output-node-format=html
var 
$config={"add":"http://188.17.143.39","go":"TRB2UWBSCQMNUG","stream":"s1","add_1":"http://85.94.1.9:8802"};

var videoid=3813;
var url_tv=true;
var outreg=/umain\('(http:\/\/[^']+)'/i;
var 
playlist=[{'file':$config['add']+'/hls/CH_R04_DOMASHNY/variant.m3u8?version=2'},{'file':'http://'+window.location.hostname+'/externals/tv-russia/smotret-tv3-online'},{'file':$config['add']+'hls/CH_R02_DOMASHNY/variant.m3u8?version=2'}];
var 
$params={top:"-80",left:"0",width:"500",height:"350",widthFr:"500",heightFr:"1500",timeout:1000,backgroundColor:"#000",backgroundPosition:"50%
 60%"};


...and I don't understand how something like 
"http://188.17.143.39/hls/CH_R04_DOMASHNY/variant.m3u8?version=2; would 
translate to...

On 2022-04-12T17:54:32+0200, Bo Berglund  wrote:
> https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1649865178~acl=%2f*~data=hdntl~hmac=5480897632ddf98d23bf870098333af5055752c54d28764b8c894711086d1266/VIDEO_1_4596000.m3u8

As the videostream doesn't work here, you're on your own on this one.

On 2022-04-12T17:54:32+0200, Bo Berglund  wrote:
> Anyway, the command above used with the latest xidel yields an erroneous URL 
> for
> the 2565 video:
> http://livedoc.cgtn.com/500d/prog_index.m3u8

That command was for Xidel 0.9.8, which you're not using anymore if I 
understand correctly.

-- 
Reino

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-12 Thread Bo Berglund 
On Tue, 12 Apr 2022 17:54:32 +0200, Bo Berglund  wrote:

>
>Anyway, the command above used with the latest xidel yields an erroneous URL 
>for
>the 2565 video:
>http://livedoc.cgtn.com/500d/prog_index.m3u8
>This is for a documentary site with wildlife footage. No MSNBC anywhere.
>
>The correct 2565 m3u8 extracted with Firefox right now is:
>http://1128480543.rsc.cdn77.org/ZzaSg9XJexGJKIy-kt2Mxg==,1649784902/1128480543/tracks-v1a1/mono.m3u8


And this is also working for the same site 2565:

http://1128480543.rsc.cdn77.org/ZzaSg9XJexGJKIy-kt2Mxg==,1649784902/1128480543/index.m3u8

It is extracted using this curl command:

curl -qSs -d
'chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11'
'http://www.freeintertv.com/myAjax/get_item_m3u8/' | sed -e
"s/^.*http\(.*\)m3u8.*$/http\1m3u8/g" 


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-12 Thread Bo Berglund 
On Tue, 12 Apr 2022 01:42:14 +0200, Reino Wijnsma  wrote:

>If you insist on using Xidel 0.9.8, then use:
>

So I am not insisting on any particular version...
I am now on the latest available.

>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
>'x:request({"post":request-combine({"url":()},{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/url,"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'
>

Anyway, the command above used with the latest xidel yields an erroneous URL for
the 2565 video:
http://livedoc.cgtn.com/500d/prog_index.m3u8
This is for a documentary site with wildlife footage. No MSNBC anywhere.

The correct 2565 m3u8 extracted with Firefox right now is:
http://1128480543.rsc.cdn77.org/ZzaSg9XJexGJKIy-kt2Mxg==,1649784902/1128480543/tracks-v1a1/mono.m3u8

And currently the correct Firefox url for 2308 is:
https://nbcnews3.akamaized.net/hls/live/723429/NBCNewsNowa418c506/clear/hdntl=exp=1649865178~acl=%2f*~data=hdntl~hmac=5480897632ddf98d23bf870098333af5055752c54d28764b8c894711086d1266/VIDEO_1_4596000.m3u8


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-12 Thread Bo Berglund 
On Sat, 9 Apr 2022 13:14:18 +0200, Reino Wijnsma  wrote:

>> $ xidel --version
>> Xidel 0.9.8
>> (20180421.6162.1f357eaaf5f3)
>
>A couple of months ago I already stated to use the latest binary from the 
>development branch:
>

So I have now installed the latest xidel binary on my Ubuntu Server 20.04.4
where all downloads happen:

$ xidel --version
Xidel 0.9.9
(20220131.8335.4e6fcea4d02e)


>> Another example page on the same site going through the same procedure:
>> http://www.freeintertv.com/view/id-2565
>
>By looking at the network-traffic you can see (or at least I can) this POST 
>request:
>
>$ curl --data-raw 
>"chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11"
> \
>"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>

This URL is used in your examples, which all work fine now that I understand
what is going on.
But this might not be the correct such URL for the 2308 video?
See below


>
>> http://www.freeintertv.com/view/id-2308
>
>This one doesn't work in my browser, but appears to work the same way:
>
>$ xidel -s "http://www.freeintertv.com/view/id-2308; -e 
>'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'
>http://livedoc.cgtn.com/500d/prog_index.m3u8
>

I have also gone ove each of your descriptive commands one by one and verified
how they work on the 2308 URL.
It seems like I get responses similar to but different from the 2565 URL for
early parts.

But when I run this command I get a response, which when used for download gets
a completely different channel:

$ xidel -s "http://www.freeintertv.com/view/id-2308; -e
'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'

output is this m3u8:
http://livedoc.cgtn.com/500d/prog_index.m3u8

It can be used with my downloader but when I record and then view the content it
shows a completely different channel...


I don't know from where this URL in the command came from (you say you can see
it...):
http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online

But it is for the 2565 video...
Maybe another is needed for 2308?


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Reino Wijnsma 
On 2022-04-11T12:25:17+0200, Bo Berglund  wrote:
> $ xidel --version
> Xidel 0.9.8
> (20180421.6162.1f357eaaf5f3)

A couple of months ago I already stated to use the latest binary from the 
development branch:

On 2022-01-09T23:07:55+0100, Reino Wijnsma  wrote:
> I'm not sure why you're trying to compile Xidel from (the wrong) source when 
> you can easily grab binaries from 
> https://sourceforge.net/projects/videlibri/files/Xidel/ (binaries from the 
> development map are recommended).

On 2022-04-11T12:25:17+0200, Bo Berglund  wrote:
> There seems to be a lot of line wraps either by the news reader I use (Free
> Agent) or the commands are not meant to be entered in a terminal...

You, as a developer, don't know the difference between a minified and 
prettified command?
I don't know what a "news reader" is, but 
https://ffmpeg.org/pipermail/ffmpeg-user/2022-April/054565.html shows my mail 
as intended.
And of course prettified commands can be entered in a terminal.

> So it prints the word "msnbc_live" as a single word and nothing gets stored on
> the disk either...

All Xidel commands in my previous post just print the output to stdout. I don't 
understand why you'd expect anything to be saved to disk.

> I tried this after making it unwrapped into a single command line, but got an
> error:
>
> xidel -s "http://www.freeintertv.com/view/id-2565; -e
> '{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"}'{"chname":
> "bXNuYmNfbGl2ZQ==","ch":
> "http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":
> "11"}
> Error:
> err:XPST0003: Unknown or unexpected operator: { (possible missing comma , or
> closing parentheses)}] )

On 2022-04-09T13:14:18+0200, Reino Wijnsma  wrote:
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e '   *opening quote of 
> extraction-query*
>   {
> "chname":string-to-base64Binary(
>   extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
> ),
> "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
> "html5":"11"
>   }
> '   *closing quote of 
> extraction-query*
> {
>   "chname": "bXNuYmNfbGl2ZQ==",
>   "ch": "http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>   "html5": "11"
> }

So, the JSON is not part of the command, but it's the *output*!

> So next I tried this last big command again after trying to get it onto a 
> single
> line (which the newsreader has wrapped...):
>
> $ xidel -s "http://www.freeintertv.com/view/id-2565; -e
> 'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")
>
> Now there is only a single > output and no command prompt until I hit Ctrl-C

You forgot to copy-paste the extraction-query closing quote.

If you insist on using Xidel 0.9.8, then use:

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
'x:request({"post":request-combine({"url":()},{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/url,"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'

I think this is enough off-topic for this mailinglist. If you have any more 
questions about Xidel, I suggest you go to:
StackOverflow - https://stackoverflow.com/questions/tagged/xidel
Mailinglist - https://sourceforge.net/p/videlibri/mailman/
(not so active) Forum - https://sourceforge.net/p/xidel/discussion

On 2022-04-11T17:46:52+0200, Robert Jeffares  wrote:
> A script using curl to download the site to a file then grep m3u8 will give
> you a list which you can then sort to select the_one_you_want then ffmpeg
> -I the_one_you_want etc

No it will not, because a POST request is required to get the video-url in this 
particular case. There's no mention of "m3u8" in the site its HTML-source.
And in addition, it's really bad practise to parse HTML with grep/sed/awk etc.

P.s. Please do not top-post on this mailinglist!

-- 
Reino

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Robert Jeffares 
I am not at home and internet in this Pacific paradise is a bit thin.
Having looked at the link I can see a bit of drilling down to the base
platform is needed. I will have a look when I get home. I have several
instances of grabbing both live and Podcasted video. In all cases you need
to get past the hand offs and embedded players to the source.

On Tue, 12 Apr 2022, 08:21 Bo Berglund,  wrote:

> On Tue, 12 Apr 2022 03:46:52 +1200, Robert Jeffares <
> jeffares.rob...@gmail.com>
> wrote:
>
> >A script using curl to download the site to a file then grep m3u8 will
> give
> >you a list which you can then sort to select the_one_you_want then ffmpeg
> >-I the_one_you_want etc
>
> Does not work...
>
> curl -o testfile http://www.freeintertv.com/view/id-2308
> grep m3u8 testfile
> ...blank output...
>
> It is more involved than this simple assumption.
> The m3u8 data appears when the player has started playing the video and
> then I
> can *manually* retrieve it using FireFox in F12 mode.
>
> But I need to automate this...
>
>
> >Bush shell script is able to manage simple cases sometimes you need more
> >grunt. Perl is a reliable option.
>
> I assume you mean bash as in the first line in such scripts:
> #!/bin/bash
>
> Bash works fine for what I do, but I need to find the right command to
> execute
> via bash.
> I already have curl code to retrieve the m3u8 data and then pass it to
> sed, but
> it won't work for this particular site.
>
> >I have a cron job which does this but use youtube-dl
>
> Youtube-dl has been abandoned for a long time now, replaced by yt-dlp.
> Youtube-dl could download live streams but only from Youtube and it could
> not be
> called with a specific timeout so when I used that a few years back I had
> to use
> the timeout function in order to make it stop at the wanted time:
>
> timeout --signal=2 67m 
>
> But that does not work with yt-dlp, which is not able to download live
> streams.
>
> >Hope this answers your question
>
> Unfortunately not.
>
>
> --
> Bo Berglund
> Developer in Sweden
>
> ___
> ffmpeg-user mailing list
> ffmpeg-user@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
>
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Bo Berglund 
On Tue, 12 Apr 2022 03:46:52 +1200, Robert Jeffares 
wrote:

>A script using curl to download the site to a file then grep m3u8 will give
>you a list which you can then sort to select the_one_you_want then ffmpeg
>-I the_one_you_want etc

Does not work...

curl -o testfile http://www.freeintertv.com/view/id-2308
grep m3u8 testfile
...blank output...

It is more involved than this simple assumption.
The m3u8 data appears when the player has started playing the video and then I
can *manually* retrieve it using FireFox in F12 mode.

But I need to automate this...


>Bush shell script is able to manage simple cases sometimes you need more
>grunt. Perl is a reliable option.

I assume you mean bash as in the first line in such scripts:
#!/bin/bash

Bash works fine for what I do, but I need to find the right command to execute
via bash.
I already have curl code to retrieve the m3u8 data and then pass it to sed, but
it won't work for this particular site.

>I have a cron job which does this but use youtube-dl

Youtube-dl has been abandoned for a long time now, replaced by yt-dlp.
Youtube-dl could download live streams but only from Youtube and it could not be
called with a specific timeout so when I used that a few years back I had to use
the timeout function in order to make it stop at the wanted time:

timeout --signal=2 67m 

But that does not work with yt-dlp, which is not able to download live streams.

>Hope this answers your question

Unfortunately not.


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Robert Jeffares 
A script using curl to download the site to a file then grep m3u8 will give
you a list which you can then sort to select the_one_you_want then ffmpeg
-I the_one_you_want etc

Bush shell script is able to manage simple cases sometimes you need more
grunt. Perl is a reliable option.

I have a cron job which does this but use youtube-dl
Hope this answers your question

Regards

Robert

On Tue, 12 Apr 2022, 02:37 Rob Hallam,  wrote:

> On Mon, 11 Apr 2022 at 09:42, Bo Berglund  wrote:
> > I have tried other linix oriented lists like Ubuntu users but got no
> replies.
>
> Did you try asking / discussing with the yt-dlp/youtube-dl communities
> as previously suggested? Figuring out how to get videos from web pages
> is their entire deal.
>
> Cheers,
> Rob
> ___
> ffmpeg-user mailing list
> ffmpeg-user@ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-user
>
> To unsubscribe, visit link above, or email
> ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".
>
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Rob Hallam 
On Mon, 11 Apr 2022 at 09:42, Bo Berglund  wrote:
> I have tried other linix oriented lists like Ubuntu users but got no replies.

Did you try asking / discussing with the yt-dlp/youtube-dl communities
as previously suggested? Figuring out how to get videos from web pages
is their entire deal.

Cheers,
Rob
___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Bo Berglund 
On Sat, 9 Apr 2022 13:14:18 +0200, Reino Wijnsma  wrote:

>By looking at the network-traffic you can see (or at least I can) this POST 
>request:
>
>$ curl --data-raw 
>"chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11"
> \
>"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>
>This can be replicated/automated with the HTML/XML/JSON parser Xidel, which 
>I've showed you earlier.

I did install it on one of my Ubuntu 20.04 machines the last time you suggested
it, here is what it tells me about its version:

$ xidel --version
Xidel 0.9.8
(20180421.6162.1f357eaaf5f3)

http://www.videlibri.de/xidel.html
by Benito van der Zander 


On the following lines I am not sure I understand what you write...
There seems to be a lot of line wraps either by the news reader I use (Free
Agent) or the commands are not meant to be entered in a terminal...


>
>The value to be used for "chname":
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
>'extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)' msnbc_live
>

I tried this, as a one-line command from xidel and ending with msnbc_live (news
reader wrapping text) but got the following output (in Ubuntu 20.04):

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e
'extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)' msnbc_live
msnbc_live

So it prints the word "msnbc_live" as a single word and nothing gets stored on
the disk either...


>"chname" base64 encoded:
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>  string-to-base64Binary(
>extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>  )
>'
>bXNuYmNfbGl2ZQ==

Here I got (again command wrapped by news reader):

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e
'string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1))'
bXNuYmNfbGl2ZQ==

So this seems to be the same output as you showed.


>The string to POST:
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>  {
>"chname":string-to-base64Binary(
>  extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>),
>"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>"html5":"11"
>  }
>'
>{
>  "chname": "bXNuYmNfbGl2ZQ==",
>  "ch": "http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>  "html5": "11"
>}
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>  request-combine(
>(),
>{
>  "chname":string-to-base64Binary(
>extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>  ),
>  "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>  "html5":"11"
>}
>  )/substring(url,2)
>'
>chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11

I tried this after making it unwrapped into a single command line, but got an
error:

xidel -s "http://www.freeintertv.com/view/id-2565; -e
'{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"}'{"chname":
"bXNuYmNfbGl2ZQ==","ch":
"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":
"11"}
Error:
err:XPST0003: Unknown or unexpected operator: { (possible missing comma , or
closing parentheses)}] )
in:
{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"}
[<- error occurs before here] {chname:

Did not try the following command but skipped down a bit:

>And the actual POST-request:
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>  x:request({
>"post":request-combine(
>  (),
>  {
>"chname":string-to-base64Binary(
>  extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>),
>
> "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>"html5":"11"
>  }
>)/substring(url,2),
>"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>  })/raw
>'
>playlist[0]['file']='http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8';
> get_item.showPlayer();


To here:

>Extract the m3u8-url:
>
>$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
>  x:request({
>"post":request-combine(
>  (),
>  {
>"chname":string-to-base64Binary(
>  extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
>),
>
> "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
>"html5":"11"
>  }
>)/substring(url,2),
>"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
>  })/extract(raw,"http.+m3u8")
>'
>http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-11 Thread Bo Berglund 
On Sat, 9 Apr 2022 13:14:18 +0200, Reino Wijnsma  wrote:

>On 2022-04-07T18:48:06+0200, Bo Berglund  wrote:
>> I want to get hold of the m3u8 URL on this site
>
>You do realize this is the FFmpeg mailinglist, right? This question has 
>nothing to do with FFmpeg.
>If you want more answers, then this is the wrong place to ask.
>

Yes I do, but I thought that getting hold of source m3u8 strings to be *used*
with ffmpeg would sort of qualify for this list...
I have tried other linix oriented lists like Ubuntu users but got no replies.

Most of my site handlers work fine but there the sites keep the m3u8 static.
There are a few which regularly change m3u8 and where I have not found a way to
dig out the m3u8 url when it does change...

I am not clever enough concerning analyzing network traffic so that is why I ask
around.

I AM much appreciating your help!

I just wish I also understood the networking stuff behind so I did not have to
ask repeatedly.

Concerning the changing sites I have added a job to refresh the URL:s 5 minutes
before the downloads start and this is where I would like to add the new site
too.
But it will not help if the changing time is in the middle of a download of
course. It seems like the changes are done more on a daily basis than very 30
minutes or so, though, so an update job might help.


-- 
Bo Berglund
Developer in Sweden

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".

Re: [FFmpeg-user] How can I extract m3u8 URL for use with ffmpeg on this site?

2022-04-09 Thread Reino Wijnsma 
Hello Bo,

On 2022-04-07T18:48:06+0200, Bo Berglund  wrote:
> I want to get hold of the m3u8 URL on this site

You do realize this is the FFmpeg mailinglist, right? This question has nothing 
to do with FFmpeg.
If you want more answers, then this is the wrong place to ask.

> Another example page on the same site going through the same procedure:
> http://www.freeintertv.com/view/id-2565

By looking at the network-traffic you can see (or at least I can) this POST 
request:

$ curl --data-raw 
"chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11"
 \
"http://www.freeintertv.com/myAjax/get_item_m3u8/;

This can be replicated/automated with the HTML/XML/JSON parser Xidel, which 
I've showed you earlier.

The value to be used for "chname":

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e 
'extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)'
msnbc_live

"chname" base64 encoded:

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
  string-to-base64Binary(
extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
  )
'
bXNuYmNfbGl2ZQ==

The string to POST:

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
  {
"chname":string-to-base64Binary(
  extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
),
"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
"html5":"11"
  }
'
{
  "chname": "bXNuYmNfbGl2ZQ==",
  "ch": "http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
  "html5": "11"
}

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
  request-combine(
(),
{
  "chname":string-to-base64Binary(
extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
  ),
  "ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
  "html5":"11"
}
  )/substring(url,2)
'
chname=bXNuYmNfbGl2ZQ%3D%3D=http%3A%2F%2Fwww.freeintertv.com%2Fexternals%2Ftv-russia%2Fsmotret-tv3-online=11

And the actual POST-request:

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
  x:request({
"post":request-combine(
  (),
  {
"chname":string-to-base64Binary(
  extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
),

"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
"html5":"11"
  }
)/substring(url,2),
"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
  })/raw
'
playlist[0]['file']='http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8';
 get_item.showPlayer();

Extract the m3u8-url:

$ xidel -s "http://www.freeintertv.com/view/id-2565; -e '
  x:request({
"post":request-combine(
  (),
  {
"chname":string-to-base64Binary(
  extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)
),

"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online;,
"html5":"11"
  }
)/substring(url,2),
"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/;
  })/extract(raw,"http.+m3u8")
'
http://1128480543.rsc.cdn77.org/SlTNwu1CSVEq9yA3i_J9Zg==,1649504141/1128480543/index.m3u8

> http://www.freeintertv.com/view/id-2308

This one doesn't work in my browser, but appears to work the same way:

$ xidel -s "http://www.freeintertv.com/view/id-2308; -e 
'x:request({"post":request-combine((),{"chname":string-to-base64Binary(extract(//meta[@property="og:image"]/@content,"([a-z_]+)\.jpg",1)),"ch":"http://www.freeintertv.com/externals/tv-russia/smotret-tv3-online","html5":"11"})/substring(url,2),"url":"http://www.freeintertv.com/myAjax/get_item_m3u8/"})/extract(raw,"http.+m3u8")'
http://livedoc.cgtn.com/500d/prog_index.m3u8

-- 
Reino

___
ffmpeg-user mailing list
ffmpeg-user@ffmpeg.org
https://ffmpeg.org/mailman/listinfo/ffmpeg-user

To unsubscribe, visit link above, or email
ffmpeg-user-requ...@ffmpeg.org with subject "unsubscribe".