Re: url no longer working as expected
Likely the case. The expense of collating all that data and presenting it to their site visitors is considerable. They use advertising to cover those costs. If the data were easily scrapable, scrapers diminish revenue, putting the resource itself at risk. Some data provides offer APIs. When you see anti-scraping effects, look for API options (I saw none there but I didn't look deeply). APIs take fewer resources to deliver, and may have strategic benefit for some data brokers. But if they have scrape-prevention and no API, they're sending a clear signal: "We need to pay our bills, please send your traffic to our page so we can do that." That said, I've come across stock APIs before, and while I don't recall many free ones there likely are some. Richard Gaskin FourthWorld.com Paul Dupuis wrote: > I get a response from Yahoos that is an html page with a 404 > information as part of it. ... > I think this is Yahoo Finance not being able to detect the > browser type and intentionally returning a 404 as a method > of deterring screen scraping. > > On 6/28/2024 1:04 PM, Hugh Senior via use-livecode wrote: ... >> Problem: >> Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; into >> any web browser and the page is displayed as expected. >> >> Use LC's URL command to access the same page direct returns a >> 404 >> put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; >> >> Anyone got any insights? ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: url no longer working as expected
Yes. put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; In the message box on 9.6.11 and 9.6.12 under Windows 11. Both return a pile of HTML text that is all the formatting and CSS linked stuff to show a "404" page. This suggests that put URL is working and it is the Yahoo server that returning a different page of HTML/CSS for the put vs when you enter the URL in a browser (Firefox in my case, where I get the Yahoo finance data for Shell, although I did have to respond to a Cookies dialog first). On 6/28/2024 2:03 PM, Bob Sneidar via use-livecode wrote: Did you try that in the message box? Bob S On Jun 28, 2024, at 10:50 AM, Paul Dupuis via use-livecode wrote: I get a response from Yahoos that is an html page with a 404 information as part of it. This happens under LC 9.6.12 and 9.6.11 I think this is Yahoo Finance not being able to detect the browser type and intentionally returning a 404 as a method of deterring screen scraping. On 6/28/2024 1:04 PM, Hugh Senior via use-livecode wrote: Platform: Windows 11, LC 9.6.12 Query: Using URL to access a web page Problem: Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; into any web browser and the page is displayed as expected. Use LC's URL command to access the same page direct returns a 404 put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; Anyone got any insights? Hugh Senior ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: url no longer working as expected
Did you try that in the message box? Bob S > On Jun 28, 2024, at 10:50 AM, Paul Dupuis via use-livecode > wrote: > > I get a response from Yahoos that is an html page with a 404 information as > part of it. > > This happens under LC 9.6.12 and 9.6.11 > > I think this is Yahoo Finance not being able to detect the browser type and > intentionally returning a 404 as a method of deterring screen scraping. > > > On 6/28/2024 1:04 PM, Hugh Senior via use-livecode wrote: >> Platform: Windows 11, LC 9.6.12 >> Query: Using URL to access a web page >> >> Problem: >> Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; into any web >> browser and the page is displayed as expected. >> >> Use LC's URL command to access the same page direct returns a 404 >> put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; >> >> Anyone got any insights? >> >> Hugh Senior >> >> >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: url no longer working as expected
I get a response from Yahoos that is an html page with a 404 information as part of it. This happens under LC 9.6.12 and 9.6.11 I think this is Yahoo Finance not being able to detect the browser type and intentionally returning a 404 as a method of deterring screen scraping. On 6/28/2024 1:04 PM, Hugh Senior via use-livecode wrote: Platform: Windows 11, LC 9.6.12 Query: Using URL to access a web page Problem: Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; into any web browser and the page is displayed as expected. Use LC's URL command to access the same page direct returns a 404 put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; Anyone got any insights? Hugh Senior ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: url no longer working as expected
I get the HTML of the page. Are you trying to open the page in a browser? Bob S > On Jun 28, 2024, at 10:04 AM, Hugh Senior via use-livecode > wrote: > > > Platform: Windows 11, LC 9.6.12 > Query: Using URL to access a web page > > Problem: > Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; into any web > browser and the page is displayed as expected. > > Use LC's URL command to access the same page direct returns a 404 > put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; > > Anyone got any insights? > > Hugh Senior > > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
url no longer working as expected
Platform: Windows 11, LC 9.6.12 Query: Using URL to access a web page Problem: Enter "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; into any web browser and the page is displayed as expected. Use LC's URL command to access the same page direct returns a 404 put url "https://uk.finance.yahoo.com/quote/SHEL.L/history/"; Anyone got any insights? Hugh Senior ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
