Re: Scraping multiple web pages help

2019-02-27 Thread Phu Sam
You need to obtain a key for API first - from https://regulationsgov.github.io/developers/ The Regulations.gov API is taking action to conserve system resources. Beginning immediately, we will limit access to one account per organization, and require approval for enabling accounts.* Please contact

Re: Scraping multiple web pages help

2019-02-18 Thread Sivan Grünberg
Hi there Drake, A quick google search revealed: - https://regulationsgov.github.io/developers/ This seems particulriy useful: - https://regulationsgov.github.io/developers/console/ And to fetch stuff from the API, there's Python requests that has a rather wonderful doc: - http://docs.pytho

Scraping multiple web pages help

2019-02-18 Thread Drake Gossi
Hello everyone, For a research project, I need to scrape a lot of comments from regulations.gov https://www.regulations.gov/docketBrowser?rpp=25&so=DESC&sb=commentDueDate&po=0&dct=PS&D=ED-2018-OCR-0064 But partly what's throwing me is the url addresses of the comments. They aren't consistent. I