Hi Nikhil, Thanks very much for your help. Gave me some clarity as to where I should be looking - . However, when I try to use the curl command, I get a couple of errors: first if I try the command you ran above which includes '--compressed' - I get an error stating that the installed libcurl version does not support this - if i try without the --compressed option, then I get an error "Protocol "'https" not supported or disabled in libcurl" - I saw online that I need to install some other software - however, don't really want to go down that path as my primary laptop is the company laptop.
Also - assuming I get this working - the next page which has the actual data - its a table with no discernible fields - How can I extract the same to a usable format? If I can get the data in Excel I can still create code to work with it - however, I am not proficient in other programming languages. Would really appreciate some help here. Regards, Abhishek On Thu, May 27, 2021 at 7:23 AM Nikhil VJ <nikhil...@gmail.com> wrote: > Hi Abhishek, > > Right-click on page -> Inspect -> opens the browser console -> Go to > Network tab , then browse around the site and check out the comings and > goings between site and server. > > I was able to get the data we're seeing in the website coming in my > command prompt (which means: one can capture this) with this simplified > cURL command: > curl 'https://nwp.imd.gov.in/blf/blf_temp/block.php' --compressed > --data-raw 'dis=22AMRAVATI' > > How to get these district codes: inspecting one page before.. > curl --compressed ' > https://nwp.imd.gov.in/blf/blf_temp/dis.php?value=22maharashtra' > > How to get the state codes: just scrape from the html of the page you > shared: https://nwp.imd.gov.in/blf/blf_temp/ > > Before my foray into python, I used to use notepad++ and libreoffice Calc > (their raw text import dialog leaves excel in the dust) to separate out the > data I needed from the html tags etc. I'd use spreadsheet formulas to even > generate command-prompt commands in bulk. Fun times. For limited jobs, > there's things you can do there in a few mins which would take hours of > coding. > > Maybe you can set these up in Octoparse (i have no experience with that as > I just roll my own code) or some other tools to get the data you need. All > the best! > > -- > Cheers, > Nikhil VJ > https://nikhilvj.co.in > > > On Thu, May 27, 2021 at 3:32 AM a.ja...@gmail.com <a.jai...@gmail.com> > wrote: > >> Hi, >> >> https://nwp.imd.gov.in/blf/blf_temp/ >> >> In the link above, we can see the 5 days weather prediction by districts >> - however, there are a lot of clicks that are required to reach any one >> particular district. Please help in identifying a method to collate all >> India Rainfall data in a single table. >> >> Tried to use Octoparse - however, either I am using it incorrectly or >> else it doesn't work in this site. any other source of the same data is >> also welcome. >> >> regards, >> Abhishek >> >> -- >> Datameet is a community of Data Science enthusiasts in India. Know more >> about us by visiting http://datameet.org >> --- >> You received this message because you are subscribed to the Google Groups >> "datameet" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to datameet+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/datameet/00fa6267-9dcd-489a-b579-58be5d62b383n%40googlegroups.com >> <https://groups.google.com/d/msgid/datameet/00fa6267-9dcd-489a-b579-58be5d62b383n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to a topic in the > Google Groups "datameet" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/datameet/gMW9xvKrSNg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > datameet+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/datameet/CAH7jeuM80boH2pdNrOkef3wQcTzFD9Yg0LVJaQL-SNr8pU_GbA%40mail.gmail.com > <https://groups.google.com/d/msgid/datameet/CAH7jeuM80boH2pdNrOkef3wQcTzFD9Yg0LVJaQL-SNr8pU_GbA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Regards, *Abhishek Jain* -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to datameet+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/datameet/CAOyXS__T5F2siTB6i9NEUpUy3Nk0xF2-euK0NPe4Dsd6hroPsg%40mail.gmail.com.
