Re: [External Email] [datameet] waterbody census data

[Nikhil VJ]

Hi,

As this repo's files are on Github LFS, some more commands may be needed
after cloning the repo to get the files.
You can see instructions for the same here:
https://stackoverflow.com/a/72830455/4355695

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Aug 2, 2024 at 11:45 AM Gunngunn  wrote:

> thank you
>
> On Thu, 1 Aug 2024 at 18:06, 'Shiv Hastawala' via datameet <
> datameet@googlegroups.com> wrote:
>
>> Woah, this seems to be a village-level data set! That's amazing, thank
>> you!
>>
>> Thanks and regards.
>>
>>
>> Yours sincerely
>>
>> *Shiv Hastawala*
>>
>> (He/His/Him)
>> Doctoral Candidate
>> Department of Economics
>> Binghamton University (State University of New York)
>>
>> Email ID: shastaw1[at]binghamton[dot]edu
>>
>> Zoom ID: 201 717 2613
>>
>> www.shivhastawala.com
>>
>>
>> On Thu, Aug 1, 2024 at 8:14 AM Gunngunn  wrote:
>>
>>> https://github.com/data014/Jal-Dharohar
>>>
>>> Please find the water body census data above processed by Chandan Kumar
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CAKm0_9maXdQP0OfAX_qndCgAbK2HpK0T82d3XL7tY1MA3E2MSw%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CAKm0_9maXdQP0OfAX_qndCgAbK2HpK0T82d3XL7tY1MA3E2MSw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAGiG-Az4S9ERkPJdqaEzGhO1ZXerrt6pgYb2CR0zp4YBvhBs0g%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAGiG-Az4S9ERkPJdqaEzGhO1ZXerrt6pgYb2CR0zp4YBvhBs0g%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAKm0_9nn%3D0eXU%2BV6odFkyDvZmW4pgf%2BOBYgcxbfc-Cc5J5oTHw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAKm0_9nn%3D0eXU%2BV6odFkyDvZmW4pgf%2BOBYgcxbfc-Cc5J5oTHw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOcG9PgSx4_MbzSckkh9Ek686iOkL8WTWpUBb5BRndfWw%40mail.gmail.com.

[datameet] PMGSY releases 7 lakh geo-tagged facilities under government open data license

[Nikhil VJ]

Hi,

Came across this:
https://omms.nic.in/Home/PMGSYRuralDataset/
PMGSY releases 7 lakh geo-tagged facilities under government open data
license

The page has FAQs about it including mentions of non-coverage of some
regions and possible errors due to mobile GPS.

I'm not sure about the exact date of this data release. If someone knows,
pls share.

Homepage of the PMGSY website from which we can navigate to this data:
https://omms.nic.in/Home/CitizenPage/

>From top menu, go to "Other Reports" -> "Facility Details". We have to
select State, District, Block.
Note that "Block" here is an internal division method of PMGSY and need not
co-incide perfectly with sub-districts.

The same site also has a lot of other data about habitation connectivity
status with population figures etc.

*Request:* If someone can gather this data into one place and share like it
was done for their roads and habitation data earlier (
https://github.com/datameet/pmgsy-geosadak),
I can then import it and add a layer of it on this site I'd made using
that, for viewing habitations / roads data on a map:
https://server.nikhilvj.co.in/pmgsy/


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMZghPUTU1J%3D8f0N18UdUQcu2q5NSZPmw63X7LwS%2B8csQ%40mail.gmail.com.

Re: [datameet] 2024 Lok Sabha election results

[Nikhil VJ]

Hi all,

Nice work Vivek!

I was scraping to catch by *timestamp* how lead margins, vote counts
change over time, from statewise results pages like this
<https://results.eci.gov.in/PcResultGenJune2024/statewiseS011.htm>  and PC-wise
results pages like this
<https://results.eci.gov.in/PcResultGenJune2024/ConstituencywiseS0124.htm> .

I've collated the data and posted it along with scraping and
collating (python) scripts on this github repo:
https://github.com/answerquest/india-elections2024-results-timewise

*Flaws in this data:*
1. Didn't catch it all from the beginning : leads-margins tally scraping
was started from around *1:50 pm,* per-candidate vote numbers scraping was
started from around *4.30pm.*
2. There would be some missed time intervals for some constituencies
sometimes some pages didn't load, script errored out due to edge cases
3. I bungled up on applying "U" prefixes for union territories so those
rows were scraped quite late.

But all in all I think it's a pretty good dataset to make time-series
viz's,
to "audit" tallies over time and detect out-of-norm additions, etc for
folks who are interested in settling some ongoing debates using data.

*Disclaimer :* I'm only sharing the data as-was at those timestamps, this
is secondary scraped data that is prone to flukes like a html tag
mis-rendering causing a bad number to come in. If you find something odd,
kindly lookup the official sources, file RTIs etc, but leave me out of it
pls.




*My compliments to Election Commission of India, in case anyone from there
is reading:*
1. It was good to have whole integers of absolute vote counts given by ECI.
Hope to see this maintained. This was a lot better than the rounded-off
fractions of vote-shares we were getting during the US 2020 elections
counting which had made it impossible to calculate the actual numbers of
votes.

2. Good website work, consistent naming of each constituency / state's
pages and consistent page structures.

3. Page-not-opening cases were there but were rare, and the chinks
disappeared from around evening onwards when the declarations were
happening and I'd expect more site visitors. On my part, I ensured my
scripts were hitting 1-at-a-time only, kept adequate time intervals etc so
that I don't bombard the server (to coders : this was intentional. Don't
suggest "fixing" it by parallel threading etc, that gets you 429'd).

4. Candidates' photos were properly organized and were instantly rendered
on all the PC-wise pages I was checking out. Which means each and every
candidate was properly tracked in the DB and their files were properly
linked and small thumbnails were kept, as opposed to past elections when
there would only be scanned pages listing all the candidates's totals. One
suggestion: converting these to .webp format will shrink the sizes and your
egress loads by around 10x.

5. Even prior to election, voters lists were quite well managed, even the
voter roll pdfs were easy to download, and it was quite easy to find our
part + serial number provided we'd done our homework (which FYI was the
only info we needed in hand apart from photo id on voting day, if you just
shared these with the officer when you entered the booth, they'd locate
your entry in 5 seconds and you would be done voting in under a minute.)

6. All in all, we've come a long way in digitization and making this data
accessible to all, Thank you for all the work done.

7. It would be great if you published some inside stories of the technical
infrastructure (server specs etc) used on 4th June for serving the website.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Jun 5, 2024 at 9:25 AM Vivek Matthew  wrote:

> Hi all,
>
> I have scraped the 2024 Lok Sabha election results from the
> results.eci.gov.in website. In case anyone is interested, you can find
> the CSV with the results attached.
>
> Once constituency-wise turnout numbers are released for phase 7, I will
> include additional columns for turnout and vote share numbers.
>
> Note that semicolon (;) is used as the column separator.
>
> Regards,
> Vivek
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/0c95b2a3-27d3-4146-8ce3-44a49ae72f6fn%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/0c95b2a3-27d3-4146-8ce3-44a49ae72f6fn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Dat

Re: [datameet] Elections 2024 mapping

[Nikhil VJ]

Thanks Arun!

Made a map!

https://server.nikhilvj.co.in/elections24/election2024-map.html

Used the same color scheme as on ECI's own map :
https://elections24.eci.gov.in/img/map.png

Screenshot:


Source code: https://github.com/answerquest/india-elections2024-map

Made it using folium lib in python this time.

Would anybody know what these attributes mean:

DOA  example: 16-3-2024(Saturday)
DOIN  example: 20-3-2024(Wednesday)
LDFN  example: 27-3-2024(Wednesday)
DSN  example: 28-3-2024(Thursday)
LDWC  example: 30-3-2024(Saturday)
DOCV  example: 4-6-2024(Tuesday)

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Apr 26, 2024 at 5:56 PM Arun Ganesh  wrote:

> Timely initiative, Nikhil.
> Managed to find a vector version of the ECI shapes. Have uploaded as a
> simplified shape and points, and includes the phases and polling dates
> https://gist.github.com/planemad/1e2b63f6b9806970db749f19980ffd25
>
> On Fri, Apr 26, 2024 at 4:42 PM Nikhil VJ  wrote:
>
>> Hi,
>> Some sharings, queries and call for information on elections 2024 (a bit
>> late!)
>>
>>
>> *1. Have the Parliamentary Constituencies seen any changes in their
>> delimitations after 2019 elections?*
>>
>> Related links:
>> Shapes:
>> https://github.com/datameet/maps/tree/master/parliamentary-constituencies
>> You can download just the folder using :
>> https://download-directory.github.io/
>>
>> For comparing with official ECI release:
>> ECI press notes page : https://elections24.eci.gov.in/eci-updates.html
>> There, at the bottom "General Election Schedule" opens a PDF which starts
>> showing maps from pg59 onwards. direct link which might not directly open
>> <https://elections24.eci.gov.in/docs/press-note-no-23.pdf>
>>
>> I've made high-res images of these pages and saved them here:
>> https://server.nikhilvj.co.in/elections24/
>> Full India map from the ECI's PDF :
>> https://server.nikhilvj.co.in/elections24/p-059-india-all-PCs-map.png
>>
>> J&K : I know there's changes. If anyone can share the new shapes of J&K
>> PCs, would be really helpful.
>>
>>
>>
>> *2. Is there an interactive map of the Parliamentary Constiuencies, with
>> election dates and other info?*
>>
>> If *not*, then one crowd-sourcing appeal:
>> a. Open this google spreadsheet and request editing access (I'll approve
>> if you don't sound like a spammer), or just do commenting:
>>
>> https://docs.google.com/spreadsheets/d/1Zj39sgQNqxN_iFRKDZ-itkJIZiELExpjwU_xB_6_C84/edit?usp=sharing
>>
>> b. Using the PDF mentioned above, find in the voting (also called
>> "polling") date for the constituency, and paste it into this sheet. You can
>> start from pg.63 onwards..
>> Some states are happening all at-a-go so those should be easier to tag.
>>
>> If yes, then pls ignore a & b and just share link to the interactive map,
>> thanks much!
>>
>>
>> *How filling the google spreadsheet can help:*
>> I have shapefile with these columns. I can merge back in the election
>> dates and publish the map.
>>
>> Here's a raw version without the dates:
>> https://server.nikhilvj.co.in/elections24/2019-PCs-simp1.html
>>
>> Disclaimer : it's directly a render of the datameet 2019 PCs data, with a
>> 100meter simplification done to reduce size. there are gaps in the shape
>> data and some areas left out. That doesn't mean those folks can't vote, pls
>> don't take it that way.
>>
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAH7jeuORDjvD7T%3DCnqb3M_8wJOTrPjqKR5BP7Lc%2B78qQ2Lh1hQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAH7jeuORDjvD7T%3DCnqb3M_8wJOTrPjqKR5BP7Lc%2B78qQ2Lh1hQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group

[datameet] Elections 2024 mapping

[Nikhil VJ]

Hi,
Some sharings, queries and call for information on elections 2024 (a bit
late!)


*1. Have the Parliamentary Constituencies seen any changes in their
delimitations after 2019 elections?*

Related links:
Shapes:
https://github.com/datameet/maps/tree/master/parliamentary-constituencies
You can download just the folder using :
https://download-directory.github.io/

For comparing with official ECI release:
ECI press notes page : https://elections24.eci.gov.in/eci-updates.html
There, at the bottom "General Election Schedule" opens a PDF which starts
showing maps from pg59 onwards. direct link which might not directly open
<https://elections24.eci.gov.in/docs/press-note-no-23.pdf>

I've made high-res images of these pages and saved them here:
https://server.nikhilvj.co.in/elections24/
Full India map from the ECI's PDF :
https://server.nikhilvj.co.in/elections24/p-059-india-all-PCs-map.png

J&K : I know there's changes. If anyone can share the new shapes of J&K
PCs, would be really helpful.



*2. Is there an interactive map of the Parliamentary Constiuencies, with
election dates and other info?*

If *not*, then one crowd-sourcing appeal:
a. Open this google spreadsheet and request editing access (I'll approve if
you don't sound like a spammer), or just do commenting:
https://docs.google.com/spreadsheets/d/1Zj39sgQNqxN_iFRKDZ-itkJIZiELExpjwU_xB_6_C84/edit?usp=sharing

b. Using the PDF mentioned above, find in the voting (also called
"polling") date for the constituency, and paste it into this sheet. You can
start from pg.63 onwards..
Some states are happening all at-a-go so those should be easier to tag.

If yes, then pls ignore a & b and just share link to the interactive map,
thanks much!


*How filling the google spreadsheet can help:*
I have shapefile with these columns. I can merge back in the election dates
and publish the map.

Here's a raw version without the dates:
https://server.nikhilvj.co.in/elections24/2019-PCs-simp1.html

Disclaimer : it's directly a render of the datameet 2019 PCs data, with a
100meter simplification done to reduce size. there are gaps in the shape
data and some areas left out. That doesn't mean those folks can't vote, pls
don't take it that way.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuORDjvD7T%3DCnqb3M_8wJOTrPjqKR5BP7Lc%2B78qQ2Lh1hQ%40mail.gmail.com.

Re: [datameet] Bangalore road widths

[Nikhil VJ]

Hi, if there are folks out there who're thinking about mapping this
on-ground, then sharing some unsolicited and untested ideas:

A real-world requirement for many logistics / transport folks is : which of
the roads are too narrow to send my truck / 4-wheeler / tempo / e-rickshaw
through?
And frankly the people living there may also appreciate if map routers
stopped sending wide bodied vehicles through their lanes.

So if you aim at that, then you can drop all the wide roads from your task
list and focus on just the narrow ones. That dataset can be built by
selective querying of OSM data. See:
https://wiki.openstreetmap.org/wiki/Key:highway

Then, in many cases it might not be the whole stretch of road, but just the
narrowest bottleneck on it that matters, as that point will decide what can
(or rather shouldn't) pass through.
So, mapping of just the narrowest points instead of lengths also cuts down
your task load and reduces the scale, complexity and skill requirements for
your operations considerably. One can take a simple photo of the point with
geo-tagging on instead of having to capture data at every X meters etc.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Apr 19, 2024 at 4:49 PM Vaidya  wrote:

> I found this map from KSRSAC.
> https://data.opencity.in/dataset/bengaluru-road-width-map
>
> Many are proposed road widths, especially in outer areas. But the ones in
> the centre of the city should be closer to reality. This data is from the
> scrapped RMP 2031.
>
> Thanks,
> Vaidya
>
>
> On Tue, Apr 16, 2024, 5:28 AM kwatra...@gmail.com 
> wrote:
>
>> Hi all,
>> Would you know of any data source which would have road width for
>> Bangalore roads?
>> OSM doesn't seem to have width tagged.
>>
>> Thanks,
>> Kartik
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/449b12d6-b315-41eb-82ff-169c65b0efaan%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/449b12d6-b315-41eb-82ff-169c65b0efaan%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAHVy6PHNGOfTZ65fY2bhRDq5La5x7ahAdUX%2B80nECf4OMdSUcg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAHVy6PHNGOfTZ65fY2bhRDq5La5x7ahAdUX%2B80nECf4OMdSUcg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMckmnSEG-tKeO4WPn-bn6UjbpNETcUwCnHa67_6CrjSg%40mail.gmail.com.

Re: [External Email] [datameet] Traffic Data for India

[Nikhil VJ]

Hi,

Rigved, thanks for the suggestion. Sudarshan,
I've saved delhi gtfs-rt (bus gps locations) data over 2019-2020 (mostly
2019) here:
https://server.nikhilvj.co.in/delhirt/

you'll see some zipped csvs, one covering each week. (.7z is a zip format
that gives good compression for text data, lookup 7zip)

Please feel free to use it as you may. Data is as-was in the realtime feed
released by https://otd.delhi.gov.in/ .

Also, just to clarify Shiv's input : No, Openstreetmap doesn't have GPS
navigation system and doesn't keep such archival data. That might be other
apps using OSM. It has gps traces uploaded by users, for use as an aid for
mapping. But most mapping doesn't rely too much on those now as there's
better quality satellite imagery around now. They're likely not to repeat
much along the same routes as the intention is to just one-time map a path.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Feb 13, 2024 at 12:26 PM rigved1...@gmail.com <
rigved12she...@gmail.com> wrote:

> One approach would be to look if someone is archiving historical GTFS
> data. (I saw something Nikhil VJ was doing for delhi a few years ago.) It
> has bus was here(lat-long) at this point of time data. With a lot of
> cleaning, I was once able to derive traffic (slow-fast) conditions for many
> roads during working hours.
> On Tuesday, February 13, 2024 at 6:32:00 AM UTC+5:30 Shiv Hastawala wrote:
>
>> Hi Sudarshan
>>
>> Disclaimer: I haven't tried it and I'm only shooting in the dark.
>>
>> Open street map (OSM) usually has archives of a lot of types of their
>> data across the web. It also has a GPS navigation system similar to Google
>> maps, so I'm assuming they would have archival data for traffic too. Maybe
>> that's a place you could look.
>>
>> If there are any other people who have concrete info on this, please
>> pitch in.
>>
>>
>> Thanks and regards.
>>
>>
>> Yours sincerely
>>
>> *Shiv Hastawala*
>>
>> (He/His/Him)
>> Doctoral Student
>> Department of Economics
>> Binghamton University (State University of New York)
>>
>> Zoom ID: 201 717 2613 <(201)%20717-2613>
>>
>> www.shivhastawala.com
>>
>>
>> On Mon, 12 Feb 2024 at 7:14 pm, Sudarshan RSA 
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I'm trying to source traffic data for India to analyze as part of an
>>> economics paper relating to air pollution that I'm trying to write. What I
>>> want is, estimates of traffic density/an adjacent measure at as granular a
>>> level as possible and preferably at the daily level. One ideal
>>> configuration would be to have gridded daily data like we do with weather
>>> variables, but I suspect that will be difficult.
>>>
>>> What is the best level at which I can find the sort of data I'm looking
>>> for? Google maps doesn't seem to have historical data for traffic.
>>>
>>> I seek your advice.
>>>
>>> Sincerely,
>>> Sudarshan
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/eaca2492-9cf3-4e0b-bcbb-430e172279bbn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/datameet/eaca2492-9cf3-4e0b-bcbb-430e172279bbn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/008cc75d-ad36-4fbc-89ab-812137b593d6n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/008cc75d-ad36-4fbc-89ab-812137b593d6n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOZgyb5hWm7KoXnbPrkgDBY9EZVYyXS3ihgDCN7W-%2BhOw%40mail.gmail.com.

Re: [datameet] Parliamentary Constituency data of India

[Nikhil VJ]

Hi Anmol,

Datameet's maps repo has PC boundaries as of 2019, and I don't know about
changes since then (if anyone knows, pls tell):
https://github.com/datameet/maps/tree/master/parliamentary-constituencies

How to obtain from latitude-longitude: Sharing a way to do in QGIS, there's
other ways in python etc.

In any case, first step would be to save that excel to CSV format.

QGIS: Layer > Add Layer > Delimited text layer > load the csv
Drag-drop the parliamentary constituencies shapefile onto the map, will be
added come as another layer of polygons.
Processing Toolbox > Join Attributes by Location
Choose the point layer at top, the polygon layer at bottom, and check "are
within" box in middle.

Run it, and after that the points layer will be having the columns from PC
layer, value being the constituency they're in.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Feb 5, 2024 at 11:36 PM Anmol Gupta  wrote:

> Hi everyone,
>
> Wanted to ask if anyone knew how to obtain the parliamentary constituency
> data of a particular location from its latitude and longitude in Excel. Any
> tutorial would help.
>
> Also, would anyone have shapefiles of updated parliamentary constituencies
> for the 2024 election? The most recent shapefiles I could find are from
> 2014 and don't account for delimitation of regions like J&K.
>
> Thank you in advance!
>
> Regards,
> Anmol
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/b0d6e3bc-5df2-4c62-b0eb-6fcc5e1b1cb3n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/b0d6e3bc-5df2-4c62-b0eb-6fcc5e1b1cb3n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNPDhHz%2Bzbg1_uhyA9%3DpsPXnL%2Bm3jfzzF4Hu42h00BtrQ%40mail.gmail.com.

Re: [datameet] Creating variable names in Devanagari

[Nikhil VJ]

Hi Sharad,

If you have to import the data into a web-based backend, then your
programmers will be most comfortable with: GeoJSON format.
Because : It's a simple text/json format which can even be opened as raw
text, and so there are many different ways to load it in a program and use
data from it.
And this will also support Devnagri columns (which become property keys
here) same as geopackage.

Another thing to look out for: Ensure that the database being used by the
backend has the same UTF-8 encoding set.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Dec 4, 2023 at 3:56 PM Sharad Lele  wrote:

> Thanks, Ujaval. Yes saving as .geopkg works (saves the field names in full
> and displays properly upon reopening). This is the only solution that has
> worked so far.
>
> But I am not sure this is going to work for us in this particular
> situation where we are then using the file in postGIS to display in a
> webGIS. Let me ask our webGIS person.
>
> Sharad
>
> On Monday, December 4, 2023 at 3:10:18 PM UTC+5:30 Ujaval Gandhi wrote:
>
>> Use the geopackage format. It's a much better modern format without the
>> limitations of shapefiles. The default encoding would be UTF-8 and it
>> should work. Tested with a new geopackage layer and it is able to create
>> column names and values in Devanagri script
>> https://i.imgur.com/2prELf6.png
>> <https://mailtrack.io/link/b7757e0a651e8afed7ee4756ff4cbf155cd0a533?url=https%3A%2F%2Fi.imgur.com%2F2prELf6.png&userId=8747767&signature=9f333ff36f736872>
>> ---
>> Ujaval Gandhi
>> Spatial Thoughts
>> www.spatialthoughts.com
>> <https://mailtrack.io/link/43de97ab1302774e567fc413805c61bb36bd8af8?url=http%3A%2F%2Fwww.spatialthoughts.com&userId=8747767&signature=904253fcbc9b4da4>
>>
>>
>>
>> On Mon, Dec 4, 2023 at 2:36 PM Sharad Lele  wrote:
>>
>>> The shapefile encoding is set to UTF-8.  But any edits to variable names
>>> only last as long as I don't save the file. Save and reopening results in
>>> going back to the same truncated/funny display as above [image:
>>> Screenshot Layer properties.jpeg]
>>>
>>> On Monday, December 4, 2023 at 2:17:41 PM UTC+5:30 Dilawar Singh wrote:
>>>
>>>> I found this
>>>> https://gis.stackexchange.com/questions/381308/how-to-change-qgis-3-default-encoding.
>>>>
>>>>
>>>> The file you are loading may have a different encoding that QGIS3. You
>>>> have to verify this manually and make sure that QGIS3 and file encoding is
>>>> the same.
>>>>
>>>> I am guessing the while QGIS3 is using utf-8, the file that you are
>>>> loading may be utf-16 encoded. Not sure if there are automated tools that
>>>> can convert UTF-16 files to UTF8 file while preserving unicode (
>>>> https://superuser.com/questions/1786434/convert-utf-16-le-to-utf-8-in-windows-via-command-line
>>>> ).
>>>>
>>>> best,
>>>>Dilawar
>>>>
>>>>
>>>>
>>>>  On Mon, 04 Dec 2023 14:05:01 +0530 *Sharad Lele
>>>> * wrote ---
>>>>
>>>> Just to clarify/muddy the situation: I am able to enter long Devanagari
>>>> text in the District/Jila column without any problems. It is the
>>>> column/field name itself that I run into problems, which may have to do
>>>> with the 8 characteri limit?
>>>>
>>>> [image: Screenshot QGIS devanagari field names.jpeg]
>>>>
>>>>
>>>> On Monday, December 4, 2023 at 1:57:04 PM UTC+5:30 Sharad Lele wrote:
>>>>
>>>> Thanks, Dilawar. The article was very interesting/useful. But for a
>>>> simpleton like me, the question then is: How do I ensure that QGIS3 in
>>>> particular is using UTF-8 encoding every time?
>>>>
>>>> Sharad
>>>>
>>>> On Monday, December 4, 2023 at 1:31:28 PM UTC+5:30 Dilawar Singh wrote:
>>>>
>>>> The answer is not going to be simple.
>>>>
>>>> Make sure that every time you save or open the file, the application
>>>> uses encoding utf-8. Microsoft uses another weird encoding called utf-16 (
>>>> https://en.wikipedia.org/wiki/UTF-16).  Stay away from utf-16 it if
>>>> you want to copy-paste anything from the web or web-based tools.
>>>>
>>>> Long answer:
>>>> https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/
&g

Re: [datameet] coal mines of India

[Nikhil VJ]

Hi, Raahil,

I'd some across these links last year of some students who were mapping
mining in OpenStreetmap. I didn't find out more about them, had just saved
the links for future exploration. Check it out:

https://tasks.hotosm.org/projects/11556
INDIA - MONITOR MINES - PAN INDIA PART 9

Linked to this:
https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/NF5GZY
Harnessing transparency initiatives to improve India's environmental
clearance process for the mineral mining sector

in the zipped dataset is a CSV with lat-longs of mines (maybe possible
locations, maybe confirmed - idk)

And a pdf report: "Detection of Surface Mining Activity" by some IITD profs:
https://drive.google.com/file/d/1UyJPbSF8LYi-16pKNR0eat1saG-fXcu5/view


Then, had found these links:
https://www.fineprint.global/visualisations/viewer/
https://doi.pangaea.de/10.1594/PANGAEA.910894

This stuff might not be directly related to coal mines, but might help you
find what you're looking for or connect with other folks who can help, so
sharing. All the best.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Sep 27, 2023 at 3:51 AM madhok...@gmail.com 
wrote:

> Hi,
>
> I am looking data on location (district, village, or GPS coordinates) of
> India's coal mines along with date of opening. Ideally, also date of
> closing for those that were shut down. If anyone knows where I might find
> this, I would be greatly appreciative.
>
> Thank you,
> Raahil
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/0d51e23e-5902-4eea-a672-6d56c1df7773n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/0d51e23e-5902-4eea-a672-6d56c1df7773n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNJheNYyz2g6HZYTFkvp%3DXHX95KNPXaDEEi73b5WJuePg%40mail.gmail.com.

Re: [datameet] Issue on using imdlib package with rasterstats to calculate zonal statistics

[Nikhil VJ]

Hi Dammalapati,

I had used imdlib python package a year back, but the data fetched using it
was vector data, not raster. It had latitude, longitude, date and value of
either rainfall in mm or temperature in degrees celsius. If you're fetching
a dataset what is different from rainfall or temperature data, then pls
specify.

Here's a few lines of code that fetches the yearly data of 1 year, converts
it to a simpler flat table in pandas dataframe and saves to CSV file:

imdlib.get_data('rain', 2000, 2000, 'yearwise')
rain1 = imdlib.open_data('rain', 2000, 2000,
'yearwise').get_xarray().to_dataframe()
rain1.to_csv('rain1.csv')

Have shared my script here:
https://github.com/answerquest/IMD-grid-data-work/blob/main/imd_grid_import/imd_import.py

Note: there were junk data placeholders for indicating null value (ie, no
value available for that location and year). Temperature: >99, Rainfall: <
-100 .

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Sep 7, 2023 at 4:28 PM Dammalapati Sai Krishna <
saikrishnadammalap...@gmail.com> wrote:

> Hi,
>
> This question is for people who have worked on Python's imdlib package.
>
> The rasters downloaded using imdlib has the following affine
> transformation:
> | 0.25, 0.00, 66.38|
> | 0.00, *0.25*, *6.38*|
> | 0.00, 0.00, 1.00|
>
> If I'm not wrong, the general standard is to define the pixel height in
> negative with top-left bounding box coordinates as X,Y. At least, the
> images downloaded from GEE follow that format.
>
> But in the array above you can see (in bold blue) that the pixel height is
> specified in the positive. And the bottom-left bounding box coordinates are
> specified (bold red).
>
> The issue I faced with this is that when I try to calculate zonal
> statistics using *rasterstats*, the positive height becomes a problem
> (negative dimensions error). I changed the transformation matrix and
> flipped the array to be able to run zonal stats.
>
> Want to know if there is a standard in defining affine transformation
> matrix and if imdlib is not using that.
>
> Thanks!
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/98c9162b-e45e-4bba-9095-be37683b7e41n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/98c9162b-e45e-4bba-9095-be37683b7e41n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNx31Bsaa1xkuUQ0eFC3tr-8CBJ1atwJU4%3DL6hi1A%2Bpdg%40mail.gmail.com.

Re: [datameet] Views on DIA, Scraping, API Restrictions

[Nikhil VJ]

ng it so
difficult to read that eventually humans won't be able to read it anymore.
Maybe we can put down our weapons, take a few steps back and communicate
that there are options available that work for both sides?


4. Load shifting
One load-shifting example I've seen in my netbanking : If I request for
some long-term account statements, instead of trying to give the data
immediately, it queues the task in backend and tells me to carry on and
come back in a few mins to download. Some other sites mail me the link when
they're done gathering the data requested. So, we could have something like
this : it distributes the load on server from peak-time spikes to the
"lazy" times later when there's not much high traffic. The institution can
stay on a cheaper infrastructure, doesn't incur higher costs, and it still
accomplishes the goal of providing data.



A mental note:
I came across a quote the other day : A government that is expected to do
everything for us, will take everything from us.
Too often I see this expectational attitude amongst folks like "everything
must be provided for, free and openly accessible!", without taking into
consideration what it all takes or the fact that they're not the only
scrapers on the planet. Or the fact that there will always be an unlimited
supply of idiots hogging up all the resources for no use other than to show
off their latest Go code's concurrency stats.
I agree that we're paying taxes, but those taxes are already accounted for
(very inefficiently, but yes), and demanding that they be used to fund all
these new web infra for giving us more free stuff leads to the same
convenient outcome : increase in our taxes. It's already happening, and
it's not sustainable. I especially don't appreciate having to pay more
taxes just for the sake of that idiot with the Go code :D.

And you can bet that if we demand that so and so government institution
make everything freely available without caring about the details, then
they will do it in the most expensive and wasteful way imaginable. The
solutions I've written above seem a lot better to me than increasing my
taxes plus creating yet another black hole in the govt budget.
I don't know exactly how we can make things work out, but if we ditched the
expectational attitude and think more as a team player with the government
being part of that team, we could go a long way.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Jul 21, 2023 at 12:18 PM Tarunima  wrote:

> Hi All,
>
> There have been a number of recent changes globally and nationally, in
> part triggered by Gen AI chat bots, that could restrict scraping. Listing
> the salient ones here:
>
>- India coming up with the Digital India Act which would bear on data
>access
>- EU releasing the Data Services Act which has provisions for
>researcher access to data (
>https://algorithmwatch.org/en/dsa-data-access-explained/)
>- Twitter and Reddit revoking API access:
>https://www.fastcompany.com/90904038/reddit-restricts-third-party-apps
>
> Does this group have any views/concerns about this?
>
> Apologies if this has already been addressed in another thread.
>
> Regards,
> Tarunima
>
>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CACj7W2mAOhV4-Qi0Y6g4BqWUFUXCWtxQpyd1xewRpxwafXEb0g%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CACj7W2mAOhV4-Qi0Y6g4BqWUFUXCWtxQpyd1xewRpxwafXEb0g%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMfUEDNwpqbvBJtt5YAzcpff9FW1stJGoDNmVJu%2BssmsA%40mail.gmail.com.

Re: [datameet] Error with GetISOTime,GetSecTime in Python

[Nikhil VJ]

Hi Shyamal,

You seem to have copied the function call without the function definition.
Every function you'll see in python code isn't a package / library of its
own. In fact, most times the package name will be something else and then
under it there will be multiple functions. Many times a custom function is
written within the program itself to take care of some task. You should
also just search the name with "python" before it on the web - if it
doesn't throw up some proper results, then it's likely not in some package
that you have to import.

When troubleshooting, it's always helpful to go back to the source where
you copied from and search for earlier places in the code where the same
name repeats.
OR when you're asking others for help, you *need* to provide full context,
including a link to the place you copied the code from, boil the code down
to absolute example values ( like: how is one supposed to know what is
edge_mrt[edge] here - a string, a datetime object, a number ?)

I'll also encourage you to sign up for a free account on *ChatGPT
(openai.com <http://openai.com>) *and first ask questions there - now
several of these questions get answered there itself with nice explanation
instead of having to post to large human groups and waiting for days for
reply. I've used it to learn a lot of things at basic level. For deeper
details, in few cases it's still prone to silly errors and we have to
cross-reference the original documentation. But it helps us get there.

And then for bare-technical stuff there is *stackoverflow*, definitely be
logged in there as well when you want to troubleshoot, and it's a better
resource for technical coding help than this group. We first get a cold
shower treatment for our incomplete questions due to the reasons above, and
after that we improve to become good at explaining the questions - so much
that in certain cases we end up solving the problem ourselves while we're
typing out the details around the question, because we now notice some
details we had missed earlier.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Jul 26, 2023 at 11:08 PM Railway Research <
researchrailways2...@gmail.com> wrote:

> Hello all,
> Have anyone of you ever faced below error with GetISOTime.
> An example of time format is below,so datetime is not working.
> Any leads on how to handle this error would be welcome.
> [image: image.png]
>
>
> [image: image.png]
> Thanks & Regards
> Shyamal Banerjee
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAAUQX0PSwWV0Q-Pb6uuFHD%3DXypEz-giJ7AKFymnr9cfoWOaeMw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAAUQX0PSwWV0Q-Pb6uuFHD%3DXypEz-giJ7AKFymnr9cfoWOaeMw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNpdQ8TLhDjaaLWQQsd8nqxDTd3ASiR3jY9pse0qhUw9g%40mail.gmail.com.

Re: [datameet] Agriculture Retail Price Query Report

[Nikhil VJ]

Hi Atanu,

Didn't see any link in your email?

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, May 2, 2023 at 10:24 AM Atanu Garai  wrote:

> Hi Community,
> We are trying to get the regular (Weekly/ Monthly) Retail Price Report for
> each district and update it. The Directorate is providing this UI to
> generate the report which is creating an ASPX file. Are you aware of any
> XML/ API/ JSON/ CSV file that is being updated for this data?
>
> Best regards,
> Atanu
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAPkreOn55%2BL%3DyfrUEi9R3WN6_VE1bBjX%2BMOQVkvmaAKrivd_HA%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAPkreOn55%2BL%3DyfrUEi9R3WN6_VE1bBjX%2BMOQVkvmaAKrivd_HA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMF7-sge%2BP6qbfb427nv_aB_zDFzhwzzgvkLAePj1XBTQ%40mail.gmail.com.

Re: [datameet] NEW DISTRICTS OF INDIA SHAPEFILE

[Nikhil VJ]

Hi Uday,

I just reproduced the steps and successfully loaded it to QGIS. no error in
the file.
Please go through all the explanations given in the thread. They've been
VERY basic.

If you're still not able to do it correctly, then you need to give some
exact details so that someone can help you. Like :
- what is size of saved file - if it's not around 39mb (some computers
calculate bytes differently (1024 vs 1000) so bit difference may be there
but explainable) - then you don't have the actual data file and you've
saved something else and are on wrong track
- what is extension of saved file - if it's not .geojson then you're on
wrong track.
- open the file in a text editor like VSCode and share the content up till
: " line 8 column 1" where you're getting the error. And if it's not
looking like:
{"type":"FeatureCollection", "features": [
{"type":"Feature","geometry":{"type":"Polygon","coordinates":[[[71.2702884145729,23

... then you're on wrong track.

Sometimes it's just a messed-up download due to network issues and simply
repeating the download again solves it.

Do try looking up online for what the .geojson format looks like, compare
etc to catch out any obvious mistakes.
If you're convinced that you're following everything properly and still
getting it wrong, then make a screen recording, upload to youtube and share
the link so you can show exact process and get feedback.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sun, Apr 30, 2023 at 10:14 PM uday...@gmail.com 
wrote:

> Hello all,
>
> QGIS throws error when i tried to open these files. Even geojson.io is
> throwing up an error  - "Invalid JSON file: SyntaxError: JSON.parse:
> unexpected character at line 8 column 1 of the JSON data". I am new to GIS
> and intersted to learn. Kindly look into this, as you have time. Any help
> is highly appreciated.
>
> Uday.
>
> On Friday, 11 November 2022 at 12:40:13 UTC+5:30 Nikhil VJ wrote:
>
>> Hi Rahul,
>>
>> Once you see the:
>>
>>> It is just opening a long list of lat and longs of feature polygon
>>>
>>
>> .. press Ctrl+S on your keyboard.
>> Save it to your computer. ensure that the file has extension as
>> ".geojson".
>>
>> That is the shapefile. There's various formats - this one is .geojson.
>> Devdatta mentioned another format which would be the .shp etc. All are
>> equally usable, all depends on your next step.
>>
>> Open QGIS or ArcGIS and drag-drop this in, you'll see it on map.
>> Or, go on https://geojson.io and drag-drop it there, you'll see it on
>> map.
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>> On Tue, Nov 8, 2022 at 11:25 AM Devdatta Tengshe 
>> wrote:
>>
>>> Rahul,
>>>
>>> This data is in GeoJSON format, and not shapefile.
>>>
>>> Once you download the data in GeoJSON format, you can then convert it to
>>> a Shapefile using a GIS tool like QGIS, or one of many online convertors.
>>>
>>> Regards,
>>> Devdatta
>>>
>>>
>>> On Tue, 8 Nov 2022 at 10:38, Rahul Dhiman 
>>> wrote:
>>>
>>>> Hello Akula. First of all thankyou for taking time to create this shape
>>>> file and making our lives easier. Please can anybody tell me how to
>>>> download the data, as when I am clicking download. It is just opening a
>>>> long list of lat and longs of feature polygon. How I can download the shape
>>>> file?
>>>>
>>>> Regards,
>>>> Rahul
>>>>
>>>> On Monday, 22 August 2022 at 08:26:49 UTC+5:30 cara.a...@berkeley.edu
>>>> wrote:
>>>>
>>>>> Thank you!!!
>>>>>
>>>>> On Mon, Jul 11, 2022, 11:31 AM akula.g...@gmail.com <
>>>>> akula.g...@gmail.com> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I have updated  INDIAN-DISTRICTS shapefile with all new districts
>>>>>> even including newly formed Andhra Pradesh districts counting  a total of
>>>>>> 755 districts.
>>>>>>
>>>>>>
>>>>>> https://github.com/datta07/INDIAN-SHAPEFILES/blob/master/INDIA/INDIA_DISTRICTS.geojson
>>>>>>
>>>>>> Regards,
>>>>>> Akula Guru Datta
>>>>>> akuladatta.github.io
>>>>>>
>>>>>> --
>>>>>>
>>>&g

Re: [datameet] Kerala hot weather

[Nikhil VJ]

Hi Chippi,

I've previously downloaded the gridded data from IMD website for all the
years,
extracted it out from that gridded format,
dropped all the placeholders they put in for null entries (like -999 and
stuff),
put it into a database,
and setup this website where one can pick a location, year and download
that data in a simple to use flat CSV format.

https://server.nikhilvj.co.in/imd_data/

Might be of help. Was done last yr, so it's uptill 2021. Should do a round
to update it sometime.

Source code for traceability is at
https://github.com/answerquest/IMD-grid-data-work

PS: Check out https://server.nikhilvj.co.in/imd_data/game.html if you'd
like to play a temperature game :)

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Mar 21, 2023 at 10:29 AM Ramaraj Palanisamy 
wrote:

> Hi,
> You can get time series data for Maximum Temperature, Minimum Temperature
> and Rainfall from the link below.
> https://www.imdpune.gov.in/lrfindex.php > Climate Monitoring > Gridded
> data archive
> Let me know if you have any difficulties.
>
> Best Regards,
>
> *Dr AP RAMARAJ*
>
>
>
>
> On Tue, 21 Mar 2023 at 05:19, Chippi Kuriakose 
> wrote:
>
>> Hi,
>>
>> Can I get temperature (highest/lowest) in kerala for last 10 years. Can
>> anyone guide me where to get it.
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CALGaVivba0gDVzHW%2BO2Wt0NpdEDD7D3bN2ONjqVwKVET5M9M3g%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CALGaVivba0gDVzHW%2BO2Wt0NpdEDD7D3bN2ONjqVwKVET5M9M3g%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CACQZBBPRbXfA%2BQ-yRURy7AO4x8hadcwExDN4Zh8hyMNouSkm5A%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CACQZBBPRbXfA%2BQ-yRURy7AO4x8hadcwExDN4Zh8hyMNouSkm5A%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNeE%3Dg3GnkEQJfLSJNQxeP669zmNgQnUX8TebZUj0UPzA%40mail.gmail.com.

[datameet] Data on socio economic effects of lockdowns / pandemic fears

[Nikhil VJ]

Hi, can anyone connect me to any kind of survey / stats data on the socio
economic effects of lockdowns / pandemic fears ?

Monthly Income loss, standard of living downgrades, losses in educational
achievement for students.

Has anybody worked on quantifying them?

Regards
Nikhil

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMQxwbF%2B3A5f4PSuwseqNq7JBmYnq9UwrJN%3DhtGEq7k-Q%40mail.gmail.com.

Re: [datameet] How I searched for the hottest villages in Kerala with Microsoft's Planetary Computer

[Nikhil VJ]

Hi Shijith,

Really good work putting so many things together, this can be a good
reference material for how to fetch and use data from these various
sources. Many different technologies / techniques brought together here.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Feb 24, 2023 at 1:30 PM Dilawar Singh 
wrote:

> Awesome stuff! Can it be extended to all Panchayats in the country easily?
>
> Just a nitpick: why not use "median" as an averaging strategy?
>
> best,
>Dilawar
>
>  On Fri, 24 Feb 2023 12:00:40 +0530 *Shijith Kunhitty
> >* wrote ---
>
> Just wanted to plug my data story on villages in Kerala and how hot they
> get.
>
> The story is at https://shijith.com/blog/kerala-panchayat-temp/
>
> And the code is at https://github.com/shijithpk/hottest-panchayats-kerala
>
> Was thinking of using Google Earth Engine for this story, but decided to
> go with Planetary Computer since it's Python-centric and I'm more familiar
> with that language.
>
> Pretty sure earth/environment science guys won't be happy with my
> methodology, but hopefully there's enough of a logic to it to accept the
> story as a basic analysis.
>
> Thanks! -shijith
>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/5e8d3b97-35c4-4de2-bca4-bef27dfdfb8en%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/5e8d3b97-35c4-4de2-bca4-bef27dfdfb8en%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/186826fd11e.84f5688a277373.8698225708209974880%40gmail.com
> <https://groups.google.com/d/msgid/datameet/186826fd11e.84f5688a277373.8698225708209974880%40gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOZjSXGOPMffbAcHHWLsuKbQg6Xs9-QMpdBziUPx4LxNg%40mail.gmail.com.

Re: [datameet] Re: Looking for list of census village names for CG and MP in Devanagari

[Nikhil VJ]

Hi Sharad,

The site you linked is quite easy to scrape with basic GET api calls (aka
you can open the url in browser also) giving the data in a proper structure
that can be directly used by a program.

*But : the data is all in English only.*

Anyways, in case you want to scrape, can get someone to do using:

Districts list:
https://www.prd.mp.gov.in/Handlers/Districts.ashx?DivisionID=0

Take district ID from there to get local bodies list:
https://www.prd.mp.gov.in/Handlers/localbodies.ashx?DistrictID=*45*
&Is_Rural=1

Take "LBID" from there to get GP Zones:
https://www.prd.mp.gov.in/Handlers/GpZones.ashx?LbId=*24319*


Note to freshers in python coding who are looking for real world use cases
to learn and apply their skills: This is a good starting project.
Make 3 nested for loops and append all the results to a dict (json) array.
At end, convert to a pandas dataframe, and output to CSV.
https://www.prd.mp.gov.in/Handlers/Districts.ashx?DivisionID=0
--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Feb 3, 2023 at 12:10 AM Sharad Lele  wrote:

> For instance, if someone can scrape the names from this website:
> https://www.prd.mp.gov.in/GramSearch/SearchPanchayat.aspx  (sequentially,
> so as to get the district, block and GP tags also)
>
> On Thursday, February 2, 2023 at 9:47:01 PM UTC+5:30 Sharad Lele wrote:
>
>> I am looking for the census village list for Chhattisgarh and Madhya
>> Pradesh (for starters) in Devanagari (Hindi script). Preferably with Census
>> 2011 codes, so that I can quickly match them to the Census dataset, which
>> is in English. But even if no codes attached, an accurate list with
>> tehsil/block and district tags in digital format (not pdf hopefully) will
>> be a big help.
>>
>> Any suggestions, folks?
>>
>> Sharad
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/4c644c88-1d3b-4e71-81a2-2e48d6c794c3n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/4c644c88-1d3b-4e71-81a2-2e48d6c794c3n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMKboJCh71j%2Bncfbg%3DSbjpxjUVW4gWHk59VZ-2SKnN_7w%40mail.gmail.com.

Re: [datameet] Re: Rivers of India shape files

[Nikhil VJ]

Hi Vaibhav,

That hydrosheds.org site has some pretty nice data! Thanks for sharing!

Data from https://www.hydrosheds.org/ downloaded, filtered for India
(roughly, lot of neighboring regions included in the rectangle) and
published as vector tiles that can be imported in QGIS, OSM iD editor etc:

Lakes polygons:
vector tile layer:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroLAKES_polys_v10_India/{z}/{x}/{y}.pbf
preview:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroLAKES_polys_v10_India/#5.17/19.971/74.424

Lake pour points: (I'm not really sure what this is but may be useful)
vector tile layer:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroLAKES_points_v10_India/{z}/{x}/{y}.pbf
preview:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroLAKES_points_v10_India/#10.3/10.0524/78.8019

Rivers:
vector tile layer:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroRIVERS_v10_India/{z}/{x}/{y}.pbf
preview:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroRIVERS_v10_India/#7.35/21.079/72.904

Basins Level 8:
vector tile layer:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroBASINS_lev08_India/{z}/{x}/{y}.pbf
preview:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroBASINS_lev08_India/#5.49/19.365/75.282

Basins Level 12:
vector tile layer:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroBASINS_lev12_India/{z}/{x}/{y}.pbf
preview:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-HydroBASINS_lev12_India/#6.96/20.54/73.371

GloRiC river classification:
vector tile layer:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-GloRiC_v10_India/{z}/{x}/{y}.pbf
preview:
https://server.nikhilvj.co.in/tileserver/data/hydrosheds.org-GloRiC_v10_India/#8/21.088/72.371

Please see the respective sections on https://www.hydrosheds.org/ for
understanding about the attributes in the data.
I'm not getting into that; just wanted to make the data accessible for
people as the huge datasets will probably not open on budget laptops.

Commands for converting the data and publishing as vector tiles shared
here: https://gist.github.com/answerquest/72e26f11d3da1069cbb02032633a71df

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Dec 6, 2022 at 10:51 AM sashik...@gmail.com 
wrote:

> Hi Vaidya,
> If you are  fine to use Google Earth Engine, please check the following
> catalog links
> https://developers.google.com/earth-engine/datasets/tags/hydrosheds
>
> with regards
> Sashi
>
> On Monday, December 5, 2022 at 8:33:19 PM UTC+5:30 vaibhav...@cropin.com
> wrote:
>
>> Hi Vaidya,
>>
>> Hydrosheds database, primarily from WWF - World Wildlife Fund is
>> available at the following link -
>> https://www.hydrosheds.org/products/gloric
>>
>> Here, you can find the vector files of all the rivers in the world.
>>
>> On Friday, December 2, 2022 at 10:44:03 AM UTC+5:30 vaidyan...@gmail.com
>> wrote:
>>
>>> Hello,
>>>
>>> Has anyone got the shapefiles for IndaWris river basins map?
>>> https://indiawris.gov.in/wris/#/riverBasins
>>> More than the basins, trying to find the map of rivers themselves.
>>>
>>> Thanks,
>>> Vaidya
>>>
>>
>> --
>> Please note that this email and all files, information, intents and
>> effects contained herein, and conveyed hereby, shall be construed as
>> strictly and absolutely confidential intellectual property of CropIn,
>> intended solely for use by the addressee. Notwithstanding anything
>> contained herein, this email shall not be construed to give rise to any
>> implied contractual relationship between CropIn and/or its representatives
>> and the recipient/s. In spite of due diligence exercised by CropIn to
>> ensure that this email and all attachments herewith are free from
>> viruses/defects, it will be the sole responsibility of the recipient to
>> proceed with caution, and CropIn and/ or its representatives shall not in
>> any way be liable for any unforeseen security breach arising thereof. If
>> this email has reached you in error, or you are not the intended recipient
>> of such email, you should delete this email along with any attachments from
>> your system, after notification to the sender. Any unauthorised
>> dissemination, disclosure and/ or use of the contents of this communication
>> to anyone for any purpose whatsoever, is strictly prohibited and punishable
>> in law.
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
&g

Re: [datameet] Converting maps in pdfs to actual shapefiles/geojsons

[Nikhil VJ]

Hi Vaidya,

You might be able to skip the manual tracing process that Devdatta linked to
<https://groups.google.com/g/datameet/c/BGB-UeJqhpA/m/SFl4OgF9AgAJ> and
pull out vectors directly, if your PDFs are made that way.

Rajesvari Parasa <https://github.com/rajesvariparasa> has successfully
turned Delhi development plan's PDF into shapefiles.

You can read the discussion about it here:
https://groups.google.com/g/datameet/c/37uN8xnWrK0/m/2axI7AB-AQAJ

Relevant snippet:

1. Extracting from PDF
- It's one of those PDFs exported from Autocad or so where you can see
multi-layers and turn them on or off.
- Rajesvari used a command-line GIS tool called "ogr2ogr" to extract
individual layers from the PDF into .geojson shapefiles.
- There was a tricky bit where the output lat-longs were all like
(716882.20,3166456.28) - a different CRS and we didn't know which.
- I used http://projfinder.com/ to find it, then Rajesvari included CRS
conversion into the ogr2ogr commands to produce proper shapefiles.
- You can see all that in this repo:
https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041 - see the
.bat file for the ogr2ogr commands



--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Nov 30, 2022 at 9:56 AM Devdatta Tengshe 
wrote:

> Hi Vaidya,
>
> This excellent post by Nikhil would be a good starting point:
> https://groups.google.com/g/datameet/c/BGB-UeJqhpA/m/SFl4OgF9AgAJ
>
> Regards,
> Devdatta
>
>
> On Wed, 30 Nov 2022 at 08:49, Vaidya  wrote:
>
>> Hi all,
>>
>> Wanted to know if it is possible to convert maps put out in pdfs to map
>> files (shapefiles, geojsons or kml).
>> This is regarding the sewage and water lines put out by BWSSB.
>> The files can be found here:
>> https://bwssb.karnataka.gov.in/info-2/BWSSB+GIS+MAP+Information/en
>>
>> Thanks,
>> Vaidya
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAHVy6PF4-%3DES_YWNgJEyi6c6G6GhMizB%2B4iL6Lm3HQjwAU8b9Q%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAHVy6PF4-%3DES_YWNgJEyi6c6G6GhMizB%2B4iL6Lm3HQjwAU8b9Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAOQOUMvSVZ6kpq3BQh8yB9FNKGeahBEjPsVnh2wOkRB7J%2BSPLQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAOQOUMvSVZ6kpq3BQh8yB9FNKGeahBEjPsVnh2wOkRB7J%2BSPLQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNN4WO_-Q89pA01MBbc%3D38%3DR_uvhcpF2uAg5TD%2BY_5OdQ%40mail.gmail.com.

Re: [datameet] NEW DISTRICTS OF INDIA SHAPEFILE

[Nikhil VJ]

Hi Rahul,

Once you see the:

> It is just opening a long list of lat and longs of feature polygon
>

.. press Ctrl+S on your keyboard.
Save it to your computer. ensure that the file has extension as ".geojson".

That is the shapefile. There's various formats - this one is .geojson.
Devdatta mentioned another format which would be the .shp etc. All are
equally usable, all depends on your next step.

Open QGIS or ArcGIS and drag-drop this in, you'll see it on map.
Or, go on https://geojson.io and drag-drop it there, you'll see it on map.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Nov 8, 2022 at 11:25 AM Devdatta Tengshe 
wrote:

> Rahul,
>
> This data is in GeoJSON format, and not shapefile.
>
> Once you download the data in GeoJSON format, you can then convert it to a
> Shapefile using a GIS tool like QGIS, or one of many online convertors.
>
> Regards,
> Devdatta
>
>
> On Tue, 8 Nov 2022 at 10:38, Rahul Dhiman 
> wrote:
>
>> Hello Akula. First of all thankyou for taking time to create this shape
>> file and making our lives easier. Please can anybody tell me how to
>> download the data, as when I am clicking download. It is just opening a
>> long list of lat and longs of feature polygon. How I can download the shape
>> file?
>>
>> Regards,
>> Rahul
>>
>> On Monday, 22 August 2022 at 08:26:49 UTC+5:30 cara.a...@berkeley.edu
>> wrote:
>>
>>> Thank you!!!
>>>
>>> On Mon, Jul 11, 2022, 11:31 AM akula.g...@gmail.com <
>>> akula.g...@gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have updated  INDIAN-DISTRICTS shapefile with all new districts even
>>>> including newly formed Andhra Pradesh districts counting  a total of 755
>>>> districts.
>>>>
>>>>
>>>> https://github.com/datta07/INDIAN-SHAPEFILES/blob/master/INDIA/INDIA_DISTRICTS.geojson
>>>>
>>>> Regards,
>>>> Akula Guru Datta
>>>> akuladatta.github.io
>>>>
>>>> --
>>>>
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>>> about us by visiting http://datameet.org
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "datameet" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to datameet+u...@googlegroups.com.
>>>>
>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/datameet/c586dac0-8513-4668-9eeb-d6181b7f1fc2n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/datameet/c586dac0-8513-4668-9eeb-d6181b7f1fc2n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/ba012d6f-1686-430f-b440-417ec03e51cbn%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/ba012d6f-1686-430f-b440-417ec03e51cbn%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAOQOUMtDEDaTWMDPLbyZ8BQERRRCRUjUwQ_nU8vOMCmQCS6uFw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAOQOUMtDEDaTWMDPLbyZ8BQERRRCRUjUwQ_nU8vOMCmQCS6uFw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuPR9LPS5SAdvggoQyf0SqZGAkwSAfOFF4FoWjkZFLn-Og%40mail.gmail.com.

Re: [datameet] Image storage, browsing and retrieval

[Nikhil VJ]

Hi Shivangi,
There's 2 pathways: self-hosted, and managed by a 3rd party.


For self-hosted, for the infra, there are some cheaper server hosting
services available: Ssdnodes, Hetzner.
I'll recco ssdnodes for most Storage+RAM for money (buy for 3 yrs, that's
where it gets wayy cheaper). Hetzner comes next and has convenient monthly
payment via paypal which works with Indian debit cards. (will not
auto-deduct, you pay). And both of these don't have hidden costs - fixed
amt each month, no usage-dependent shock bills. (Ssdnodes on 3 yr plan - no
monthly paying actvity for 3 yrs!)

Con: this path is self-managed, can take more in-house effort for
maintenance, defense against attacks etc. And there is a fixed based cost
irrespective of whether ur using it to the full extent or not.

After infra, if we see the actual software that will manage the images,
then nextcloud as mentioned is one option.
Sharing another one: https://min.io/  -> it's made to be an open source
alternative to Amazon S3 buckets / Azure blob storage etc. Offers features
on par with those services, but self-hosted. I have successfully run one
instance of it on my server without much effort. But have not heavily used
it yet.


*Managed services:*
Azure Blob storage, Google Cloud storage, Amazon AWS are some options.

I have worked with Azure blob professionally and its quite good, integrates
well with python and nodejs code. We can also make certain containers
(equiv to folders - sort of) public-view, so you can have absolute URLs of
all the files and publish those in all places. The links will be
direct-download so can be directly embedded, not like you have to open a
google driver folder in a new tab or something.

But i wasn't the one paying so don't directly know what that is like. From
all the expense reports it seems like a v.small cost for a lot of files.
I've explored their pricing page and it looks fully pay-as-you-go, and very
cheap until you reach a v.high amt of usage. Could not find a fixed base
cost or other such red flag.

So I'll recco this option. Don't have to spend resources on server setup
and management; can start using the service from get-go. If some launch
gets delayed a few months, then you won't be incurring high costs during
that time. If you have a ton of images saved but in daily usage very few
are actually downloaded/uploaded, then it's a lower cost due to that also.
You can choose an India server location of the service, so minimal latency
(I've experienced no lag with several thousand files fetched as fast as my
net would allow).
With Azure
<https://azure.microsoft.com/en-in/products/storage/storage-explorer/>
Storage Explorer
<https://azure.microsoft.com/en-in/products/storage/storage-explorer/> tool
I'm able to see/manage all the stored files in an online folder and
directly upload there, so I don't have to rely on scripts.

---
All the best.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Nov 3, 2022 at 1:14 PM Thejesh GN  wrote:

> - Nextcloud photos is great. You could also use Nextcloud for other
> document formats, so can work as all in one document management tool
> - You can also check  https://photoprism.app - It can also do some google
> photos like things like maps, people tagging etc. Its self hosted and FOSS
>
>
>
> Thej
> --
> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
> http://thejeshgn.com
> GPG ID :  0xBFFC8DD3C06DD6B0
>
>
> On Thu, 3 Nov 2022 at 12:49, Shivangi Desai 
> wrote:
>
>> Thanks Dilawar for the information.
>> I will look into it too.
>>
>> Regards,
>> Shivangi
>>
>> On Thu, Nov 3, 2022 at 10:54 AM Dilawar Singh 
>> wrote:
>>
>>> https://github.com/nextcloud/photos may just fit the bill.
>>>
>>> - Self hosted (+/- depending on how you look at it). I run one on an AWS
>>> $5/month instance and it is enough for our small org right now. we don't
>>> use it for photos though.
>>> - it may not be hard to link google form to Nextcloud (but Zapier
>>> doesn't have integration, so what do i know..).
>>> - Have an Android app. Possibly iOS as well. You have to tag the photos.
>>> Or arrange them in different folders.
>>>
>>> Nextcloud is a very popular solution and a lot of plugins. You may to
>>> browse through them to check if something suitable it here. Before you plan
>>> to self host on a dedicated server, you can try it on a shared instance.
>>> Its easy to install but hard to upgrade (used to break often).
>>> My MilesWeb shared account cost me ~ Rs. 2k per year.
>>>
>>> best,
>>>Dilawar
>>>
>>>
>>>  On Thu, 03 Nov 2022 10:43:16 +0530 *Shivangi Desai
>>> >* wrote ---
>&

[datameet] Data on Refugee camps in India

[Nikhil VJ]

Hello,

Where can I get data about refugee camps, refugee populations in India?
Both past and present.

Looking primarily for locations.

But beyond that, population numbers, demographics, timeline, reasons for
being there, origins would also be helpful.

Has anyone mapped these?

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNAwMYCemJ2SgTiW5yPBuyUjuuAkfApnGK8bTtLqJQGVg%40mail.gmail.com.

Re: [datameet] Are all public funded datasets open by default under GODL license?

[Nikhil VJ]

Hi, just for clarification, can anyone who knows abt it pls elaborate on
what is meant by "negative list" ?

I'm assuming right now that it means certain datasets that have to be
exempted from disclosure due to national security purposes. But if it's
different pls tell.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sat, Oct 29, 2022 at 11:54 AM sreeram kandimalla <
kandimalla.sree...@gmail.com> wrote:

>
> NSDI/DST did say in the second RTI that the negative list they published
> as part of the new spatial guidelines is their negative list for NDSAP( if
> that helps ).
>
>
>
> On Sat, 29 Oct, 2022, 8:24 am Arun Ganesh,  wrote:
>
>> Thank you Sreeram for taking the effort to clarify this using RTI.
>>
>> Small props to the information officer for constructing new sentences for
>> each response instead of copy pasting the same one.
>>
>> My takeaway is that NDSAP implementation has stalled because there does
>> not seem to be a deadline for departments to create a negative list of
>> datasets. Which further means that one cannot assume GODL.
>>
>> With spatial data the new geospatial data regulations at least clarifies
>> the negative list. Knowing if we can use the SoI village boundaries
>> under GODL since its unrestricted might be a start and could set a
>> precedent for other SoI datasets. Would asking this through RTI be
>> effective?
>>
>> On Fri, Oct 28, 2022 at 9:27 AM sreeram kandimalla <
>> kandimalla.sree...@gmail.com> wrote:
>>
>>>
>>> Also another RTI filed asking for the negative lists available with DST
>>> for NDSAP got me this response. I thought people here would be interested
>>> in knowing.
>>>
>>> [image: Screenshot 2022-10-25 at 4.05.02 PM.png]
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Oct 28, 2022 at 6:44 PM sreeram kandimalla <
>>> kandimalla.sree...@gmail.com> wrote:
>>>
>>>> So, I filed an RTI( my first ) regarding this with DST, which after
>>>> being kicked around between DST and MeitY was rejected saying the questions
>>>> weren't clear enough. So, I went around looking for more clear questions
>>>> from people who actually had some clue of how to interpret these things and
>>>> found this interesting discussion about the same question.  -
>>>> https://commons.wikimedia.org/wiki/Template_talk:GODL-India#Can_the_user_site_assume_GODL%3F
>>>>
>>>>
>>>> The discussion seemed dead without having ever reached a conclusion.
>>>> But, there was some information there about what to ask for in an RTI and I
>>>> filed another RTI with those questions, this time I sent it to both the
>>>> departments and they promptly forwarded it to Ministry of Railways, which
>>>> then pushed it back to DST. In the end I got a response from the 
>>>> "e-Governance
>>>> Group under MeitY"
>>>>
>>>> Here is the response I got -
>>>>
>>>> ---
>>>>
>>>> 1) Is GODL valid if originally used outside of data.gov.in?
>>>>
>>>> Reply:GODL is a Gazette Notified document (
>>>> https://data.gov.in/sites/default/files/Gazette_Notification_OGDL.pdf)
>>>> and it does apply on the data sets published under NDSAP and through the
>>>> OGD Platform.
>>>>
>>>> 2) Is GODL valid for data formats outside Section 3.2 of Implementation
>>>> Guidelines?
>>>>
>>>> Reply:Section 3.2 of NDSAP recommends that data has to publish on OGD
>>>> platform in data formats prevalent in Government. The data formats outside
>>>> the above said section are not allowed to be publish on OGD platform. The
>>>> Government Open Data License (GODL) does apply on the data sets published
>>>> under NDSAP and through the OGD Platform.
>>>>
>>>> 3) Is section 3.2 of Implementation Guidelines actually being bypassed?
>>>>
>>>> Reply:Section 3.2 of NDSAP recommends that data has to publish on OGD
>>>> platform in data formats prevalent in Government.
>>>>
>>>> 4) Is Implementation Guidelines mandatory or best practice?
>>>>
>>>> Reply:The Implementation Guidelines for National Data Sharing and
>>>> Accessibility Policy (NDSAP) details out recommendations and operational
>>>> procedures for the purpose of implementation of NDSAP by government
>>>> agencies.
>>>>
>>&g

Re: [datameet] Pune Ward-wise population data

[Nikhil VJ]

Hi Sritama,

Govt site is here: https://www.pmc.gov.in/en/pmc-final-prabhag-rachna-2022
(in case that goes down, here is archive.org mirror
<https://web.archive.org/web/20220727115259/https://www.pmc.gov.in/en/pmc-final-prabhag-rachna-2022>
)

There are various PDF docs published here apart from the maps.
Some of them would be holding the population stats you're looking for.
Annexure 5 seems to have wardwise totals. I'm not sure if that's the
population numbers or voters count.
Some online pdf-to-excel converters or Tabula / Excalibur tool should help
make that into excel. (If you get it done and proper, pls share, I'll try
to include that metadata in the shapefiles)

In case you're also looking for the shapefiles of the wards,
https://github.com/answerquest/pune-2022-wards
Also published on Datameet's maps github alongwith prior years' wards:
https://github.com/datameet/Municipal_Spatial_Data/tree/master/Pune

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Oct 24, 2022 at 1:48 PM Anindya Mozumdar  wrote:

> Dear Sritama,
>
> You can check out -
> https://www.censusindia.co.in/towns/pune-population-pune-maharashtra-802814
>
> Thank you.
> Regards,
> Anindya
> (https://aydnina.com)
>
> On 20.10.2022 00:25, Sritama Chaudhury wrote:
> > Hello all,
> >
> > Where can I find ward-wise population data for Pune.
> >
> > Best regards,
> > Sritama
> >
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/3e7ebdc81138d9b18b149c8b0742800a%40posteo.net
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMUgniHXaB%2BYh-D6nVG%3D9F8uWF1hJ-0cYiHMr9h0yPDDQ%40mail.gmail.com.

Re: [datameet] Rainfall data

[Nikhil VJ]

Hi Pankaj,

Added in the webpage now:

And even after that, for some locations there just isn't data for certain
years or dates in the original dataset. Those slots were filled with junk
values in the .grd files like -999 - they've been sanitised out from here.


You just need to keep looking for a location that gives you data and let go
of the ones that don't, that is all. Not much anybody can do if IMD itself
didn't archive it.

Please also see the other instructions in the page.
0.5 grid : select accordingly on the map.

For Assam : you'll have to select the grid points coming within assam, and
try for them.

For bulk work, Generate 1 link, then copy the same structures to create
more links with the year value in the URL changed.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Oct 14, 2022 at 9:12 PM Pankaj pandey  wrote:

> Dear Nikhil I am trying to download  temperature data  for 2000 of Assam
> and getting this massage: {"success":false,"message":"No data for give
> location / year"}. What is problem Please how to download temperature data
> at 0.5 degree for all grid point of assam
>
> On Fri, Oct 14, 2022 at 5:41 PM Nikhil VJ  wrote:
>
>> Hi All,
>>
>> Got a request regd need to simplify this IMD grid data fetching process
>> for folks, and that gave me impetus to complete setting up a website where
>> individual location + year data can be downloaded as simple flat CSV (which
>> opens in Excel / LibreOffice Calc).
>>
>> So here it is: *https://server.nikhilvj.co.in/imd_data/
>> <https://server.nikhilvj.co.in/imd_data/>*
>>
>> There's a map and year selector to help choose, after that a link is
>> generated for downloading. Open that in browser or do wget/curl
>> on commandline, and it'll download a CSV with proper naming.
>>
>> Sample link:
>>
>> https://server.nikhilvj.co.in/imd_data/API/fetchData?year=2010&lat=25.25&lon=76.25&which=all
>>
>> You are *encouraged to scrape* : use the format above to compose your
>> own link. (Hint: Excel formulas like: =A2&B2&"?year="&C2 are perfect for
>> this)
>>
>> With a caveat: Given that it's not some fancy server that can take huge
>> loads, there's a cruel and harsh rate limiter in place which will allow
>> only 1 hit per 10 secs per IP address for anonymous users.
>>
>> If you want to bypass that, there's an email-otp login process given
>> which will add a token and let you do 5 hits per second, and that won't be
>> restricted by IP address.
>> This would be useful in institutions where multiple users on the same
>> network may want to fetch data at the same time.
>> ex: a school/college hands-on session in climate data visualization where
>> the host as usual forgets to pre-download the data
>> So you can scrape and I can keep my server from crashing - both sides win.
>>
>> Source code is at https://github.com/answerquest/IMD-grid-data-work (see
>> imd_data_api folder)
>>
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>>
>> On Wed, Oct 12, 2022 at 8:30 AM Nikhil VJ  wrote:
>>
>>> Hi Pankaj,
>>>
>>> The place initially referred to is archival data - by definition it's
>>> historic and not realtime.
>>> What I do see on https://imdpune.gov.in/lrfindex.php is "Gridded Data
>>> Real Time" -> you might want to explore that.
>>> Over there, Daily 0.5x0.5 grid temp data links are there.
>>> You can select a single date there and download the file.
>>>
>>> I don't know right now how to open it, though. imdlib library that I've
>>> used - I didn't see anything about daily data.
>>> Here's its documentation:
>>> https://imdlib.readthedocs.io/en/latest/Usage.html
>>>
>>> I've posted a query about it in the lib's github :
>>> https://github.com/iamsaswata/imdlib/issues/17 , let's see if the
>>> author responds. Perhaps some inspection of the source code also might
>>> yield a way to open the daily data.
>>>
>>> PS: Next time do initiate a separate conversation with a different
>>> subject line - as here you have asked about temperature data but the
>>> subject reads rainfall data. Becomes less discoverable over the open
>>> internet or after a long period of time.
>>>
>>> --
>>> Cheers,
>>> Nikhil VJ
>>> https://nikhilvj.co.in
>>>
>>>
>>> On Thu, Oct 6, 2022 at 11:56 PM Pankaj pandey 
>>

Re: [datameet] Rainfall data

[Nikhil VJ]

Hi All,

Got a request regd need to simplify this IMD grid data fetching process for
folks, and that gave me impetus to complete setting up a website where
individual location + year data can be downloaded as simple flat CSV (which
opens in Excel / LibreOffice Calc).

So here it is: *https://server.nikhilvj.co.in/imd_data/
<https://server.nikhilvj.co.in/imd_data/>*

There's a map and year selector to help choose, after that a link is
generated for downloading. Open that in browser or do wget/curl
on commandline, and it'll download a CSV with proper naming.

Sample link:
https://server.nikhilvj.co.in/imd_data/API/fetchData?year=2010&lat=25.25&lon=76.25&which=all

You are *encouraged to scrape* : use the format above to compose your own
link. (Hint: Excel formulas like: =A2&B2&"?year="&C2 are perfect for this)

With a caveat: Given that it's not some fancy server that can take huge
loads, there's a cruel and harsh rate limiter in place which will allow
only 1 hit per 10 secs per IP address for anonymous users.

If you want to bypass that, there's an email-otp login process given which
will add a token and let you do 5 hits per second, and that won't be
restricted by IP address.
This would be useful in institutions where multiple users on the same
network may want to fetch data at the same time.
ex: a school/college hands-on session in climate data visualization where
the host as usual forgets to pre-download the data
So you can scrape and I can keep my server from crashing - both sides win.

Source code is at https://github.com/answerquest/IMD-grid-data-work (see
imd_data_api folder)


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Oct 12, 2022 at 8:30 AM Nikhil VJ  wrote:

> Hi Pankaj,
>
> The place initially referred to is archival data - by definition it's
> historic and not realtime.
> What I do see on https://imdpune.gov.in/lrfindex.php is "Gridded Data
> Real Time" -> you might want to explore that.
> Over there, Daily 0.5x0.5 grid temp data links are there.
> You can select a single date there and download the file.
>
> I don't know right now how to open it, though. imdlib library that I've
> used - I didn't see anything about daily data.
> Here's its documentation:
> https://imdlib.readthedocs.io/en/latest/Usage.html
>
> I've posted a query about it in the lib's github :
> https://github.com/iamsaswata/imdlib/issues/17 , let's see if the author
> responds. Perhaps some inspection of the source code also might yield a way
> to open the daily data.
>
> PS: Next time do initiate a separate conversation with a different subject
> line - as here you have asked about temperature data but the subject reads
> rainfall data. Becomes less discoverable over the open internet or after a
> long period of time.
>
> --
> Cheers,
> Nikhil VJ
> https://nikhilvj.co.in
>
>
> On Thu, Oct 6, 2022 at 11:56 PM Pankaj pandey  wrote:
>
>> Thanks Akshay for sharing the code. Actually I want to download tmax and
>> tmin data at 0.5 degree not at 1 degree for 2000 to 2021. As I want to
>> dowanload  gridded data  real time Download Daily Maximum Temperature
>> data ( 0.5 x 0.5 degree) Binary File (Real-time). In this page only the
>> select data option is available. Should i  replace year with data  in code.
>> It is requested please modify code for this case
>>
>> with regards
>> p.k.pANDEY
>>
>>
>>
>> On Thu, Oct 6, 2022 at 11:30 PM Akshay Kumar 
>> wrote:
>>
>>> Dear Sir
>>> Pl check this link.
>>>
>>> https://github.com/Manisht9/imdData
>>>
>>>
>>> On Thu, 6 Oct 2022 at 10:55 PM, Pankaj pandey 
>>> wrote:
>>>
>>>> Dear  Nikhil  how can batch download data,
>>>> https://imdpune.gov.in/lrfindex.php. This page give data wise data.
>>>> how can download  suppose 20 year data of temperature. Any python code to
>>>> do this to bulk download  in csv. Please update any code will very much
>>>> neede
>>>>
>>>> With Regards
>>>> P.K.Pandey
>>>>
>>>>
>>>> On Thu, Oct 6, 2022 at 7:28 PM Nikhil VJ  wrote:
>>>>
>>>>> Hi Georgelio, Vastav,
>>>>>
>>>>> While this link is down:
>>>>> https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html
>>>>>
>>>>> Upon browsing around the IMD site I found under Climate Monitoring in
>>>>> top menu, https://imdpune.gov.in/lrfindex.php
>>>>> Side menu: Gridded Data Archive -> All the data is there and available
>>>>> just fi

Re: [datameet] Rainfall data

[Nikhil VJ]

Hi Pankaj,

The place initially referred to is archival data - by definition it's
historic and not realtime.
What I do see on https://imdpune.gov.in/lrfindex.php is "Gridded Data Real
Time" -> you might want to explore that.
Over there, Daily 0.5x0.5 grid temp data links are there.
You can select a single date there and download the file.

I don't know right now how to open it, though. imdlib library that I've
used - I didn't see anything about daily data.
Here's its documentation: https://imdlib.readthedocs.io/en/latest/Usage.html

I've posted a query about it in the lib's github :
https://github.com/iamsaswata/imdlib/issues/17 , let's see if the author
responds. Perhaps some inspection of the source code also might yield a way
to open the daily data.

PS: Next time do initiate a separate conversation with a different subject
line - as here you have asked about temperature data but the subject reads
rainfall data. Becomes less discoverable over the open internet or after a
long period of time.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Oct 6, 2022 at 11:56 PM Pankaj pandey  wrote:

> Thanks Akshay for sharing the code. Actually I want to download tmax and
> tmin data at 0.5 degree not at 1 degree for 2000 to 2021. As I want to
> dowanload  gridded data  real time Download Daily Maximum Temperature
> data ( 0.5 x 0.5 degree) Binary File (Real-time). In this page only the
> select data option is available. Should i  replace year with data  in code.
> It is requested please modify code for this case
>
> with regards
> p.k.pANDEY
>
>
>
> On Thu, Oct 6, 2022 at 11:30 PM Akshay Kumar 
> wrote:
>
>> Dear Sir
>> Pl check this link.
>>
>> https://github.com/Manisht9/imdData
>>
>>
>> On Thu, 6 Oct 2022 at 10:55 PM, Pankaj pandey 
>> wrote:
>>
>>> Dear  Nikhil  how can batch download data,
>>> https://imdpune.gov.in/lrfindex.php. This page give data wise data. how
>>> can download  suppose 20 year data of temperature. Any python code to do
>>> this to bulk download  in csv. Please update any code will very much neede
>>>
>>> With Regards
>>> P.K.Pandey
>>>
>>>
>>> On Thu, Oct 6, 2022 at 7:28 PM Nikhil VJ  wrote:
>>>
>>>> Hi Georgelio, Vastav,
>>>>
>>>> While this link is down:
>>>> https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html
>>>>
>>>> Upon browsing around the IMD site I found under Climate Monitoring in
>>>> top menu, https://imdpune.gov.in/lrfindex.php
>>>> Side menu: Gridded Data Archive -> All the data is there and available
>>>> just fine.
>>>>
>>>>
>>>> So, taking offline my data dump to save my server's bandwidth.
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>> Nikhil VJ
>>>> https://nikhilvj.co.in
>>>>
>>>>
>>>> On Thu, Oct 6, 2022 at 6:38 PM Nikhil VJ  wrote:
>>>>
>>>>> Wow, Dead Internet Theory vibes 🤐
>>>>>
>>>>> Ok, I've moved all the gridded data file copies that I had at my end
>>>>> to: https://server.nikhilvj.co.in/dump/imd_grid/
>>>>>
>>>>> Disclaimer: they were download from  source
>>>>> <https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html>
>>>>> on 13th July 2022, and their sha256 checksums are at:
>>>>>
>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_rain.txt
>>>>>
>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmax.txt
>>>>>
>>>>> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmin.txt
>>>>>
>>>>> Beyond that digital trail, no guarantee on integrity of the data from
>>>>> my side. Use at own risk.
>>>>>
>>>>> This code snippet should help you turn a file into a flat CSV table:
>>>>>
>>>>> import imdlib
>>>>> rain1 = imdlib.open_data('rain', 2010, 2010,
>>>>> 'yearwise').get_xarray().to_dataframe()
>>>>> rain2 = rain1[rain1['rain'] > -100].reset_index()
>>>>> rain2.to_csv('dump.csv',index=False)
>>>>>
>>>>> --
>>>>> Cheers,
>>>>> Nikhil VJ
>>>>> https://nikhilvj.co.in
>>>>>
>>>>>
>>>>> On Thu, O

Re: [datameet] Rainfall data

[Nikhil VJ]

Hi Georgelio, Vastav,

While this link is down:
https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html

Upon browsing around the IMD site I found under Climate Monitoring in top
menu, https://imdpune.gov.in/lrfindex.php
Side menu: Gridded Data Archive -> All the data is there and available just
fine.


So, taking offline my data dump to save my server's bandwidth.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Oct 6, 2022 at 6:38 PM Nikhil VJ  wrote:

> Wow, Dead Internet Theory vibes 🤐
>
> Ok, I've moved all the gridded data file copies that I had at my end to:
> https://server.nikhilvj.co.in/dump/imd_grid/
>
> Disclaimer: they were download from  source
> <https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html> on
> 13th July 2022, and their sha256 checksums are at:
>
> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_rain.txt
>
> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmax.txt
>
> https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmin.txt
>
> Beyond that digital trail, no guarantee on integrity of the data from my
> side. Use at own risk.
>
> This code snippet should help you turn a file into a flat CSV table:
>
> import imdlib
> rain1 = imdlib.open_data('rain', 2010, 2010,
> 'yearwise').get_xarray().to_dataframe()
> rain2 = rain1[rain1['rain'] > -100].reset_index()
> rain2.to_csv('dump.csv',index=False)
>
> --
> Cheers,
> Nikhil VJ
> https://nikhilvj.co.in
>
>
> On Thu, Oct 6, 2022 at 10:53 AM Akshay Kumar 
> wrote:
>
>>  IMD_CSV.7z
>> <https://drive.google.com/file/d/1kYrptA1jYDhrAE5OiNQSQQrcpncQaRo_/view?usp=drive_web>
>>  Clipped_rain_INDIA.7z
>> <https://drive.google.com/file/d/1JbZmrSJGMK87Ze6X2MPRWkt_SDw2I5TN/view?usp=drive_web>
>> Hi
>> Please find the attached shapefiles of the IMD Gridded rainfall dataset
>> clipped for India for the years 1990 to 2021. Also, I am attaching the csv
>> files of rainfall datasets, which were further used for creating shapefiles
>> of corresponding years. Please let me know if it solves your purpose.
>> You can clip out Himachal from the Indian shapefile.
>>
>>
>>
>> *Profound RegardsAkshay Kumar*
>>
>> *LinkedIn <https://www.linkedin.com/in/akshay-kumar-a68572118/>*
>> *+91-8198834098*
>>
>> P Please don't print this e-mail unless you really need to. Go Green.
>> *Every 3000 A4 Size papers costs 1 tree.*
>>
>>
>> On Thu, Oct 6, 2022 at 10:39 AM georg...@gmail.com 
>> wrote:
>>
>>> Thank you Nikhil, As Vastav said that link is no longer available. Is
>>> there any chance that you have collected the data? I want data for Himachal
>>> from 2000 - 2021
>>> Thank you
>>>
>>> On Thursday, 6 October 2022 at 03:11:55 UTC+5:30
>>> f201...@hyderabad.bits-pilani.ac.in wrote:
>>>
>>>> Hi Nikhil
>>>> It seems like the link used to get data no longer works. The government
>>>> has likely made the data inaccessible. Did you download all the files by
>>>> any chance? Would it be possible to upload those in google drive?
>>>>
>>>> Thanks
>>>> Vastav Ratra
>>>>
>>>> On Wednesday, October 5, 2022 at 3:58:10 AM UTC+2 nikh...@gmail.com
>>>> wrote:
>>>>
>>>>> Hi Georgelio,
>>>>>
>>>>> I was able to extract the rainfall data just fine.
>>>>> Bit of programming is needed to get rid of junk data, and we need to
>>>>> work our way thru lot of files for multi-yr. Here is a repo with work done
>>>>> on it.
>>>>>
>>>>> https://github.com/answerquest/IMD-grid-data-work
>>>>>
>>>>> I made a viz of the data for a location nr my home in Pune, and the
>>>>> figures appear to corelate with local history (Panshet flood of 1962 for
>>>>> example shows up vividly). But I'd really like to know what the original
>>>>> raw data was using which the gridded data was obtained.
>>>>>
>>>>> I've loaded it all into a DB; should be able to get out specific data
>>>>> needed, let me know if you need for a particular location(s) and years
>>>>> range.
>>>>>
>>>>> Is there a use case for uploading a mirror dataset to zenodo or kaggle
>>>>> or someplace in simpler CSV form? (Big upload so better if there is a
>>>>> demand for it. Or

Re: [datameet] Rainfall data

[Nikhil VJ]

Wow, Dead Internet Theory vibes 🤐

Ok, I've moved all the gridded data file copies that I had at my end to:
https://server.nikhilvj.co.in/dump/imd_grid/

Disclaimer: they were download from  source
<https://imdpune.gov.in/Clim_Pred_LRF_New/Grided_Data_Download.html> on
13th July 2022, and their sha256 checksums are at:
https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_rain.txt
https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmax.txt
https://github.com/answerquest/IMD-grid-data-work/blob/main/sha256_checksum_tmin.txt

Beyond that digital trail, no guarantee on integrity of the data from my
side. Use at own risk.

This code snippet should help you turn a file into a flat CSV table:

import imdlib
rain1 = imdlib.open_data('rain', 2010, 2010,
'yearwise').get_xarray().to_dataframe()
rain2 = rain1[rain1['rain'] > -100].reset_index()
rain2.to_csv('dump.csv',index=False)

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Oct 6, 2022 at 10:53 AM Akshay Kumar  wrote:

>  IMD_CSV.7z
> <https://drive.google.com/file/d/1kYrptA1jYDhrAE5OiNQSQQrcpncQaRo_/view?usp=drive_web>
>  Clipped_rain_INDIA.7z
> <https://drive.google.com/file/d/1JbZmrSJGMK87Ze6X2MPRWkt_SDw2I5TN/view?usp=drive_web>
> Hi
> Please find the attached shapefiles of the IMD Gridded rainfall dataset
> clipped for India for the years 1990 to 2021. Also, I am attaching the csv
> files of rainfall datasets, which were further used for creating shapefiles
> of corresponding years. Please let me know if it solves your purpose.
> You can clip out Himachal from the Indian shapefile.
>
>
>
> *Profound RegardsAkshay Kumar*
>
> *LinkedIn <https://www.linkedin.com/in/akshay-kumar-a68572118/>*
> *+91-8198834098*
>
> P Please don't print this e-mail unless you really need to. Go Green.
> *Every 3000 A4 Size papers costs 1 tree.*
>
>
> On Thu, Oct 6, 2022 at 10:39 AM georg...@gmail.com 
> wrote:
>
>> Thank you Nikhil, As Vastav said that link is no longer available. Is
>> there any chance that you have collected the data? I want data for Himachal
>> from 2000 - 2021
>> Thank you
>>
>> On Thursday, 6 October 2022 at 03:11:55 UTC+5:30
>> f201...@hyderabad.bits-pilani.ac.in wrote:
>>
>>> Hi Nikhil
>>> It seems like the link used to get data no longer works. The government
>>> has likely made the data inaccessible. Did you download all the files by
>>> any chance? Would it be possible to upload those in google drive?
>>>
>>> Thanks
>>> Vastav Ratra
>>>
>>> On Wednesday, October 5, 2022 at 3:58:10 AM UTC+2 nikh...@gmail.com
>>> wrote:
>>>
>>>> Hi Georgelio,
>>>>
>>>> I was able to extract the rainfall data just fine.
>>>> Bit of programming is needed to get rid of junk data, and we need to
>>>> work our way thru lot of files for multi-yr. Here is a repo with work done
>>>> on it.
>>>>
>>>> https://github.com/answerquest/IMD-grid-data-work
>>>>
>>>> I made a viz of the data for a location nr my home in Pune, and the
>>>> figures appear to corelate with local history (Panshet flood of 1962 for
>>>> example shows up vividly). But I'd really like to know what the original
>>>> raw data was using which the gridded data was obtained.
>>>>
>>>> I've loaded it all into a DB; should be able to get out specific data
>>>> needed, let me know if you need for a particular location(s) and years
>>>> range.
>>>>
>>>> Is there a use case for uploading a mirror dataset to zenodo or kaggle
>>>> or someplace in simpler CSV form? (Big upload so better if there is a
>>>> demand for it. Or, better if someone from the institution itself does it).
>>>> I love the multi-dimensional array format they've used, but the junk values
>>>> undo the storage savings (a .7z of flattened CSV may be smaller), and it's
>>>> tougher to access for folks not in the know.
>>>>
>>>> - Nikhil
>>>> https://nikhilvj.co.in
>>>>
>>>> On Tue, Oct 4, 2022, 15:37 georg...@gmail.com 
>>>> wrote:
>>>>
>>>>> Hello all,
>>>>>  Gridded rainfall data for India in IMD is not working. I need
>>>>> rainfall data for Himachal from 2010 to 2022.
>>>>> Thanks
>>>>>
>>>>> --
>>>>> Datameet is a community of Data Science enthusiasts in India. Know
>>>>> more about us by visiti

Re: [datameet] Reverse Geocoding

[Nikhil VJ]

Hi,

In my Pmgsy habitations project i had implemented this in an api call, for
the block boundaries data got from pmgsy. Openapi link:

https://server.nikhilvj.co.in/pmgsy/docs#/geospatial/blockFromMap_API_blockFromMap__lat___lon__get

Of course, the boundaries shared by Pmgsy aren't to be taken as official
admin boundaries as told by themselves, so pls don't.

I'm on course to add in more datasets and expand this api, esp recent SOI
village boundaries and toponyms data.

We could make a centralised server, but then limitations kick in like rate
limiting that will make it useless for folks who want to do bulk
operations.

Scaling up can only happen if users are willing to pay, but that makes it
another paid service, and the output quality won't be refined enough to
justify.

For the folks needing bulk ops, how about we make a dockerised
recipe-program that people can start up on their systems?

-Nikhil
https://nikhilvj.co.in


On Tue, Oct 4, 2022, 21:18 Arun Ganesh  wrote:

> One quick option using a hosted service is the Mapbox tilequery API
> https://docs.mapbox.com/playground/tilequery/ which offers 100k free
> requests per month. Once you upload the data, you can use the tilequery
> endpoint as an API to request details of the vector feature at any
> coordinate.
>
> On Tue, Oct 4, 2022 at 5:55 AM sreeram kandimalla <
> kandimalla.sree...@gmail.com> wrote:
>
>> This exists from bhuvan at https://bhuvan-app1.nrsc.gov.in/api/ . Was
>> mentioned in a recent OSM telegram conversation. Only works for Andhra
>> Pradesh and Karnataka right now.
>>
>> On Tue, 4 Oct, 2022, 1:22 pm Deepak Sharda,  wrote:
>>
>>> You can directly get it working from a postgres table. Use pincode layer
>>> available with datameet GitHub and run a query point in polygon.
>>>
>>>
>>> it will work
>>>
>>> So steps
>>>
>>> 1. get pincode layer
>>> 2. update in any postgres install postgis
>>> 3. run a query ST_intersects(polygon.geom ,
>>> ST_SetSrid(ST_Make_point(long,lat),4326)
>>>
>>> 3. use any language python , javascript to make a connection
>>>
>>>
>>>
>>> On Tue, 4 Oct, 2022, 12:07 pm Dammalapati Sai Krishna, <
>>> saikrishnadammalap...@gmail.com> wrote:
>>>
 Hey Shivangi,

 I was thinking on creating an API for this. Let me know if you would be
 interested to work together on this! Or share if you get to know this
 service already exists :)

 Regards,
 Sai

 Sent from my iPhone

 On 04-Oct-2022, at 11:47 AM, Shivangi Desai 
 wrote:

 
 Hey All,

 Wanted to check with you all what is the general consensus on doing
 Reverse Geocoding (for points in mostly rural parts of India)?

 Should we create a self hosted geo server (which exposes REST API) to
 do this? or
 Is it better to use any already available and reliable online REST API
 service?

 We would need boundary hierarchy to be returned, i.e. State, District,
 Block, Village/City etc.

 Any recommendations?

 Thanks & Regards,
 Shivangi

 --
 Datameet is a community of Data Science enthusiasts in India. Know more
 about us by visiting http://datameet.org
 ---
 You received this message because you are subscribed to the Google
 Groups "datameet" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to datameet+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/datameet/CAB%3DRKFvq9_tNkArdNQqjuf9BKihCY_ub4bBHws0P6jYXJH0JzA%40mail.gmail.com
 
 .

 --
 Datameet is a community of Data Science enthusiasts in India. Know more
 about us by visiting http://datameet.org
 ---
 You received this message because you are subscribed to the Google
 Groups "datameet" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to datameet+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/datameet/5F3D41F8-12FF-4A13-92DF-85BFBBAB691F%40gmail.com
 
 .

>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CA%2B4SsQB1YKpN8LWUmMjDFSkxz8nL-rjsdcJ4jajtjF8XmB9c_A%40mail.gmail.com
>>> 

Re: [datameet] Rainfall data

[Nikhil VJ]

Hi Georgelio,

I was able to extract the rainfall data just fine.
Bit of programming is needed to get rid of junk data, and we need to work
our way thru lot of files for multi-yr. Here is a repo with work done on
it.

https://github.com/answerquest/IMD-grid-data-work

I made a viz of the data for a location nr my home in Pune, and the figures
appear to corelate with local history (Panshet flood of 1962 for example
shows up vividly). But I'd really like to know what the original raw data
was using which the gridded data was obtained.

I've loaded it all into a DB; should be able to get out specific data
needed, let me know if you need for a particular location(s) and years
range.

Is there a use case for uploading a mirror dataset to zenodo or kaggle or
someplace in simpler CSV form? (Big upload so better if there is a demand
for it. Or, better if someone from the institution itself does it). I love
the multi-dimensional array format they've used, but the junk values undo
the storage savings (a .7z of flattened CSV may be smaller), and it's
tougher to access for folks not in the know.

- Nikhil
https://nikhilvj.co.in

On Tue, Oct 4, 2022, 15:37 georg...@gmail.com  wrote:

> Hello all,
>  Gridded rainfall data for India in IMD is not working. I need rainfall
> data for Himachal from 2010 to 2022.
> Thanks
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/b6a669b9-6b30-41d6-bc22-cb960814bc6en%40googlegroups.com
> 
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNOoh4gkO99jM4ArmRxOhXa6ngH_tyZqDtEv81EhbOuzw%40mail.gmail.com.

Re: [datameet] Karnataka Polling Station Boundaries scraped from KSRSAC

[Nikhil VJ]

Hi Vivek,

Great work!
it looks like the big files are all in Github LFS

Example: content of Karnataka.shp :
version https://git-lfs.github.com/spec/v1
oid sha256:e32b9cfaa855993cabba3de1f860b248cac98cb53b1a5bf50b8913dda33ad044
size 106013972

Can you tell how to download that, and add the same in the repo's readme ?

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sun, Jul 31, 2022 at 5:44 AM Vivek Matthew 
wrote:

> Hello folks,
>
> I had scraped the polling station boundaries for Karnataka from the KSRSAC
> site (https://kgis.ksrsac.in/pollinginfo/). The data is up on this GitHub
> repo: https://github.com/Vonter/india-polling-stations
>
> There are a total of 60366 polling stations. On the repo there is a
> Karnataka.geojson with the boundaries for all the 60366 polling stations
> but it is a huge 156MB file. So I have also divided it into separate
> GeoJSONs for each of the 224 Assembly Constituencies (~1MB each) and each
> of the 31 districts (~6MB each) in Karnataka.
>
> I'll soon be adding Parliamentary Constituency level GeoJSONs. Converting
> the GeoJSONs into other formats is also in the pipeline. If anyone knows of
> other states that have similar sites which could be scraped to generate
> polling station boundary GeoJSONs, I would be happy to try and do so.
>
> When I was searching for Karnataka polling station boundary geodata a
> few days back (out of interest in more granular population density geodata)
> I didn't find it anywhere online. So I hope that someone else out there
> searching for it will find this data useful now that it has been scraped
> from KSRSAC.
>
> Regards,
> Vivek
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAH8O8LdHebEeQcRWLjXf6t2UK9JHAZ10yR8O_MseD9ji5kygYw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAH8O8LdHebEeQcRWLjXf6t2UK9JHAZ10yR8O_MseD9ji5kygYw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuO-Y5_rFumEqTCbgqm3pNoWVH%3Dv7oeZQ-ssL_2efMbG6w%40mail.gmail.com.

Re: [datameet] Re: NEW DISTRICTS OF INDIA SHAPEFILE

[Nikhil VJ]

Hi Vaibhav,

So the link you shared is for humans to load a page in the browser, that
hosts the file from where they can click download to download the actual
file.

To get a direct download link that your software can use, right-click the
download button and copy that link.

You can also just download the full file to your system and then make your
software read that, to make things simpler.

Do navigate your way around the github repo to the latest version of the
file, lest you keep using an older commit after the file has been updated.
(see the link - if you see weird chars inside, then you're on a specific
commit that may or may not be the latest version)

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Jul 25, 2022 at 1:03 AM VAIBHAV PURI  wrote:

> Hi
>
> I am facing a similar issue as Gurpreet. I tried adding a new vector layer
> under protocol, type: GeoJSON and source as permalink but received the
> following error:
>
> Invalid Data Source:
> https://github.com/datta07/INDIAN-SHAPEFILES/blob/31af6045eb62fcf082b59b54da3e7c05b121b121/INDIA/INDIA_DISTRICTS.geojson
> is not a valid or recognized data source.
>
> How to resolve this? or something wrong being done at my end to fetch the
> district file?
>
> Regards,
> Vaibhav
>
> On Mon, Jul 25, 2022 at 12:10 AM Akshit Shah 
> wrote:
>
>> Hi  Akula Guru Datta,
>>
>> I cross checked the district file with the latest LGD district list and
>> found there are 3 new districts formed recently which are missing from the
>> file. Can you please add them. The District are namely
>> 1. Tamulpur, Assam
>> 2. Pakyong, Sikkim
>> 3. Soreng, Sikkim
>>
>> you can find the jurisdiction in the government order for these
>> respective districts uploaded on the LGD website. Attaching the same here
>> for ease of reference.
>>
>> Warm Regards,
>>
>> *Akshit Shah*
>>
>>
>> On Tue, Jul 12, 2022 at 10:28 PM Akula Guru Datta <
>> akula.guruda...@gmail.com> wrote:
>>
>>> Hi Sai Krishna,
>>>
>>> I made this myself by gathering various resources.
>>> Thanks for your info on new district Tamulpur was missing. I will add it
>>> soon.
>>>
>>> Regards,
>>> Akula Guru Datta
>>> akuladatta.github.io
>>>
>>> On Tue, Jul 12, 2022 at 6:35 PM Dammalapati Sai Krishna <
>>> saikrishnadammalap...@gmail.com> wrote:
>>>
>>>> Hello Guru Datta, Thank you very much for this effort!
>>>>
>>>> Would you be able to share the source of this data? That would help us
>>>> in using this data with proper quotes.
>>>>
>>>> Also, as per the Local Government Directory
>>>> <https://lgdirectory.gov.in/> there are 756 districts. Just want to
>>>> bring it to your notice if you haven't come across this yet.
>>>>
>>>> Thanks again!
>>>> Sai Krishna
>>>>
>>>> On Tuesday, July 12, 2022 at 12:01:11 AM UTC+5:30 akula.g...@gmail.com
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I have updated  INDIAN-DISTRICTS shapefile with all new districts even
>>>>> including newly formed Andhra Pradesh districts counting  a total of 755
>>>>> districts.
>>>>>
>>>>>
>>>>> https://github.com/datta07/INDIAN-SHAPEFILES/blob/master/INDIA/INDIA_DISTRICTS.geojson
>>>>>
>>>>> Regards,
>>>>> Akula Guru Datta
>>>>> akuladatta.github.io
>>>>>
>>>> --
>>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>>> about us by visiting http://datameet.org
>>>> ---
>>>> You received this message because you are subscribed to a topic in the
>>>> Google Groups "datameet" group.
>>>> To unsubscribe from this topic, visit
>>>> https://groups.google.com/d/topic/datameet/4kuQecxfEv8/unsubscribe.
>>>> To unsubscribe from this group and all its topics, send an email to
>>>> datameet+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/datameet/4dcde696-df5a-4868-8ae4-3fd5e389ba3dn%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/datameet/4dcde696-df5a-4868-8ae4-3fd5e389ba3dn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by v

Re: [datameet] NEW DISTRICTS OF INDIA SHAPEFILE

[Nikhil VJ]

Hi Gurpreet,

Just checked; the INDIA_DISTRICTS.geojson file is loading on QGIS fine at
my end. It's big, though.
The data is already split up by states in the same github repo - you can go
into the states folder:
https://github.com/datta07/INDIAN-SHAPEFILES/tree/master/STATES

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Jul 19, 2022 at 4:26 AM Gurpreet Singh 
wrote:

> Hi Akula
>
> Thanks for sharing the file. I am unable to upload it on Q-GIS. Can you
> please help me on this.
> Or if you have shape file of the districts, you might like to share.
>
>
> *Gurpreet Singh*
>
> PhD - Food and Agribusiness (IIM-Ahmedabad)
> M.A. Rural Development, Tata Institute of Social Sciences
> B.Sc. Agriculture, G.B.P.U.A.&T., Pantnagar
>
>
> “Stretching his hand out to catch the stars, he forgets the flowers at
> his feet” ~ Jeremy Bentham
>
>
>
> On Tue, Jul 12, 2022 at 12:01 AM akula.g...@gmail.com <
> akula.guruda...@gmail.com> wrote:
>
>> Hi all,
>>
>> I have updated  INDIAN-DISTRICTS shapefile with all new districts even
>> including newly formed Andhra Pradesh districts counting  a total of 755
>> districts.
>>
>>
>> https://github.com/datta07/INDIAN-SHAPEFILES/blob/master/INDIA/INDIA_DISTRICTS.geojson
>>
>> Regards,
>> Akula Guru Datta
>> akuladatta.github.io
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/c586dac0-8513-4668-9eeb-d6181b7f1fc2n%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/c586dac0-8513-4668-9eeb-d6181b7f1fc2n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAO3Acsh8dbPWS%3DsUzoFdm9KwW8Gu73wF1Aw3omhPfvp7mBCxrg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAO3Acsh8dbPWS%3DsUzoFdm9KwW8Gu73wF1Aw3omhPfvp7mBCxrg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuN49VHdsq-tMLruT-RvTjKeKcU1vfi9KFNvKqOdFPmTkQ%40mail.gmail.com.

[datameet] IMD Pune Gridded weather data processing

[Nikhil VJ]

Hi,

(replying to this thread
<https://groups.google.com/g/datameet/c/FrLXpbpRnHg/m/FWJgNL5ZAQAJ> with a
changed subject line)
I searched on net for how to extract the imdb gridded data (Sai Kirshna
thanks for ur program too)
And I found this video: https://www.youtube.com/watch?v=40Dvj6MwGTs by
OpenGeoLab <https://www.youtube.com/c/OpenGeoLab> channel on youtube (done
some pretty good training work in GIS!)

Mentions: "imdblib <https://imdlib.readthedocs.io/en/latest/>" python
package.. and it looks like that one takes care of a lot of the complexity
for us! Big thanks to Saswati Nandi <https://github.com/iamsaswata> who
authored it.

With a bit of looking up
<https://docs.xarray.dev/en/stable/user-guide/pandas.html#dataset-and-dataframe>
I was able to bring the data out to a simple flat table format that we all
know and love.

Sharing the code of a sample extraction here:
https://github.com/answerquest/IMD-grid-data-work/blob/main/2022-07-13%20IMD%20Gridded%20data%20extraction%20sample%20flow.ipynb

I took out some data for nearby Pune, and have done a quick viz in this
google doc:
https://docs.google.com/spreadsheets/d/13j3kMvGdr52_dX-4nG9YDFCu3ZO_sYadFmYeiZJVgWo/edit?usp=sharing

*Observations:*
1. There is a lot of junk data in there : for 1950s yrs that I saw it was
over 60%. Fixed values like -999 are filled in where there wasn't any
reading for that particular place and date - probably because the data
system being used couldn't work with nulls. So it's important to get rid of
the junk data points before moving forward.
2. If I 7z-zipped the flat table csv after removing the junk data, and
whadyaknow, the result is smaller. (tip for folks managing this department
in IMD. Also, check out HDF5 format)
3. There's 3 data items: Max temp, min temp, rainfall.
4. Temp data is resolved to .5 lat-long points and available from 1950.
Rain data is resolved to .25 lat-long points and available from 1901.

*Next steps:*
As usual I'm going to setup a week-long script (going year by year only,
with plenty of pauses so that it doesn't overload the IMD site, mind!),
extract it all and load it up into a Postgresql DB / API etc for accessing.

*Would anyone out there like to collaborate on a visualization?*
We have data that is in point lat-long form, and is temporal with date-wise
resolution.
So, map-based combined with time series and/or animation would be good.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Jul 8, 2022 at 8:00 AM Dammalapati Sai Krishna <
saikrishnadammalap...@gmail.com> wrote:

> The method to acquire this data and to aggregate it at district levels can
> be taken from here:
> https://github.com/d-saikrishna/GIS_RemoteSensing/blob/master/Covid_Env/Data%20Collection%20and%20Pre-Processing.ipynb
>
> Let me know if you need more help.
>
> Sai Krishna
>
> On 08-Jul-2022, at 12:53 AM, Jayshree Dutta  wrote:
>
> 
> Hello, I'm doing my Ph.D in Chennai and currently visiting researcher in
> Austria. I need India's district wise data on Temperature (avg, or min
> -max), humidity,  wind velocity and total sun shine hours. Can anyone help
> me getting these datasets? Atleast any of them? Anytime after 2015 sound be
> fine, and of course more recent the better!
>
> With regards
> Jay
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAFFWLbCs3vAtVmdp1u6yfjhp3Rwvod0N6axt5_m_o-ONWWUJuQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAFFWLbCs3vAtVmdp1u6yfjhp3Rwvod0N6axt5_m_o-ONWWUJuQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/BDDF6844-ECF4-4B08-A048-BAC7F1F549AC%40gmail.com
> <https://groups.google.com/d/msgid/datameet/BDDF6844-ECF4-4B08-A048-BAC7F1F549AC%40gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because yo

Re: [datameet] india building footprints

[Nikhil VJ]

Hi All,

India data from Microsoft's ML Buildings Footprints data release is
deployed on both vector and raster tile layers:
https://server.nikhilvj.co.in/buildings1/
This is a TileServer GL instance.

Recipe for how this was done and the configs etc reqd is shared here:
https://github.com/answerquest/maptiles_recipe_buildings
This process may be useful for working with other large datasets also.

To overlay on JOSM, pls use this PNG layer:
https://server.nikhilvj.co.in/buildings1/styles/basic/{z}/{x}/{y}.png

For raster overlay in browser, pls use webp format which is smaller and
faster:
https://server.nikhilvj.co.in/buildings1/styles/basic/{z}/{x}/{y}.webp

For vector tiles (will work in QGIS also), pls use this:
https://server.nikhilvj.co.in/buildings1/data/india_buildings_z14/{z}/{x}/{y}.pbf

There was a lot of troubleshooting needed for getting the raster output
working - big thanks to Deepak Sharda for help in that.

Meanwhile, I've also completed importing the data to a postgis DB. If
anyone wants data for a limited region, pls reach me on nikhil.js [at]
gmail.com

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, May 26, 2022 at 7:07 PM Deepak Sharda  wrote:

> To make folder structure mbtile you might need to add ''-e''   in the
> recipe.
>
> or tile is made you can use Mbutil to create folder structure from single
> file.
>
>
> To your second query to create building footprints with rater tiles there
> are two ways two approach it.
>
> You can create raster tiles from geotiff by using tiles2py in folder
> structure.
>
> But in your case you can use Tileserver-gl what it does is it can host
> vector tiles but serve them as raster.
>
> You can check config file and change as per your requirements such as
> 256x256 or 512x512.
>
>
> let me know if you still face issue.
>
> Note - Go by vector tiles its fast and you can give 3D effect  as you have
> seen in our app.
>
>
> Thanks and Regards
>
> On Thu, 26 May, 2022, 6:39 pm Nikhil VJ,  wrote:
>
>> Hi Deepak,
>>
>> Thanks for the suggestion.
>>
>> I came across tippecanoe again : https://github.com/mapbox/tippecanoe
>> and finally seeing a use case for it. So, while the DB is still being
>> loaded, started off tippecanoe program on the original .geojsonl.
>>
>> The program was quite fast - in about 6 hrs it had generated a 1.4GB
>> .mbtiles file and it didn't even take too much RAM.
>>
>> Then, with the help of https://github.com/maptiler/tileserver-php:
>> This vector tiles URL is working at my end in QGIS:
>>
>> https://server.nikhilvj.co.in/buildings/tileserver-php/tileserver.php?/index.json?/buildings-z13/{z}/{x}/{y}.pbf
>>
>> I was looking for options to generate raster .png etc tiles, but haven't
>> found one yet.
>>
>> I also saw an option in tippecanoe to output .pbf tiles to a folder
>> instead of a single .mbtiles file. Did a couple of runs, but that hasn't
>> worked out.
>>
>> Meanwhile, Sreeram-K posted in OSM-India telegram group: folks at
>> mapwith.ai have incorporated the data and are now showing it in their
>> rapID editor alongwith the AI roads data, as suggestions for openstreetmap
>> editing. One link:
>>
>> https://mapwith.ai/rapid#background=none&datasets=fbRoads,msBuildings&disable_features=boundaries&map=16.17/13.09066/80.27361
>>
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>>
>>
>> On Tue, May 24, 2022 at 1:48 PM Deepak Sharda 
>> wrote:
>>
>>> Hello Nikhil,
>>>
>>> I would definitely love to collobrate. The best i would suggest is to
>>> make Mbtiles and host them as data is not going to change a lot so most of
>>> the time static.
>>>
>>> Rendering from hosted Mbtiles will not only fast but easy to handle on
>>> small web server.
>>>
>>> How to proceed.
>>>
>>> First creation of Mbtiles
>>> second Hosting of Mbtiles
>>> Third  Styling at client side.
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Tue, 24 May, 2022, 1:03 pm Nikhil VJ,  wrote:
>>>
>>>> Hi Bhibhash,
>>>>
>>>> Short answer: No.
>>>> There is no metadata. I took the top 1000 lines and visualized it : the
>>>> shapes were spread all over the country and there wasn't any order in them.
>>>>
>>>> With an intern's help I've started import of them into a portable
>>>> postgresql DB (dockerized with a persistent volume) on my webserver.
>>>> Started y

Re: [datameet] india building footprints

[Nikhil VJ]

Hi Deepak,

Thanks for the suggestion.

I came across tippecanoe again : https://github.com/mapbox/tippecanoe
and finally seeing a use case for it. So, while the DB is still being
loaded, started off tippecanoe program on the original .geojsonl.

The program was quite fast - in about 6 hrs it had generated a 1.4GB
.mbtiles file and it didn't even take too much RAM.

Then, with the help of https://github.com/maptiler/tileserver-php:
This vector tiles URL is working at my end in QGIS:
https://server.nikhilvj.co.in/buildings/tileserver-php/tileserver.php?/index.json?/buildings-z13/{z}/{x}/{y}.pbf

I was looking for options to generate raster .png etc tiles, but haven't
found one yet.

I also saw an option in tippecanoe to output .pbf tiles to a folder instead
of a single .mbtiles file. Did a couple of runs, but that hasn't worked out.

Meanwhile, Sreeram-K posted in OSM-India telegram group: folks at mapwith.ai
have incorporated the data and are now showing it in their rapID editor
alongwith the AI roads data, as suggestions for openstreetmap editing. One
link:
https://mapwith.ai/rapid#background=none&datasets=fbRoads,msBuildings&disable_features=boundaries&map=16.17/13.09066/80.27361


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in



On Tue, May 24, 2022 at 1:48 PM Deepak Sharda  wrote:

> Hello Nikhil,
>
> I would definitely love to collobrate. The best i would suggest is to make
> Mbtiles and host them as data is not going to change a lot so most of the
> time static.
>
> Rendering from hosted Mbtiles will not only fast but easy to handle on
> small web server.
>
> How to proceed.
>
> First creation of Mbtiles
> second Hosting of Mbtiles
> Third  Styling at client side.
>
>
> Thanks
>
>
>
> On Tue, 24 May, 2022, 1:03 pm Nikhil VJ,  wrote:
>
>> Hi Bhibhash,
>>
>> Short answer: No.
>> There is no metadata. I took the top 1000 lines and visualized it : the
>> shapes were spread all over the country and there wasn't any order in them.
>>
>> With an intern's help I've started import of them into a portable
>> postgresql DB (dockerized with a persistent volume) on my webserver.
>> Started yesterday and it's 10 million shapes in; from size comparisons
>> it'll probably be around 79 million shapes, so should be imported in a
>> week's time.
>>
>> I've set an auto-increment primary key column in the DB table, so once
>> imported at least the data can be pulled out in batches.
>>
>> There are 2 paths ahead once I've imported it:
>> ---
>>
>> *1. Serving vector tiles*
>> I want to serve MVT vector tiles from this DB - and then have a map
>> display the same. That way one can directly have this stuff render on a map
>> as they browse it - fetching only one small region at a time and hopefully
>> not being too slow.
>>
>> Finding some references here:
>> https://blog.jawg.io/how-to-make-mvt-with-postgis/
>> https://www.crunchydata.com/blog/dynamic-vector-tiles-from-postgis
>>
>> https://medium.com/@frederic.rodrigo/web-mapping-comparing-vector-tile-servers-from-postgres-postgis-405055e69084
>>
>> I'm new to this MVT business, so if anybody wants to collaborate pls feel
>> free to jump in.
>>
>> --
>>
>> *2. Tag with region ids*
>> Add columns to the DB table and tag each shape by state, district ids etc
>> using Within operations either in the DB or in python-geopandas.
>>
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>>
>> On Tue, May 24, 2022 at 10:14 AM Bibhash Nath 
>> wrote:
>>
>>> Thanks for sharing.
>>> Do we have state-wise datasets? It seems quite large to process on my
>>> computer.
>>>
>>> Best,
>>> Bibhash
>>>
>>> On Wed, May 18, 2022 at 4:58 AM Nikhil VJ  wrote:
>>>
>>>> Thanks Justin for sharing!
>>>>
>>>> I've downloaded and extracted the India.geojsonl on a webserver.
>>>>
>>>> ref about .geojsonl : https://www.interline.io/blog/geojsonl-extracts/
>>>> -> pretty useful! One can loop through a huge file without having to
>>>> load it all into RAM.
>>>>
>>>> top lines look like:
>>>> {"type": "Feature", "properties": {},"geometry": {"type":
>>>> "Polygon","coordinates": [[[83.06380515611697, 25.34167404697847],
>>>> [83.06380909901775, 25.341635591519122], [83.06386494585949,
>>>> 25.341640268589657], [83.06386100295869, 25.341678724047526],
>&g

Re: [datameet] india building footprints

[Nikhil VJ]

Hi Bhibhash,

Short answer: No.
There is no metadata. I took the top 1000 lines and visualized it : the
shapes were spread all over the country and there wasn't any order in them.

With an intern's help I've started import of them into a portable
postgresql DB (dockerized with a persistent volume) on my webserver.
Started yesterday and it's 10 million shapes in; from size comparisons
it'll probably be around 79 million shapes, so should be imported in a
week's time.

I've set an auto-increment primary key column in the DB table, so once
imported at least the data can be pulled out in batches.

There are 2 paths ahead once I've imported it:
---

*1. Serving vector tiles*
I want to serve MVT vector tiles from this DB - and then have a map display
the same. That way one can directly have this stuff render on a map as they
browse it - fetching only one small region at a time and hopefully not
being too slow.

Finding some references here:
https://blog.jawg.io/how-to-make-mvt-with-postgis/
https://www.crunchydata.com/blog/dynamic-vector-tiles-from-postgis
https://medium.com/@frederic.rodrigo/web-mapping-comparing-vector-tile-servers-from-postgres-postgis-405055e69084

I'm new to this MVT business, so if anybody wants to collaborate pls feel
free to jump in.

--

*2. Tag with region ids*
Add columns to the DB table and tag each shape by state, district ids etc
using Within operations either in the DB or in python-geopandas.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, May 24, 2022 at 10:14 AM Bibhash Nath  wrote:

> Thanks for sharing.
> Do we have state-wise datasets? It seems quite large to process on my
> computer.
>
> Best,
> Bibhash
>
> On Wed, May 18, 2022 at 4:58 AM Nikhil VJ  wrote:
>
>> Thanks Justin for sharing!
>>
>> I've downloaded and extracted the India.geojsonl on a webserver.
>>
>> ref about .geojsonl : https://www.interline.io/blog/geojsonl-extracts/
>> -> pretty useful! One can loop through a huge file without having to load
>> it all into RAM.
>>
>> top lines look like:
>> {"type": "Feature", "properties": {},"geometry": {"type":
>> "Polygon","coordinates": [[[83.06380515611697, 25.34167404697847],
>> [83.06380909901775, 25.341635591519122], [83.06386494585949,
>> 25.341640268589657], [83.06386100295869, 25.341678724047526],
>> [83.06380515611697, 25.34167404697847]]]}}
>> {"type": "Feature", "properties": {},"geometry": {"type":
>> "Polygon","coordinates": [[[87.87555977691633, 22.397660095199],
>> [87.8754256865811, 22.397658576750985], [87.87542690191553,
>> 22.397566835386712], [87.87556099225075, 22.39756835383578],
>> [87.87555977691633, 22.397660095199]]]}}
>> ...
>> So we have just basic polygons, one for each building, no properties or
>> categorization.
>>
>> Here are some ideas on what to do with this:
>>
>> 1. Postgresql DB:
>> - Load all of these into a PostGreSQL DB
>> - Setup an api that will take a lat/lon and give all shapes within 1km
>> radius
>> - Next possible api: send a bounding polygon and get all buildings in that
>>
>> 2. Split up by district or lower level admin boundaries:
>> - Load the admin boundaries into a Postgresql DB
>> - Loop through each line (ie each building)
>> - Find out which place it's in by ST_Within query
>> - Dump it into a separate .geojsonl for that place
>> - We've now split it into multiple smaller files that are more usable,
>> can be loaded up by OSM mappers etc.
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>>
>> On Wed, May 18, 2022 at 3:51 AM Justin 
>> wrote:
>>
>>> [image: lg.png]
>>>
>>> https://github.com/microsoft/GlobalMLBuildingFootprints
>>>
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com
>>> <https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --

[datameet] PMGSY Habitations <> OSM rural places data comparison app

[Nikhil VJ]

Hi All,

I've created an application where we can compare PMGSY and OSM rural
Habitations data:
https://server.nikhilvj.co.in/pmgsy/



Right-click on the map to load all habitations for that area (what they
call Block).
If you scroll below the first table, You'll find an OSM button - click it
to fetch rural places from OSM.
The app will pull those from overpass, then compare with the Habitations
data to show nearby and far OSM places.

There's also a feedback layer built : You can submit feedback for a PMGSY
Habitation or an OSM place.
(Input side is ready; output side still work to do)
You can copy the browser URL for a permalink - can share it with others and
they'll land exactly where you are.
Example:
https://server.nikhilvj.co.in/pmgsy/?S=20&D=293&B=447&O=N#12/22.2023/76.6530
(habitations in a lake?)

There is an "Edit in OSM" button below the map - clicking that will open up
main website OSM Editor, with the block's boundary loaded as guide so that
you can map.

There's more we can do with this - one idea is to make this a place where
we can link up PMGSY and OSM ids and have feedback flow both ways.
Another is to vet the data and filter out the Habitations that are suitable
candidates for a bulk import into OSM.
Scope for integrations too - for example, I just included a cool SOI
maps background published by https://ramseraph.github.io/opendata/maps/SOI/

Inviting inputs and collaborations. And the bottom of the application page
has a way to contributie if you'd like to support this and such projects.

API doc: https://server.nikhilvj.co.in/pmgsy/docs
Source code: https://github.com/answerquest/pmgsy_osm_comparison
Tech stack: Backend: Python3 FastAPI
DB: PostGreSQL, same one as in
https://github.com/answerquest/import_geosadak

I've also run an analysis on every state > district > block and created a
stats CSV which gives numbers of habitations, OSM places, near/far breakup
etc. You'll find it in the repo.
This project is a work in progress, so if you're reading this a week or
month later, things may have changed.

More Screenshots:








--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOM6gOvS%3D8OO0T-z4kTb9oTpHeu26wr%2B_vOqZ-wruiNng%40mail.gmail.com.

Re: [datameet] india building footprints

[Nikhil VJ]

Thanks Justin for sharing!

I've downloaded and extracted the India.geojsonl on a webserver.

ref about .geojsonl : https://www.interline.io/blog/geojsonl-extracts/
-> pretty useful! One can loop through a huge file without having to load
it all into RAM.

top lines look like:
{"type": "Feature", "properties": {},"geometry": {"type":
"Polygon","coordinates": [[[83.06380515611697, 25.34167404697847],
[83.06380909901775, 25.341635591519122], [83.06386494585949,
25.341640268589657], [83.06386100295869, 25.341678724047526],
[83.06380515611697, 25.34167404697847]]]}}
{"type": "Feature", "properties": {},"geometry": {"type":
"Polygon","coordinates": [[[87.87555977691633, 22.397660095199],
[87.8754256865811, 22.397658576750985], [87.87542690191553,
22.397566835386712], [87.87556099225075, 22.39756835383578],
[87.87555977691633, 22.397660095199]]]}}
...
So we have just basic polygons, one for each building, no properties or
categorization.

Here are some ideas on what to do with this:

1. Postgresql DB:
- Load all of these into a PostGreSQL DB
- Setup an api that will take a lat/lon and give all shapes within 1km
radius
- Next possible api: send a bounding polygon and get all buildings in that

2. Split up by district or lower level admin boundaries:
- Load the admin boundaries into a Postgresql DB
- Loop through each line (ie each building)
- Find out which place it's in by ST_Within query
- Dump it into a separate .geojsonl for that place
- We've now split it into multiple smaller files that are more usable, can
be loaded up by OSM mappers etc.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, May 18, 2022 at 3:51 AM Justin  wrote:

> [image: lg.png]
>
> https://github.com/microsoft/GlobalMLBuildingFootprints
>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/68c0ecc4-5b07-4eb9-a69d-ae6e5ad6832bn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuM_yRYLDRir7vYjT7JVoP-J%2BkWAEkbp_U84QSb82AX3aA%40mail.gmail.com.

Re: [datameet] Delhi Metro

[Nikhil VJ]

Hi Vaibhav,

Where all have you already looked for initial info, and over and above that
what is missing that are you still looking for?

Possible sources for metadata:
https://www.delhimetrorail.com/
https://en.wikipedia.org/wiki/Delhi_Metro
https://delhi-metro.net/
https://www.wikidata.org/wiki/Q3278367 (explore around from there)

For geospatial data, you can get into the data extraction side of
Openstreetmap - the open-data alternative to google maps which lets people
extract specific data out of it.
https://overpass-turbo.eu/
If you navigate to NCR region and open Wizard and put in "railway=station",
you get a lot of hits. But these may also include the regular railway
stations, so you can select a positive result, see more tags there and
refine your search further.
if we change the query in the Wizard box to : railway=station and
network="Delhi Metro"
..then it gets better.

Similarly to get the metro tracks you'll need to search by other tags.
Find out relevant tags by which you can extract data:
https://wiki.openstreetmap.org/wiki/Metro_Mapping

Once you've got a good query, can export the data to a shapefile for
working on.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Mar 25, 2022 at 1:40 AM VAIBHAV PURI  wrote:

> Hi
>
> I am looking for recent most vector data of Delhi Metro along with
> meta-data like Stations info, year of operation etc.
>
> Need it for an ongoing research
> Any leads would be appreciated.
>
> Regards,
> Vaibhav
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/8f1e7f08-6acf-4a20-8c4a-8f344139b446n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/8f1e7f08-6acf-4a20-8c4a-8f344139b446n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOsyzxqbYQRn69PDggmQsM%3D2REHSZNw4n2qUbWZMw2bWg%40mail.gmail.com.

Re: [datameet] Re: Flood plain maps for Maharashtra

[Nikhil VJ]

Hi,

Replying on the original subject, this gov site has river basin maps. From
"flood plain" I first assumed it's about rivers and monsoon flooding; not
the sea.

https://mwrra.org/maps/
-

On the sea level rise matter, a prerequisite dataset for creating the
coastal flooding scenarios you guys want, is : sea level rise numbers over
time. So, locating that data - simple yearwise numbers probably - might be
useful. One can take those and generate all the maps they need using the
mentioned elevation data etc.

What I've seen so far is interactive simulations where people key in how
much they want to raise the levels and then the program gives output. When
I found one such site some years ago, I had to set the value sky high to
see some real action - but it was totally unrealistic.

some links related from a quick web search:
https://en.wikipedia.org/wiki/Past_sea_level
https://en.wikipedia.org/wiki/Sea_level_rise
https://www.e-education.psu.edu/earth107/node/1506

https://coast.noaa.gov/slr/ - this simulator (for USA coasts only) starts
at 1ft, which is as per the numbers on the wikipedia article would be
multiple centuries in the future. Not sure if humans will still be doing
coastal tourism / real estate purchases at that point. We're seeing
populations of millions moving between continents in present day - it looks
a little too far-fetched to assume the affected peoples covered in these
maps won't move at all over the span of centuries.A once in a decade storm
generally causes far more sea surge and temporary inundation, so one can
expect that peoples and governments involved will have/need arrangements
going above and beyond these sea level rise extents. Don't want to be rude,
but I'm seeing a gap between the buzz and reality here.

And since this topic has been going around for over 2 decades now or more,
it would be great to see a comparison between then-projected sea level rise
versus actual sea level rise, to catch if projection models have under/over
estimated anything and make corrections.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Mar 1, 2022 at 2:12 AM Supriya Krishnan 
wrote:

> @Archi and @Parth > I am also working on urban planning and climate change.
> I am looking for sea level rise maps (as a GIS) for the Mumbai
> metropolitan region coast. Any leads would be great.
> Thank you in advance.
>
> Best,
> Supriya
>
> On Tuesday, April 10, 2018 at 1:28:31 AM UTC+2 archip...@gmail.com wrote:
>
>> Hi Parth,
>> I don't know if you are still looking for this data. I used sea level
>> rise data for whole of Indian coastline. Though this data is a lil dated,
>> the AR-5 of IPCC didn't change the extent or depth of inundation for the
>> same. I can dig them out for you. Do let me know!
>>
>> Best,
>> Archi
>>
>>
>> On Wednesday, 25 October 2017 03:19:25 UTC+5:30, pad...@cornell.edu
>> wrote:
>>>
>>> Hello,
>>>
>>> My name is Parth, and I am a graduate student at Cornell University. We
>>> are focusing on coastal tourism in the state of Maharashtra, and I am
>>> having trouble finding the GIS data, for areas which might be affected due
>>> to Sea level rise.
>>> Would anybody be able to help me out?
>>> Thank you!
>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/f2353e15-9a73-4f0d-9627-3ccc0a3a78d7n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/f2353e15-9a73-4f0d-9627-3ccc0a3a78d7n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNvTm4L2XHLHKsBdhCXoRLPi8w9OiBy5SDQkWBgvUE_aw%40mail.gmail.com.

Re: [datameet] MOU with Ministry of Rural Development ?

[Nikhil VJ]

Hi,
I've imported the data to a PostGreSQL DB on my webserver which is hosted
on SSDNodes - I think it'll be useful to have it in that form to be able to
do more with it.

I've shared the import script and steps in this repo if anyone wants to
load it into their own DB:
https://github.com/answerquest/import_geosadak

Some data inconsistencies found:
States having BLOCK_ID values in boundaries shapefile which does not have
matching Master data entry:
TamilNadu - 3
JammuAndKashmir - 3
These blocks do have road data etc but no entry in master data excel:
8108, 8105, 8106, 8063, 8034, 8041

States having BLOCK_ID=0 in their block boundary shapefile:
Punjab - 1
WestBengal - 3
I've imported these boundaries with randomly assigned block ids. We'll need
to match them to actual block id.

Limited offer: If someone wants to query my DB then pls contact me on
nikhil.js (at) gmail.com for credentials (limited capacity and bandwidth,
note).

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Feb 23, 2022 at 10:32 AM Thejesh GN  wrote:

> @Arun Ganesh   @Devdatta Tengshe
>  @Naveen Francis   and @Justin
> Meyers 
>
> Can we add this data to the main maps repo at ->
> https://github.com/datameet/pmgsy-geosadak
> At some point we can have a mini site with all the attributes and layer
> descriptions similar to the maps project that we have.
> http://projects.datameet.org/maps/
>
>
>
> Thej
> --
> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
> http://thejeshgn.com
> GPG ID :  0xBFFC8DD3C06DD6B0
>
>
> On Tue, 22 Feb 2022 at 20:16, Naveen Francis  wrote:
>
>> Thanks Harsh, Thej and Arun.
>>
>> Awesome !!!
>>
>> On Monday, 14 February, 2022 at 3:12:09 pm UTC+5:30 Thejesh GN wrote:
>>
>>>
>>>
>>> I was going to write a longer blog post this week, Dev :)
>>>
>>> The short answer is:
>>>
>>> DataMeet is one of the parties signing MoU as a community partner.
>>>
>>> DataMeet's role will be to suggest changes to datasets if required, help
>>> the open data community use data, and create how-tos. Take the feedback to
>>> send it to them with our suggestions.
>>>
>>> We are going to sign a non-commercial MOU with them. I will share the
>>> copy once I have the final version and the blog post that I am thinking
>>> about.
>>>
>>> The initial dataset would include.
>>> - Data for 10 lakh habitations
>>> - 25+ lakh km rural roads
>>> - 50,000+ km of PMGSY-III proposals
>>>
>>> Data will be released under Government's Open Data License.
>>>
>>> We are yet to get the complete picture of all the datasets and details
>>> of it. But that's the work we will do as part of this community engagement.
>>> Feb 22 is just the day to kick start it.
>>> --
>>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>>> http://thejeshgn.com
>>> GPG ID :  0xBFFC8DD3C06DD6B0
>>>
>>>
>>> On Mon, 14 Feb 2022 at 14:05, Devdatta Tengshe 
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I see that the Ministry of Rural Development is going to release some
>>>> data on 22nd Feb, and is also going to sign some MOU with datameet:
>>>>
>>>> https://gavs.webex.com/gavs/j.php?RGID=r1b499f15a791c966b6debf693bbe01a8
>>>>
>>>> What kind of data can we expect? Can someone share what the MOU is
>>>> going to be about ?
>>>>
>>>> Regards,
>>>> Devdatta Tengshe
>>>>
>>>>
>>>> --
>>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>>> about us by visiting http://datameet.org
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "datameet" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to datameet+u...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/datameet/3a567b1c-8df3-4a1c-8ddd-8b2ea640ecf0n%40googlegroups.com
>>>> <https://groups.google.com/d/msgid/datameet/3a567b1c-8df3-4a1c-8ddd-8b2ea640ecf0n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> T

Re: [datameet] Cadastral Maps for Gujarat

[Nikhil VJ]

Hi Meeta,

I had got these for some villages in Maharashtra some years ago; I don't
know about Gujarat but there could be a similar office and process there;
I'll guess Gandhinagar.

1. Walk into Settlement Commissioner And Director Of Land Records office :
https://goo.gl/maps/TsNSTHvKrHwrrQqj9
2. Tell someone you want to buy cadastral maps for so and so villages under
so and so Talukas (or what the sub-district is called in Guj) under so and
so District.
3. They'll give you a bill with the costs and send you to another floor for
making payment (digital only - only debit card was accepted when I had
gone, hope they're accepting UPI now) and getting a challan (official
receipt).
4. Cost was a few hundred bucks per village.
5. Get the challan, come back here and give it to them.
6. They will give a CD with .dwg files saved in it. Maybe one CD per
village.
7. Then take on the much harder task of finding someone who has a working
CD drive ;)

PS: No, there won't be any customization in the medium, as there are strict
rules from up top. No, they won't just email it to you even if the file is
a few KBs only, not even if their own gmail account is open right there on
the same PC that is writing the CD. Just take the CDs quietly and get out
of there.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Mar 2, 2022 at 10:03 PM Meeta Gupta  wrote:

> Hello Everyone,
>
> Can anyone please share the source/dataset (if worked on) for the
> cadastral maps (shapefile format) for Gujarat state. To be specific, I am
> looking for cadastral maps for the districts Banaskantha, Mahesana and
> Patan.
>
> Any leads will be highly appreciated.
>
> Regards,
>
> Meeta Gupta
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAOHZ-thUxYtQ_5YmX5nsVCuOQL%3DSBV1_et-BOi6_%2B8zYF3ifFg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAOHZ-thUxYtQ_5YmX5nsVCuOQL%3DSBV1_et-BOi6_%2B8zYF3ifFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMUZK0OKs%2B6tS14DXX0LwyjQ-X8t-bgNcvFtqpKLc4aXQ%40mail.gmail.com.

Re: [datameet] AUTOCAD files to be made into shapefiles, how?

[Nikhil VJ]

Hi, Rigved,

With PDFs exported from autocad, Rajesvari Parasa had worked out a method
shared here:
https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041

Example ogr2ogr command:
ogr2ogr -f "GEOJSON"  Boundaries_DDA_ZONE_Boundary_polyline.geojson
draftplan.pdf Layers_Boundries_DDA_ZONE_Boundary_polyline -s_srs EPSG:32643
-t_srs EPSG:4326 --config OGR_PDF_READ_NON_STRUCTURED YES

It seems ogr2ogr supports dwg format as well:
https://gdal.org/drivers/vector/dwg.html
But I guess some R&D would be needed to get the exact command.

I'm reckoning the ogr2ogr command wraps in some of the conversions details
in Nishad's thread.

when I import them in QGIS they show up at 0,0 lat long

If the shapes once imported into QGIS have retained their geometry, then
we're in luck. Rajesvari was facing the same issue, except in her case the
shapes were many times bigger than the world map.
We then used this site: http://projfinder.com/ (not ssl so bear with bsnl
popups and reload)

Enter a lat-long value from your shape, then move the map to the actual
location where that point should be. You'll get a list of possible CRS
projections. Try them out by varying the *-s_src* argument in the ogr2ogr
command above.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Mar 2, 2022 at 10:22 AM Nishadh K A  wrote:

> Check this thread,
> https://groups.google.com/g/datameet/c/cnPMEQ7Cr6Q/m/cA3vJlrCAAAJ
> The  steps followed are *Autocad DXF->Geojson->Geometry
> conversion(line/polygon on need basis)->geo referencing the geometry*
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/7ead5994-6506-454f-93c8-efa43e405ed2n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/7ead5994-6506-454f-93c8-efa43e405ed2n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuM9FSuP-z%2BQc_%3D_bxnH75HyXe3v2YJqsC4D5XAGJm3-cQ%40mail.gmail.com.

Re: [datameet] Re: Web Scraping Mission Antyodaya website

[Nikhil VJ]

Hi,

I don't think Selenium is required - this looks like it can be done with
just varying the request payload of one POST api call.
POST api call to URL:
https://missionantyodaya.nic.in/preloginVillageInfrastructureReports2020.html
the POST request content type is application/x-www-form-urlencoded

at *state level*, request payload is like:
stateCode: 27
stateName: MAHARASHTRA
districtCode:
districtName:
blockCode:
blockName:
gpCode:
gpName:

It* district level* it becomes:
stateCode: 27
stateName: MAHARASHTRA
districtCode: 469
districtName: AURANGABAD
blockCode:
blockName:
gpCode:
gpName:

then *block level*:
stateCode: 27
stateName: MAHARASHTRA
districtCode: 469
districtName: AURANGABAD
blockCode: 4315
blockName: KHULTABAD
gpCode:
gpName:

then* GP level:*
stateCode: 27
stateName: MAHARASHTRA
districtCode: 469
districtName: AURANGABAD
blockCode: 4315
blockName: KHULTABAD
gpCode: 170584
gpName: BODKHA

If in python, one can use Beautifulscrape to capture the table data as well
as get the (code + name) pairs for the next level.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Feb 4, 2022 at 1:42 PM Sanjay Bhangar 
wrote:

> Piyush -
>
> You could write a python (or your preferred language) script that just
> requests the HTML, parses it, and follows the hierarchy, without using
> selenium. This could be a bunch of work as the site doesn't use regular
> links with GET requests, but rather when you click on a state in the table,
> it uses Javascript to fill up hidden form fields with the state code, etc.
> and then does a form submit, causing a POST request to be made with those
> values.
>
> For eg. you can see the links in the table have an onClick handler like 
> "selectState(2,'HIMACHAL
> PRADESH','preloginDistrictInfrastructureReports2020.html')" .
>
> Then, in the javascript, you can see the selectState function defined like
> so:
>
> function selectState(stateCode,stateName,action){ 
>   $("#stateCode").val(stateCode); 
>   $("#stateName").val(stateName); 
>   $("#reportForm").attr('action', action);
>   $("#reportForm").submit();
>
> }
>
> In this JS file:
> https://missionantyodaya.nic.in/resources/antyodaya/js/custom/prelogin/reports/preloginReport.js
>
> So this will make a POST request to
> preloginDistrictInfrastructureReports2020.html
> with stateCode=2, stateName=HIMACHAL PRADESH
>
> Similarly, there are different onCick handlers defined for selecting
> districts, etc. that you can follow down to see what URLs they are calling
> with what parameters. And in theory, you could write some HTML parsing code
> and some regex to go through the items in each table, parse out the
> parameters and URLs to call, and follow things down.
>
> So, in theory you could write this without mucking around with selenium,
> but it also seems like a lot more work than if the site was structured
> "normally" with unique URLs and GET requests.
>
> For the page numbering, this seems okay: the HTML outputs all the items
> across all the pages, and then the actual pagination on the page is purely
> client-side javascript - so if you were to read the HTML on the page via
> python or so, you would just get all the items in the table without having
> to worry about pagination.
>
> Unfortunately, this does seem like a lot of work and I don't really have
> the time to do anything, but it seemed like an interesting problem and I
> was curious so I took a look. Hope it could help a bit.
>
> All the best,
> Sanjay
>
> On Fri, Feb 4, 2022 at 1:03 PM Piyush Kumar 
> wrote:
>
>> Could folks here suggest how to go about this?
>>
>>
>> https://missionantyodaya.nic.in/preloginStateInfrastructureReports2020.html
>>
>> When we click this link, we get data on village-level infrastructure put
>> within multiple HTML tables across many pages (separated into state, dist.,
>> block etc.)
>>
>> Suppose I want to scrape data upto the village level for a particular
>> state, is there any way I can get it done without too much back and forth
>> over Selenium webdriver? Please note that to access village level data you
>> have to go through a nested hierarchy of links (gram panchyt within block,
>> which is within a district and so on). To make matters more complicated,
>> the pages have also not been numbered.
>>
>> Can someone in the know help me figure this out?
>>
>> Thanks in advance
>> Piyush
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to t

Re: [datameet] pre-delimitation (<2008) assembly constituency shapefiles for UP

[Nikhil VJ]

Hi Chandrashekhar,

http://projfinder.com/ - this tool can help you find the correct projection.

Take a point from your shape - mark on the map where it should be - it will
give you a probable list of projections.
Next, set that as your shapefile's projection in QGIS. Then, export it to
another shapefile, with projection option changed to normal lat-longs which
is EPSG:4326.
That should convert everything to normal lat-longs.

Ref:
https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041#crs-conversion


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Dec 30, 2021 at 11:21 AM Chandrashekhar Raman <
chandrashekhar.ra...@gmail.com> wrote:

> thanks Devdatta, i am using the one shared by Dileep last time around, the
> shape is perfect just that the latitude and longitude seems to be scaled
> down , so in my case UP shows up small and almost next to the equator.
> i was told  this is something to do with choosing the right projection and
> can be done with QGIS, but i do not know how. much appreciate any ideas or
> links
>
> chandrashekhar
>
> On Thu, Dec 30, 2021 at 9:27 AM Devdatta Tengshe 
> wrote:
>
>> It is available in this thread:
>> https://groups.google.com/g/datameet/c/tYMSW1sbjwg/m/alfUTFpSDwAJ
>>
>>
>> Regards,
>> Devdatta
>>
>>
>> On Wed, 29 Dec 2021 at 19:28, Chandrashekhar Raman <
>> chandrashekhar.ra...@gmail.com> wrote:
>>
>>> Hello All, can anyone point to  pre-delimitation AC shapefiles for Uttar
>>> Pradesh.
>>>
>>> thanks
>>> Chandrashekhar
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CACSGm8ez0JwwM%3DxXxSLmyjoSpLux5y6J1_m3qYirNc3EZY5X5A%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CACSGm8ez0JwwM%3DxXxSLmyjoSpLux5y6J1_m3qYirNc3EZY5X5A%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAOQOUMuEbXcGRRjpR6dtEA_j%3DE9kw6M%2ByFTd%3DWTvd4rpeUBbVw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAOQOUMuEbXcGRRjpR6dtEA_j%3DE9kw6M%2ByFTd%3DWTvd4rpeUBbVw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CACSGm8fDjSSSwruChTchPBYbM0NhgCy%2BDayHvUJ-M4ArMd9YxA%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CACSGm8fDjSSSwruChTchPBYbM0NhgCy%2BDayHvUJ-M4ArMd9YxA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOEBcW%3DctguQqq8x79Vj_dw6A9vcLX1s7x%2BMTYcrkZEzw%40mail.gmail.com.

Re: [datameet] Re: bihar pre delimitation AC shapefiles

[Nikhil VJ]

Hi David,

This is in response to your second question about the projection the shapes
are in.
This might help to find out: http://projfinder.com/
Have to start with identifying a spot in the shapefile that you can
recognise and locate on a real map. Center the map in the website there,
and key in the co-ords for that spot in the shapefile. The website then
gives you a listing of most likely projections.

We used that tool here:
https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Oct 26, 2021 at 1:09 AM David Sorge  wrote:

> Thanks so much for sharing this! Three quick questions:
>
> 1) Do you have the link within the ECI website?
> I'd love to be able to link back to it, check for additional metadata,
> etc. Unfortunately, ECI is a pretty difficult website to navigate (for me
> at least -- maybe I just need to learn how to use it better.)
>
> 2) Do you know what projection the files are in?
> I'd like to transform it so it fits with data in a Kalianpur 1975/India
> Zone IIa Projection, but to get it there, I need to figure out what
> projection the data starts in. Were there any clues about this on the site
> where the files originated?
>
> 3) Is there a well-known fix for the Uttarakhand issue?
> I was able to do a quick and imprecise fix in python using geopandas and
> shapely:
>
> ```
> import pandas as pd
> import geopandas as gpd
> from shapely.affinity import translate
>
> AC_path = Path('geocoding/AC_Data/States')
> gdfs = []
> for file in AC_path.glob('*/*.shp'):
> gdf = gpd.read_file(file)
> gdfs.append(gdf)
>
> AC_data = pd.concat(gdfs)
> mask = AC_data.ST_CODE == 'S28'
>
> AC_data.loc[mask, 'geometry'] = \
> AC_data[AC_data.ST_CODE == 'S28'].geometry \
> .apply(lambda geom:
> translate(geom, xoff=-14.3468, yoff=0.1394))
> ```
>
> However, this results in only an approximate fix, with some boundary
> problems as shown in the plot here. The fact that Uttarkhand looks just a
> bit small for the space probably has to do with issues with the projection.
>
>
>
> This fix will work well enough for my purposes, I think, but I'd be
> interested to know about another solution if one exists.
>
> Thanks so much!
>
> David
>
>
> On Friday, October 30, 2015 at 2:47:20 AM UTC-4 cadv wrote:
> The predetermination data was available fro download from the Election
> commission website itself.
> I had kept it safely.
>
> Here it is. There is a known problem about the placement of Uttarakhand in
> one of the sets.
>
>
> On Wednesday, October 28, 2015 at 11:43:58 AM UTC+5:30, Chandrashekhar
> Raman wrote:
>  does anyone have access to pre-delimitation assembly constituency
> shapefiles of Bihar.. i.e 2005 backwards. much appreciate if can be shared.
>
> thanks
> cs
>
>
> On Friday, October 30, 2015 at 2:47:20 AM UTC-4 cadv...@gmail.com wrote:
>
>> The predetermination data was available fro download from the Election
>> commission website itself.
>> I had kept it safely.
>>
>> Here it is. There is a known problem about the placement of Uttarakhand
>> in one of the sets.
>>
>>
>> On Wednesday, October 28, 2015 at 11:43:58 AM UTC+5:30, Chandrashekhar
>> Raman wrote:
>>>
>>>  does anyone have access to pre-delimitation assembly constituency
>>> shapefiles of Bihar.. i.e 2005 backwards. much appreciate if can be shared.
>>>
>>> thanks
>>> cs
>>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/ab6e8db6-2c9f-43ab-a83f-583c820eb983n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/ab6e8db6-2c9f-43ab-a83f-583c820eb983n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMmO%2BpOSD89C5PSoNMX%2BCoEgCLatOtd4V7itdPKKC2qBg%40mail.gmail.com.

Re: [datameet] Re: How to link Vehicles to Trips for RT-GTFS?

[Nikhil VJ]

Hi Yash,

Here is a site where lot of GTFS related solutions are collected together;
its maintained by a network of people working in the sector:
https://github.com/CUTR-at-USF/awesome-transit

Mentioned there is one tool being used by a lot of folks in other countries
for figuring out the trip_id and other stuff: http://thetransitclock.org/
It's not a simple solution. Their mailing list will give a good picture of
what all it takes.

What Rigved has mentioned is unfortunately the situation for most bus
transit systems, and it's unfortunate seeing that a lot of investment and
buzz went into fitting buses with GPS, but they didn't bother looking into
how they will update which route number the vehicle is on.
It would possible to set it up if one can get a buy-in from the bus agency.

Here's what in my opinion would be a lowest effort way to do it:
- Make a mobile app for depot managers. Every depot logs in here.
- Have them input the bus's unique number (usually a serial number painted
on the bus, or the license plate number), and the route the bus is going
on, the departure time and submit.
- This information is sent by the app to the backend system where the GPS
data is also flowing in
- Now this backend system is able to attach route_id with each vehicle and
change it when the route changes.
- It'll be possible for backend to determine which trip_id under that route
this is on with some programming work put in.

At present, bus depots maintain registers where they write in which bus is
going on which route and which driver and conductor. There is also a lot of
paperwork later for compiling the data together to submit up the management
chain. This intervention would be attractive if this writing business was
replaced by digital medium and it also removes most of the paperwork burden.

But at the same time a level of flexibility and autonomy needs to be
maintained. You shouldn't have the HQ breathing down their necks and
micro-managing things. The Depot managers are weighing in a ton of tacit
real world parameters when making decisions about which bus to send on
which route, including human requirements of the personnel, street level
complications etc. The dynamic decision making process goes a long way in
optimizing the transit agency's utilization, efficiency, costs etc and it
would be tragic to lose that all in a blind technocratic digitization rush.

Extra: I have a net negative faith in the AI/ML lobby's advertised promises
to take care of this stuff. I trust the depot managers far higher and
believe working with them will give better results than trying to replace
them. But that's just an unqualified personal opinion.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Sep 7, 2021 at 9:28 AM rigved1...@gmail.com <
rigved12she...@gmail.com> wrote:

> Hi Yash,
> Logically, there should ideally be a device in bus/app on mobile where the
> conductor/driver manually selects the Trip Id and indicates it is started.
> In reality, I have not seen this implemented in any Indian RT-GTFS feeds.
> As a result, GPS locations almost never match the trip paths. I have seen
> situations where the morning TRIP_ID is carried through entire day with the
> bus moving in entire city.
> If you find a workable solution and implement the feed, please let me
> know. A clean RT-GTFS dataset would be goldmine of information.
>
> On Monday, September 6, 2021 at 9:41:19 AM UTC+5:30 yas...@gmail.com
> wrote:
>
>> Hi,
>>
>> If we have a GTFS feed and want to set up an RT-GTFS, we would need to
>> maintain a mapping between a GPS-fitted vehicle and the corresponding Trips
>> it is scheduled to make. The location updates come from a vehicle. The
>> Vehicle ID and the latest location is thus known. However, this needs to be
>> linked to a Trip so that RT-GTFS shows the status of the Trip. What tools
>> are generally used to make this mapping?
>>
>> -Yash
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/b2056a46-9aac-46bb-b570-85551c0dc4f7n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/b2056a46-9aac-46bb-b570-85551c0dc4f7n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" g

Re: [datameet] Re: Delhi draft DP pdf turned into a map (and you can do it too)

[Nikhil VJ]

Hi All,

Thanks for the +1s.
Got an input about making it printable. So, did it.

[image: print.png]

Click the print icon on left sidebar to open a page where you can adjust
various settings and take a printout.

Only the map portion will be printed. You can increase the height/width
pixels and then print to A0/A1/A2 size etc to take large high-res prints.
(and I have no idea what setting makes the perfect print, if you find out
then pls share.)
Background tiles : I've used from leaflet providers.
<https://leaflet-extras.github.io/leaflet-providers/preview/> Pls lookup
legalities yourself.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sun, Jul 18, 2021 at 12:52 AM Ujaval Gandhi 
wrote:

> Nikhil and Rajeswari - really impressive work! Thanks for sharing the
> process for GeoPDF extraction and hosting them on a nice static map. Will
> definitely be pointing people to your repo.
> [image: Logo] <https://spatialthoughts.com/>
> Ujaval Gandhi
> Spatial Thoughts
> mobile: +91-8095684687
> email: uja...@spatialthoughts.com
> [image: LinkedIn icon] <https://www.linkedin.com/in/spatialthoughts/>  [image:
> Twitter icon] <https://twitter.com/spatialthoughts>
>
>
>
> On Sat, Jul 17, 2021 at 7:12 PM Ganesh Babu  wrote:
>
>> This is a very important missing part in the existing information
>> dissemination and feedback collection components of master plan
>> development in Indian cities. I am planning to try something similar for
>> Chennai when they eventually release master plan 3.0
>>
>> In case you aren't aware, a bunch of like-minded non-governmental
>> organisations are working on a campaign called Main Bhi Dilli
>> <https://www.mainbhidilli.com/>  to make planning in Delhi more
>> representative and inclusive by engaging citizens in the 2041 Master Plan
>> process. I think your online tool can be a valuable component of their
>> efforts too. Potentially their combined resources can help tackle some of
>> the issues you raised in the input collection component. Their contact
>> information is on the website, in case you can't reach them, I can put you
>> in touch with one of the organisers if you are interested.
>>
>> Fantastic work again!
>>
>> On Thu, 15 Jul 2021 at 20:51, Arun Ganesh  wrote:
>>
>>> This is fantastic work Nikhil! Love how it's parametrized for easy
>>> configuration.
>>>
>>> The backend part for citizen inputs is especially interesting and would
>>> really complete the whole loop for so many such geospatial applications.
>>> The moderation of such content is definitely a pain and something i keep
>>> thinking about. It may be helpful just to have it hosted as a demo for the
>>> platform.
>>>
>>> On Thu, Jul 15, 2021 at 2:29 PM Naraina Damle 
>>> wrote:
>>>
>>>> GOOD
>>>>
>>>> On Tuesday, July 13, 2021 at 8:41:14 AM UTC+5:30 nikh...@gmail.com
>>>> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> Sharing about a recent collaboration I did with Rajesvari Parasa to
>>>>> turn a development plan PDF into an web based interactive map. Spinoffs
>>>>> from here may be useful to the data community.
>>>>>
>>>>> Here's the output:
>>>>> https://draftmpd41.github.io/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Sharing what went into it:
>>>>>
>>>>> 1. Extracting from PDF
>>>>> - It's one of those PDFs exported from Autocad or so where you can see
>>>>> multi-layers and turn them on or off.
>>>>> - Rajesvari used a command-line GIS tool called "ogr2ogr
>>>>> <https://duckduckgo.com/?t=lm&q=ogr2ogr&ia=web>" to extract
>>>>> individual layers from the PDF into .geojson shapefiles.
>>>>> - There was a tricky bit where the output lat-longs were all like
>>>>> (716882.20,3166456.28) - a different CRS and we didn't know which.
>>>>> - I used http://projfinder.com/ to find it, then Rajesvari included
>>>>> CRS conversion into the ogr2ogr commands to produce proper shapefiles.
>>>>> - You can see all that in this repo:
>>>>> https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041 -
>>>>> see the .bat file for the ogr2ogr commands
>>>>>
>>>>>
>>>>> 2. Poor person's GIS system
>>>>> - I cobbled together a static website (html + vanilla javascript) that
>

Re: [datameet] Karnataka - Administrative boundaries by KSRSAC.

[Nikhil VJ]

Wow, amazing step forward by Karnataka govt, kudos to the folks who pushed
to make this happen!

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Jul 30, 2021 at 10:11 AM H. S. Sudhira  wrote:

> Dear all,
>
> I am sure most of you have learnt that KSRSAC has put out the
> administrative boundaries of Karnataka in SHP and KML file formats. In case
> you missed it, do check it out here:
> K-GIS (karnataka.gov.in)
> <https://ksrsac.karnataka.gov.in/kgis/downloads.aspx>
>
> Please see related posts here:
>
> https://www.linkedin.com/posts/ksrsac-gok_ksrsac-kgis-gis-activity-6825748090587361280-vb2K
> https://twitter.com/ksrsac_gok/status/1419981543557144582
>
> A key question is on the licensing of this data. Ideally, it should be
> under the Open Government Data License (as per NDSAP). Hope they mention it
> explicitly.
>
> Regards,
> Sudhira.
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CALhej%3D%3D_DMNhC9ejkQjMXi0eq2tBYPoQJypkuODJKtb0_7KMYw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CALhej%3D%3D_DMNhC9ejkQjMXi0eq2tBYPoQJypkuODJKtb0_7KMYw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOSARtzWNJQ4UFODjrF-HdgKRHyVK9KKVCDvd7xibZAJQ%40mail.gmail.com.

Re: [datameet] Re: Pune Villages Map

[Nikhil VJ]

Hi Rushikesh,

https://mrsac.gov.in/MRSAC/map/map
Maharashtra Remote Sensing Application Centre (MRSAC) has published most
district->talukas and taluika->villages as geo-pdfs.

And you could use..
https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Jul 28, 2021 at 1:16 PM Nikhil VJ  wrote:

> Hi Rushikesh,
>
> Lots of data but zero guarantees, pls compare with official sources and
> verify at your own end before proceeding. We worked on this a few years ago
> and have left it here.
>
> https://github.com/datameet-pune/datameet-pune.github.io/wiki/Maharashtra-Villages-mapping
>
> --
> Cheers,
> Nikhil VJ
> https://nikhilvj.co.in
>
>
> On Wed, Jul 28, 2021 at 12:00 PM Sharad Lele 
> wrote:
>
>> Rather:
>>
>> https://www.cse.iitb.ac.in/~pocra/MahaCensus_shapefile_data1.2/MaharashtraCensus.html
>>
>> On Wednesday, July 28, 2021 at 12:00:06 PM UTC+5:30 Sharad Lele wrote:
>>
>>> Check out:
>>>
>>>
>>> https://www.cse.iitb.ac.in/~pocra/MahaCensus_shapefile_data1.2/Attribute.html
>>>
>>> On Wednesday, July 28, 2021 at 2:14:20 AM UTC+5:30
>>> rushikesh...@gmail.com wrote:
>>>
>>>> Hi, does anyone have pune villages shapefile or maharashtra villages
>>>> shapefiles ?
>>>>
>>>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/5d4929e4-f513-450e-986e-c42725b6c0d1n%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/5d4929e4-f513-450e-986e-c42725b6c0d1n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuObAa179pB%3D8eGB2JYZNVTin7N1Bax%2BfEftpW4T1CJ7LQ%40mail.gmail.com.

Re: [datameet] Re: Pune Villages Map

[Nikhil VJ]

Hi Rushikesh,

Lots of data but zero guarantees, pls compare with official sources and
verify at your own end before proceeding. We worked on this a few years ago
and have left it here.
https://github.com/datameet-pune/datameet-pune.github.io/wiki/Maharashtra-Villages-mapping

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Jul 28, 2021 at 12:00 PM Sharad Lele  wrote:

> Rather:
>
> https://www.cse.iitb.ac.in/~pocra/MahaCensus_shapefile_data1.2/MaharashtraCensus.html
>
> On Wednesday, July 28, 2021 at 12:00:06 PM UTC+5:30 Sharad Lele wrote:
>
>> Check out:
>>
>>
>> https://www.cse.iitb.ac.in/~pocra/MahaCensus_shapefile_data1.2/Attribute.html
>>
>> On Wednesday, July 28, 2021 at 2:14:20 AM UTC+5:30 rushikesh...@gmail.com
>> wrote:
>>
>>> Hi, does anyone have pune villages shapefile or maharashtra villages
>>> shapefiles ?
>>>
>>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/5d4929e4-f513-450e-986e-c42725b6c0d1n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/5d4929e4-f513-450e-986e-c42725b6c0d1n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMNCeGSDp29xFGBZjXho-9Aq2QSEFzaFVAM4kQRniWM1Q%40mail.gmail.com.

[datameet] Delhi draft DP pdf turned into a map (and you can do it too)

[Nikhil VJ]

Hi All,

Sharing about a recent collaboration I did with Rajesvari Parasa to turn a
development plan PDF into an web based interactive map. Spinoffs from here
may be useful to the data community.

Here's the output:
https://draftmpd41.github.io/




Sharing what went into it:

1. Extracting from PDF
- It's one of those PDFs exported from Autocad or so where you can see
multi-layers and turn them on or off.
- Rajesvari used a command-line GIS tool called "ogr2ogr
<https://duckduckgo.com/?t=lm&q=ogr2ogr&ia=web>" to extract individual
layers from the PDF into .geojson shapefiles.
- There was a tricky bit where the output lat-longs were all like
(716882.20,3166456.28) - a different CRS and we didn't know which.
- I used http://projfinder.com/ to find it, then Rajesvari included CRS
conversion into the ogr2ogr commands to produce proper shapefiles.
- You can see all that in this repo:
https://github.com/draftmpd41/layers_draft_delhi_master_plan_2041 - see the
.bat file for the ogr2ogr commands


2. Poor person's GIS system
- I cobbled together a static website (html + vanilla javascript) that
mimics those geoserver-driven multiple-layers-groups sites like we see at
Bhuvan and various municipal corporations' GIS portals.
- Code: https://github.com/draftmpd41/draftmpd41.github.io
- Parameterised the whole thing: By editing a small JS file and CSV table,
you can repurpose this to make it yours.
- The site is static, as in: no backend. Can run it anywhere.
- Loads layers only when user clicks on them first time. So, site loads
quickly even if data is heavy.
- Disclaimer: if you don't want your site visitors grabbing all the data,
then please go the geoserver way. This one's for open sharing (F12 >
Network > XHR)


3. Developed but not deployed: Citizens inputs layer
- Here, I added in a citizens' inputs form to capture geo-located comments.
The comments go to a backend program where they're saved in a basic MySQL
DB.
- We haven't activated this because there aren't any people on board to
admin the thing (moderation, spam prevention etc - nightmare).
- But the setup is simple and can be useful for others who want to do this,
so sharing it.
- Frontend with additional section and inputs layer:
https://github.com/draftmpd41/frontend-with-inputs
- Backend code: https://github.com/draftmpd41/draftmpd41_backend (in
python3)
- It's setup such that frontend and backend can be hosted separately.
- Disclaimer. It's simple and basic. No user accounts or spam prevention.
Moderation would have to be through phpMyAdmin on the backend server.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuO9FPhfgNCMvBgAhqaw-74CNAkKRjNpDV3Me6TXzSHsYw%40mail.gmail.com.

Re: [datameet] IT Rules

[Nikhil VJ]

Hi,

Can we get links to the actual text of the rules, any detailed explanation
to them? Better to be informed by source than by media one-liners.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Jun 10, 2021 at 1:17 PM Thejesh GN  wrote:

> Friends,
>
> Our friends at Hasgeek are doing a survey.  I think it's highly relevant
> for all of us. Please participate.
>
> _
>
>
> Privacy Mode and Hasgeek are conducting a survey with developers about
> their awareness and opinion about the IT Rules. As a developer or someone
> who runs a community, do you view these Rules as impacting your work? Do
> you have concerns/questions about these Rules and other government
> policies?
>
>
> Take the survey from
>
> https://www.surveymonkey.com/r/ITRulesSurvey
>
>
> The survey will take 5-8 minutes to complete.
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAABnYsVDdTmrrOQGsx_%2Brw%3D1OBNO9Vo%3DnXK3ka4FsRYbHPGopA%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAABnYsVDdTmrrOQGsx_%2Brw%3D1OBNO9Vo%3DnXK3ka4FsRYbHPGopA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOhp6MmCZHW%3DDud9ULR4n75bkOTX2aWsNoyyUq1k6eA1w%40mail.gmail.com.

Re: [datameet] Help with Indian Time Use Survey 2019 Dataset

[Nikhil VJ]

Hi Jagannath,

What is this about? whose time use?
Please give some explanation that will work for a general audience that
isn't in your specific field.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, May 26, 2021 at 5:29 AM rjagann...@gmail.com <
rjagannath1...@gmail.com> wrote:

> Hi Datameet,
>
> I had a question about the Indian TUS 2019.
>
> The 2019 data has indicator variables to understand whether multiple and
> simultaneous activities were conducted in a particular time slot. I noticed
> that in a given time slot where multiple activities were conducted, there
> is only information about the major activity, with values being missing
> for the other activities (as you can see in the screenshots below).
> [image: unnamed.png]
> [image: unnamed (1).png]
> This is how I am interpreting the missing values:
>
>- If  *multiple activity = Yes* for an activity, all subsequent
>activities in the time slot are considered to be multiple activities
>- If *simultaneous activity = Yes*, *all *subsequent activities in the
>time slot were performed simultaneously with it
>- If *simultaneous activity = No*, *none *of the subsequent
>activities in the time slot were performed simultaneously
>
> *This means that it is not possible for some activities conducted in a
> particular time slot to be simultaneous while some are not.*
>
> I wanted to confirm whether our understanding is correct since I have not
> come across any documentation regarding this.
>
> Best,
> Jagannath R
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/ec6c5c56-0e8b-4ed6-a106-ff197024f590n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/ec6c5c56-0e8b-4ed6-a106-ff197024f590n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOT8UoTZip%2BqcA2x8UDqq19b%3Dj-C-41Fbb-nrrwSDkPwA%40mail.gmail.com.

Re: [datameet] Extract district wise rainfall prediction data from NWP site

[Nikhil VJ]

Hi Abhishek,

Right-click on page -> Inspect -> opens the browser console -> Go to
Network tab , then browse around the site and check out the comings and
goings between site and server.

I was able to get the data we're seeing in the website coming in my command
prompt (which means: one can capture this) with this simplified cURL
command:
curl 'https://nwp.imd.gov.in/blf/blf_temp/block.php' --compressed
--data-raw 'dis=22AMRAVATI'

How to get these district codes: inspecting one page before..
curl --compressed '
https://nwp.imd.gov.in/blf/blf_temp/dis.php?value=22maharashtra'

How to get the state codes: just scrape from the html of the page you
shared: https://nwp.imd.gov.in/blf/blf_temp/

Before my foray into python, I used to use notepad++ and libreoffice Calc
(their raw text import dialog leaves excel in the dust) to separate out the
data I needed from the html tags etc. I'd use spreadsheet formulas to even
generate command-prompt commands in bulk. Fun times. For limited jobs,
there's things you can do there in a few mins which would take hours of
coding.

Maybe you can set these up in Octoparse (i have no experience with that as
I just roll my own code) or some other tools to get the data you need. All
the best!

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, May 27, 2021 at 3:32 AM a.ja...@gmail.com 
wrote:

> Hi,
>
> https://nwp.imd.gov.in/blf/blf_temp/
>
> In the link above, we can see the 5 days weather prediction by districts -
> however, there are a lot of clicks that are required to reach any one
> particular district. Please help in identifying a method to collate all
> India Rainfall data in a single table.
>
> Tried to use Octoparse - however, either I am using it incorrectly or else
> it doesn't work in this site. any other source of the same data is also
> welcome.
>
> regards,
> Abhishek
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/00fa6267-9dcd-489a-b579-58be5d62b383n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/00fa6267-9dcd-489a-b579-58be5d62b383n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuM80boH2pdNrOkef3wQcTzFD9Yg0LVJaQL-SNr8pU_GbA%40mail.gmail.com.

[datameet] Data on lockdown deaths?

[Nikhil VJ]

Hi,

Is there anyone out there cataloguing deaths in India caused by lockdowns?
At some point we ought to ask which one kills more people, right?

Undiagnosed and untreated cancer, chronic illnesses, heart attacks,
malnutrition due to no money in pocket or due to unavailability of means to
get care to those that need it : these are caused or exacerbated by
lockdowns. Suicide and alcoholism owing to 21st century untouchability
practices as well (aka social distancing).
Aside from locking up the population in tiny houses with poor circulation,
and having everybody congregate in a few places during limited opening
hours for essentials - which are fantastic ways to... do what, who knows.
Also, here in Pune all flat owners have been hit with Rs.5k or more
"scrapping of concession" in property tax - Increased taxes along with
decreased or halted income will be another long term side-effect of
lockdown solutions passionately advocated by people who only have hammers
in their toolboxes.

Anybody keeping counts on lockdown induced deaths? Or is the maniacal
PCR-positive numbers chase months after even WHO (and scores of sensible
doctors before them) had announced that they're unreliable and must be
accompanied with clinical diagnosis confirmation to qualify as an actual
case, the only game in data-town?

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMJj3oh_0h2pUWLAWHSrt96qoYeXE19cVF-a1aHaAiCQA%40mail.gmail.com.

Re: [datameet] Routing challenge: Covery every path optimally

[Nikhil VJ]

Thank you Julien, Ujaval, Dilawar for your excellent responses.

It's a relief to learn that this is not one of those "will take ages to
compute" ones.
Fascinating, but still hauntingly just beyond my technical reach.

Dilawar : talk is cheap : Agreed! Here's some sample data:
https://files.nikhilvj.co.in/routing/pune_peth1.gpkg

It's a peth (old city) area in Pune. Screenshot:


Created using HOT Export tool:
https://export.hotosm.org/en/v3/exports/26662965-94cb-4f38-a476-d52ecc159d8d

Desired output: .geojson Polyline shape (or equivalent) that traverses the
whole area.

Possible variants:
- 1 surveyor only
- N surveyors

(if N is difficult then ditch it and let's do 1 surveyor only; can make
multiple grids as per Ujawal's recco.)

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, May 12, 2021 at 1:21 AM Dilawar Singh 
wrote:

> Solutions to such problems can be found in operational research literature.
>
> Have a look at min-flow max-cut problems in graph theory which are suited
> to solve this kind of optimization problem (
> https://networkx.org/documentation/stable/reference/algorithms/generated/networkx.algorithms.flow.minimum_cut.html).
> This problem can be converted into an instance of this problem after
> suitable transformations. Another approach could be finding a balanced
> partition of a graph (gomory-hu etc).
>
> I don't see a computational challenge here since the size of the problem
> is likely to be a few hundred places and thousands on possible routes.
> Maybe a brute force algo with do the job as well.
>
> PS: Talk is cheap! If you show me the data, I can show you the code.
>
> Dilawar Singh, Ph.D.
> LinkedIn <https://www.linkedin.com/in/dilawar-singh-ph-d-44b81b194/> ORCID
> <https://orcid.org/-0002-4645-3211> Github
> <https://github.com/dilawar>
>
>
> On Tue, May 11, 2021 at 6:14 PM Ujaval Gandhi 
> wrote:
>
>> A colleague of mine solved a similar problem using the OpenRouteService
>> API. They used k-means clusters to determine the initial distribution of
>> points between surveyors and the optimization API
>> <https://openrouteservice.org/dev/#/api-docs/optimization/post> for
>> solving optimal routing between the points.
>>
>> Another less glamorous but maybe a more practical solution: Overlay a
>> grid and count the length of roads inside each grid. Assign grids to each
>> surveyor. You can add distance from starting point in the calculation as
>> well. I have run field operations before, and a grid-based approach is
>> usually more manageable than a complex 'start here and walk the streets in
>> this order'.
>>
>> Good luck!
>> [image: Logo] <https://spatialthoughts.com/>
>> Ujaval Gandhi
>> Spatial Thoughts
>> mobile: +91-8095684687
>> email: uja...@spatialthoughts.com
>> [image: LinkedIn icon] <https://www.linkedin.com/in/spatialthoughts/>  
>> [image:
>> Twitter icon] <https://twitter.com/spatialthoughts>
>>
>>
>>
>> On Tue, May 11, 2021 at 9:09 AM Nikhil VJ  wrote:
>>
>>> Hi All,
>>>
>>> Wishing everyone good health, stability and pragmatism in these times.
>>> I came across a certain technical problem statement pertaining to ground
>>> survey planning in a target area:
>>>
>>> Given X ground surveyors,
>>> Create X routes that start from one location,
>>> Cover all the existing roads and pedestrian pathways in the target area
>>> (obtained from OpenStreetMap data),
>>> Such that each path is walked over at least once.
>>> Balance the distance amongst the routes so that no one gets the brunt of
>>> the tasks.
>>>
>>> Variant 1: Multiple starting locations allowed.
>>>
>>> Reaching out to check if anyone has experience working this out? It
>>> seems like a common/recurring challenge that can use a common solution.
>>>
>>> I'm checking out OSMNX, but not finding a usable example yet over there.
>>>
>>> One idea: Plot a point at say every 50 meters along all the paths.
>>> Inspect and adjust manually at intersections etc. Then run a travelling
>>> salesman type algorithm on it to ensure that each point has been covered at
>>> least once.
>>>
>>> Another idea: Create a user interface to assist a person to work out the
>>> solution manually - make selections, plot the routes and see the result,
>>> tweak the selections and try again. Less glamorous but possibly more
>>> effective than chasing behind exotic algorithms.
>>>
>>> One base dataset required for such pr

[datameet] Routing challenge: Covery every path optimally

[Nikhil VJ]

Hi All,

Wishing everyone good health, stability and pragmatism in these times.
I came across a certain technical problem statement pertaining to ground
survey planning in a target area:

Given X ground surveyors,
Create X routes that start from one location,
Cover all the existing roads and pedestrian pathways in the target area
(obtained from OpenStreetMap data),
Such that each path is walked over at least once.
Balance the distance amongst the routes so that no one gets the brunt of
the tasks.

Variant 1: Multiple starting locations allowed.

Reaching out to check if anyone has experience working this out? It seems
like a common/recurring challenge that can use a common solution.

I'm checking out OSMNX, but not finding a usable example yet over there.

One idea: Plot a point at say every 50 meters along all the paths. Inspect
and adjust manually at intersections etc. Then run a travelling salesman
type algorithm on it to ensure that each point has been covered at least
once.

Another idea: Create a user interface to assist a person to work out the
solution manually - make selections, plot the routes and see the result,
tweak the selections and try again. Less glamorous but possibly more
effective than chasing behind exotic algorithms.

One base dataset required for such problems is: distance matrix. Another:
way to map on-road route between any two points, lots of times. I've got
those sorted out using OSRM, so no worries on that front.

Please forward this to colleges / students that might be looking for such
problem statements to take up. I can setup an official internship if
required.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuPsR63qjLvXzHw6iOkRCotXhw5jX175gKVJcks3sLPN9Q%40mail.gmail.com.

Re: [datameet] Setting up a similar COVID info portal as COVID Pune for Ahmedabad

[Nikhil VJ]

I would not want to clog their systems at this point.

In another thread on this group, this site has been shared:
https://stopcoronatn.github.io

Simple webpage (actually just a markdown readme), embedding a publish
output of google docs. I'll recommend this approach.
Whether the doc is being updated automatedly or manually, bottom line is
the google doc is v.easy to update.

You can fork it: https://github.com/stopcoronatn/stopcoronatn.github.io
Replace the iframe content and other links, and have it up for your side
any time. Please name your repo as "something.github.io" - that way when
you activate github pages in your repo's settings, it'll be directly
deployed on "something.github.io" url.

Keeping aside the technicals, do you have the data and people to update it?
That I feel needs to be the focus, and for that the other region's team
can't help.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Apr 22, 2021 at 10:30 AM Ravikant P  wrote:

> Hi,
>
> Can anyone help us get in touch with team handling covidpune.com?
>
> Best, Ravikant
>
> On Sat, Apr 17, 2021, 5:41 PM Ravikant P  wrote:
>
>>
>> Dear Kush,
>>
>> For technical aspects, could you get in touch with covid pune team? Most
>> probably it's the same team who has set up https://covidnashik.com/ as
>> well. Their technical team might have the bandwidth to set-up and maintain
>> a similar portal for Ahmedabad.
>>
>> I could not locate their contact phone number, but email is is mentioned
>> on the portal: feedb...@covidpune.com
>>
>> Best, Ravikant
>>
>>
>> On Sat, Apr 17, 2021 at 4:46 PM Khush Vachhrajani <
>> khush.vachhraj...@gmail.com> wrote:
>>
>>> Dear all,
>>>
>>> Hope you and your loved ones are well and healthy.
>>>
>>> Is there anyone in this community who is based in Ahmedabad and who is
>>> willing to work on a covid info portal for Ahmedabad on similar lines as 
>>> COVID
>>> Pune <https://covidpune.com/>?
>>>
>>> Please respond to this email or ping me on whatsapp - 9426465070. I do
>>> not have the technical abilities to build a portal but happy to coordinate
>>> with a team of volunteers in best of my capacity.
>>>
>>> Thanks,
>>> Khush
>>>
>>> --
>>>
>>> *Khush Vachhrajani*
>>>
>>> *National Resource Person*
>>>
>>> Social Accountability Forum for Action & Research
>>>
>>>
>>>
>>> Phone: 9426465070  Mobile: 8209119415
>>>
>>> Web: safar-india.org
>>>
>>> Email: khush.vachhraj...@gmail.com
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CANyX%2Bxdax-mnZusrdtG3F6ugf1ft9rcHebqKJ-6ohV1msF89Qg%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CANyX%2Bxdax-mnZusrdtG3F6ugf1ft9rcHebqKJ-6ohV1msF89Qg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CALt%2BO6wBbip2o%3D5k2V21VZm35kzZo6ybb-GeSTsRoi7Tj2Zq%2BQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CALt%2BO6wBbip2o%3D5k2V21VZm35kzZo6ybb-GeSTsRoi7Tj2Zq%2BQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNs6KR_3EZJhynXVuhe9BF_nmHh2PcZm5mFBGDxAyM1HA%40mail.gmail.com.

Re: [datameet] ANN: Opening of 7,00,000+ Rural Points of Interests Data

[Nikhil VJ]

Hi Harsh,

The PMGSY site is dizzyingly full of data! Kudos and gratitude to all the
people who have been working on it and the govt / elected officials who
supported its release to the public. Sets a great benchmark / precedent.

Even apart from the data itself, the hierarchy in the dropdown selects is
valuable too as people can use that for mapping so many other things in
other fields.

I'm not able to see geo-tagging in the sections I'm checking out. Can you
guide pls?

Suggestion : Make short screen recording videos on youtube showing how to
use the site. There's a lot of free tools and sites for it, but if zoom is
already there then one can start a call with recording on and screen-share
and do the job.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Mar 1, 2021 at 9:03 PM Arun Ganesh  wrote:

>
>>
>> Does anyone know how to get this data ported to OSM (if at all that's
>> possibility)?
>>
>>
> Importing data to OSM is possible, but since it will have to be conflated
> with any existing data, it will require quite a bit of data
> preparation with many volunteers. An example of a recent import was the
> data.gov.in health facilities dataset
> https://wiki.openstreetmap.org/wiki/India_Health_Facilities_Import . An
> overview of the import process is here:
> https://wiki.openstreetmap.org/wiki/Import/Guidelines . If the data
> quality is not consistent and requires manual cleanup, going for an import
> might be a lot of effort.
>
> That said, the PMGSY data is quite valuable and can add a lot of missing
> info into OSM for rural areas. It would make sense to start a conversation
> with the OSM community on ideas and how to take this forward. A good way to
> begin is by sending an intro email to the mailing list
> https://lists.openstreetmap.org/listinfo/talk-in and starting a
> conversation on it on the telegram group https://t.me/OSMIndia . There
> are quite a few folks experienced with OSM imports who can help out.
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CA%2BGKQr2BRKc619HFpEMffimJHuEgrdSnRJUg6ekW%3Dup368P5Bw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CA%2BGKQr2BRKc619HFpEMffimJHuEgrdSnRJUg6ekW%3Dup368P5Bw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuPO4Xu-PCapZ75a7FY_v%3D8W4-H9z7z%3DdSO0jfZcm2oWgw%40mail.gmail.com.

Re: [datameet] Different type of geojson?

[Nikhil VJ]

Hi Alpha,

The file is loading just fine on mapshaper.org, and looking at the raw text
it seems fine.

Regarding further technicalities of your project, gis.stackexchange.com may
be a more in scope forum.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Feb 19, 2021 at 10:47 AM Alpha Beta  wrote:

> So I'm trying to use d3 to draw a map of geojson. But the geojson is not
> quite as listed in the examples. Is there a functional way you know of to
> draw the paths. The geojson file I'm trying to use it :
> https://projects.datameet.org/indian_village_boundaries/ga/
>
> You can download directly from here:
> https://github.com/datameet/indian_village_boundaries/raw/master/ga/ga.geojson
>
> I can't seem to get the map plotted
>
> The geojson fiile in this case is not as simply listed as the examples. Do
> I have to create a new geojson? if so how? Or do I use some d3 functional
> method to draw the paths?
>
> Because I intend on allowing the user to select to draw the boundaries
> just for the districts or only sub-districts etc.. all while keeping the
> outer boundary of the whole map of goa in constant display.
>
> Basically allowing the user to click throught the hierarchy of displaying
> first District -> sub-district -> etc... and backwards.
>
> Regards
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/96bd65ce-e7f6-484c-9932-dd56e8db2676n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/96bd65ce-e7f6-484c-9932-dd56e8db2676n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuN%2BgHRpus0%3D%3DC%3DYAoVy83YZpKR0Ygp2JdPh147UhWpDqA%40mail.gmail.com.

Re: [datameet] Query regarding Linking CSV tables for Pipeline Route alignment in QGIS

[Nikhil VJ]

Hi Jasvinder,

Some sample data may help. People working in other fields may not know what
you mean by your specific terms but may know the process that you need to
do.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Jan 11, 2021 at 4:38 PM Jasvinder Singh 
wrote:

> For a Pipeline Alignment I have TP (Turning point Data - Coords &
> Chainages linked to TP numbers) in one Table.
> In another Table I have certain road crossings with data columns
> indicating crossing laying between TP numbers and also intermediate
> chainage between those two TP's.
> How can I get coords for the road Crossings.
>
> Newbie here in the field of GIS.
>
> Regards,
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/3c31a9fd-34c2-4edd-a441-92ba64f441cdn%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/3c31a9fd-34c2-4edd-a441-92ba64f441cdn%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuPyV-3C2wNSuOXH00D0NKTiyERZ4UXsm3DoDaTy-10tEg%40mail.gmail.com.

[datameet] What is Covid RT-PCR test's Ct (Cycle Threshold) limit in India?

[Nikhil VJ]

Hi All,

I wanted to know if the Indian government or related agencies have set any
standard limit in a crucial setting in the Covid tests called "Cycle
Threshold".

Are the folks collecting stats on covid cases - having any data on cycle
values for those cases?

Have you or someone you know been diagnosed as a Covid-19 case? What was
the cycle value in your test result? Does the lab that did the test share
this data?

Sharing an article and some excerpts from it:

https://www.msn.com/en-us/health/medical/experts-us-covid-19-positivity-rate-high-due-to-too-sensitive-tests/ar-BB18wE8B
Experts: US COVID-19 positivity rate high due to 'too sensitive' tests

"With a cutoff of 35, about half of those tests would no longer qualify as
positive. About 70 percent would no longer be judged positive if the cycles
were limited to 30.
In Massachusetts, from 85 to 90 percent of people who tested positive in
July with a cycle threshold of 40 would have been considered negative if
the threshold were 30 cycles, Mina said. "
"The Food and Drug Administration said that it does not specify the cycle
threshold ranges used to determine who is positive and 'commercial
manufacturers and laboratories set their own.'"
"The CDC said its own calculations suggest its extremely hard to detect a
live virus in a sample above a threshold of 33 cycles. "


This was one - there's many more if I search for "RT-PCR test cycle
threshold value covid" on duckduckgo. (tip: google search is broken when it
comes to anything controversial. Proverbial case of overprotective mother
suffocating the child in the quest to protect it. Take the same query and
run it in Bing, Duckduckgo etc also.)

Looking into this I'm seeing an analogy with vectorizing raster satellite
imagery : Your software can easily fill the whole thing up with false
positives, or can produce no result at all. Lot of fine tuning is required
to get the "perfect setting" that minimizes the false positives and false
negatives, and you often never reach a perfect setting that didn't have any
mistakes. It's not a hard yes/no thing. You invariably need manual
intervention (and even with AI interventions we're seeing problems), and it
frustrates the hell out of people who assumed this technology thing is a
silver bullet.

Inviting people with better knowledge on this topic to correct me: My
understanding is that there is an exponential (maybe doubling, maybe some
other factor) change from one Ct value to the other. To go from 33 to 40,
would be.. well, non-trivial.

So one set of data needed is : What are the Ct limits being used in current
testing? Is there a single value standardized by the government? If there
is variation, then who decides?

Another set of data that can be just as useful : Have these Ct limits been
changed since the pandemic began almost a year ago? How have they changed,
and is there any co-relation between that and the Covid+ case counts? Is it
possible to "explode" / "reign in" a pandemic by merely altering this
setting without any ground level realities changing? If yes, then why is
talk about it missing from the mainstream discourse?


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuM%2BFh7nsNGzaF5PkTxR-VUA%2B9vce7%3DwCRWcxiVnKYsKQQ%40mail.gmail.com.

Re: [datameet] Guidance on digitizing boundaries

[Nikhil VJ]

Hi Gufran,

I had done georeferencing in QGIS of Pune's latest (2017) ward maps when
they were published:
https://github.com/datameet-pune/datameet-pune.github.io/tree/master/maps

It's been used by lot of folks since then as I guess official digital
shapefiles were never published though internally the corporation does seem
to be using their own. There is an obvious disclaimer that it was manually
done by volunteer and I've included the original maps and links for
referencing - use at own risk, no guarantees. Corrections in case anything
is wrong are most welcome.

I've found three ways to do the first crucial step of *geo-referencing* the
pdf / image map:

There is one traditional way there which involves more work, but has lot
more options like if your pdf/image is relatively distorted in proportions
compared to actual map then you can do things to rectify. Sharing its link:
https://www.qgistutorials.com/en/docs/3/georeferencing_basics.html
This involves clicking and marking common points between your image/pdf map
and the actual map.

There is another quicker plugin in QGIS I had found later, which uses a
different and much quicker approach, but works well only if your pdf/jpg is
already good in proportions and doesn't have too much distortion.
https://gvellut.github.io/FreehandRasterGeoreferencer/
This involves "floating" the pdf/image over the map and resizing, rotating
it till it fits properly.

Then, there is a website called mapwarper that uses the the same approach
as first solution but more streamlined (albeit with fewer power options):
http://mapwarper.net/
Once the "warping" onto map is done, you can download the georeferenced TIF
and load it on QGIS. Or you can publish your map, copy the tile layer
string from the export section and import in QGIS as a XYZ tiles map layer.

Once the georeferencing is complete, you have a image layer properly fixed
to lat-long map, which moves and zooms with the map. Now you can create an
empty polygons shapefile and start manually tracing the boundaries.

Tips from digitizing experience:
1. It's good to use Snap settings to make your boundaries snap to each
other properly so you don't leave out slivers of no-mans-land or overlaps
between wards. I found 20 pixels snap to be good.
2. These ward boundaries oftentimes follow along canals and roads. So, you
can add a map satellite view background layer and keep it semi-transparent
to assist you.
3. Bring in any other boundaries file (like district etc) which might help
you start from the outside.
4. If using the common points method, try to have a good spread of points.
Avoid keeping any 3 consecutive points in a line, as small errors can get
magnified. Crossroads/chowks are good places to pin.

Some links and video online that can guide through this process:
https://www.qgistutorials.com/en/docs/digitizing_basics.html
https://www.cadlinecommunity.co.uk/hc/en-us/articles/36291898-QGIS-How-do-you-Snap-and-Trace-when-Digitising-
https://www.youtube.com/results?search_query=digitizing+map+boundaries

All the best, and do share when done - it'll be good to see more ward
boundaries come up from MH!

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Dec 28, 2020 at 10:30 AM Digvijay Bendrikar Shinde <
digvijaybendrikarshi...@gmail.com> wrote:

> Thanks Ar Akshit,
>
> MRSAC don't have corporation ward wise boundaries PDF, they have revenue
> village and outer boundaries of towns and cities... But within the cities
> details are Missing.
>
> Thanks about Karnataka link, will check.
>
> Regards
> Digvijay
> (PhD scholar at CTARA, IIT Bombay)
>
> On Sun, Dec 27, 2020, 4:26 PM Akshit Shah  wrote:
>
>> Hi Gufran and Digvijay,
>>
>> You can get the pdf copy of maps from the following portal:
>> https://mrsac.gov.in/MRSAC/map/map
>>
>> you can either import pdf in AutoCAD and separate the admin boundary
>> layers > georeference them > and convert them to shapefile. your
>> digitisation effort will be saved if the pdf has vector elements in it.
>>
>> for Karnataka State there is a portal where you can find Admin boundary
>> upto Town, wards and village level:
>> https://kgis.ksrsac.in/kgis/downloads.aspx#
>>
>> On Sun, Dec 27, 2020 at 7:08 AM Gufran Pathan 
>> wrote:
>>
>>> Hi team,
>>>
>>> I am trying to digitize ward boundaries for the city of Akola
>>> (Maharashtra) - create shape files so that I can visualize them along with
>>> associated data. I'm reaching out to get some guidance on how one would go
>>> about doing this? What does the process look like at a high-level? Does one
>>> get physical / PDF copies of the map with boundaries and then 'digitize'
>>> it? What tools are typically used in the digitization pro

Re: [datameet] Discord

[Nikhil VJ]

Are there distributed alternatives to emails which has better support of
structured knowledge management?


*Discourse* forum? https://www.discourse.org/
I've seen some pretty good networking happening on this - ODK Forum
<https://forum.getodk.org/> (for mobile based survey tools) is one example.
It's open source with lot of open integrations
<https://www.discourse.org/integrations> and active coding community
working on it. I've set a test one up recently without too much trouble on
a 4GB hetzner server that costs under rs.1k/mo. You need to map your domain
to it and have smtp email working  - I was able to do it with a bigrock.in
domain. There would be scaling up challenges of course, but can be handled.
(more usage = more capability to ramp up). Search works well.


--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sat, Dec 5, 2020 at 1:13 AM Dilawar Singh 
wrote:

> Slack is not an ideal platform for collecting open knowledge or data. In
> addition to what Thajesh has already mentioned, slack is a vendor lock-in
> which should be a big NO. Moreover, If I am not wrong, they don't let you
> browse post 10k messages in the free plan. Personally, I can live without
> one less account.
>
> Not to mention that there is a non-zero probability they will be around
> after 5 or 10 years. The same argument applies to any centralized platform.
> Are there distributed alternatives to emails which has better support of
> structured knowledge management?
>
> best,
> Dilawar Singh, Ph.D.
> LinkedIn <https://www.linkedin.com/in/dilawar-singh-ph-d-44b81b194/> ORCID
> <https://orcid.org/-0002-4645-3211> Github
> <https://github.com/dilawar>
>
>
> On Sat, Dec 5, 2020 at 12:35 AM shaily sparsha 
> wrote:
>
>> Agreed for slack.
>>
>> On Fri, Dec 4, 2020 at 1:32 AM BNP Analytics 
>> wrote:
>>
>>> Agreed.
>>> A slack or discord channel will be good.
>>> Mainly as they may support better code sharing.
>>>
>>>
>>> On Tue, Dec 1, 2020 at 5:27 AM Akshay Aditya 
>>> wrote:
>>>
>>>> Just a message to everyone.
>>>> Instead of the usual format of emailing on the group or commenting on
>>>> the groups. Should we consider doing this on discord?, we would be able to
>>>> share information more quickly and threads would be easier to follow.
>>>> Sometimes I’m looking for something and it’s a little hard to find them
>>>> because of the email format.
>>>>
>>>> Let me know what you all think.
>>>>
>>>> --
>>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>>> about us by visiting http://datameet.org
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "datameet" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to datameet+unsubscr...@googlegroups.com.
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/datameet/250EDAA2-4F8B-4498-A481-6F141700A5A5%40gmail.com
>>>> .
>>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CAFJrGLGwg0zRONi%2BoEeLFz96DTJOSsw1ZbfTf2qyfr%3DV-a2j3Q%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CAFJrGLGwg0zRONi%2BoEeLFz96DTJOSsw1ZbfTf2qyfr%3DV-a2j3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>>
>>
>> Regards,
>> SHAILY SPARSHA
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAHnqzP0K0sTiS60GU5G5EDY21-_SZrRWY_9NsQfsvfQndAeZUw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAHnqzP0K0sTiS60GU5G5EDY21-_SZ

Re: [datameet] Electoral Rolls Karnataka - Request for resources

[Nikhil VJ]

Hi,

Just to update, I got in touch with Mr.PG and we have setup a workflow on a
cloud server and it's chugging along nicely.

What the program does is cool - it implements a python library: ocrmypdf in
bulk mode.

This description from their docs is what it's mainly doing:
OCRmyPDF adds an optical character recognition (OCR) text layer to scanned
PDF files, allowing them to be searched.

I made some tweaks to PG's program, have put it on github here:
https://github.com/answerquest/bulk_pdf_OCR/

I think it may be useful at other places too.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Nov 24, 2020 at 12:59 AM Anirudh K 
wrote:

> Hi all,
>
> The Chief Electoral Officer - Karnataka has published a new version of
> Electoral Rolls. These are image based PDFs that have to be converted to
> text based PDFs.
>
> There is a need for additional compute resources to convert these large
> files. If anyone would like help with this, the process would entail
> running a python script (already made) on Google Colab and sharing the
> output folder on Google Drive. A more technical description of the process
> is detailed below.
>
> Please reach out to bhat...@gmail.com (or call PG Bhat - 9900141232) to
> help out with this project, or in case of any queries.
>
> The full process:
>
>1. Create a shared folder on Drive called 'ERMS' and give edit access
>to bhat...@gmail.com.
>2. He will create 3 subfolders:
>   - *Code* - This will contain the script. There is no need for any
>   software to be installed locally.
>   - *Image files* - This houses the image files
>   - *Text files* - where the script will write the results
>3. Run the script on Colab (free account). The text files can then be
>downloaded from the drive folder
>
> Thank you for considering this request.
>
> Regards,
> Anirudh
>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/be9e4621-03a6-4e7e-8dfd-51ab93478b4en%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/be9e4621-03a6-4e7e-8dfd-51ab93478b4en%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNTvXDhKG64jhwJ94GY6U_EVyxFupdtOnajbQfYmK_pqA%40mail.gmail.com.

Re: [datameet] Re: Data Scrapping from e-courts website

[Nikhil VJ]

Hi Peter,

Can you share a sample instruction (click this -> click that) or link on
how to reach a place on the website where we can see a listing under the
IPC code?


About KritiDev - do you mean KrutiDev?

There's converters available now to convert from legacy ascii fonts (where
we would use a custom font to make A's glyph look like one akshar and B
look like another akshar and so on) to unicode (where different languages
have their own char code and co-exist).

I found various websites on searching online for "hindi to unicode
converter", but also there's this open source collection of htmls contain
javascripts that I have used to work with earlier:
https://sites.google.com/site/hindifontconverters/files. Has simple web
page files with javascripts to do the conversions.

A budget document I was working with 5 yrs back had its own version of
legacy font - I hacked into one javascript here, added in new mappings and
customised my own converter.

Sorry to digress but just sharing in case the legacy font thing was being a
blocker to anyone. Also if someone wants to build a full solution out of
this that takes say word docs and converts to unicode without losing
formatting and can bring in some resources - let me know. I didn't have the
skills to programmatically work with office docs 5 yrs ago; I do now.

And there was one surprise finding related to this: I've found that legacy
fonts survive the journey through pdfs better than unicode. So if an
institution insists on sharing documents as pdf, I'd rather have them stick
to their old legacy fonts and use one of these converter tools at my end to
get the text out into unicode.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Nov 9, 2020 at 12:41 PM pmay...@gmail.com 
wrote:

> Hi Lovish,
> My experience (only for district courts in MP) is that scraping is *not*
> possible. It *is* possible to look for all cases in which a specific IPC
> offence is involved (e.g. 376(D) Gang rape). But to find out what happened
> in each case, you must go to each *seriatim*, check what decision was
> made by the court and--if you're lucky--access the judgement made in the
> case. In MP, those judgements are in Hindi, rendered in KritiDev.
> I've written a paper looking at some rape cases. Feel free to contact me
> directly.
> best wishes,
> Peter Mayer
> On Monday, November 9, 2020 at 1:35:09 PM UTC+10:30 Lovish Sharma wrote:
>
>> Hi,
>>
>> I am working as an associate for an NGO working in the field of crimes
>> against women. Currently I am doing research on crimes against women in
>> prominent cities. For that, I need to scrap the data from e-courts website, 
>> *https://districts.ecourts.gov.in/
>> <https://districts.ecourts.gov.in/>* .
>>
>> Kindly help me with that.
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/c9029026-4bf0-464b-ada8-6c4964911afen%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/c9029026-4bf0-464b-ada8-6c4964911afen%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOu34SqofuCuKkjLmoGvLzavpuYkE3UqgQ1xLbNSDsQJQ%40mail.gmail.com.

Re: [datameet] Data Scrapping from e-courts website

[Nikhil VJ]

Hi Lovish,

Is there any link on this site where you are able to navigate through
different data items? I only see pages where you have to input certain case
code etc.
Please share more details with directions and examples - links or
screenshots - of what you need.
Then maybe somebody might be able to help.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Nov 9, 2020 at 8:35 AM Lovish Sharma 
wrote:

> Hi,
>
> I am working as an associate for an NGO working in the field of crimes
> against women. Currently I am doing research on crimes against women in
> prominent cities. For that, I need to scrap the data from e-courts website, 
> *https://districts.ecourts.gov.in/
> <https://districts.ecourts.gov.in/>* .
>
> Kindly help me with that.
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/4c1b2005-8594-4afe-ba88-a8e7147a5798n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/4c1b2005-8594-4afe-ba88-a8e7147a5798n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOpmAiHPc_T63qHD%2BzGu3U9_N-MaQ6GhSvt612X6UDMBg%40mail.gmail.com.

Re: [datameet] I need WPF application with open source world map and it need to display location based up on lat and long values

[Nikhil VJ]

Hi Saikumar,

No idea what you meant by WPF.

If you want to create a browse-able map on a web page having a *map* as the
background and your data as the foreground layer, that you can then share
the link with others to see,
Take a look at leafletjs.com . It's a free JavaScript library using which
you can set up a map on a webpage having any free background like
openstreetmap, and show your data on top. Another one like it is
openlayers. These don't require any sign up anywhere and there are no usage
quota limits on JS side.

The background layer - you can choose which one to use. Some are free, some
require signing up and will charge if usage goes high. Check out this site
for free layers: http://leaflet-extras.github.io/leaflet-providers/preview/

If you're using QGIS software to compose your map, then there are plugins
to export it as webpage which you can then host online. See
https://plugins.qgis.org/plugins/qgis2web/
--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Oct 22, 2020 at 11:57 AM saikumar dandla 
wrote:

> Thank you so much for your valuable response.
>
> On Thursday, October 22, 2020 at 9:33:13 AM UTC+5:30 deeps...@gmail.com
> wrote:
>
>> This might Help.
>>
>> Here's a number of projects that might save you some work:
>>
>> OpenSource:
>>
>>-
>>
>> http://www.codeproject.com/Articles/238551/WPF-Map-App-WPF-meets-Google-Geocoding-Static-Maps
>>- http://xamlmapcontrol.codeplex.com/
>>- http://greatmaps.codeplex.com/
>>- http://wpfsharpmapcontrols.codeplex.com
>>
>>
>> On Thu, 22 Oct, 2020, 12:41 am saikumar dandla, 
>> wrote:
>>
>>> urgent please any one can suggest me best possible solution
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/0b655f4d-85e2-4b5a-8a8a-4f4175442808n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/datameet/0b655f4d-85e2-4b5a-8a8a-4f4175442808n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/cb2471f3-7401-4724-91c3-5e4403864b43n%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/cb2471f3-7401-4724-91c3-5e4403864b43n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuON8g10bt6GQ0wVfoFPubGESnvOMrrugecN-J_65yQjyg%40mail.gmail.com.

Re: [datameet] IMD rainfall data to csv file

[Nikhil VJ]

Hi Uday,

Great, giving the actual structure always helps. Now, do share how you get
this data from IMD so that others can get to it (or for their place / time)
too.

Generally to be able to work with such data we need to un-pivot it and
bring all the values into one column.

Towards that, I have created a basic unpivoting tool sometime back which
might help:
https://answerquest.github.io/table-unpivot.html

It requires properly formatted tables though, so you would have to do some
work on your data before you can use this. I won't be able to help
regarding that, maybe someone else can. I think the place name and lat-long
need to be brought in as columns too, so it won't be as simple as deleting
them off.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Tue, Sep 8, 2020 at 6:58 PM Uday Bhaskar Reddy 
wrote:

> Hi Nikhil,
>
> Attaching a sample file as suggested.
>
> Regards,
> Uday.
>
> On Tue, Sep 8, 2020 at 6:57 AM Nikhil VJ  wrote:
>
>> Hi Uday,
>> Do share all relevant links, sample data whenever possible.
>>
>> Regards
>> Nikhil
>>
>> On Tue, Sep 8, 2020, 01:30 Uday Bhaskar Reddy 
>> wrote:
>>
>>> Hi all,
>>>
>>> Could someone kindly share the easiest way to make a CSV file from the
>>> rainfall data shared by IMD in a text file. The problem is the data is
>>> divided into stations with the header containing the station name, location
>>> and coordinates not conforming to the format of the rainfall data.
>>>
>>> Thank you for your time,
>>> Uday.
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CAE9YNv%3DJH79LU%3DFFd%3D%3DYucQV9%3DAzEBwddA-gPQfb2Q8%2BZsk5%3Dg%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CAE9YNv%3DJH79LU%3DFFd%3D%3DYucQV9%3DAzEBwddA-gPQfb2Q8%2BZsk5%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAH7jeuMz3nh272Lcxj1DCvWGUQJCThKbG9qWMhazKf-92985Bw%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAH7jeuMz3nh272Lcxj1DCvWGUQJCThKbG9qWMhazKf-92985Bw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAE9YNvndfMjgkH4cw5Ga9cNx3A4vqDQ02mT%2B-xq8Z2U4XHLuXg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAE9YNvndfMjgkH4cw5Ga9cNx3A4vqDQ02mT%2B-xq8Z2U4XHLuXg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOJiGrHEqXjTsmHGC9AcBLSznz-bLMHTsrHKKJMhma0OQ%40mail.gmail.com.

Re: [datameet] IMD rainfall data to csv file

[Nikhil VJ]

Hi Uday,
Do share all relevant links, sample data whenever possible.

Regards
Nikhil

On Tue, Sep 8, 2020, 01:30 Uday Bhaskar Reddy  wrote:

> Hi all,
>
> Could someone kindly share the easiest way to make a CSV file from the
> rainfall data shared by IMD in a text file. The problem is the data is
> divided into stations with the header containing the station name, location
> and coordinates not conforming to the format of the rainfall data.
>
> Thank you for your time,
> Uday.
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAE9YNv%3DJH79LU%3DFFd%3D%3DYucQV9%3DAzEBwddA-gPQfb2Q8%2BZsk5%3Dg%40mail.gmail.com
> 
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMz3nh272Lcxj1DCvWGUQJCThKbG9qWMhazKf-92985Bw%40mail.gmail.com.

Re: [datameet] TN Election Form 20

[Nikhil VJ]

Hi,

I think data regarding carpentry should be shared even if it is carpenters
that are requesting it. Heck, the carpenters are likely the ones who can
supply the demand that encourages open carpentry data availability. Mode of
sharing can be encouraged to be kept public so that the other carpenters
can also access it and it isn't monopolized.

Example: Don't message it to that carpenter personally, put it on GitHub or
something.

In fact, we might have a situation where some rich carpenter has put in a
ton of money into acquiring that data already and is sitting pretty with a
monopolistic advantage. Sharing the same openly might just help level the
playing field between the competing carpenters. Heck, that's usually the
case.

Whatever the motivations of individual carpenters, increasing the sum total
of open carpentry data will always bring more benefits to the carpentry
profession and those affected by it moving forward. After all, haven't your
local bakery's cakes been getting tastier since recipes started being
shared openly?

Because the monopolizers most likely already have it and have had the
resources to acquire it long before any open data folks came on to the
scene, it's silly to imagine that little you or little me can in any way
hinder those folks by closing down access to everyone - that approach only
entrenches the monopoly and turns us into enablers of the monopoly.

If you agree in principle with the above then feel free to apply it to
other professions.

Regards
Nikhil VJ
https://nikhilvj.co.in

Full disclosure: no association with any carpenters ;)

On Mon, Aug 31, 2020, 18:05 deepak srinivasan  wrote:

> Hi Arul Prakasam
>
> You probably should put a full disclosure that you belong to DMK and
> possibly even share your use case.
>
> While free data sharing from open data sources needs to be encouraged
> amongst enthusiasts, sharing electoral data to members of political parties
> seeking to influence election results should not be supported
>
> regards
> deepak
>
> On Sun, Aug 30, 2020 at 7:39 PM Rtn Arul Prakasam T 
> wrote:
>
>> Hi Friends, I would like to analysis TN Election Result data so I need
>> 2016 and 2018 Form 20 data CSV format any have this data kindly support to
>> me
>>
>> Arul Prakasam T
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/95017783-c535-4eed-9831-36ed2ebbc339n%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/95017783-c535-4eed-9831-36ed2ebbc339n%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> regards
> deepak
> 9962962016
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAEcN9mFxwc7tj6J1Jm6Md1xoBmT5JvZ6hWTiABZkY9669OO1Rw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAEcN9mFxwc7tj6J1Jm6Md1xoBmT5JvZ6hWTiABZkY9669OO1Rw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuM4HGMMBs-gyW_xPnnCFvN%3DJGVbOeHpPNQR1jXUgoQyhw%40mail.gmail.com.

Re: [datameet] Help with R logic - near similar name

[Nikhil VJ]

Hi Ram,

I'm not sure about R, but if you have the list in an excel / csv then
OpenRefine can help you iron it all out in a jiffy. Check out this article
I've written that explains the flow for this particular task:
http://datameet.org/2018/06/13/openrefine-bus-stop/

OpenRefine is a tool made for non-coders to clean up messy data. Site:
https://openrefine.org/

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Aug 26, 2020 at 6:21 AM m...@ncf-india.org 
wrote:

> Hi Ram
>
> In addition to the helpful suggestions made above, here are some
> R-specific pointers:
> — stringr is an extremely helpful package with which to do most of the
> string manipulation actions (whitespace removal, tokenisation, regex
> matching) recommended above.
> — you may also need a package that helps you compute ‘distances’ between
> the strings you are comparing. stringdist is one such package. However,
> with Indian names, I found some of the phonetic distance algorithms
> (rogerroot, soundex) in the phonics package much more helpful.
>
> Hope this helps! Good luck!
> Madhu
>
> On Wednesday, 26 August 2020 at 00:48:45 UTC+5:30 sudat...@gmail.com
> wrote:
>
>> Hi Ram,
>>
>> Faced with similar issues, the following worked for me -
>>
>> 1. Make everything lower or upper case using tolower/ toupper
>> 2. Grep to match the common pattern of name
>>
>> Best,
>> Sudatta
>>
>> On Aug 25, 2020, at 7:52 AM, Rahul Gupta  wrote:
>>
>> Hi Ram,
>>
>> Not sure if there is something very similar to FuzzyWuzzy (Python) in R.
>> But you can try this link
>> https://astrostatistics.psu.edu/su07/R/html/base/html/agrep.html
>>
>> It is similar kind of approximate string matching. You can set your own
>> threshold criteria and filter data accordingly.
>>
>> On Tue, 25 Aug, 2020, 8:09 pm rammano...@gmail.com, 
>> wrote:
>>
>>> Hi,
>>>
>>> I have collected hospital data from multiple sources. However, each
>>> source have different name. Trying to clean list with no duplicates. I am
>>> using R and couldn't resolve with stringdist_join . Appreciate you
>>> suggesting some approach.
>>>
>>> For example, Guntur (A.P) is listed with following names. Can we mark
>>> (or eliminate) duplicate?
>>>
>>> Example 1
>>> SANKARA EYE HOSPITAL(GUNTUR)
>>> SANKARA EYE HOSPITAL
>>> SANKARA EYE HOSPITAL ( A UNIT OF SRI KANCHI KAMA KOTI MEDICAL TRUST)
>>>
>>>
>>> Example 2
>>> ASHIRWAD HEART HOSPITAL ( GHATKOPAR )
>>> Ashirwad Heart Hospital
>>> ASHIRWAD HEART HOSPITAL ( GHATKOPAR )
>>> Ashirwad Heart Hospita-Ghatkopar
>>>
>>> Thanks
>>> Ram
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+u...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/19ee8101-84ec-42b0-974a-43035b5902f1n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/datameet/19ee8101-84ec-42b0-974a-43035b5902f1n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+u...@googlegroups.com.
>>
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAKxLuZeB5_2K4Td%3DP8-_AjFob9Wp2Vc9jic649HD%2BV1itEpYfg%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAKxLuZeB5_2K4Td%3DP8-_AjFob9Wp2Vc9jic649HD%2BV1itEpYfg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> htt

Re: [datameet] Covid-19 death records

[Nikhil VJ]

Hi,

The hard way, but also the easy way, may be to collect data from people
themselves, with all the obvious disclaimers brought in.
Towards that, setting up a data collection site might be a good prospect.
There, relatives of the deceased who *want* their stories to be shared in
the hopes it may help others, can post and declare what they're sharing it
as open data.

Additionally, I'd also really like to see data collection from Covid-19
survivors. Particularly, what helped in their case, in whatever level of
detail they want to share.

There has been too much dirtying of the waters by official orgs. WHO is,
I'm very sorry, not the most reliable source. The unipolar credibility
worldview that has been adopted by us has been to our collective detriment.
Whether lockdowns worked at all or not is under question because places
that didn't impose strict lockdowns like Sweden, Japan, South Dakota
haven't turned into giant body-piles (and no, I will not accept an A-to-B
numbers comparison - the rationale for lockdowns was that if not, we will
have mass die-offs. So I will evaluate the effectiveness of lockdowns only
on whether the non-lockdown place has bodies piling up or not as originally
claimed. Anything else - doesn't justify forced lockdowns that we knew
going in will kill a lot of people too and irreparably push entire
generations into poverty and leave a lot more people without the resources
to protect themselves from the disease). Hospitals and agencies that have
been given incentives to report as many Covid-19 cases and deaths as
possible, have, predictably and unfortunately, done what were incentivised
to do. And data visualization folks have, I'm sorry again, gone in for
numbers orgy and it's feeling sick now.

There have been cases in other countries of people dying from motor
accidents or gunshot wounds being marked as Covid-19 deaths, of
official govt bodies having to retract and downgrade their own
published figures after scandals erupted. There have been nurses speaking
out about patients being put on wrong treatments that aggravated their
conditions and no relatives being around them to stop it. There have been
medical practitioners with equal or more experise than those on the
authority side, and with actual success track records in treating covid-19
patients, questioning official policies. Instead of addressing the concerns
raised, the tech giants have gone into full-on Church-vs-Galileo mode and
exponentially increased the collective distrust by proactively censoring
anyone daring to diverge from what CNN believes to be true. I would frankly
trust authorities more when they earn my trust than when they dictate it.

If we want to help in even reducing the trust deficit and encourage
alienated people to begin trusting official authorities again, I think
qualitative voluntary data collection from kin of deceased and from
survivors will be important. So that the anecdotes build up confirm the
official line. And if they don't, well, it may help make sure we have
better official sources later on.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Mon, Aug 17, 2020 at 1:14 AM Thejesh GN  wrote:

> May be RTI?
>
> Regards,
> Thej
>
> Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್
> http://thejeshgn.com
>
> On Mon, 17 Aug, 2020, 1:00 AM Hemant Desai, 
> wrote:
>
>> Some records are necessary. However in absence of official communication,
>> where can such data be sourced from.
>>
>>
>>
>> On Sun, 16 Aug, 2020, 2:19 PM Thejesh GN,  wrote:
>>
>>> Dear All,
>>>
>>> 49k+ people are dead due to COVID19 in India, no one knows who they are.
>>> In a year we will forget their stories and it will be just one single
>>> number. I was wondering if we can collect
>>>
>>>
>>> Date of death
>>> State
>>> District
>>> Gender
>>> Age
>>> Name (not sure of this, due to privacy. We can discuss)
>>> Reference link
>>>
>>> - So that we have enough data to analyse
>>> - They are recorded publicly just like deaths of soldiers in war or
>>> attacks
>>>
>>>
>>> Deaths and births are usually public documents. But i don't see MoHFW
>>> publishing this data. Do you have any idea how to go about?
>>>
>>>
>>>
>>> Regards,
>>> Thej
>>>
>>> Thejesh GN ⏚ ತೇಜೇಶ್ ಜಿ.ಎನ್
>>> http://thejeshgn.com
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receivin

Re: [datameet] Data Story: In 6 years, forest land the size of Nagaland diverted

[Nikhil VJ]

Hi,

I posted on their github repo and they've replied with lot more details and 
opportunities for collaboration if anyone is interested. There seems to be 
a map created by IndiaSpend in this regard. You can join in the 
conversation there: 
https://github.com/pratikunterwegs/forest-clearance-india/issues/1 


Regards
Nikhil VJ


On Saturday, July 4, 2020 at 11:26:18 PM UTC+5:30, Nikhil VJ wrote:
>
> It would be great if there was a publicly available map of all projects.
>
> This seems to be the repo mentioned: 
> https://github.com/pratikunterwegs/forest-clearance-india 
>
> (the news article as is typical of Indian media outlets makes sure to 
> never link to the sources they're talking about - I got this on searching 
> the web)
>
> --
> Cheers,
> Nikhil VJ
> https://nikhilvj.co.in
>
>
> On Fri, Jul 3, 2020 at 4:54 PM Thejesh GN <> wrote:
>
>> Forest land nearly equal to the size of Nagaland has been approved for 
>> diversion between 2014 and 2020 or is pending to be approved for various 
>> infrastructure and developmental projects such as mining, irrigation, and 
>> regularisation of encroachments, according to an analysis by Vijay Ramesh, 
>> a PhD student at Columbia University.
>>
>> *Ramesh has analysed 48,000 clearance proposals using the R programming 
>> language (the raw data is available on GitHub). He sourced the forest 
>> clearance data from the Union environment ministry’s Parivesh website, 
>> which has details of all clearances divided into two sections—before 2014 
>> and after 2014.*
>>
>>
>>
>> https://www.hindustantimes.com/india-news/in-6-years-forest-land-the-size-of-nagaland-diverted/story-GzksCVWeGfMaEWoGVxMLXO.html
>>
>>
>> Thej
>> --
>> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
>> http://thejeshgn.com
>> GPG ID :  0xBFFC8DD3C06DD6B0
>>
>> -- 
>> Datameet is a community of Data Science enthusiasts in India. Know more 
>> about us by visiting http://datameet.org
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/datameet/CAABnYsUSyaaHaxf2bB%2BpnrRv3W6VbRysO-HsiAUTmqNB0ck9%3Dg%40mail.gmail.com
>>  
>> <https://groups.google.com/d/msgid/datameet/CAABnYsUSyaaHaxf2bB%2BpnrRv3W6VbRysO-HsiAUTmqNB0ck9%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/e9df82e7-d489-4c24-bd79-3c6734ac9a3ao%40googlegroups.com.

Re: [datameet] Data Story: In 6 years, forest land the size of Nagaland diverted

[Nikhil VJ]

It would be great if there was a publicly available map of all projects.

This seems to be the repo mentioned:
https://github.com/pratikunterwegs/forest-clearance-india

(the news article as is typical of Indian media outlets makes sure to never
link to the sources they're talking about - I got this on searching the web)

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Fri, Jul 3, 2020 at 4:54 PM Thejesh GN  wrote:

> Forest land nearly equal to the size of Nagaland has been approved for
> diversion between 2014 and 2020 or is pending to be approved for various
> infrastructure and developmental projects such as mining, irrigation, and
> regularisation of encroachments, according to an analysis by Vijay Ramesh,
> a PhD student at Columbia University.
>
> *Ramesh has analysed 48,000 clearance proposals using the R programming
> language (the raw data is available on GitHub). He sourced the forest
> clearance data from the Union environment ministry’s Parivesh website,
> which has details of all clearances divided into two sections—before 2014
> and after 2014.*
>
>
>
> https://www.hindustantimes.com/india-news/in-6-years-forest-land-the-size-of-nagaland-diverted/story-GzksCVWeGfMaEWoGVxMLXO.html
>
>
> Thej
> --
> Thejesh GN *⏚* ತೇಜೇಶ್ ಜಿ.ಎನ್
> http://thejeshgn.com
> GPG ID :  0xBFFC8DD3C06DD6B0
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAABnYsUSyaaHaxf2bB%2BpnrRv3W6VbRysO-HsiAUTmqNB0ck9%3Dg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAABnYsUSyaaHaxf2bB%2BpnrRv3W6VbRysO-HsiAUTmqNB0ck9%3Dg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNypSGbAdifjPjdRnY_8Yc5no4nZ__B4cChyZuypZADTg%40mail.gmail.com.

Re: [datameet] Request for Road Traffic Density data 2019-2020

[Nikhil VJ]

Hi Divyansh,

Some steering questions seeing that this is a research thing:

- What are the methods in which such data can be recorded and collected?

- What are legalities surrounding each of these methods?

- What kinds of entities have capability to collect such data?

- Are there any entities under above list that are known to be doing this
collection?

- Are there any entities from above list that might be able to share this
collected data with you?

- If none of above are directly openly sharing, what do you need to do to
get them to share it with you?

- If above questions could not be answered satisfactorily, then a more
basic question: Does the data exist at all in the way it's originally
required and under terms that you can obtain it?

- If not, then would it be worth to pivot from locating source of needed
data to creating / collating / collecting needed data?

- What would one need to do it? Technology, resources, time, know-how?

- Are there others looking for the same / similar things as you are, and
are there opportunities to form alliances for a common goal?

- If so, then what is it that you can bring to the table?

- Is it better to initiate such an effort, or join into one that is already
ongoing?

Myself I've got no clue, but this might help you take steps towards your
goal. Oftentimes I've observed that the thing needed comes into existence
only after questions for it are asked and then gears are switched from
expecting a result to participating in forming it. All the best!

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Thu, Jun 18, 2020 at 10:29 PM Divyansh Deshmukh 
wrote:

> Hello,
>
> I, Divyansh Deshmukh, am a research student in IIIT Hyderabad under Dr.
> Sachin Chaudhari (SPCRC Lab, IIIT Hyderabad). We here, at IIIT Hyderabad
> are doing a research project on air pollution and are in need for the road
> traffic density data.
>
> *Data required - Traffic density data from 01/01/2019 to 14/06/2020.
> Cities - Hyderabad, Bangalore and Delhi. Hourly average preferable. *
>
> The data required is extremely important for us for betterment and
> development of the cities. It would be really helpful if the data required
> is provided.
>
> Kindly consider my request.
>
> Looking forward to hearing from you.
>
> Thanks & Regards
> Divyansh Deshmukh
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAPPcYWOX2f5%2BLAitoydr1Vp_TjvZgGRAHrgs29b0CbNY90_A2g%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAPPcYWOX2f5%2BLAitoydr1Vp_TjvZgGRAHrgs29b0CbNY90_A2g%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMXiXr-%3DvZF6Sbbe6txNdaQXya7zcppOCyQawp_W%3DCc0A%40mail.gmail.com.

Re: [datameet] Get Smart Route to Avoid Containment Zone

[Nikhil VJ]

Hi Souvik,

Good work!

Suggestion on UI end: can you find a way to capture clicks or right-clicks
on the map and let the user add to/from addresses that way, other than
typing?
That might also save on place api usage.

I've made custom context menus / click-to-pin feature in Leaflet maps, not
sure how its done in google maps but I've seen such implementations
elsewhere.
Here's a link from a quick search:
https://stackoverflow.com/questions/7168394/google-map-v3-context-menu#10957262

Another for-later suggestion: Is your routing running on google's api or
other? If other, then it might be a good bet to change the ui to leaflet
and avoid google's usage costs that hit when visitor count increases.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, May 13, 2020 at 12:29 PM Souvik Sankar Mitra <
souviksankar2...@gmail.com> wrote:

> I have developed a smart navigation application to avoid containment zone.
> It gives a safe route between source and destination which does not go
> through any containment zone. Till now it only available for Kolkata and
> Howrah but it can be implemented for other cities also.
>
> You can check it here - http://shampa.covidrespons.in/
> For more information you can check out this video -
> https://www.youtube.com/watch?v=UtUaCDmJV_M
>
> Hope you like it.
> Thank you.
>
> Souvik Sankar Mitra
> M.Tech (Geoinformatics) , Indian Institute of Remote Sensing
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/41afc5f7-aafe-4451-9d94-1d09ab15feb3%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/41afc5f7-aafe-4451-9d94-1d09ab15feb3%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuNOtpm4YnnYte0fU0bEy1obngMiPFs6gxzgUh%3Da%3D4%2BcPg%40mail.gmail.com.

Re: [datameet] Re: Facebook Road AI data!!!!!!!!!!

[Nikhil VJ]

Hi Devdatta, Ujaval

Thanks! Both ways look good. I got the ogr2ogr way working, but QGIS
processing toolbox way looks promising too.

Hi Deepak,
That would be an unfair question to ask as the intention for FB to share
the dataset is "this is what some of our programs have come up with by
looking at lots of satellite imagery data etc. It might help you map roads
that may have been missed so far, please see and check for yourself".

So this is at an initial stage in a very long process. The reason why such
initiatives exist is because there is no definitive "all rural roads"
proper source.
And whether the data does help or not may change from place to place with
local condtions like soil color, tree cover etc impacting the ability of
the programs in interpreting satellite imagery. So there can't even be any
definite answer at centralized level - it may be helpful at some place and
not at others - people will have to dig in and find out.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Apr 29, 2020 at 6:36 PM Deepak Sharda  wrote:

> thank a lot.
>
> one more question. As osm doesnt have so many rural roads  in its data for
> India.
>
> Does this file has all India rural road Data kindly tell is it worth.
>
>
> u being so kind and helpful
>
>
> thank you
>
>
>
> On Tue, 28 Apr, 2020, 10:51 pm Justin, 
> wrote:
>
>> @Deepak, open it, and walk away for about 30 minutes. It will open. Try
>> to have your area of interested centered on your screen, zoom into it. Then
>> select and export the section you need, otherwise it is a bit difficult to
>> work with.
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/d6563043-1155-494b-8c85-1bbe20d94cbe%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/d6563043-1155-494b-8c85-1bbe20d94cbe%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CA%2B4SsQBLaOqJAB%2BeOTpLe7czE4uB4PZkY%2BLzkzxOK%2B7ebX8uUQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CA%2B4SsQBLaOqJAB%2BeOTpLe7czE4uB4PZkY%2BLzkzxOK%2B7ebX8uUQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMH2MVwmuBimUg_eo2w9LR9KTXZaBOuHPJOnHXNA-nndA%40mail.gmail.com.

[datameet] Re: Facebook Road AI data!!!!!!!!!!

[Nikhil VJ]

I'm reading more about the .gpkg format here:

http://switchfromshapefile.org/#alternatives

Question: If I have lat-long bounds for a smaller regions, then how do 
extract that much part of data (one feature starting inside and continuing 
outwards is ok, no need to chop)... without having to load up the complete 
file into memory ?

I understand this format is SQLite at its core. And that is known for 
enabling us to query it without having to load the full data into memory.
I've been using osmconvert for extracting from .pbf, but afaik it doesn't 
support .gpkg formats.

Regards
Nikhil VJ
https://nikhilvj.co.in


On Tuesday, April 28, 2020 at 10:51:03 PM UTC+5:30, Justin wrote:
>
> @Deepak, open it, and walk away for about 30 minutes. It will open. Try to 
> have your area of interested centered on your screen, zoom into it. Then 
> select and export the section you need, otherwise it is a bit difficult to 
> work with. 
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/014a86b6-ed55-4018-890e-7dcbf412dc38%40googlegroups.com.

Re: [datameet] any help with geocoding and reverse geocoding without api or from open source

[Nikhil VJ]

Hi Deepak,

I was digging up regarding this recently and tried one option that didn't
work out; sharing my notes:

Starting rule : Cannot rely on 3rd party service; needs to be self-hosted;
use OSM country extract from https://download.geofabrik.de/ as base data.

Starting point:

OSM wiki page listing several different software for searching places:

https://wiki.openstreetmap.org/wiki/Search_engines


Osmnames + Sphinxsearch :

This seemed easier to start off so tried it. Once I got it depoyed, didn't
get any proper results even for city names. I'm guessing the parameters may
require fine-tuning. And I couldn't understand why they are involving
wikipedia. It seemed like the search was relying more on data from
wikipedia than from OSM.

Didn't explore on it further. (Spare time burned up!)


Nominatim related leads, where I want to dig into in next sprint:

https://www.cascadia-analytics.com/2017/10/01/nominatim-docker.html

https://github.com/mediagis/nominatim-docker


Pelias seems promising too, but we need a proper elasticsearch deployment
to run it and that takes up quite some memory.


-


My personal observations, may be inaccurate:

We should treat geocoding and reverse geocoding as two different
applications.


Reverse geocoding would be much simpler to do, since you just need to put
in the lat-longs and cough up the closest place names.


Geocoding involves complicated textual search, especially given the
variations in ways place names are stored. There has to be high flexibility
like making up for spelling differences. So elasticsearch seems like a good
fit for it. Then, geo-constraining searches is very important here. We need
ways to progressively narrow down the search areas and cut out false
positives. Letting the user draw a box on the map to search in, for
example, can greatly increase effectiveness of results rather than
searching the whole country / planet data.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sun, Apr 19, 2020 at 8:26 AM Bhanu K  wrote:

> Hi Deepak,
>
> Few options depending on the tool:
>
> - if Google Sheets is an option, Geocode by Awesome Table
> <https://gsuite.google.com/u/0/marketplace/app/geocode_by_awesome_table/904124517349?hl=en&pann=sheets_addon_widget>
> is a good add-on.
> - if Microsoft Excel is an option, geocode-excel
> <https://github.com/gramener/geocode-excel> is an offline version.
> - Nominatim API <https://nominatim.org/release-docs/develop/api/Overview/>
> can be used in a language of your choice.
>
> thanks,
> Bhanu
>
>
> On Sat, Apr 18, 2020 at 5:00 PM Deepak Sharda 
> wrote:
>
>> i googled and get this site address which is open source of geocoding and
>> reverse geocoding but i am not expert in node.js  could you help .
>>
>>
>> is it helpful for data relating to india.
>>
>>
>> there was a file aattached as sample data for india
>>
>> https://github.com/pelias/pelias
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/a672e807-c8dd-4dba-8d79-f407aa076cb1%40googlegroups.com
>> <https://groups.google.com/d/msgid/datameet/a672e807-c8dd-4dba-8d79-f407aa076cb1%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> --
> Bhanu
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CADbFdW4_1DrFTrQvpTdBuBmW2KMmm3Kxr8D84nmKK9Viwt6fmg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CADbFdW4_1DrFTrQvpTdBuBmW2KMmm3Kxr8D84nmKK9Viwt6fmg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOFsm2NgkWWQt_582OW7UGithgqFAp02-hFC4%3D%3DWO_0eQ%40mail.gmail.com.

Re: [datameet] URL COVID-19 live

[Nikhil VJ]

Hi Atul,

Great work done!

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Apr 1, 2020 at 7:12 PM Atul Panday 
wrote:

> Dear All
>
> Please Follow the url for live update.
>
> https://app.powerbi.com/view?r=eyJrIjoiNGM4NjQ4NTEtYmNmZS00NDRjLTljNzQtNGJjODJiYzA3YWYzIiwidCI6IjU2NTczMmI1LWE5YjYtNDZiYS05YTdmLWZmMWRhZTJmNTEyZiIsImMiOjEwfQ%3D%3D
>
> this data is refresh every 5 minutes.
>
> any query please ping me +917024457369
>
> Kind regards
>
> --
>
>
> *Atul PandayEmail: greatatulpan...@gmail.com
> Mob: 7024457369*
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAEN4%2BHeOTadZ7VX%2BCMggafndQ1BG0-L7ZwY-42-bBcwTf5d1Pg%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAEN4%2BHeOTadZ7VX%2BCMggafndQ1BG0-L7ZwY-42-bBcwTf5d1Pg%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMt07Y-Q6d-9Ct1Q4T0jZEhkBpCUwDwnh3wPCjSzh-ZRA%40mail.gmail.com.

Re: [datameet] Please have a look: COVID- 19 Dashboard

[Nikhil VJ]

Hi Digvijay,

Good work!
Feedback: One of the graphs is showing error at my end: viz above this
heading isn't loading:

Timeline and Top 10 Ranking of States by the number of COVID-19 confirmed
cases (Indians and Foreign Nationals combined)
--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Sat, Mar 28, 2020 at 8:54 PM Digvijay Bendrikar Shinde <
digvijaybendrikarshi...@gmail.com> wrote:

> Hello All,
>
> As we are realising the imminent threat of COVID- 19, I am hereby sharing
> a Covid-19 DataVisualization dashboard developed by a group of students,
> postdocs, faculty, and researchers from the Nutrition Group@IITB, CTARA,
> Climate Studies, and GISE Lab, Computer Science and Engineering, IIT Bombay.
>
> A district-level GIS map is its main feature. It is updated real-time as
> per MHFW data.
>
> https://bit.ly/Covid19-India-District-DataVizualization
>
> Please give your feedback and share this in your professional network.
>
> You may also like to see the work of Nutrition group, IIT-Bombay-
>
> http://www.iitbnutritiongroup.in/
>
> Regards,
> Digvijay Bendrikar-Shinde
> PhD Scholar
> CTARA, IIT Bombay
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CA%2BsGXO%2BaPv6-en7qKMWebccOk4fRn-TKH70q3_cPDLN04%3DL%2BJw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CA%2BsGXO%2BaPv6-en7qKMWebccOk4fRn-TKH70q3_cPDLN04%3DL%2BJw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuM5S%3DzabaaPjdp1tpzOMiaiSxEd7BqPoxXUcS5hObjm_g%40mail.gmail.com.

Re: [datameet] GPU consultant

[Nikhil VJ]

Hi, do expand on what is a GPU - others may know of the same thing by a
different name.

On my end i want to explore if it can be possible to string many simple
computers (or raspberry Pi's) in parallel and do parallel processing of a
particular program task which is low on memory needs but takes time to
process and where many concurrent requests are coming. I'm looking for
details on how exactly the management is done.

One simplification: all the machines would be on LAN with specific ip
addresses, and the program is loaded as API endpoint, so running it means
just pinging that ip address over the LAN and waiting for callback.

Application: in a typical office there are so many CPUs lying idle or just
running basic email n excel, why not use them.
This may be way different than the GPU thing but I don't know.

On Fri, Jan 3, 2020, 12:27 Sanjay Chitnis  wrote:

> What is the application? If it is ML,  https://mlperf.org/ is THE
> resource.
>
> On Fri, Jan 3, 2020 at 12:07 PM kunal s  wrote:
>
>> Hi, is anyone aware of a consultant/company which can help us setup a GPU
>> in-house in India (delhi) , getting slightly wary of cloud GPU costs.
>>
>> Please let me know, if you have any contacts.
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CALgbiVh9tTFMyVvLmkX49WkiZpGnt-MfXHjSmMbYaMRFcXCLKA%40mail.gmail.com
>> 
>> .
>>
>
>
> --
> Sanjay Chitnis
> http://www.google.com/profiles/sanjay.chitnis
> http://www.linkedin.com/in/sanjaychitnis
>
>
> *"You are what your deep, driving desire is. As your desire is, so is your
> will. As your will is, so is your deed. As your deed is, so is your
> destiny." – Brihadaranyaka Upanishad IV. 4.5*
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAOhib6mOU%2B0fgcdAu%2BLSoZhE5y1nvZUFX5J_N8JmCO7EcsRvVg%40mail.gmail.com
> 
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuPvS5Ba9tyK3xXPjNL00uqBBSX7gguG24VB-bBavp-9Og%40mail.gmail.com.

Re: [datameet] findmygov.in and the new graphql interface

[Nikhil VJ]

Hi Vikas,

Nice work! Still have to check it all out, but quickly want to share
regarding one of the issues you mentioned:

OpenRefine is a useful tool to resolve spelling differences in repeating
and messy data.
Here is a write-up on it:
http://datameet.org/2018/06/13/openrefine-bus-stop/

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in



On Sat, Dec 7, 2019 at 12:02 PM ヴィカス ヤダヴァ (vikas yadav) 
wrote:

> Hi all.
>
> I'd like to introduce findmygov.in, working on it for about two years.
> Its main objectives are to present contact info of elected and appointed
> officers across the country, across all departments, state, central and in
> all Indian languages (not auto translated). There are few more objectives
> <https://findmygov.in/site/about>. Since there is not any easy way to get
> it, mostly comes down to how much time I have to manually feed the data.
> The newly SWaaS platform which all districts websites are shifting towards,
> should make it little easier. Need to sit and make a parser. Data is also
> gathered through RTI. Data of recent election candidates and full results
> of LS and Assembly polls is there but not exposed yet. During LS elections,
> candidates for every constituency were shown with pics and links to their
> affidavits. I missed the train for on-going Jharkhand assembly elections.
> The homepage of findmygov.in asks you location and shows everything about
> that point.
>
> All elected reps of assembly, parliament constituencies and few municipal
> bodies are mapped (using polygons from Datameet). Tamil Nadu also has
> polygons till village level (Panchayat Raj from bhuvan GIS). Actually I
> have full PR polygon data for entire country but not mapped yet due to lack
> of time and the spellings mismatch prevents automatic linking. Spellings is
> the most common problem working with nationwide data across ministries and
> departments.
>
> In last few days I setup a graphql interface
> <https://findmygov.in/graphql> to expose some of this data. It is using
> the graphiql UI so you can see the docs and try out. It is just blocked by
> captcha wall. All graphql data also has updated timestamp, so you can
> verify the freshness. Govt department contact info stays the same only the
> officer may change. Even twitter handles of reps and officers is supported
> (but data is mostly missing for now). Polygons are not exposed in the
> graphql yet but easy to add them.
>
> findmygov.in is self-funded not-for-profit experiment. I'm just a coder
> who cannot design a website.
>
> Some query examples (there is a Docs link on the top right to learn more
> about types):
>
> query {
>   states {
> name
> elections {
>   name
> }
>   }
> }
>
> ...
> divisions {
>   name
>   districts {
> name
>   }
> }
> ...
>   districts {
> name
> officers(desig: DEPUTYCOMMISSIONER) {
>   name,
>   phone,
>   email
> }
>   }
>
> Please share you remarks. Any features, bugs or corrections are most
> welcome.
> Thanks,
> Vikas
>
> thevikas.com
> <https://thevikas.com/?utm_source=gmail&utm_medium=footer&utm_campaign=c11>
>  - *Find My Government* - findmygov.in <https://goo.gl/onYnWL>
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CAFyZHpcBY7PMYjfPmHpQd%2BgTYAtSn%3DwP%2BD-ztM1ZOc%3DbmcY3Mw%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CAFyZHpcBY7PMYjfPmHpQd%2BgTYAtSn%3DwP%2BD-ztM1ZOc%3DbmcY3Mw%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuMMDPc6z%2B3M8qkS-%3DyHxr4aGKnrO6%2BkN4ig-O2qL4V41A%40mail.gmail.com.

Re: [datameet] India Post Office Data

[Nikhil VJ]

Hi Devdatta,

Nice find!
It's all dated 2007, so it's possible the program got phased out in the
intervening 12 years, or it didn't transfer over to next official in charge
or so.

The attached letter from Dept of Posts, explaining that they were
identifying institutions with v.high mail volume and allotting individual
pincodes to them, indicates there may be more such codes created.

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Dec 4, 2019 at 10:33 AM Devdatta Tengshe 
wrote:

> I have finally found my Whale!!
>
> For a Long time, I have been searching for an example of a Pincode, which
> delivers to only a specific Government Office/Institution. I finally found
> this link to a circular on the ICAR site, which says that pincode 110114
> has been assigned to it.
> https://icar.org.in/files/circulars/new_pin.pdf
>
> Strangely enough, this pincode is nowhere to be found. It is not present
> in the All India Pincode Directory on data.gov.in, nor on the India Posts
> Site, nor on the Bhuvan Site. It not present even in the Contact us Section
> of the ICAR website.
>
> Why it is not present, is one of the many unsolved mysteries of the Indian
> Government.
>
> Regards,
> Devdatta Tengshe
>
>
> On Friday, 10 May 2019 17:31:12 UTC+5:30, Devdatta Tengshe wrote:
>>
>> And One Post office can serve multiple pincodes (I've heard this from
>>>> Employees of the Postal Department, but haven't found an example of it)
>>>>
>>> This is quite odd. Would be good to confirm this.
>>>
>>
>> I was told by an old Post master,  that certain Government Buildings had
>> their own Pincodes, and hence the post office which delivered to that
>> building, as well as the area around that building, delivered post to
>> multiple pincodes.
>>
>> This is either a myth, or an historic fact, but I haven't seen any
>> example of this.
>>
>>
>> Regards,
>> Devdatta
>>
>>
>> On Fri, May 10, 2019 at 5:02 PM Arun Ganesh  wrote:
>>
>>> There used to be a hackpad
>>> https://datameet.hackpad.com/PINCODE-Hell.-M4hPFJVV2Gm with a lot of
>>> pincode research which unfortunately seems to be lost.
>>>
>>> Unique pincodes are assigned to an area that has substantial postal
>>> deliveries. A pincode area could be as small as a university with a single
>>> post office, or an entire district with multiple post offices.
>>>
>>> For each pincode, only one post office serves as the delivery office,
>>> where all post destined for the pincode lands, and is further routed to one
>>> of the non delivery post offices in the same pincode based on address.
>>>
>>> > And One Post office can serve multiple pincodes (I've heard this from
>>> Employees of the Postal Department, but haven't found an example of it)
>>>
>>> This is quite odd. Would be good to confirm this.
>>>
>>> --
>>> Datameet is a community of Data Science enthusiasts in India. Know more
>>> about us by visiting http://datameet.org
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "datameet" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to datameet+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/datameet/CA%2BGKQr3y63iPy9bNWcWGTKT8vPpa_-6VbCLXGZ6jLxjY1MHrTQ%40mail.gmail.com
>>> <https://groups.google.com/d/msgid/datameet/CA%2BGKQr3y63iPy9bNWcWGTKT8vPpa_-6VbCLXGZ6jLxjY1MHrTQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/de95dd91-1de3-405f-97b1-f208df543498%40googlegroups.com
> <https://groups.google.com/d/msgid/datameet/de95dd91-1de3-405f-97b1-f208df543498%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOycAo24tLspvgYwznU%2Bxz%2BESARX7VsFg0g5Ds0jkcgEA%40mail.gmail.com.

[datameet] Pune Urban Data Exchange (PUDX) Datathon

[Nikhil VJ]

Pune Urban Data Exchange (PUDX) Datathon

https://cps.iisc.ac.in/pudx/

Registration Ends: Nov 22

Lot of interesting live data being made available through API / python SDK.

Check out this example python notebook (a program with commands and output 
and documentation shown). It is pulling in air quality data and rendering 
it as some visualizations.

https://github.com/iudx/pyIUDX/blob/master/examples/pyIUDX_sample_usecases.ipynb

Regards
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/5ab41657-9500-488f-937c-7f56a10d141c%40googlegroups.com.

[datameet] Pune Urban Data Exchange (PUDX) Datathon

[Nikhil VJ]

Pune Urban Data Exchange (PUDX) Datathon

https://cps.iisc.ac.in/pudx/

Registration Ends: Nov 22

Lot of interesting live data being made available through API / python SDK.

Check out this example python notebook (a program with commands and output
and documentation shown). It is pulling in air quality data and rendering
it as some visualizations.
https://github.com/iudx/pyIUDX/blob/master/examples/pyIUDX_sample_usecases.ipynb

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuOfTi34kRO4z%3DOXHdDMEHv1vrDOrU7EOP%2Be0bdVaO1YDQ%40mail.gmail.com.

Re: [datameet] Help in extracting Mumbai's Development Plan 2034 as Shapefile

[Nikhil VJ]

Hi Abraham,

This screencast might help.

https://youtu.be/bD80VSO1qiQ

Regards
Nikhil

On Wednesday, October 23, 2019 at 5:35:59 PM UTC+5:30, Abraham Daniel Raj 
wrote:
>
> Hi Nikhil Vj,
>
>
> After turning on a layer on the map. How to catch and save the response. 
> Can you inform?
>
>
>
> With regards,
> Daniel
>
>
> On Wed, Oct 23, 2019 at 2:31 PM Nikhil VJ > 
> wrote:
>
>> Hi, the second website shared 1991 DP:
>> http://www.loginmumbai.org/map.html 
>> <http://www.loginmumbai.org/map.html#>
>> .. is vector!
>>
>> Right-click > Inspect > Network > XHR . Turn on a layer on the map. Catch 
>> and save the response.
>>
>> Here's one site that helps convert the json to csv:
>> https://json-csv.com/
>>
>> --
>> Cheers,
>> Nikhil VJ
>> https://nikhilvj.co.in
>>
>>
>> On Wed, Oct 23, 2019 at 9:38 AM Devdatta Tengshe > > wrote:
>>
>>> This is coming from an ArcGIS Based Service at: 
>>> https://agsmaps.mcgm.gov.in/server/rest/services/Development_Plan_2034/MapServer/
>>>
>>> It is possible to scrape the data from here, but we need to know what 
>>> layers to scrape. Unfortunately, their naming convention is non-existent, 
>>> and unless we have some detailed knowledge on what each layer means, it 
>>> will be a futile exercise, even if we get the data.
>>>
>>> Regards,
>>> Devdatta
>>>
>>>
>>> On Wed, Oct 23, 2019 at 1:23 AM akshit Shah >> > wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks Nikhil VJ for underlining the fact that we are still far away 
>>>> from free data regime prevailing in western countries. I already 
>>>> approached 
>>>> head of Development Plan department at MCGM and their typical reply was 
>>>> that "we can not share this data due to security reasons !!, even to share 
>>>> this with MMRDA (Mumbai Metropolitan Region Development Authority) we have 
>>>> signed non-disclosure bond and stuff. 
>>>>
>>>> I don't understand what threat can a publicly available development 
>>>> plan could pose to authorities !!!  just because its in a vector format. 
>>>> 26/11 happened even before the GIS era in India. Rationally, the amount of 
>>>> help in urban planning research through such dataset is far more greater 
>>>> than any security threat (that you guys may think of). but it seems our 
>>>> bureaucrats doesn't understand such things (may be because 
>>>> Architecture/Planning is not available as an optional subject).
>>>>
>>>> Coming to the extraction point, there are multiple long routes to 
>>>> achieve the objective of getting vector shapes. I have tried one of them 
>>>> by 
>>>> geo-referencing the one PDF tile using AutoCAD Map & then exporting it 
>>>> layer by layer to QGIS. but I could not achieve accurate scale and 
>>>> geo-referencing. Also the task is too tedious considering the scale of 
>>>> city, the development plan is divided into more than 150 tiles which will 
>>>> take too long to accomplish.
>>>>
>>>> Alternatively, every concerned individual can populate landuse data in 
>>>> OSM for a specific region, this way it can be made available in vector. 
>>>> But 
>>>> the manual input for metadata is also a time consuming challenge.
>>>>
>>>> Lastly we need some other idea to cut short this task, as authority is 
>>>> not going to handover it so easily.
>>>>
>>>> On Mon, Oct 21, 2019 at 11:42 PM Nikhil VJ >>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Great to know they're finally moving from PDFs to zoomable maps! One 
>>>>> thing I had really hoped for was for people to pin-point exact locations 
>>>>> in 
>>>>> the DP and talk about that, instead of just making grand reports with 
>>>>> overall stats.
>>>>>
>>>>> So again the data you see on this site is coming from a map server 
>>>>> that pre-renders the data and sends stuff across as broken down image 
>>>>> tiles.
>>>>>
>>>>> Precisely to prevent site visitors from scraping the original vector 
>>>>> data.
>>>>>
>>>>> I'm guessing the bureaucracy would have consented to this move (from 
>>>

Re: [datameet] Help in extracting Mumbai's Development Plan 2034 as Shapefile

[Nikhil VJ]

Hi, the second website shared 1991 DP:
http://www.loginmumbai.org/map.html <http://www.loginmumbai.org/map.html#>
.. is vector!

Right-click > Inspect > Network > XHR . Turn on a layer on the map. Catch
and save the response.

Here's one site that helps convert the json to csv:
https://json-csv.com/

--
Cheers,
Nikhil VJ
https://nikhilvj.co.in


On Wed, Oct 23, 2019 at 9:38 AM Devdatta Tengshe 
wrote:

> This is coming from an ArcGIS Based Service at:
> https://agsmaps.mcgm.gov.in/server/rest/services/Development_Plan_2034/MapServer/
>
> It is possible to scrape the data from here, but we need to know what
> layers to scrape. Unfortunately, their naming convention is non-existent,
> and unless we have some detailed knowledge on what each layer means, it
> will be a futile exercise, even if we get the data.
>
> Regards,
> Devdatta
>
>
> On Wed, Oct 23, 2019 at 1:23 AM akshit Shah 
> wrote:
>
>> Hi,
>>
>> Thanks Nikhil VJ for underlining the fact that we are still far away from
>> free data regime prevailing in western countries. I already approached head
>> of Development Plan department at MCGM and their typical reply was that "we
>> can not share this data due to security reasons !!, even to share this with
>> MMRDA (Mumbai Metropolitan Region Development Authority) we have signed
>> non-disclosure bond and stuff.
>>
>> I don't understand what threat can a publicly available development plan
>> could pose to authorities !!!  just because its in a vector format. 26/11
>> happened even before the GIS era in India. Rationally, the amount of help
>> in urban planning research through such dataset is far more greater than
>> any security threat (that you guys may think of). but it seems our
>> bureaucrats doesn't understand such things (may be because
>> Architecture/Planning is not available as an optional subject).
>>
>> Coming to the extraction point, there are multiple long routes to
>> achieve the objective of getting vector shapes. I have tried one of them by
>> geo-referencing the one PDF tile using AutoCAD Map & then exporting it
>> layer by layer to QGIS. but I could not achieve accurate scale and
>> geo-referencing. Also the task is too tedious considering the scale of
>> city, the development plan is divided into more than 150 tiles which will
>> take too long to accomplish.
>>
>> Alternatively, every concerned individual can populate landuse data in
>> OSM for a specific region, this way it can be made available in vector. But
>> the manual input for metadata is also a time consuming challenge.
>>
>> Lastly we need some other idea to cut short this task, as authority is
>> not going to handover it so easily.
>>
>> On Mon, Oct 21, 2019 at 11:42 PM Nikhil VJ  wrote:
>>
>>> Hi,
>>>
>>> Great to know they're finally moving from PDFs to zoomable maps! One
>>> thing I had really hoped for was for people to pin-point exact locations in
>>> the DP and talk about that, instead of just making grand reports with
>>> overall stats.
>>>
>>> So again the data you see on this site is coming from a map server that
>>> pre-renders the data and sends stuff across as broken down image tiles.
>>>
>>> Precisely to prevent site visitors from scraping the original vector
>>> data.
>>>
>>> I'm guessing the bureaucracy would have consented to this move (from
>>> PDFs to actual map) only under strict conditions that it must not be
>>> scrape-able.
>>>
>>> All the best.. if somebody does find a way to break through though, I
>>> can imagine the whole thing will be taken down and that will be
>>> regrettable, because this is a really good leg-up for the urban planning
>>> sector in India and I'd like to see the same happen in Pune as well.
>>> Between PDFs and this I prefer this even if both options aren't open.
>>>
>>> A longer approach could be to share examples from Europe/US of cities
>>> releasing such full data openly and persuade/petition these folks with
>>> that. Again, all the best for that. And when one day some municipal
>>> corporation in India does do this, we should congratulate and praise them
>>> like anything so that others are encouraged too.
>>>
>>> Yet another approach, and given the target area is Mumbai, this might
>>> actually have a chance: get a price from the bureaucrats for releasing this
>>> data publicly, and collectively crowdfund it. Just consider the combined
>>> purses of all real estate companie

Re: [datameet] Help in extracting Mumbai's Development Plan 2034 as Shapefile

[Nikhil VJ]

Hi,

Great to know they're finally moving from PDFs to zoomable maps! One thing
I had really hoped for was for people to pin-point exact locations in the
DP and talk about that, instead of just making grand reports with overall
stats.

So again the data you see on this site is coming from a map server that
pre-renders the data and sends stuff across as broken down image tiles.

Precisely to prevent site visitors from scraping the original vector data.

I'm guessing the bureaucracy would have consented to this move (from PDFs
to actual map) only under strict conditions that it must not be scrape-able.

All the best.. if somebody does find a way to break through though, I can
imagine the whole thing will be taken down and that will be regrettable,
because this is a really good leg-up for the urban planning sector in India
and I'd like to see the same happen in Pune as well. Between PDFs and this
I prefer this even if both options aren't open.

A longer approach could be to share examples from Europe/US of cities
releasing such full data openly and persuade/petition these folks with
that. Again, all the best for that. And when one day some municipal
corporation in India does do this, we should congratulate and praise them
like anything so that others are encouraged too.

Yet another approach, and given the target area is Mumbai, this might
actually have a chance: get a price from the bureaucrats for releasing this
data publicly, and collectively crowdfund it. Just consider the combined
purses of all real estate companies and high profile ngo's operating in
this region that would want this. If someone from the relevant govt office
is seeing this, c'mon guys, give a quote!

Regards
Nikhil VJ

On Mon, Oct 21, 2019, 21:51 Aditi Nair  wrote:

> I would also be interested in this accessing this information.
>
> On Mon, Oct 21, 2019 at 8:13 AM akshit Shah 
> wrote:
>
>> Dear Datameet Memebrs,
>>
>> I want your help in extracting Development Plan 2034 of Mumbai from their
>> web portal <https://dpremarks.mcgm.gov.in/dp2034/>. I want the whole
>> Landuse with plot boundaries and its metadata (CTS no., DP Remarl,
>> Reservation, etc.) in GSI as a vector shapefile. Can anyone please help me
>> in doing so?
>>
>> Alternatively, the Development Plan 1991 of Mumbai has been hosted by
>> UDRI on their portal <http://www.loginmumbai.org/map.html#>. Can we
>> extract this as well?
>>
>> Thank you in advance.
>>
>> --
>> Warm Regards,
>>
>> *Ar. Akshit Shah*
>> | M.Tech Student, RCGSIDM, IIT-KGP |
>> | B.Arch, VNIT Nagpur |
>> 
>> *Indian Institute of Technology*
>> Kharagpur, West Bengal,
>> 721302, India
>> +91 3222 255221
>> IIT KGP <http://www.iitkgp.ac.in/> | Linkedin
>> <http://www.linkedin.com/in/Ar-Akshit-Shah>
>>
>> --
>> Datameet is a community of Data Science enthusiasts in India. Know more
>> about us by visiting http://datameet.org
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to datameet+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/datameet/CAJHzfxW%3D6NmFO6NnBxoCwE8Awt4OggtThZDrin0V5J3MTrsE9w%40mail.gmail.com
>> <https://groups.google.com/d/msgid/datameet/CAJHzfxW%3D6NmFO6NnBxoCwE8Awt4OggtThZDrin0V5J3MTrsE9w%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
> --
>
> *ADITI NAIR*
> *MS DESIGN AND URBAN  ECOLOGIES*
> PARSONS SCHOOL OF DESIGN
> SCHOOL OF DESIGN STRATEGIES
>
> naira...@newschool.edu
> <https://ww3.newschool.edu/edu-assets/marketing-communication/email-signature.html#>
>
> M 914 (410) 1981
> [image: The New School]
>
> --
> Datameet is a community of Data Science enthusiasts in India. Know more
> about us by visiting http://datameet.org
> ---
> You received this message because you are subscribed to the Google Groups
> "datameet" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to datameet+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/datameet/CADsa7XddMkbW_66cVUt%2B2d78TdX-09FOUy6hfnLNy4L59huL6A%40mail.gmail.com
> <https://groups.google.com/d/msgid/datameet/CADsa7XddMkbW_66cVUt%2B2d78TdX-09FOUy6hfnLNy4L59huL6A%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/CAH7jeuO_DzMx5SnPuqegPjP0JXRo-kp2CEA2kiaxTPctQh1ZZA%40mail.gmail.com.

[datameet] Re: Help in data extraction

[Nikhil VJ]

Hi Sudhansu,

Apologies, no answers from my end, but maybe I can help understand your 
challenge better.
The page in question seems to be loading all the map information as tile 
images from a dedicated map server, similar to : https://mapserver.org/
It does not load up the data as vector shapefiles.

People who have to publish maps on website, tend to go the server way 
specifically to prevent users like us from scraping this data while still 
letting us see it.
If there's other stuff like WMS etc then I don't know about that. I can 
just confirm that the data isn't being loaded as shapefiles or vector data, 
hence not immediately scrape-able.
Also: I'm having trouble accessing the website today. Had seen it a couple 
of days ago.

Regards
Nikhil


On Wednesday, October 16, 2019 at 7:59:41 AM UTC+5:30, Sudhansu S. Rath 
wrote:
>
> Dear datameet members,
>
> I was searching for some raster and GIS data related to Bhubaneswar city. 
> I came across this site ( http://www.bhubaneswarone.in/Bhubaneswarone/ 
> )
>  
> created by the city development authority. Can anyone help me in the 
> extraction of a few layers from this site in GIS usable format.??
>
> Regards
> Sudhansu
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/695fb19e-5dcf-497d-b11d-1910a3840443%40googlegroups.com.

Re: [datameet] Police Station Map and Mapping Information

[Nikhil VJ]

Hi Shruti,

Sharing possible ways that I know of to find and build the data you're 
looking for from the generally available map platforms. Sentences in 
brackets give more background info for people new to the subject. Note: I 
am adding an important caveat at the bottom.


*OpenStreetMap or OSM* (the all-open commons alternative to google maps 
that you'll find members of this community earnestly pitching for) way:
Relevant tag/key-value pair : "amenity=police" . Link: 
https://wiki.openstreetmap.org/wiki/Tag:amenity%3Dpolice 
Query on Overpass Turbo (the "backend" of Openstreetmap and the quickest 
way to search download specific data): https://overpass-turbo.eu/s/LX6


*Google Places API* (the more complicated "backend" of Google maps which 
while having more coverage for now, is way more restricted both technically 
and legally and the reason why you'll find members of this community 
earnestly pitching for moving to OSM / enriching OSM instead )
... lists "police" as one of the POI types than can be searched and 
retrieved programmatically.
Link: https://developers.google.com/places/supported_types

Note: Since the term "poiice" is generic, the direct search may also throw 
up places that aren't police stations, eg: police training institutes. It's 
safe to assume any dataset you get will be messy and will need considerable 
manual work on it to get to an acceptable stage. 


*Caveat:* We cannot expect full coverage or any proper answers about extent 
of coverage (The usual answer to "does this cover everything?" is "Nobody 
knows!") because that requires some official institution putting the data 
out there, and if there had been one then one wouldn't need to search on 
general map platforms. So what you get here is typically a perpetual 
work-in-progress cobbling-together of data by ordinary people volunteering 
time and effort. It will probably never be "complete" but it's much better 
than nothing. The remedy to lack of coverage is to start contributing to 
the common well and get others to do the same.


Regards
Nikhil VJ
Pune


On Friday, August 30, 2019 at 10:23:58 PM UTC+5:30, Ajayakumar A wrote:
>
> u can search in google map ( type it  as *police stations*, first u hv to 
> zoom the area, then u can type, if u search it for wide area in google map, 
> u will not get full data.so u hv to locate some area and search the police 
> station) and make it as kml files, then u can process it in Arcgis. U can 
> prepare the map also.
>
> On Tue, Aug 27, 2019 at 8:12 PM Shruti Lakhtakia  > wrote:
>
>> Hi,
>>
>> I am a Ph.D. student working on crime and development in India, and I am 
>> looking for:
>>
>> (1) a map of all the police stations in India, as well as their 
>> jurisdictions,
>> (2) information on how police stations can be mapped to the assembly 
>> constituencies of their respective states.
>>
>> Please do let me know if you have any experience working on this, or any 
>> suggestions on how this information can be collected.
>>
>> Thank you!
>>
>> Sincerely,
>> Shruti Lakhtakia
>>
>> -- 
>> Datameet is a community of Data Science enthusiasts in India. Know more 
>> about us by visiting http://datameet.org
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "datameet" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to data...@googlegroups.com .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/datameet/570bafb6-783a-4313-8de4-2088e162c6c2%40googlegroups.com
>>  
>> <https://groups.google.com/d/msgid/datameet/570bafb6-783a-4313-8de4-2088e162c6c2%40googlegroups.com?utm_medium=email&utm_source=footer>
>> .
>>
>
>
> -- 
> -.
> Ajayakumar A
>
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/d1340698-a326-4e84-9dd3-514fe620d7ab%40googlegroups.com.

[datameet] Pune + Pimpri-Chinchwad bus schedules GTFS data published

[Nikhil VJ]

Hi,

The Pune Open Data Portal (http://opendata.punecorporation.org/) has 
published bus schedules of PMPML (the bus system plying in Pune, Pimpri 
Chinchwad and surrounding areas), in the global standard static GTFS format.


I have deployed a visualization of the same using an open source 
application called ScheduleViewer, here:
http://server.nikhilvj.co.in:5040/


Regards, Nikhil VJ
Pune, India
https://nikhilvj.co.in

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/b9d0b443-3534-45f1-a4bb-c82306a35ec9%40googlegroups.com.

[datameet] Re: Kochi Data

[Nikhil VJ]

Hi, replying to a specific question about Kochi that was asked (and hence 
nearly lost!) in a different thread, on this pre-existing thread from the 
past carrying a better subject line.

Through an event conducted by Kochi Metro earlier this year, some datasets 
were shared. This general boundary of Kochi, named "GCDA" was shared. I 
don't think it has ward boundaries; it's got the full combined boundary. 
Attached. This might help in georeferencing and tracing the ward boundaries 
inside. 
*Disclaimer* : I got it in a zip with other files; no idea about authors 
and no assurances on accuracy.


Also, around the same event this mailing list for Kochi open data has been 
set up; I don't know about the participation but hey at least there's some 
people from Kochi on there. 
https://groups.google.com/forum/#!forum/kochiopendataforum 


Regards
Nikhil VJ
https://nikhilvj.co.in


On Monday, August 19, 2013 at 4:39:39 PM UTC+5:30, PRAKASH PS wrote:
>
> Hi all,
>
>
> Is there any source from which I can get shapefiles of Kochi proposed 
> metro stations and maps of Kerala State?
>
>
> Thank you
> Prakash
>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/41097975-f21e-46aa-a301-b291442ccf72%40googlegroups.com.
<>

[datameet] OpenGovDataHack Aug-Sep 2019

[Nikhil VJ]

Quoting from website:
"National Informatics Centre (NIC) in association with Internet & Mobile 
Association of India (IAMAI) is planning to conduct the Hackathon/Coding 
Challenge – #OpenGovDataHack 2019, for Entrepreneurs, Innovators, Start-ups 
and MSME to create unique and innovative service delivery Applications to 
foster data driven innovation. The aim is to look for local solutions to 
the local problems by the local entrepreneurs.

This Hackathon is to be conducted across four locations of India namely 
Raipur (Chhattisgarh), Kozhikode (Kerala), Rajkot (Gujarat) and Chandigarh 
(UT). It aims to Support & Showcase potentially great Ideas/talent from 
across the country by reaching out to their City."

read more on https://event.data.gov.in/event/opengovdatahack2019/

Cross-posting from OSM-India Telegram group where Naveen Francis has posted 
this.

-Nikhil VJ, Pune, India

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/datameet/549aa918-aa7d-4b57-b270-9931f37b6aa6%40googlegroups.com.