Re: [datameet] Data of Indian Railways

[Nikhil VJ]

Hi,  

We might be able to get some data from Train On Map site of Indian Railways?

https://enquiry.indianrail.gov.in/ntes/trainOnMap.jsp

(good to see the map is working again.. last few months it had been down 
after google api changes)


The browser console is yielding quite some stuff.

One JS file there is holding some data:

https://enquiry.indianrail.gov.in/ntes/js/stnCodesWithNamesArrayStr.js


(note: you may see weird chars.. that's unicode hindi, save the file 
locally and you'll see the chars coming proper. )

--

Another dataset we could get from here: Train (routes) data (unique code 
and name), with names in Hindi too.
See this API URL, I tried with wget and am getting results without having 
to do any cookie sessions etc:
https://enquiry.indianrail.gov.in/ntes/SearchTrain?trainNo=22

Change last arg for different results.. It gives max 30 results.

I changed the output to CSV by this process; it can be scripted: 
1. Ran though https://www.freeformatter.com/json-formatter.html which put 
quotes around all the keys. (But one could script this too?)
2. There is a "function(){..." line which needs to be taken care of. 
3. In advanced text editor, did Find+Replace-All on the following terms:
[search string] >> [replace with] 
function(){return _LANG==="en-us"?" >> "
":" >> ", "trainNameHindi": "
"}, >> ",
(don't remove quotes)

4. Now it becomes valid JSON. OpenRefine recognizes it and converts it to 
tabular form, and even this site converts to CSV: https://konklone.io/json/



Regards
Nikhil VJ Pune, India

On Friday, January 4, 2019 at 9:22:12 PM UTC+5:30, Srihari Thalla wrote:
>
> Thanks for the mention Arun!
>
> I have now updated the crawler - removing tabs, unwanted newlines, leading 
> and trailing spaces in the data columns.
>
> Here is the latest links to download:
> JSON: 
> https://api.apify.com/v1/execs/7t9roKQ9yp6T8ZnpR/results?format=json=1=1
> CSV: 
> https://api.apify.com/v1/execs/7t9roKQ9yp6T8ZnpR/results?format=csv=1=1
>
> Hope this helps!
>
> @Jasvinder I think one solution to extract locations for the stations is 
> via Overpass using the station codes and combining them to the spreadsheet.
>
> -- Srihari
>
>
> On Fri, 4 Jan 2019 at 12:43, Jasvinder Singh  > wrote:
>
>> Dear Arun,
>> Exactly the type of simple data sheet that newbies can understand. 
>> However how location (Coordinates) is linked in this file?
>>
>> On Fri, Jan 4, 2019 at 12:11 PM Arun Ganesh > > wrote:
>>
>>> Spreadsheet if anyone wants to explore: 
>>> https://docs.google.com/spreadsheets/d/1AFwl_5cB9qD39VWNox1LoeL3tGaGB22f7p4vc7IyMqY/edit#gid=0
>>>
>>> There are 16,770 station entries of which 11,660 seem to be currently 
>>> operational according to the expiry date of 2999. 
>>>
>>> Filtering out goods stations, there are 9835 entries. This still seems 
>>> to include a few yards and cabins that are not legitimate stations. Also 
>>> noticed quite a few spelling and formatting issues in the names. The 
>>> station codes look correct. Some amount of manual cleanup is needed on this 
>>> list. 
>>>
>>> The official number of stations according to IR is 7349 stations (as of 
>>> 2017) 
>>> 
>>> and 1817 halts/block huts (2013) 
>>> .
>>>  
>>>  
>>>
>>> On Fri, Jan 4, 2019 at 11:53 AM Arun Ganesh >> > wrote:
>>>
 Beauty of the internet, crawler got done by Srihari: 
 https://twitter.com/sriharithalla/status/1080801313707896837

 JSON data: 
 https://api.apify.com/v1/execs/TsBwnYutP5u9FCKp5/results?format=json=1

 I'm in the process of doing a little bit of cleanup using openrefine 
 and will share on a spreadsheet.

 On Fri, Jan 4, 2019 at 10:01 AM Jasvinder Singh >>> > wrote:

> Dear All,
>
> Not all the members are familiar with intricacies of the data 
> collection for such projects. Since this seems to be a crowd sourcing 
> endeavour, I suggest that the basic data collection protocol be 
> enumerated 
> for newbies so that they can also contribute data which can then be put 
> in 
> proper format by professionals. 
>
> Regards,
>
> Jasvinder Singh
>
> On Tue, Dec 25, 2018 at 10:32 AM Nikhil VJ  > wrote:
>
>> Hi folks,
>>
>> There's a project afoot in the OpenStreetMap and Wikidata communities 
>> to get together Indian Railways data.
>>
>> One major part of it: Properly mapping all the railway stations of 
>> India, and ensuring they have wikidata entries.
>>
>> Here's a wiki page set up for it: 
>> https://www.wikidata.org/wiki/Wikidata:WikiProject_Indian_Railways 
>>
>> I'm cross-posting from OpenStreetMap India Telegram 
>>  group:
>>
>> (Arun Ganesh): There 

Re: [datameet] Mapping of Hyderabad Bus Stops and Routes

[Nikhil VJ]

Hi Friends, 

We had a great mapping session yesterday. Most valuable was the tacit local 
information of the participants.
We're changing venue and timings today :

6th Jan : continuation of Hyderabad Bus Routes mapping and discussion about 
project
3rd Floor, Abhyaas (opp JNTU gate, behind ICICI bank)
1pm to 6pm (break around 3pm)


Regards
Nikhil VJ

On Friday, January 4, 2019 at 3:27:32 PM UTC+5:30, Nikhil VJ wrote:
>
> Hi Folks,
>
> Apologies for the late post, but please share this with your friends in 
> Hyderabad. We're conducting a Mapathon for mapping Hyderabad's Bus Routes.
>
> 5th, 6th Jan 2019, JNTU College, 10am to 6pm. Participants can also come 
> on just one of the days.
>
>
> 
>
> Direct link to poster: https://i.imgur.com/7dvSfIt.jpg
>
>
> Regards
> Nikhil VJ, Pune, India
>
>
> On Tuesday, December 11, 2018 at 4:39:34 PM UTC+5:30, Nikhil VJ wrote:
>>
>> Hi Srihari,
>>
>> Session was good! Our main purpose was to announce about the project and 
>> invite the larger GIS community in Hyderabad to come in and get involved, 
>> share ideas or even take up some tasks if they feel inclined. So that 
>> invite remains open - pls contact me on me [at] nikhilvj.co.in if anyone 
>> wants to get involved and is ready for grunt work (I prefer Hyderabad-based 
>> folks because you'll need to know those places, but there are opportunities 
>> for folks outside too).
>>
>> I'm very excited about the possibility of detecting common patterns 
>> between the routes, suggested by one of the participants whose group works 
>> with Random Forest. If we can bring common pattern detection and editing to 
>> routes management it'll mean a major time-saving in editing routes. 
>> (Example use case: one more stop is added between two existing ones. The 
>> change should get rippled across all the routes/trips that ply there.)
>>
>> I've copied links gathered at the session and written some detailed 
>> explanations on this swecha forum post (finally seeing a discourse forum 
>> that's India-specific), will follow up over there with more stuff so follow 
>> it there. The data entry and route mapping apps we've developed can be 
>> applied to other places too. I haven't published their code yet, hope to do 
>> so soon.
>>
>>
>> https://discuss.swecha.org/t/9-dec-18-session-on-bus-routes-mapping-project-for-hyderabad/218
>>
>>
>> Regards
>> Nikhil VJ
>> Pune, India
>>
>>

-- 
Datameet is a community of Data Science enthusiasts in India. Know more about 
us by visiting http://datameet.org
--- 
You received this message because you are subscribed to the Google Groups 
"datameet" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to datameet+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.