Hi Rahul and everyone, Thanks for clarifying the timezone, Rahul, and thank you both for the kind words.
I've added another useful layer to the database: AM/PM and full local timestamps for 330,118 of the 363,507 trips, using the related exports already in the shared collection. Indian Standard Time is now explicit, with UTC equivalents included for anyone combining this with other time-series data. The enriched edition is here: https://drive.google.com/file/d/1EpSQ4hrUt3Nw80nETayj4ip9N4f7dMfS/view This is the full standalone package, about 42 MB. Download and extract it, open goa-mining.sqlite, and use the same trip_evidence view as before. The added fields are also in the JSON export, and the README explains them. No separate merge is needed. All original trips, quantities, source references and supporting tables are retained. The remaining 33,389 trips still have their original dates and printed clocks; unresolved AM/PM is left blank rather than guessed. Daily totals remain unchanged. The earlier download remains available too. I hope this makes the next stage of analysis easier. Happy to help with useful next questions as the group explores it. Bryan CivicDataForge [email protected] On Sat, Sep 5, 2026 at 1:20 AM Rahul Basu <[email protected]> wrote: > Hi Bryan > > All the time stamps are Indian Standard Time. > > Rahul > Today is the first day of the rest of your life ! > > > On Sat, Sep 5, 2026 at 8:28 AM Rahul Basu <[email protected]> wrote: > >> Wow. That's amazing. And so quick.Thanks a ton Bryan. >> >> Rahul >> Today is the first day of the rest of your life ! >> >> >> On Sat, Sep 5, 2026 at 3:43 AM Bryan <[email protected]> wrote: >> >>> Hi Rahul and everyone, >>> >>> I've converted the LAQ 167 archive into a SQLite database, with >>> compressed JSON records and smaller JSON tables alongside it: >>> >>> https://drive.google.com/file/d/1Npw5WK7eOCr7hKzRiirjT0lgBGLDRcKS/view >>> >>> The ZIP is about 31 MB. Download and extract it, then open >>> goa-mining.sqlite in a SQLite viewer and select the trip_evidence view. The >>> README explains the tables, and QUERIES.sql includes starting queries. This >>> is a free community contribution, with no paid service needed to use the >>> file. >>> >>> What's in it: >>> - 363,507 individual trips from all ten Annexure D PDFs, covering 55,420 >>> pages. >>> - All 798 published daily route totals, plus block/lessee, monthly >>> production, dispatch and route-permission context from A/B/C. >>> - Source PDF names, page references and hashes, so a record can be >>> traced back to the original document. >>> >>> The trip-level counts and ore quantities reconcile exactly with all 798 >>> published daily route totals. There are no duplicate trip IDs or parsing >>> exceptions. I also checked every cell in 290 rows across 36 sampled pages >>> with a second PDF parser; that independent check is sampled, not every page. >>> >>> The actual trip dates range from 6 July 2024 to 19 August 2026, with >>> different cutoffs by block. The September answer date is not a claim that >>> every block is current to September. Block II has contextual records but no >>> trip PDF in Annexure D. >>> >>> All trips match a published route-permission entry using the route text, >>> without fuzzy matching. The package also flags a 0.01 difference in one >>> stated production total and four differing commencement-date statements >>> across annexures, retaining both versions. The printed clocks lack >>> AM/PM/timezone, so I haven't turned them into precise timestamps or claimed >>> hourly violations. AQI remains a linked companion source, not an inferred >>> route-to-sensor or causal match. >>> >>> If a row looks wrong, a PDF/page reference or trip ID will let us check >>> it precisely. I hope this saves the group a substantial amount of >>> preparation work. >>> >>> Bryan >>> CivicDataForge >>> [email protected] >>> On Friday, September 4, 2026 at 3:03:12 AM UTC-4 Rahul Basu wrote: >>> >>>> On mining transport, there's a big set of data as part of the >>>> Legislative Assembly Questions. Basically all the trips for major mining >>>> leases, routes, etc. It is in multiple PDFs, would greatly appreciate it if >>>> anyone can process these into databases. >>>> http://static.goavidhansabha.gov.in/goalpub/vr/167_01092026.zip >>>> >>>> Thanks in advance. >>>> >>>> Rahul >>>> Today is the first day of the rest of your life ! >>>> >>>> >>>> On Wed, Sep 2, 2026 at 2:56 PM Rahul Basu <[email protected]> wrote: >>>> >>>>> Friends, esp Sanjay & Bryan, >>>>> >>>>> I've organised the data better. There's a bunch of sub-folders in this >>>>> folder >>>>> <https://drive.google.com/drive/folders/1crvN-kiKZUYgFeu7PZwsjyhcYc99Q4ZA?usp=sharing>. >>>>> In each folder, I've tried to create a doc file that provides some >>>>> background. >>>>> >>>>> The mining transport data that Sanjay worked on is in Mining >>>>> Transport - Pub >>>>> <https://drive.google.com/drive/folders/1GTurKNqV02XI3_79zlDpQXZhf9w_C2bR?usp=sharing>. >>>>> I've added some older data to this folder, as well as newer data (could be >>>>> some overlap in periods). >>>>> >>>>> BTW, the work around the AQI data logger >>>>> <https://groups.google.com/g/datameet/c/k5H0xQqwSc4/m/AUD3QsYnDAAJ> by >>>>> Arun is basically air quality data from stations established specifically >>>>> to monitor air quality along these transportation routes. >>>>> >>>>> Please point out any additional information that may be useful for >>>>> analysis. >>>>> >>>>> Looking forward to the insights of the community. >>>>> >>>>> Rahul >>>>> Today is the first day of the rest of your life ! >>>>> >>>>> >>>>> On Sun, Jun 14, 2026 at 11:24 AM Rahul Basu <[email protected]> >>>>> wrote: >>>>> >>>>>> Friends, >>>>>> >>>>>> Thanks for the interest in this, unexpected for me. I now have to >>>>>> sort through the data and figure out how to organise it for public use. >>>>>> Give me some time on this. >>>>>> >>>>>> For the moment, here's two datasets I received under RTI around >>>>>> transport permits. >>>>>> >>>>>> *Background*: Goa runs an ore accounting software, which basically >>>>>> accounts for mineral ore at different locations. To move ore from one >>>>>> point >>>>>> to another, a transport permit is required. This is usually for a decent >>>>>> quantity, say 10,000 tons, to move from Mine X to Jetty Y. Then there are >>>>>> truck trips (usually 10 tons each), and the trucks are monitored real >>>>>> time. >>>>>> There are also trips by barge from the jetty to the ship, and some rail >>>>>> as >>>>>> well. When all the trips against the permit are completed, there's >>>>>> usually >>>>>> some difference, which gets adjusted. The software provider has changed >>>>>> from Megasoft to GEL. GEL software is called Bhumija. >>>>>> >>>>>> Mining was in three major phases: pre-2012 ban; 2015-2018; and 2024 >>>>>> onwards. Ore is mined in Goa, most exported, some consumed in Goa, some >>>>>> shipped by barge to JSW Dolvi, Maharashtra (recent). Ore is also imported >>>>>> into Goa from overseas. Some ore mined in Karnataka and Maharashtra also >>>>>> moves through Goa, mostly for export but occasionally for use in steel >>>>>> plants in Goa. >>>>>> >>>>>> The first folder >>>>>> <https://drive.google.com/drive/folders/1zeV-zquaHPq8pm6DkTjUHdIr9C_T-IBL?usp=sharing> >>>>>> has >>>>>> transport permits. Lots of them. From 1-Apr-2020-2025 >>>>>> >>>>>> The second folder >>>>>> <https://drive.google.com/drive/folders/1rx21ExPmuWpWAd4HA6BgISARTVAzjsF1?usp=sharing> >>>>>> is around closing stock also has some version of the manuals for the >>>>>> software. I had asked for: >>>>>> >>>>>> a) Manuals of the Bhumija software >>>>>> b) Guidelines / SOPs / instruction for monitoring ore through Bhumija >>>>>> c) In excel format, for each major mineral ore transport trip after >>>>>> 25-Mar-2024, permit number, carrier registration number, the source >>>>>> location with time and weight at the source weighbridge and the >>>>>> destination >>>>>> location with time and weight >>>>>> d) In excel format, closing stock of major mineral ore as of >>>>>> 31-Mar-2024 and 31-Mar-2025 for all locations in Bhumija >>>>>> >>>>>> I have a few questions from the data: >>>>>> 1) Does the ore balance at any location go negative at any point in >>>>>> time? >>>>>> 2) Overall statistics - major input locations, major transport >>>>>> routes, total quantities, etc. >>>>>> 3) Is the data complete? Is any further information needed / >>>>>> desireable? >>>>>> >>>>>> More questions will come as other data sets get added >>>>>> >>>>>> Please feel free to work with this while I try to set up a better >>>>>> data repository. Email me directly if you have questions on the data in >>>>>> the >>>>>> two folders. >>>>>> >>>>>> Warmly >>>>>> Rahul >>>>>> On Saturday, June 13, 2026 at 8:59:42 PM UTC+5:30 Natasha kalra wrote: >>>>>> >>>>>>> Hi Rahul, >>>>>>> >>>>>>> Thank you for sharing this. The dataset sounds extremely valuable, >>>>>>> especially given the combination of RTI-based information, government >>>>>>> records, GIS layers, and transport-chain data linking leases, jetties, >>>>>>> and >>>>>>> shipments. >>>>>>> >>>>>>> I work in the broader sustainability, environmental governance, and >>>>>>> data-driven policy space, and I would be interested in understanding the >>>>>>> scope of the datasets and the kinds of analyses you are looking to >>>>>>> undertake. I am affiliated with the Institute for Social and >>>>>>> Economic Change (ISEC), Bengaluru, a premier interdisciplinary research >>>>>>> institute engaged in policy-oriented research, consultancy, and >>>>>>> capacity-building across areas such as sustainability, environmental >>>>>>> governance, urban development, and social transformation. Through our >>>>>>> work, >>>>>>> we collaborate with government agencies, corporates, development >>>>>>> organizations, and communities to develop evidence-based and scalable >>>>>>> solutions to contemporary challenges. >>>>>>> >>>>>>> It would be great to connect and discuss the available data, current >>>>>>> status of data cleaning and organization, and potential avenues for >>>>>>> collaboration. Please let me know a convenient time for a call. >>>>>>> >>>>>>> Looking forward to learning more. >>>>>>> >>>>>>> Warm regards, >>>>>>> >>>>>>> Natasha >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Dr. Natasha Kalra >>>>>>> >>>>>>> Environmental Sustainability Consultant >>>>>>> >>>>>>> Initiator- BaansCraft >>>>>>> >>>>>>> https://www.instagram.com/baanscraft/ >>>>>>> >>>>>>> M: + 91 99167 70495 <+91%2099167%2070495> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sat, Jun 13, 2026 at 6:57 PM bibhu prasad mishra < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi Rahul, >>>>>>>> >>>>>>>> I am very interested in collaborating on this. As I am from Odisha, >>>>>>>> which is a major mineral hub, I believe this is a great opportunity to >>>>>>>> work >>>>>>>> with this data. >>>>>>>> >>>>>>>> I would love to help with the data manipulation and visualization, >>>>>>>> especially as you look toward the second phase involving Odisha. >>>>>>>> Please let >>>>>>>> me know how we can get started. >>>>>>>> >>>>>>>> >>>>>>>> Thanks and Regards >>>>>>>> *Bibhu Prasad Mishra* >>>>>>>> *Civil Engineer,* >>>>>>>> *B.tech, M.tech.* >>>>>>>> 7894761150 <078947%2061150>, 9937512121 <099375%2012121> >>>>>>>> >>>>>>>> On Sat, Jun 13, 2026, 6:28 PM Arun Ganesh <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Rahul, this is a very deep topic and requires many hands to dig >>>>>>>>> through (pun intended). is it possible to share links to some of the >>>>>>>>> collected data so it can be explored? >>>>>>>>> >>>>>>>>> Just extracting data from PDF tables requires a lot of effort and >>>>>>>>> it would be best to keep the data open so it can get maximum >>>>>>>>> interest. Or >>>>>>>>> are there reasons to not make this public given its sensitive nature? >>>>>>>>> >>>>>>>>> On Sat, Jun 13, 2026 at 12:58 PM Dr. C.Gajendran < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Dear Your email find interest to me; please contact me at >>>>>>>>>> 9443368980 <094433%2068980>. We shall discuss further. >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> >>>>>>>>>> Dr. C. GAJENDRAN B.E (Civil) M.E >>>>>>>>>> (Env.Eng.),Ph.D.,D.R.T.M.,D.I.S.,FIE(Ind.)., >>>>>>>>>> IGS., ISG., >>>>>>>>>> Vice Principal | Director – IQAC | Professor & Head, Civil >>>>>>>>>> Engineering | Vice President IIC >>>>>>>>>> *Jawaharlal College of Engineering and Technology, Palakkad* >>>>>>>>>> >>>>>>>>>> Mobile & Whatsup! : +91-9443368980 <+91%2094433%2068980> >>>>>>>>>> skype: c_gajendran >>>>>>>>>> Google Site <https://sites.google.com/view/drgajendran/home> >>>>>>>>>> ORCHID : 0000-0002-2765-7075 >>>>>>>>>> Vidwan <https://vidwan.inflibnet.ac.in/profile/96162> >>>>>>>>>> Scopus Author ID: 57359210200 >>>>>>>>>> <http://www.scopus.com/inward/authorDetails.url?authorID=57359210200&partnerID=MN8TOARS> >>>>>>>>>> >>>>>>>>>> Face book <https://www.facebook.com/cgajendran/> : >>>>>>>>>> Twitter : dr_gajendran >>>>>>>>>> Instagram : dr.gajendran >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sat, Jun 13, 2026 at 12:52 PM Rahul Basu <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi friends, >>>>>>>>>>> >>>>>>>>>>> I work on mining in India, primarily Goa. Over time, a lot of >>>>>>>>>>> data has been collected through various public sources and RTI. >>>>>>>>>>> Much of it >>>>>>>>>>> is in excel, a part is in GIS form. Some of the excel is old >>>>>>>>>>> government >>>>>>>>>>> style, needs openrefine to cleanup. >>>>>>>>>>> >>>>>>>>>>> Lots of interesting analysis can be done (for example >>>>>>>>>>> <https://ecoinsee.org/journal/ojs/index.php/ees/article/view/1878/355>). >>>>>>>>>>> Some of data includes detailed information on transport of >>>>>>>>>>> minerals from leases to jetties to ships. >>>>>>>>>>> >>>>>>>>>>> I'm not very good at data manipulation / data visualization and >>>>>>>>>>> was looking for collaborators. >>>>>>>>>>> >>>>>>>>>>> If anyone is interested, please ping me. >>>>>>>>>>> >>>>>>>>>>> PS. Odisha seems to have good mining data as well, though much >>>>>>>>>>> more data collection is needed. Can be looked at in a second phase. >>>>>>>>>>> >>>>>>>>>>> Warmly >>>>>>>>>>> Rahul >>>>>>>>>>> Today is the first day of the rest of your life ! >>>>>>>>>>> Goa Foundation <http://goafoundation.org> | The Future We Need >>>>>>>>>>> <http://thefutureweneed.org> | Goenchi Mati Movement >>>>>>>>>>> <https://linktr.ee/goenchimati> >>>>>>>>>>> https://orcid.org/0000-0002-1218-878X >>>>>>>>>>> The Supreme Court and Intergenerational Equity: The Goa Mining >>>>>>>>>>> Case >>>>>>>>>>> <https://www.banyantreebookstore.com/product-page/the-supreme-court-of-india-and-intergenerational-equity-the-goa-mining-case> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Datameet is a community of Data Science enthusiasts in India. >>>>>>>>>>> Know more about us by visiting http://datameet.org >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "datameet" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To view this discussion visit >>>>>>>>>>> https://groups.google.com/d/msgid/datameet/CAN8Wyx_6JXrvcYQnPSw7j%3DKZg2JEWSsB66ur4SRBGaZrsJuDRQ%40mail.gmail.com >>>>>>>>>>> <https://groups.google.com/d/msgid/datameet/CAN8Wyx_6JXrvcYQnPSw7j%3DKZg2JEWSsB66ur4SRBGaZrsJuDRQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>>> . >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Datameet is a community of Data Science enthusiasts in India. >>>>>>>>>> Know more about us by visiting http://datameet.org >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "datameet" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion visit >>>>>>>>>> https://groups.google.com/d/msgid/datameet/CAMFo2vtYsS6B%3DMs_du2eE8H3%2BYqTjcUfopNhXp0%3De_37fWqqLA%40mail.gmail.com >>>>>>>>>> <https://groups.google.com/d/msgid/datameet/CAMFo2vtYsS6B%3DMs_du2eE8H3%2BYqTjcUfopNhXp0%3De_37fWqqLA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>>>> more about us by visiting http://datameet.org >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "datameet" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion visit >>>>>>>>> https://groups.google.com/d/msgid/datameet/CA%2BGKQr1ph9XXnZ%3Dv5qC1ikSafjUqLu9gMCJcyqE6QX_FZ5gOHg%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/datameet/CA%2BGKQr1ph9XXnZ%3Dv5qC1ikSafjUqLu9gMCJcyqE6QX_FZ5gOHg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>>> more about us by visiting http://datameet.org >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "datameet" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> >>>>>>> To view this discussion visit >>>>>>>> https://groups.google.com/d/msgid/datameet/CAN%3DkOZnAQb-N%3D%3DLg5kfTwCgxhG2oaorB8p3PGdcXopJnMONiFQ%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/datameet/CAN%3DkOZnAQb-N%3D%3DLg5kfTwCgxhG2oaorB8p3PGdcXopJnMONiFQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>> more about us by visiting http://datameet.org >>>>>> --- >>>>>> You received this message because you are subscribed to a topic in >>>>>> the Google Groups "datameet" group. >>>>>> To unsubscribe from this topic, visit >>>>>> https://groups.google.com/d/topic/datameet/WLHGLnl5IX0/unsubscribe. >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>> [email protected]. >>>>>> To view this discussion visit >>>>>> https://groups.google.com/d/msgid/datameet/c67f7584-fcbc-4a1c-83ba-dff8b0e08085n%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/datameet/c67f7584-fcbc-4a1c-83ba-dff8b0e08085n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> Datameet is a community of Data Science enthusiasts in India. Know more >>> about us by visiting http://datameet.org >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "datameet" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/datameet/WLHGLnl5IX0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/datameet/fd96996b-f85d-4bc5-8174-384c56885192n%40googlegroups.com >>> <https://groups.google.com/d/msgid/datameet/fd96996b-f85d-4bc5-8174-384c56885192n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to a topic in the > Google Groups "datameet" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/datameet/WLHGLnl5IX0/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/datameet/CAN8Wyx9D2Zunt0Lxy77kg2QgPiy-qACRSk66DKet5pVrAiHojQ%40mail.gmail.com > <https://groups.google.com/d/msgid/datameet/CAN8Wyx9D2Zunt0Lxy77kg2QgPiy-qACRSk66DKet5pVrAiHojQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/datameet/CAFb%3Dd3iSTWZaC%2BRjyWhMEnScT16hyy3MxwFK4sNc%3DGfvDbLRZw%40mail.gmail.com.
