Hi Rahul and everyone, I've converted the LAQ 167 archive into a SQLite database, with compressed JSON records and smaller JSON tables alongside it:
https://drive.google.com/file/d/1Npw5WK7eOCr7hKzRiirjT0lgBGLDRcKS/view The ZIP is about 31 MB. Download and extract it, then open goa-mining.sqlite in a SQLite viewer and select the trip_evidence view. The README explains the tables, and QUERIES.sql includes starting queries. This is a free community contribution, with no paid service needed to use the file. What's in it: - 363,507 individual trips from all ten Annexure D PDFs, covering 55,420 pages. - All 798 published daily route totals, plus block/lessee, monthly production, dispatch and route-permission context from A/B/C. - Source PDF names, page references and hashes, so a record can be traced back to the original document. The trip-level counts and ore quantities reconcile exactly with all 798 published daily route totals. There are no duplicate trip IDs or parsing exceptions. I also checked every cell in 290 rows across 36 sampled pages with a second PDF parser; that independent check is sampled, not every page. The actual trip dates range from 6 July 2024 to 19 August 2026, with different cutoffs by block. The September answer date is not a claim that every block is current to September. Block II has contextual records but no trip PDF in Annexure D. All trips match a published route-permission entry using the route text, without fuzzy matching. The package also flags a 0.01 difference in one stated production total and four differing commencement-date statements across annexures, retaining both versions. The printed clocks lack AM/PM/timezone, so I haven't turned them into precise timestamps or claimed hourly violations. AQI remains a linked companion source, not an inferred route-to-sensor or causal match. If a row looks wrong, a PDF/page reference or trip ID will let us check it precisely. I hope this saves the group a substantial amount of preparation work. Bryan CivicDataForge [email protected] On Friday, September 4, 2026 at 3:03:12 AM UTC-4 Rahul Basu wrote: > On mining transport, there's a big set of data as part of the Legislative > Assembly Questions. Basically all the trips for major mining leases, > routes, etc. It is in multiple PDFs, would greatly appreciate it if anyone > can process these into databases. > http://static.goavidhansabha.gov.in/goalpub/vr/167_01092026.zip > > Thanks in advance. > > Rahul > Today is the first day of the rest of your life ! > > > On Wed, Sep 2, 2026 at 2:56 PM Rahul Basu <[email protected]> wrote: > >> Friends, esp Sanjay & Bryan, >> >> I've organised the data better. There's a bunch of sub-folders in this >> folder >> <https://drive.google.com/drive/folders/1crvN-kiKZUYgFeu7PZwsjyhcYc99Q4ZA?usp=sharing>. >> >> In each folder, I've tried to create a doc file that provides some >> background. >> >> The mining transport data that Sanjay worked on is in Mining Transport - >> Pub >> <https://drive.google.com/drive/folders/1GTurKNqV02XI3_79zlDpQXZhf9w_C2bR?usp=sharing>. >> >> I've added some older data to this folder, as well as newer data (could be >> some overlap in periods). >> >> BTW, the work around the AQI data logger >> <https://groups.google.com/g/datameet/c/k5H0xQqwSc4/m/AUD3QsYnDAAJ> by >> Arun is basically air quality data from stations established specifically >> to monitor air quality along these transportation routes. >> >> Please point out any additional information that may be useful for >> analysis. >> >> Looking forward to the insights of the community. >> >> Rahul >> Today is the first day of the rest of your life ! >> >> >> On Sun, Jun 14, 2026 at 11:24 AM Rahul Basu <[email protected]> wrote: >> >>> Friends, >>> >>> Thanks for the interest in this, unexpected for me. I now have to sort >>> through the data and figure out how to organise it for public use. Give me >>> some time on this. >>> >>> For the moment, here's two datasets I received under RTI around >>> transport permits. >>> >>> *Background*: Goa runs an ore accounting software, which basically >>> accounts for mineral ore at different locations. To move ore from one point >>> to another, a transport permit is required. This is usually for a decent >>> quantity, say 10,000 tons, to move from Mine X to Jetty Y. Then there are >>> truck trips (usually 10 tons each), and the trucks are monitored real time. >>> There are also trips by barge from the jetty to the ship, and some rail as >>> well. When all the trips against the permit are completed, there's usually >>> some difference, which gets adjusted. The software provider has changed >>> from Megasoft to GEL. GEL software is called Bhumija. >>> >>> Mining was in three major phases: pre-2012 ban; 2015-2018; and 2024 >>> onwards. Ore is mined in Goa, most exported, some consumed in Goa, some >>> shipped by barge to JSW Dolvi, Maharashtra (recent). Ore is also imported >>> into Goa from overseas. Some ore mined in Karnataka and Maharashtra also >>> moves through Goa, mostly for export but occasionally for use in steel >>> plants in Goa. >>> >>> The first folder >>> <https://drive.google.com/drive/folders/1zeV-zquaHPq8pm6DkTjUHdIr9C_T-IBL?usp=sharing> >>> has >>> transport permits. Lots of them. From 1-Apr-2020-2025 >>> >>> The second folder >>> <https://drive.google.com/drive/folders/1rx21ExPmuWpWAd4HA6BgISARTVAzjsF1?usp=sharing> >>> >>> is around closing stock also has some version of the manuals for the >>> software. I had asked for: >>> >>> a) Manuals of the Bhumija software >>> b) Guidelines / SOPs / instruction for monitoring ore through Bhumija >>> c) In excel format, for each major mineral ore transport trip after >>> 25-Mar-2024, permit number, carrier registration number, the source >>> location with time and weight at the source weighbridge and the destination >>> location with time and weight >>> d) In excel format, closing stock of major mineral ore as of 31-Mar-2024 >>> and 31-Mar-2025 for all locations in Bhumija >>> >>> I have a few questions from the data: >>> 1) Does the ore balance at any location go negative at any point in time? >>> 2) Overall statistics - major input locations, major transport routes, >>> total quantities, etc. >>> 3) Is the data complete? Is any further information needed / desireable? >>> >>> More questions will come as other data sets get added >>> >>> Please feel free to work with this while I try to set up a better data >>> repository. Email me directly if you have questions on the data in the two >>> folders. >>> >>> Warmly >>> Rahul >>> On Saturday, June 13, 2026 at 8:59:42 PM UTC+5:30 Natasha kalra wrote: >>> >>>> Hi Rahul, >>>> >>>> Thank you for sharing this. The dataset sounds extremely valuable, >>>> especially given the combination of RTI-based information, government >>>> records, GIS layers, and transport-chain data linking leases, jetties, and >>>> shipments. >>>> >>>> I work in the broader sustainability, environmental governance, and >>>> data-driven policy space, and I would be interested in understanding the >>>> scope of the datasets and the kinds of analyses you are looking to >>>> undertake. I am affiliated with the Institute for Social and Economic >>>> Change (ISEC), Bengaluru, a premier interdisciplinary research institute >>>> engaged in policy-oriented research, consultancy, and capacity-building >>>> across areas such as sustainability, environmental governance, urban >>>> development, and social transformation. Through our work, we collaborate >>>> with government agencies, corporates, development organizations, and >>>> communities to develop evidence-based and scalable solutions to >>>> contemporary challenges. >>>> >>>> It would be great to connect and discuss the available data, current >>>> status of data cleaning and organization, and potential avenues for >>>> collaboration. Please let me know a convenient time for a call. >>>> >>>> Looking forward to learning more. >>>> >>>> Warm regards, >>>> >>>> Natasha >>>> >>>> >>>> >>>> >>>> Dr. Natasha Kalra >>>> >>>> Environmental Sustainability Consultant >>>> >>>> Initiator- BaansCraft >>>> >>>> https://www.instagram.com/baanscraft/ >>>> >>>> M: + 91 99167 70495 <+91%2099167%2070495> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Sat, Jun 13, 2026 at 6:57 PM bibhu prasad mishra < >>>> [email protected]> wrote: >>>> >>>>> Hi Rahul, >>>>> >>>>> I am very interested in collaborating on this. As I am from Odisha, >>>>> which is a major mineral hub, I believe this is a great opportunity to >>>>> work >>>>> with this data. >>>>> >>>>> I would love to help with the data manipulation and visualization, >>>>> especially as you look toward the second phase involving Odisha. Please >>>>> let >>>>> me know how we can get started. >>>>> >>>>> >>>>> Thanks and Regards >>>>> *Bibhu Prasad Mishra* >>>>> *Civil Engineer,* >>>>> *B.tech, M.tech.* >>>>> 7894761150 <078947%2061150>, 9937512121 <099375%2012121> >>>>> >>>>> On Sat, Jun 13, 2026, 6:28 PM Arun Ganesh <[email protected]> wrote: >>>>> >>>>>> Hi Rahul, this is a very deep topic and requires many hands to dig >>>>>> through (pun intended). is it possible to share links to some of the >>>>>> collected data so it can be explored? >>>>>> >>>>>> Just extracting data from PDF tables requires a lot of effort and it >>>>>> would be best to keep the data open so it can get maximum interest. Or >>>>>> are >>>>>> there reasons to not make this public given its sensitive nature? >>>>>> >>>>>> On Sat, Jun 13, 2026 at 12:58 PM Dr. C.Gajendran <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Dear Your email find interest to me; please contact me at 9443368980 >>>>>>> <094433%2068980>. We shall discuss further. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Dr. C. GAJENDRAN B.E (Civil) M.E >>>>>>> (Env.Eng.),Ph.D.,D.R.T.M.,D.I.S.,FIE(Ind.)., >>>>>>> IGS., ISG., >>>>>>> Vice Principal | Director – IQAC | Professor & Head, Civil >>>>>>> Engineering | Vice President IIC >>>>>>> *Jawaharlal College of Engineering and Technology, Palakkad* >>>>>>> >>>>>>> Mobile & Whatsup! : +91-9443368980 <+91%2094433%2068980> >>>>>>> skype: c_gajendran >>>>>>> Google Site <https://sites.google.com/view/drgajendran/home> >>>>>>> ORCHID : 0000-0002-2765-7075 >>>>>>> Vidwan <https://vidwan.inflibnet.ac.in/profile/96162> >>>>>>> Scopus Author ID: 57359210200 >>>>>>> <http://www.scopus.com/inward/authorDetails.url?authorID=57359210200&partnerID=MN8TOARS> >>>>>>> >>>>>>> Face book <https://www.facebook.com/cgajendran/> : >>>>>>> Twitter : dr_gajendran >>>>>>> Instagram : dr.gajendran >>>>>>> >>>>>>> >>>>>>> On Sat, Jun 13, 2026 at 12:52 PM Rahul Basu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi friends, >>>>>>>> >>>>>>>> I work on mining in India, primarily Goa. Over time, a lot of data >>>>>>>> has been collected through various public sources and RTI. Much of it >>>>>>>> is >>>>>>>> in excel, a part is in GIS form. Some of the excel is old government >>>>>>>> style, >>>>>>>> needs openrefine to cleanup. >>>>>>>> >>>>>>>> Lots of interesting analysis can be done (for example >>>>>>>> <https://ecoinsee.org/journal/ojs/index.php/ees/article/view/1878/355>). >>>>>>>> >>>>>>>> Some of data includes detailed information on transport of >>>>>>>> minerals from leases to jetties to ships. >>>>>>>> >>>>>>>> I'm not very good at data manipulation / data visualization and was >>>>>>>> looking for collaborators. >>>>>>>> >>>>>>>> If anyone is interested, please ping me. >>>>>>>> >>>>>>>> PS. Odisha seems to have good mining data as well, though much more >>>>>>>> data collection is needed. Can be looked at in a second phase. >>>>>>>> >>>>>>>> Warmly >>>>>>>> Rahul >>>>>>>> Today is the first day of the rest of your life ! >>>>>>>> Goa Foundation <http://goafoundation.org> | The Future We Need >>>>>>>> <http://thefutureweneed.org> | Goenchi Mati Movement >>>>>>>> <https://linktr.ee/goenchimati> >>>>>>>> https://orcid.org/0000-0002-1218-878X >>>>>>>> The Supreme Court and Intergenerational Equity: The Goa Mining Case >>>>>>>> <https://www.banyantreebookstore.com/product-page/the-supreme-court-of-india-and-intergenerational-equity-the-goa-mining-case> >>>>>>>> >>>>>>>> -- >>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>>> more about us by visiting http://datameet.org >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "datameet" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion visit >>>>>>>> https://groups.google.com/d/msgid/datameet/CAN8Wyx_6JXrvcYQnPSw7j%3DKZg2JEWSsB66ur4SRBGaZrsJuDRQ%40mail.gmail.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/datameet/CAN8Wyx_6JXrvcYQnPSw7j%3DKZg2JEWSsB66ur4SRBGaZrsJuDRQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>> more about us by visiting http://datameet.org >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "datameet" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion visit >>>>>>> https://groups.google.com/d/msgid/datameet/CAMFo2vtYsS6B%3DMs_du2eE8H3%2BYqTjcUfopNhXp0%3De_37fWqqLA%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/datameet/CAMFo2vtYsS6B%3DMs_du2eE8H3%2BYqTjcUfopNhXp0%3De_37fWqqLA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>> more about us by visiting http://datameet.org >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "datameet" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion visit >>>>>> https://groups.google.com/d/msgid/datameet/CA%2BGKQr1ph9XXnZ%3Dv5qC1ikSafjUqLu9gMCJcyqE6QX_FZ5gOHg%40mail.gmail.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/datameet/CA%2BGKQr1ph9XXnZ%3Dv5qC1ikSafjUqLu9gMCJcyqE6QX_FZ5gOHg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>> more about us by visiting http://datameet.org >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "datameet" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> >>>> To view this discussion visit >>>>> https://groups.google.com/d/msgid/datameet/CAN%3DkOZnAQb-N%3D%3DLg5kfTwCgxhG2oaorB8p3PGdcXopJnMONiFQ%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/datameet/CAN%3DkOZnAQb-N%3D%3DLg5kfTwCgxhG2oaorB8p3PGdcXopJnMONiFQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> Datameet is a community of Data Science enthusiasts in India. Know more >>> about us by visiting http://datameet.org >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "datameet" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/datameet/WLHGLnl5IX0/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion visit >>> https://groups.google.com/d/msgid/datameet/c67f7584-fcbc-4a1c-83ba-dff8b0e08085n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/datameet/c67f7584-fcbc-4a1c-83ba-dff8b0e08085n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/datameet/fd96996b-f85d-4bc5-8174-384c56885192n%40googlegroups.com.
