Wow. That's amazing. And so quick.Thanks a ton Bryan. Rahul Today is the first day of the rest of your life !
On Sat, Sep 5, 2026 at 3:43 AM Bryan <[email protected]> wrote: > Hi Rahul and everyone, > > I've converted the LAQ 167 archive into a SQLite database, with compressed > JSON records and smaller JSON tables alongside it: > > https://drive.google.com/file/d/1Npw5WK7eOCr7hKzRiirjT0lgBGLDRcKS/view > > The ZIP is about 31 MB. Download and extract it, then open > goa-mining.sqlite in a SQLite viewer and select the trip_evidence view. The > README explains the tables, and QUERIES.sql includes starting queries. This > is a free community contribution, with no paid service needed to use the > file. > > What's in it: > - 363,507 individual trips from all ten Annexure D PDFs, covering 55,420 > pages. > - All 798 published daily route totals, plus block/lessee, monthly > production, dispatch and route-permission context from A/B/C. > - Source PDF names, page references and hashes, so a record can be traced > back to the original document. > > The trip-level counts and ore quantities reconcile exactly with all 798 > published daily route totals. There are no duplicate trip IDs or parsing > exceptions. I also checked every cell in 290 rows across 36 sampled pages > with a second PDF parser; that independent check is sampled, not every page. > > The actual trip dates range from 6 July 2024 to 19 August 2026, with > different cutoffs by block. The September answer date is not a claim that > every block is current to September. Block II has contextual records but no > trip PDF in Annexure D. > > All trips match a published route-permission entry using the route text, > without fuzzy matching. The package also flags a 0.01 difference in one > stated production total and four differing commencement-date statements > across annexures, retaining both versions. The printed clocks lack > AM/PM/timezone, so I haven't turned them into precise timestamps or claimed > hourly violations. AQI remains a linked companion source, not an inferred > route-to-sensor or causal match. > > If a row looks wrong, a PDF/page reference or trip ID will let us check it > precisely. I hope this saves the group a substantial amount of preparation > work. > > Bryan > CivicDataForge > [email protected] > On Friday, September 4, 2026 at 3:03:12 AM UTC-4 Rahul Basu wrote: > >> On mining transport, there's a big set of data as part of the Legislative >> Assembly Questions. Basically all the trips for major mining leases, >> routes, etc. It is in multiple PDFs, would greatly appreciate it if anyone >> can process these into databases. >> http://static.goavidhansabha.gov.in/goalpub/vr/167_01092026.zip >> >> Thanks in advance. >> >> Rahul >> Today is the first day of the rest of your life ! >> >> >> On Wed, Sep 2, 2026 at 2:56 PM Rahul Basu <[email protected]> wrote: >> >>> Friends, esp Sanjay & Bryan, >>> >>> I've organised the data better. There's a bunch of sub-folders in this >>> folder >>> <https://drive.google.com/drive/folders/1crvN-kiKZUYgFeu7PZwsjyhcYc99Q4ZA?usp=sharing>. >>> In each folder, I've tried to create a doc file that provides some >>> background. >>> >>> The mining transport data that Sanjay worked on is in Mining Transport >>> - Pub >>> <https://drive.google.com/drive/folders/1GTurKNqV02XI3_79zlDpQXZhf9w_C2bR?usp=sharing>. >>> I've added some older data to this folder, as well as newer data (could be >>> some overlap in periods). >>> >>> BTW, the work around the AQI data logger >>> <https://groups.google.com/g/datameet/c/k5H0xQqwSc4/m/AUD3QsYnDAAJ> by >>> Arun is basically air quality data from stations established specifically >>> to monitor air quality along these transportation routes. >>> >>> Please point out any additional information that may be useful for >>> analysis. >>> >>> Looking forward to the insights of the community. >>> >>> Rahul >>> Today is the first day of the rest of your life ! >>> >>> >>> On Sun, Jun 14, 2026 at 11:24 AM Rahul Basu <[email protected]> wrote: >>> >>>> Friends, >>>> >>>> Thanks for the interest in this, unexpected for me. I now have to sort >>>> through the data and figure out how to organise it for public use. Give me >>>> some time on this. >>>> >>>> For the moment, here's two datasets I received under RTI around >>>> transport permits. >>>> >>>> *Background*: Goa runs an ore accounting software, which basically >>>> accounts for mineral ore at different locations. To move ore from one point >>>> to another, a transport permit is required. This is usually for a decent >>>> quantity, say 10,000 tons, to move from Mine X to Jetty Y. Then there are >>>> truck trips (usually 10 tons each), and the trucks are monitored real time. >>>> There are also trips by barge from the jetty to the ship, and some rail as >>>> well. When all the trips against the permit are completed, there's usually >>>> some difference, which gets adjusted. The software provider has changed >>>> from Megasoft to GEL. GEL software is called Bhumija. >>>> >>>> Mining was in three major phases: pre-2012 ban; 2015-2018; and 2024 >>>> onwards. Ore is mined in Goa, most exported, some consumed in Goa, some >>>> shipped by barge to JSW Dolvi, Maharashtra (recent). Ore is also imported >>>> into Goa from overseas. Some ore mined in Karnataka and Maharashtra also >>>> moves through Goa, mostly for export but occasionally for use in steel >>>> plants in Goa. >>>> >>>> The first folder >>>> <https://drive.google.com/drive/folders/1zeV-zquaHPq8pm6DkTjUHdIr9C_T-IBL?usp=sharing> >>>> has >>>> transport permits. Lots of them. From 1-Apr-2020-2025 >>>> >>>> The second folder >>>> <https://drive.google.com/drive/folders/1rx21ExPmuWpWAd4HA6BgISARTVAzjsF1?usp=sharing> >>>> is around closing stock also has some version of the manuals for the >>>> software. I had asked for: >>>> >>>> a) Manuals of the Bhumija software >>>> b) Guidelines / SOPs / instruction for monitoring ore through Bhumija >>>> c) In excel format, for each major mineral ore transport trip after >>>> 25-Mar-2024, permit number, carrier registration number, the source >>>> location with time and weight at the source weighbridge and the destination >>>> location with time and weight >>>> d) In excel format, closing stock of major mineral ore as of >>>> 31-Mar-2024 and 31-Mar-2025 for all locations in Bhumija >>>> >>>> I have a few questions from the data: >>>> 1) Does the ore balance at any location go negative at any point in >>>> time? >>>> 2) Overall statistics - major input locations, major transport routes, >>>> total quantities, etc. >>>> 3) Is the data complete? Is any further information needed / desireable? >>>> >>>> More questions will come as other data sets get added >>>> >>>> Please feel free to work with this while I try to set up a better data >>>> repository. Email me directly if you have questions on the data in the two >>>> folders. >>>> >>>> Warmly >>>> Rahul >>>> On Saturday, June 13, 2026 at 8:59:42 PM UTC+5:30 Natasha kalra wrote: >>>> >>>>> Hi Rahul, >>>>> >>>>> Thank you for sharing this. The dataset sounds extremely valuable, >>>>> especially given the combination of RTI-based information, government >>>>> records, GIS layers, and transport-chain data linking leases, jetties, and >>>>> shipments. >>>>> >>>>> I work in the broader sustainability, environmental governance, and >>>>> data-driven policy space, and I would be interested in understanding the >>>>> scope of the datasets and the kinds of analyses you are looking to >>>>> undertake. I am affiliated with the Institute for Social and Economic >>>>> Change (ISEC), Bengaluru, a premier interdisciplinary research institute >>>>> engaged in policy-oriented research, consultancy, and capacity-building >>>>> across areas such as sustainability, environmental governance, urban >>>>> development, and social transformation. Through our work, we collaborate >>>>> with government agencies, corporates, development organizations, and >>>>> communities to develop evidence-based and scalable solutions to >>>>> contemporary challenges. >>>>> >>>>> It would be great to connect and discuss the available data, current >>>>> status of data cleaning and organization, and potential avenues for >>>>> collaboration. Please let me know a convenient time for a call. >>>>> >>>>> Looking forward to learning more. >>>>> >>>>> Warm regards, >>>>> >>>>> Natasha >>>>> >>>>> >>>>> >>>>> >>>>> Dr. Natasha Kalra >>>>> >>>>> Environmental Sustainability Consultant >>>>> >>>>> Initiator- BaansCraft >>>>> >>>>> https://www.instagram.com/baanscraft/ >>>>> >>>>> M: + 91 99167 70495 <+91%2099167%2070495> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Sat, Jun 13, 2026 at 6:57 PM bibhu prasad mishra < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Rahul, >>>>>> >>>>>> I am very interested in collaborating on this. As I am from Odisha, >>>>>> which is a major mineral hub, I believe this is a great opportunity to >>>>>> work >>>>>> with this data. >>>>>> >>>>>> I would love to help with the data manipulation and visualization, >>>>>> especially as you look toward the second phase involving Odisha. Please >>>>>> let >>>>>> me know how we can get started. >>>>>> >>>>>> >>>>>> Thanks and Regards >>>>>> *Bibhu Prasad Mishra* >>>>>> *Civil Engineer,* >>>>>> *B.tech, M.tech.* >>>>>> 7894761150 <078947%2061150>, 9937512121 <099375%2012121> >>>>>> >>>>>> On Sat, Jun 13, 2026, 6:28 PM Arun Ganesh <[email protected]> wrote: >>>>>> >>>>>>> Hi Rahul, this is a very deep topic and requires many hands to dig >>>>>>> through (pun intended). is it possible to share links to some of the >>>>>>> collected data so it can be explored? >>>>>>> >>>>>>> Just extracting data from PDF tables requires a lot of effort and it >>>>>>> would be best to keep the data open so it can get maximum interest. Or >>>>>>> are >>>>>>> there reasons to not make this public given its sensitive nature? >>>>>>> >>>>>>> On Sat, Jun 13, 2026 at 12:58 PM Dr. C.Gajendran <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Dear Your email find interest to me; please contact me at >>>>>>>> 9443368980 <094433%2068980>. We shall discuss further. >>>>>>>> >>>>>>>> Regards, >>>>>>>> >>>>>>>> Dr. C. GAJENDRAN B.E (Civil) M.E >>>>>>>> (Env.Eng.),Ph.D.,D.R.T.M.,D.I.S.,FIE(Ind.)., >>>>>>>> IGS., ISG., >>>>>>>> Vice Principal | Director – IQAC | Professor & Head, Civil >>>>>>>> Engineering | Vice President IIC >>>>>>>> *Jawaharlal College of Engineering and Technology, Palakkad* >>>>>>>> >>>>>>>> Mobile & Whatsup! : +91-9443368980 <+91%2094433%2068980> >>>>>>>> skype: c_gajendran >>>>>>>> Google Site <https://sites.google.com/view/drgajendran/home> >>>>>>>> ORCHID : 0000-0002-2765-7075 >>>>>>>> Vidwan <https://vidwan.inflibnet.ac.in/profile/96162> >>>>>>>> Scopus Author ID: 57359210200 >>>>>>>> <http://www.scopus.com/inward/authorDetails.url?authorID=57359210200&partnerID=MN8TOARS> >>>>>>>> >>>>>>>> Face book <https://www.facebook.com/cgajendran/> : >>>>>>>> Twitter : dr_gajendran >>>>>>>> Instagram : dr.gajendran >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Jun 13, 2026 at 12:52 PM Rahul Basu <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi friends, >>>>>>>>> >>>>>>>>> I work on mining in India, primarily Goa. Over time, a lot of data >>>>>>>>> has been collected through various public sources and RTI. Much of >>>>>>>>> it is >>>>>>>>> in excel, a part is in GIS form. Some of the excel is old government >>>>>>>>> style, >>>>>>>>> needs openrefine to cleanup. >>>>>>>>> >>>>>>>>> Lots of interesting analysis can be done (for example >>>>>>>>> <https://ecoinsee.org/journal/ojs/index.php/ees/article/view/1878/355>). >>>>>>>>> Some of data includes detailed information on transport of >>>>>>>>> minerals from leases to jetties to ships. >>>>>>>>> >>>>>>>>> I'm not very good at data manipulation / data visualization and >>>>>>>>> was looking for collaborators. >>>>>>>>> >>>>>>>>> If anyone is interested, please ping me. >>>>>>>>> >>>>>>>>> PS. Odisha seems to have good mining data as well, though much >>>>>>>>> more data collection is needed. Can be looked at in a second phase. >>>>>>>>> >>>>>>>>> Warmly >>>>>>>>> Rahul >>>>>>>>> Today is the first day of the rest of your life ! >>>>>>>>> Goa Foundation <http://goafoundation.org> | The Future We Need >>>>>>>>> <http://thefutureweneed.org> | Goenchi Mati Movement >>>>>>>>> <https://linktr.ee/goenchimati> >>>>>>>>> https://orcid.org/0000-0002-1218-878X >>>>>>>>> The Supreme Court and Intergenerational Equity: The Goa Mining Case >>>>>>>>> <https://www.banyantreebookstore.com/product-page/the-supreme-court-of-india-and-intergenerational-equity-the-goa-mining-case> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>>>> more about us by visiting http://datameet.org >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "datameet" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion visit >>>>>>>>> https://groups.google.com/d/msgid/datameet/CAN8Wyx_6JXrvcYQnPSw7j%3DKZg2JEWSsB66ur4SRBGaZrsJuDRQ%40mail.gmail.com >>>>>>>>> <https://groups.google.com/d/msgid/datameet/CAN8Wyx_6JXrvcYQnPSw7j%3DKZg2JEWSsB66ur4SRBGaZrsJuDRQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>>> more about us by visiting http://datameet.org >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "datameet" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion visit >>>>>>>> https://groups.google.com/d/msgid/datameet/CAMFo2vtYsS6B%3DMs_du2eE8H3%2BYqTjcUfopNhXp0%3De_37fWqqLA%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/datameet/CAMFo2vtYsS6B%3DMs_du2eE8H3%2BYqTjcUfopNhXp0%3De_37fWqqLA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>>> more about us by visiting http://datameet.org >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "datameet" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion visit >>>>>>> https://groups.google.com/d/msgid/datameet/CA%2BGKQr1ph9XXnZ%3Dv5qC1ikSafjUqLu9gMCJcyqE6QX_FZ5gOHg%40mail.gmail.com >>>>>>> <https://groups.google.com/d/msgid/datameet/CA%2BGKQr1ph9XXnZ%3Dv5qC1ikSafjUqLu9gMCJcyqE6QX_FZ5gOHg%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> Datameet is a community of Data Science enthusiasts in India. Know >>>>>> more about us by visiting http://datameet.org >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "datameet" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> >>>>> To view this discussion visit >>>>>> https://groups.google.com/d/msgid/datameet/CAN%3DkOZnAQb-N%3D%3DLg5kfTwCgxhG2oaorB8p3PGdcXopJnMONiFQ%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/datameet/CAN%3DkOZnAQb-N%3D%3DLg5kfTwCgxhG2oaorB8p3PGdcXopJnMONiFQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> Datameet is a community of Data Science enthusiasts in India. Know more >>>> about us by visiting http://datameet.org >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "datameet" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/datameet/WLHGLnl5IX0/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion visit >>>> https://groups.google.com/d/msgid/datameet/c67f7584-fcbc-4a1c-83ba-dff8b0e08085n%40googlegroups.com >>>> <https://groups.google.com/d/msgid/datameet/c67f7584-fcbc-4a1c-83ba-dff8b0e08085n%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > Datameet is a community of Data Science enthusiasts in India. Know more > about us by visiting http://datameet.org > --- > You received this message because you are subscribed to a topic in the > Google Groups "datameet" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/datameet/WLHGLnl5IX0/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion visit > https://groups.google.com/d/msgid/datameet/fd96996b-f85d-4bc5-8174-384c56885192n%40googlegroups.com > <https://groups.google.com/d/msgid/datameet/fd96996b-f85d-4bc5-8174-384c56885192n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Datameet is a community of Data Science enthusiasts in India. Know more about us by visiting http://datameet.org --- You received this message because you are subscribed to the Google Groups "datameet" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion visit https://groups.google.com/d/msgid/datameet/CAN8Wyx_C%2ByZVGEYWxUp8Fix10BMTryX1BuEkMn_eJHmY3M2FXg%40mail.gmail.com.
