Hi Anand, I left some comments on the SPI PR [3616].
>From your javadoc on ReportIdToken, I can see that you have given some thought to time-series backends for this data. Do you have any concrete implementations in mind? What do you think might be a good fit for storage other than JDBC / RDBMS? [3616] https://github.com/apache/polaris/pull/3616 Thanks, Dmitri. On Thu, Jan 29, 2026 at 1:49 PM Anand Kumar Sankaran < [email protected]> wrote: > Hi Dmitri, > > Thank you for taking the time to review. I had pulled out the schema in > its own PR (based on comments) here: > https://github.com/apache/polaris/pull/3523 > > I will update the document with the SPI and carve out another PR that > focuses on the SPI. > > — > Anand > > *From: *Dmitri Bourlatchkov <[email protected]> > *Date: *Thursday, January 29, 2026 at 10:43 AM > *To: *[email protected] <[email protected]> > *Cc: *Anand Kumar Sankaran <[email protected]> > *Subject: *Re: Polaris Telemetry and Audit Trail > > This Message Is From an External Sender > This message came from outside your organization. > Report Suspicious > <https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZAGr2cumY1cAlUeoPrU2eNUPz9Wxq2PaG1t99vEN3v60gucxC3Qsk-xK09RndcW46WtNjMMjQZbaQ495h1D1g293F_uqLhPM8gDMlA6xRmo5b5uqrSh2v7QKHrhQ$> > > Hi Anand, > > Thanks for making a proposal doc and starting this thread. Apologies for > late comments. > > As you know I already reviewed a couple of related PRs, which had a > smaller impact on the codebase, but now, I believe, we're coming to the > core of this feature and it might need a deeper discussion. > > For the sake of awareness of other project members, I'd like to > highlight some aspects by email. > > * Storing scan metrics in RDBMS in general looks like a > reasonable approach to me. However, Polaris Persistence is an extensible > mechanism. Other backends are possible, including privately developed > backends. > > From this POV, I believe it might be preferable to start not with an RDBMS > schema, but with a java SPI outlining expected read/write operations and > the data model. This will make it easier to assess impact on other > Persistence implementations. Naturally, Polaris service code will have to > be able to work seamlessly regardless of the backend impl. (including the > do-nothing impl.). > > I see that some connection between the SQL schema and java code is made in > PR [3385], but it does not appear to offer a clean SPI that could be > implemented by different Persistence backends. Would it be ok from your > perspective to make another PR with just SPI code? > > * Regarding ad-hoc SQL queries - it is certainly a valid use case for > custom code to query the Polaris database directly. However, those ad-hoc > queries may require certain indexes, which may not have a relevant use case > in OSS code. In that case, I believe it would be preferable to manage the > extra indexes in custom code (not in Polaris). Whether this is the case or > not currently, I cannot say with certainty, so apologies if this is a false > alarm... Having the java SPI available would be a huge aid to reasoning > about the proposed SQL schema, I think. > > That said, I do not mean to complicate the development of this feature, > just trying to structure it in a way that is hopefully easy to understand > (including myself), maintain and expand later. If I missed something, > please feel free to point it out. > > [3385] https://github.com/apache/polaris/pull/3385 > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$> > > Thanks, > Dmitri. > > On Wed, Jan 14, 2026 at 10:05 PM Anand Kumar Sankaran via dev < > [email protected]> wrote: > > Hi Yufei, > > Gave commenting privileges to all. > > Thanks. > > — > Anand > > From: Yufei Gu <[email protected]> > Date: Wednesday, January 14, 2026 at 6:13 PM > To: [email protected] <[email protected]> > Cc: Anand Kumar Sankaran <[email protected]> > Subject: Re: Polaris Telemetry and Audit Trail > > This Message Is From an External Sender > This message came from outside your organization. > Report Suspicious< > https://us-phishalarm-ewt.proofpoint.com/EWT/v1/Iz9xO38YGHZK!YhNDZABkHi1B6hyPVYUK0T-K5j4Aoqdrlu07UjBpJcpjOsz7Ie7d_DhPl-ywi50dfQz4O6LL-AHLCOQTQ0X7Le8yD2cF0ogquGFB8Aw8uQLgB0RL6Ezpa-pLDPA4ziXA$ > > > > Hi Anand, thanks for the design doc and PR. Can you allow comments in the > doc so that people can chime in? Thanks! > > Yufei > > > On Sat, Jan 10, 2026 at 9:09 AM Anand Kumar Sankaran via dev < > [email protected]<mailto:[email protected]>> wrote: > Hi all > > My first PR for adding AWS STS Session Tags support for credential vending > was merged. https://github.com/apache/polaris/pull/3327 > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkklJ4Xf9Y$> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3327__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3IZAJGqY$ > > > > I have been working with Prashant on an end-to-end telemetry and audit > trail tracking for Polaris. It is documented here. > https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0 > <https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkk8MVJPlY$> > < > https://urldefense.com/v3/__https://docs.google.com/document/d/1Ehzvi5RNPs4hChkBFI6VD23myEqm-7sWW3d2kjmuYj8/edit?tab=t.0__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl3hYqTlcg$ > > > > Based on Prashant’s initial feedback (parity with Apache Gravitino for > metrics reports), I have an initial PR here. > https://github.com/apache/polaris/pull/3385 > <https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkyZhI0OE$> > < > https://urldefense.com/v3/__https://github.com/apache/polaris/pull/3385__;!!Iz9xO38YGHZK!8vlQ2_OqjmZNIpbrTTluFFNKFCEeM87nr95MP5pMpMu5fFWVSoBZ3eb6-C2xbAGCG-ry5eaCO-pFLxl39opCznI$ > > > > I will be reaching out to various folks for review. I am open to breaking > down the PR into smaller logical groups of PR if it helps. > > I request you to look at the Google doc and let me know what you think. > > — > Anand > > > > -- > Dmitri Bourlatchkov > Senior Staff Software Engineer, Dremio > Dremio.com > <https://urldefense.com/v3/__https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkiOwjJc0$> > / > Follow Us on LinkedIn > <https://urldefense.com/v3/__https://www.linkedin.com/company/dremio__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkCLjBQ2M$> > / > Get Started > <https://urldefense.com/v3/__https://www.dremio.com/get-started/__;!!Iz9xO38YGHZK!_dbBfwEnZEuPp_8qIuBDoYfWRgCZKiPDMg9ouDltqsHhr9hqCHVlF9IedSkNVBDWupbneU7inC9YK-THL-lTjWkkoKQUMII$> > > > The Agentic Lakehouse > *The only lakehouse built for agents, managed by agents* > > > -- Dmitri Bourlatchkov Senior Staff Software Engineer, Dremio Dremio.com <https://www.dremio.com/?utm_medium=email&utm_source=signature&utm_term=na&utm_content=email-signature&utm_campaign=email-signature> / Follow Us on LinkedIn <https://www.linkedin.com/company/dremio> / Get Started <https://www.dremio.com/get-started/> The Agentic Lakehouse The only lakehouse built for agents, managed by agents
