Hi, I’m not trying to block incremental OpenLineage scaffolding, but I think we should make explicit what the first merges do and do not settle.
I’m OK with #4705, and possibly #4826, moving forward if they are explicitly non-committal scaffolding. My concern is that these merges should not be read as consensus on the local lineage backend, query model, persistence semantics, or JDBC schema. In particular, I do not think #4826 should be read as settling how the SPI maps to JDBC schema evolution, NoSQL backends, or backend capability differences. I also think the local-store path needs explicit usability and operational criteria before schema work merges: what queries are expected to be efficient, what graph size or retention window we are designing for, what write/read load Polaris should absorb, and what happens when lineage volume exceeds those bounds. Without that, we may merge a schema that is technically workable for small examples but unclear for real lineage volume, query latency, pruning, and backend load. Those need a separate decision before #4827 or equivalent schema/persistence work merges. Robert [4705] https://github.com/apache/polaris/pull/4705 [4826] https://github.com/apache/polaris/pull/4826 [4827] https://github.com/apache/polaris/pull/4827 On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < [email protected]> wrote: > Hi Dmitri, > > Thanks for the comments. Replying point-by-point: > > > re: How to use this data > > I agree, we should list this in the document itself so it does not get > lost. Let me put something in the appendix for now and link it to the > relevant section so we don't lose the document's flow. Please take a look > there. > > > re: Link to OL Spec? > > https://openlineage.io/apidocs/openapi/ > > > re: Does OL persistence have to be limited to JDBC? > > No, this is the LineagePersistent Contract (from earlier in this thread, > linked again for comfort): > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > . > This was made to ensure that different persistence layers can implement a > single interface for Lineage information. > > > re: Does OL data have to be located in the same database as MetaStore > data? > > I don't think there is a hard requirement for this. Happy to discuss if > anyone else thinks differently! > > > I believe Polaris should allow downstream projects to opt in/out of > including > OL REST APIs and related persistence code. This is the general approach > we're following for the Metrics API proposal. > > Understood. I think this is an implementation detail and you've marked it > on the PR as well. Let us take care of this discussion there. > > Best, > Adnan Hemani > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi All, > > > > Commenting in general. > > > > I see a few PRs have been opened related to OpenLineage. This is fine. > > > > However, I believe we need to review some project-wide approaches related > > to adding this feature (among other features). > > > > 1) Does OL persistence have to be limited to JDBC? > > > > 2) Does OL data have to be located in the same database as MetaStore > data? > > > > 3) I believe Polaris should allow downstream projects to opt in/out of > > including OL REST APIs and related persistence code. This is the general > > approach we're following for the Metrics API proposal [1] [4115] > > > > These points are not in opposition to supporting OL in Polaris. These are > > merely technical concerns, which I think would benefit Polaris as a whole > > if designed with flexibility and modularity in mind upfront. > > > > WDYT? > > > > [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > Thanks, > > Dmitri. > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > [email protected]> wrote: > > > > > Hi all, > > > > > > I wanted to ensure that the OpenLineage proposal I previously posted > in a > > > different thread [1] was actually being found, given that it was deep > > into > > > the thread. I request the community to review this proposal so we can > > > potentially start implementation. > > > > > > Proposal: > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > Best, > > > Adnan Hemani > > > > > > [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > >
