Hi All, I believe PR [4705] establishes a fairly isolated module structure for further OL work and does not imply much about the feature itself. I think it can be merged in its current state, provided the community agrees on the further feature direction.
Regarding the specific Persistence implementation, from my POV the following question also needs to be addressed: The proposal doc [1] assumes that the job edges are aggregated away from the OL input data and that the resulting inter-dataset graph is persisted with some retention characteristics. This is being discussed in the comment thread rooted in the "Abstract" section (at the phrase: "maintains a directed graph of dataset and field relationships"). However, I did not see a clear specification of this behaviour in the main doc body. I think this needs to be discussed in more detail, either in the doc or (preferably) in this email thread. More specifically: * How / when such retention is performed. * How queries relate to data retention. Do they only "see" the latest state of the inter-dataset graph or do they see several such graphs changing over time (within the retention window)? Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL files for OL tables. I support this approach. I believe this will allow for a smoother evolution of the OL schema with respect to MetaStore and other schemas. Additionally this approach reduces the impact of OL changes on users who elect not to enable this feature. [1] https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o [4705] https://github.com/apache/polaris/pull/4705 [4827] https://github.com/apache/polaris/pull/4827 Thanks, Dmitri. On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <[email protected]> wrote: > Hi, > > I’m not trying to block incremental OpenLineage scaffolding, but I think we > should make explicit what the first merges do and do not settle. > > I’m OK with #4705, and possibly #4826, moving forward if they are > explicitly > non-committal scaffolding. > > My concern is that these merges should not be read as consensus on the > local > lineage backend, query model, persistence semantics, or JDBC schema. > > In particular, I do not think #4826 should be read as settling how the SPI > maps > to JDBC schema evolution, NoSQL backends, or backend capability > differences. > > I also think the local-store path needs explicit usability and operational > criteria before schema work merges: what queries are expected to be > efficient, > what graph size or retention window we are designing for, what write/read > load > Polaris should absorb, and what happens when lineage volume exceeds those > bounds. > > Without that, we may merge a schema that is technically workable for small > examples but unclear for real lineage volume, query latency, pruning, and > backend load. > > Those need a separate decision before #4827 or equivalent > schema/persistence > work merges. > > Robert > > [4705] https://github.com/apache/polaris/pull/4705 > [4826] https://github.com/apache/polaris/pull/4826 > [4827] https://github.com/apache/polaris/pull/4827 > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < > [email protected]> wrote: > > > Hi Dmitri, > > > > Thanks for the comments. Replying point-by-point: > > > > > re: How to use this data > > > > I agree, we should list this in the document itself so it does not get > > lost. Let me put something in the appendix for now and link it to the > > relevant section so we don't lose the document's flow. Please take a look > > there. > > > > > re: Link to OL Spec? > > > > https://openlineage.io/apidocs/openapi/ > > > > > re: Does OL persistence have to be limited to JDBC? > > > > No, this is the LineagePersistent Contract (from earlier in this thread, > > linked again for comfort): > > > > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > > . > > This was made to ensure that different persistence layers can implement a > > single interface for Lineage information. > > > > > re: Does OL data have to be located in the same database as MetaStore > > data? > > > > I don't think there is a hard requirement for this. Happy to discuss if > > anyone else thinks differently! > > > > > I believe Polaris should allow downstream projects to opt in/out of > > including > > OL REST APIs and related persistence code. This is the general approach > > we're following for the Metrics API proposal. > > > > Understood. I think this is an implementation detail and you've marked it > > on the PR as well. Let us take care of this discussion there. > > > > Best, > > Adnan Hemani > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > Commenting in general. > > > > > > I see a few PRs have been opened related to OpenLineage. This is fine. > > > > > > However, I believe we need to review some project-wide approaches > related > > > to adding this feature (among other features). > > > > > > 1) Does OL persistence have to be limited to JDBC? > > > > > > 2) Does OL data have to be located in the same database as MetaStore > > data? > > > > > > 3) I believe Polaris should allow downstream projects to opt in/out of > > > including OL REST APIs and related persistence code. This is the > general > > > approach we're following for the Metrics API proposal [1] [4115] > > > > > > These points are not in opposition to supporting OL in Polaris. These > are > > > merely technical concerns, which I think would benefit Polaris as a > whole > > > if designed with flexibility and modularity in mind upfront. > > > > > > WDYT? > > > > > > [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > > > Thanks, > > > Dmitri. > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > > [email protected]> wrote: > > > > > > > Hi all, > > > > > > > > I wanted to ensure that the OpenLineage proposal I previously posted > > in a > > > > different thread [1] was actually being found, given that it was deep > > > into > > > > the thread. I request the community to review this proposal so we can > > > > potentially start implementation. > > > > > > > > Proposal: > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > > > Best, > > > > Adnan Hemani > > > > > > > > [1] https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > > > > > >
