You’re thinking about this the same way that I do — first agree what is the desired behavior of a so-called ’system field’.
I have questions like: * Does a system field appear in all tables (relations, including tables derived via a query) or only base tables? * If there are many system fields (hundreds), how do we keep the plan of a manageable size? * Do system fields belong to the whole query (as CURRENT_DATE does), or just a record (table alias)? I think the notion of a field that is present in the row but is not included in the expansion of ‘*’ or ‘alias.*’ would be useful. It would be fairly easy to implement because it only affects star expansion (and therefore only affects the validator and sql-to-rel-converter, not the planner). Julian > On Oct 27, 2023, at 12:45 PM, Gian Merlino <g...@apache.org> wrote: > > I'm looking at adding system fields to certain tables in Druid: they should > be referenceable by name but shouldn't show up through star-expansion. For > example, we'd like to expose the name of the data file currently being read > as "__filename". > > I'm trying to figure out if this is something we can do with Calcite right > now or if we need to do some Calcite changes first. > > I did find a couple of things, but AFAICT they are not exactly what I'm > looking for: > > - There is discussion in CALCITE-2755 about treating Elasticsearch's _id > field as a system field, but it looks like the _id field is a key in a map > rather than a top-level field, and its visibility is managed at runtime by > ElasticsearchProject rather than being handled at the validator. > > - In SqlValidatorImpl expandStar, there's a variable "includeSystemVars" > that is set to false when called by SqlToRelConverter. Seems promisingly > named, although it doesn't seem to do anything as far as I can tell. I'm > wondering if I'm missing something, or if this is an incomplete feature. > It's also not clear to me how the validator would figure out which fields > are system fields anyway, since I don't see an obvious place to mark them > in a RelDataType, which I think is how the validator learns about available > fields from its tables. > > Happy to do a contribution to Calcite if one is needed. > > Thanks, > Gian
