Thanks everyone for chiming in. I'm not too well versed in the history of Iceberg. So I searched the list and the spec for precedent on storing information purely for debugging purposes. As far as I can tellthere is none. The closest I found is the rule that all columns must be written to data files even when redundant with manifest metadata, which is explicitly a backup "in case of corruption or bugs in the metadata layer". Please correct me if I'm wrong here. If not, this means requiring the _pos column for debugging purposes only would be a first.
I lean towards removing it. But if that is not possible, I’d prefer to build on top of an earlier proposal from Andrei and would suggest making it optional. Then we could state following requirements: - column files MUST adhere to the same positional alignment as the base file - _pos is never needed to resolve a read - writers MAY materialize it, for example when they cannot guarantee alignment, or for troubleshooting - when absent, readers derive the position from the row ordinal - when present and disagreeing with the ordinal, the file is corrupt, rather than there being a second legal read path Best Marco On Fri, Aug 28, 2026 at 10:00 PM Ryan Blue <[email protected]> wrote: > Conceptually the value we need is not the row's position in the column > update file, it is the row's position in the original data file. We expect > them to be equal, but it's an unrecoverable and undetectable problem if you > don't store the row position and a writer omits deleted rows. I think > that's the debugging that Marco mentioned. > > I'd prefer to keep this since I think that Parquet should be able to store > these efficiently. But that's not what happens today so this isn't a > strongly held opinion. > > Ryan > > On Thu, Aug 27, 2026 at 6:54 AM Leonid Lygin via dev < > [email protected]> wrote: > >> Glad to see that everyone (up to this point) agrees! >> >> Should we ratify the decision in the doc itself? >> >> On Wed, Aug 26, 2026 at 6:15 PM Anurag Mantripragada >> <[email protected]> wrote: >> > >> > The addition of `_pos` column was not a remnant of previous design, but >> a conscious choice we made during one of the column updates sync. I'm in >> favor of removing it if it makes implementation easier. >> > >> > ~ Anurag >> > >> > On Wed, Aug 26, 2026 at 7:31 AM Gábor Kaszab <[email protected]> >> wrote: >> >> >> >> Yes, the question is whether we want to persist _pos into the column >> files. We earlier concluded that even though we went with the dense >> representation we persist _pos. However, I think re-opening the question is >> reasonable, because that's just extra noise ATM, and we shouldn't write >> that field into column files. Reading the _pos column would still work >> regardless if we include the base file or not to the query. >> >> >> >> Best Regards, >> >> Gabor >> >> >> >> >> >> Russell Spitzer <[email protected]> ezt írta (időpont: 2026. >> aug. 26., Sze, 16:21): >> >>> >> >>> What is the actual argument here? I think having a persisted field >> doesn't make sense since we expect perfect alignment. We would expect >> reading the file in isolation with the metadata _pos column should still >> work right? >> >>> >> >>> If we are just discussing removing a persisted value, I'm in favor of >> that. >> >>> >> >>> On Wed, Aug 26, 2026 at 8:37 AM Gábor Kaszab <[email protected]> >> wrote: >> >>>> >> >>>> I hear you, and I share the same opinion. If we don't need such a >> field then it's just extra unnecessary complexity to write it. I'm not >> entirely convinced on the debugging use of the _pos field. Would be >> beneficial to reduce unnecessary noise and confusion by not adding the _pos >> field. >> >>>> >> >>>> Let's discuss this on the next sync! In the meantime, opinions are >> welcome here too. >> >>>> >> >>>> Thanks, >> >>>> Gabor >> >>>> >> >>>> Andrei Tserakhau via dev <[email protected]> ezt írta >> (időpont: 2026. aug. 26., Sze, 15:32): >> >>>>> >> >>>>> +1 on this question. >> >>>>> >> >>>>> Right now `_pos` column feels more like debug leftovers, it bring >> some confusion for read-side weather it's expected to be readed or not. >> >>>>> >> >>>>> I think removing it would make implementation easier. >> >>>>> >> >>>>> Best, >> >>>>> Andrei >> >>>>> >> >>>>> On Wed, Aug 26, 2026 at 2:42 PM Leonid Lygin via dev < >> [email protected]> wrote: >> >>>>>> >> >>>>>> Thanks for the quick response! >> >>>>>> >> >>>>>> My biggest concern with `_pos` is not performance but rather >> clarity >> >>>>>> and implementation divergence: >> >>>>>> >> >>>>>> 1. including `_pos` is redundant, and (at least for me) provokes a >> >>>>>> re-read of the row alignment section — "why include `_pos` if files >> >>>>>> are fully aligned?"; >> >>>>>> 2. having `_pos` fully duplicate the row position, there are two >> >>>>>> different legal ways to implement reads: either positionally, or >> using >> >>>>>> `_pos`. >> >>>>>> >> >>>>>> On Wed, Aug 26, 2026 at 2:35 PM Gábor Kaszab < >> [email protected]> wrote: >> >>>>>> > >> >>>>>> > Hey All, >> >>>>>> > >> >>>>>> > Thanks for bringing this up! (for me the initial mail went to >> spam, though...) >> >>>>>> > >> >>>>>> > Technically, with the dense representation we don't really need >> the _pos column in the column files, unless for troubleshooting. While >> checking the row counts is good, if they don't match we might get a better >> understanding on what the writer missed writing if we had the _pos col, >> also the order could be verified. >> >>>>>> > >> >>>>>> > Apart from debugging, I think either way is just fine. An >> additional detail to consider is that according to my experiments, there >> isn't really any storage cost for writing the _pos with delta encoding >> (e.g. with Parquet V2). So the conclusion was that since it comes for free, >> and might help for debugging, why not write it. >> >>>>>> > >> >>>>>> > Should we reopen this question? Any further feedback is welcome. >> >>>>>> > >> >>>>>> > Best Regards, >> >>>>>> > Gabor >> >>>>>> > >> >>>>>> > Leonid Lygin via dev <[email protected]> ezt írta >> (időpont: 2026. aug. 26., Sze, 14:10): >> >>>>>> >> >> >>>>>> >> Definitely agree that including `_pos` raises questions. >> >>>>>> >> >> >>>>>> >> If "debugging" is to be understood as figuring out if the >> column files >> >>>>>> >> have gaps -- just checking the row counts is good enough for >> that. Is >> >>>>>> >> there a lot to be gained from figuring out where exactly the >> gap is >> >>>>>> >> occurring? >> >>>>>> >> >> >>>>>> >> On Mon, Aug 24, 2026 at 1:57 PM Marco Kroll >> >>>>>> >> <[email protected]> wrote: >> >>>>>> >> > >> >>>>>> >> > Hi all, >> >>>>>> >> > >> >>>>>> >> > I just saw the agenda [1] for tomorrow's (2026-08-25) sync >> and want to +1 the `_pos` column topic. >> >>>>>> >> > My understanding is that this column exists for two reasons: >> >>>>>> >> > 1. debugging >> >>>>>> >> > 2. detect if writers skipped deleted rows >> >>>>>> >> > >> >>>>>> >> > My take is that using the dense Null filled representation >> addresses both of these issues. >> >>>>>> >> > It implicitly encodes the position, very much like for >> deletion vectors and since all rows need to be present, comparing the row >> count of the base file with the column file can be used to verify that all >> rows were written. >> >>>>>> >> > >> >>>>>> >> > The main thing to add to the doc would be that the row order >> must be identical to the base file. >> >>>>>> >> > >> >>>>>> >> > Best >> >>>>>> >> > Marco >> >>>>>> >> > >> >>>>>> >> > [1]: >> https://docs.google.com/document/d/1Bd7JVzgajA8-DozzeEE24mID_GLuz6iwj0g4TlcVJcs/edit?tab=t.jvm7iiiulf8q#heading=h.rbisiun18esp >> >
