On Sun, Nov 24, 2019 at 8:07 PM Fan Liya <liya.fa...@gmail.com> wrote: > > Hi Wes, > > I agree with you that this is a data representation issue. > > My point is that, data representation and data operation are closely > related. > As far as this issue is concerned, if we allow several values in the union > vector to be mapped to the same value in the underlying vector, it is > possible that when we modify one value in the union vector, the other value > is also modified, which is unexpected.
Right, but Arrow columnar data is immutable, so any mutation operations are application/implementation-level concerns and should not influence the specification documents. Implementations need to be aware of the implications of the specification, of course. > This is a problem with our current specification, because our > vectors/arrays provide set/write APIs. > So we may need a "coherency protocol" to define the behavior (e.g. copy on > write) when trying to modify a shared value, IMO. It's an application/implementation-level concern so I think it would need to be addressed separately from clarifying the specification. > > Best, > Liya Fan > > On Sat, Nov 23, 2019 at 3:31 AM Wes McKinney <wesmck...@gmail.com> wrote: > > > hi Liya, > > > > I don't understand your point -- we are strictly discussing data > > representation here I believe. From a data representation perspective, > > there is no conflict with repeated or non-monotonic offset values. > > > > On Fri, Nov 22, 2019 at 1:49 AM Fan Liya <liya.fa...@gmail.com> wrote: > > > > > > This is an interesting question. > > > IMO, to support repeated values, we also need to design a "coherency > > > protocol", to avoid the scenario where once a value is witten, the change > > > is propagated to another slot unexpectedly. > > > > > > Best, > > > Liya Fan > > > > > > On Fri, Nov 22, 2019 at 1:34 PM Micah Kornfield <emkornfi...@gmail.com> > > > wrote: > > > > > > > Hmm, I also thought the intention was monotonically increasing. I can't > > > > think of a strong reason one way or another. If the argument about > > code to > > > > do random access is the same in all cases, is there any benefit to > > forcing > > > > any order at all? Memory prefetching? > > > > > > > > On Thu, Nov 21, 2019 at 11:48 AM Wes McKinney <wesmck...@gmail.com> > > wrote: > > > > > > > > > hi Antoine, > > > > > > > > > > It's a good question. > > > > > > > > > > The intent when we wrote the specification was to be strictly > > > > > monotonic, but there seems nothing especially harmful about relaxing > > > > > the constraint to allow for repeated values or even non-monotonicity > > > > > (strict or otherwise). For example, if we had the union > > > > > > > > > > ['a', 'a', 'a', 0, 1, 'b', 'b'] > > > > > > > > > > then this could be represented as > > > > > > > > > > type_ids: [0, 0, 0, 1, 1, 0, 0] > > > > > offsets: [0, 0, 0, 0, 1, 1, 1] > > > > > child[0]: ['a', 'b'] > > > > > child[1]: [0, 1] > > > > > > > > > > or > > > > > > > > > > type_ids: [0, 0, 0, 1, 1, 0, 0] > > > > > offsets: [1, 1, 1, 0, 1, 0, 0] > > > > > child[0]: ['b', 'a'] > > > > > child[1]: [0, 1] > > > > > > > > > > What do others think? Either way some clarification in the > > > > > specification would be useful. Because the code used to do random > > > > > access is the same in all cases, I feel weakly supportive of removing > > > > > constraints on the offsets. > > > > > > > > > > - Wes > > > > > > > > > > On Thu, Nov 21, 2019 at 9:04 AM Antoine Pitrou <anto...@python.org> > > > > wrote: > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > I'd like some clarification on the spec and intent for dense > > arrays. > > > > > > > > > > > > Currently, it is specified that offsets of a dense union are "in > > order > > > > / > > > > > > increasing" (*). However, it is not obvious whether repeated > > values > > > > are > > > > > > allowed or not. > > > > > > > > > > > > I suspect the intent is to avoid having people exploit unions as > > some > > > > > > kind of poor man's dictionaries. Also, perhaps some optimizations > > are > > > > > > possible if monotonic or strictly monotonic indices are assumed? > > But I > > > > > > don't know the history behind the union type. > > > > > > > > > > > > Regards > > > > > > > > > > > > Antoine. > > > > > > > > > > > > > > > > > > (*) https://arrow.apache.org/docs/format/Columnar.html#dense-union > > > > > > > > > > >
