Same as Uwe.
Regards Antoine. Le 11/07/2019 à 14:05, Uwe L. Korn a écrit : > Hello Liya, > > I'm quite -1 on this type as Arrow is about efficient columnar structures. We > have opened the standard also to matrix-like types but always keep the > constraint of consecutive memory. Now also adding types where memory is no > longer consecutive but spread in the heap will make the scope of the project > much wider (It seems that we then just turn into a general serialization > framework). > > One of the ideas of a common standard is that some need to make compromises. > I think in this case it is a necessary compromise to not allow all kind of > string representations. > > Uwe > > On Thu, Jul 11, 2019, at 6:01 AM, Fan Liya wrote: >> Hi all, >> >> >> We are thinking of providing varchar/varbinary vectors with a different >> memory layout which exists in a wide range of systems. The memory layout is >> different from that of VarCharVector in the following ways: >> >> >> 1. >> >> Instead of storing (start offset, end offset), the new layout stores >> (start offset, length) >> 2. >> >> The content of varchars may not be in a consecutive memory region. >> Instead, it can be in arbitrary memory address. >> >> >> Due to these differences in memory layout, it incurs performance overhead >> when converting data between existing systems and VarCharVectors. >> >> The above difference 1 seems insignificant, while difference 2 is difficult >> to overcome. However, the scenario of difference 2 is prevalent in >> practice: for example we store strings in a series of memory segments. >> Whenever a segment is full, we request a new one. However, these memory >> segments may not be consecutive, because other processes/threads are also >> requesting/releasing memory segments in the meantime. >> >> So we are wondering if it is possible to support such memory layout in >> Arrow. I think there are more systems that are trying to adopting Arrow, >> but are hindered by such difficulty. >> >> Would you please give your valuable feedback? >> >> >> Best, >> >> Liya Fan >>
