Hi, My past experience with fixing timestamps was that no fix was going to work if even one of major SQL engines of the Hadoop stack disagreed with the approach and was not willing to implement it. For this reason, we can't just add a specification to Hive, we need an agreement from said communities.
Indeed, there are more projects than these three that deal with timestamps, but these are the ones that deal with them on the SQL level and this proposal is about the semantics of the SQL timestamp types. I have planned to write a small summary from the file format perspective as well and send it to affected groups. I had Avro, Parquet, ORC, Arrow and Kudu in mind. Based on your suggestion, I will add Iceberg to that list. While I agree that a Google Doc would not be adequate for the final version of the plan, I think it is a better tool for doing the review and the design discussion than the individual mailing lists, for the following reasons: - It allows separate discussions around separate parts of the proposal and these discussion can happen in context (they are tied to specific parts of the document). - It allows adding suggestions to the proposal, in-context and immediately visible to everyone. - Most importantly, it is equally accessible to the Hive, Spark and Impala communities, therefore allows a real cross-component discussion. Br, Zoltan On Fri, Jan 11, 2019 at 12:10 AM Owen O'Malley <owen.omal...@gmail.com> wrote: > > ---------- Forwarded message --------- > From: Owen O'Malley <owen.omal...@gmail.com> > Date: Thu, Jan 10, 2019 at 3:09 PM > Subject: Re: [DISCUSS] Consistent Timestamps across Hadoop > To: Zoltan Ivanfi <z...@cloudera.com> > > > No, that isn't right. > > The discussion for Apache projects needs to happen in the open and not the > private google doc that isn't archived at Apache. > > Three is a severe underestimate of the projects that care about timestamps. > The Apache projects that care about parts of that document are: > > Avro > Hive > Iceberg > Impala > ORC > Parquet > Spark > > That said, Hive needs to make its decisions about what the semantics of Hive > should be. Impala, Iceberg, and Spark may make separate choices. Avro, ORC, > and Parquet need their bindings for each engine need to agree with the > semantics for that engine. > > My point is that Hive should have a page that describes its current semantics > with respect to timestamps, but those discussions need to happen on the Hive > list and result in documents in the Hive wiki. Hive can't tell other projects > what to do, but by clarifying their semantics it makes inter-operation > better. In my opinion, Spark SQL should move to local date time semantics for > timestamp. But they should want to do that to make themselves more compatible > with the SQL standard. Clearly Hive can't force them to change their > semantics. > > .. Owen > > > On Thu, Jan 10, 2019 at 8:04 AM Zoltan Ivanfi <z...@cloudera.com> wrote: >> >> Hi, >> >> Once we are through the discussion phase and hopefully have reached >> agreement, I support moving the document to a more permanent place. >> I'm unsure about what the best place would be though. Since it needs >> the agreement of three communities, it does not strictly belong to any >> single one of them (although the Hive Metastore is certainly a central >> component in this ecosystem, so we could put it in the Hive >> documentation based on that). I am also uncertain about whether we >> should use a wiki page, because it is too easily editable and after >> reaching an agreement it should not be modified without asking or >> notifying the same communities again. >> >> Is there a documentation space where modifications are subject to >> review? Or can we protect a wiki page to achieve that? I'm open to >> your suggestions. >> >> Thanks, >> >> Zoltan >> >> >> On Wed, Jan 9, 2019 at 9:32 PM Owen O'Malley <owen.omal...@gmail.com> wrote: >> > >> > From an Apache point of view, we really need to move this document and the >> > discussion to the Apache wiki and mailing lists. >> > >> > Did you want to take a first pass at moving it to Hive's wiki? >> > >> > .. Owen >> > >> > On Tue, Dec 11, 2018 at 10:40 AM Zoltan Ivanfi <z...@cloudera.com> wrote: >> >> >> >> Hi Owen, >> >> >> >> Thanks, I think your email contains a great summary of the problems >> >> tackled in the proposal. I would like highlight two particular topics >> >> from the discussion that we are having in the comments (details can be >> >> read in the document): >> >> >> >> It seems that we have agreement on the desired semantics of the more >> >> explicit SQL types. In particular, I was glad to hear that the TIMESTAMP >> >> WITH LOCAL TIME ZONE type that is already implemented in Hive is supposed >> >> to have Instant semantics. (In fact, it already does have Instant >> >> semantics, but it also has additional time zone information that is >> >> unused at this moment, and I wasn't sure whether that will be utilized, >> >> changing the semantics, or whether the semantics will remain and the >> >> superflous time zone data will be removed.) >> >> We are still discussing what is the best course of action to take with >> >> the plain TIMESTAMP type, which behaved differently in different file >> >> formats in Hive 2 and was made to behave the same way in a >> >> compatibility-breaking manner in Hive 3. My take on this type is that it >> >> has already been used to write huge amounts of data and for this reason >> >> we should restore its Avro- and Parquet-specific incosistent behaviour >> >> (possibly controlled by a feature flag), so that legacy data remains >> >> readable and legacy workarounds remain functional. The new, more explicit >> >> SQL types will provide a clear migration path away from the messy >> >> TIMESTAMP type. >> >> >> >> All in all, I feel that we are converging towards a common goal and I >> >> have high hopes that the more explicit timestamp types will have much >> >> better interoperability and consistency across different Hadoop SQL >> >> engines. >> >> >> >> Thanks, >> >> >> >> Zoltan >> >> >> >> >> >> On Mon, Dec 10, 2018 at 7:54 PM Owen O'Malley <owen.omal...@gmail.com> >> >> wrote: >> >>> >> >>> Thank you for starting this discussion. Clearly the Hive semantics on >> >>> timestamp are very messed up, but has been moving in the right direction >> >>> of becoming more SQL standard compliant. I'm pulling this discussion >> >>> back to the list rather than the personal GoogleDoc, which isn't very >> >>> collaborative. >> >>> >> >>> I like your breakdown of the semantics: >> >>> >> >>> Instant - point in time that will appear different depending on the >> >>> reader time zone >> >>> LocalDateTime - consistent hour and minute regardless of the reader time >> >>> zone. >> >>> OffsetDateTime - consistent hour and minute with the offset of the >> >>> writer time zone >> >>> >> >>> The SQL standard has: >> >>> >> >>> Timestamp & Timestamp without time zone = LocalDateTime >> >>> Timestamp with time zone = OffsetDateTime >> >>> >> >>> Hive 2 had very confused semantics for timestamp: >> >>> >> >>> When storage was ORC, text, or RCFile with a text serde it was >> >>> LocalDateTime >> >>> When storage was Avro, Parquet, or RCFile with a binary serde it was >> >>> Instant >> >>> >> >>> Hive 3.1 has moved toward the SQL standard extended with Oracles' >> >>> timestamp with local time zone: >> >>> >> >>> Timestamp = LocalDateTime >> >>> Timestamp with local time zone = Instant >> >>> >> >>> This leaves us with a few problems: >> >>> >> >>> The Hive bindings to Parquet and Avro don't handle timestamps correctly. >> >>> ORC doesn't support timestamps with local time zone. I start working on >> >>> it in ORC-189. >> >>> We don't have timestamp with time zone support. >> >>> >> >>> .. Owen >> >>> >> >>> On Thu, Dec 6, 2018 at 7:55 AM Marta Kuczora >> >>> <kuczo...@cloudera.com.invalid> wrote: >> >>>> >> >>>> Hi Hive Community, >> >>>> >> >>>> I would like to share the following document on our "Consistent >> >>>> Timestamp >> >>>> types in Hadoop" plans for review. >> >>>> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit >> >>>> >> >>>> With this plan we would like to get an agreement on consistent timestamp >> >>>> behavior on Hive, Spark and Impala and in order to achieve this, we are >> >>>> sharing this document with all three communities. >> >>>> >> >>>> Please review and comment, any feedback is much appreciated! >> >>>> >> >>>> Regards, >> >>>> Marta
