On Wed, Aug 23, 2017 at 4:23 PM, Rajan Dhabalia <rdhaba...@apache.org> wrote:
> > because this field might be accessed when processing every message. the > overhead in key/value property is much higher than an int64 field. > > I have the same concern as Andrews mentioned. > But a very quick implementation question: properties > <https://github.com/apache/incubator-pulsar/blob/master/ > pulsar-common/src/main/proto/PulsarApi.proto#L47> > field is also part of message-metadata only. > So, I believe in your usecase, we want to skip metadata-deserialization and > idea is to read only "EventTime" field (field-12) without deserializing > message-metada using ByteBuf-offset (eg: read last int64 using end of > metadata-index).? Don't you think it will prevent us to add additional > String-field in future because then it will be hard to know "EventTime" > index? > Ah, not. In protobuf, the key/value properties are list of key/value pairs. If you want to look up a `event-time` property, you either do a sequential lookup or put the key/value pairs back into a hashmap and look up. Either approach isn't really efficient if getEventTime needs to happen on every message. Also, it is going to introduce additional overhead for keys - every message needs to carry a key `eventtime`. Hope this make sense. > > Thanks, > Rajan > > > On Wed, Aug 23, 2017 at 3:56 PM, Sijie Guo <guosi...@gmail.com> wrote: > > > On Wed, Aug 23, 2017 at 3:19 PM, Sahaya Andrews <andr...@apache.org> > > wrote: > > > > > If this is not going to be used by the broker for anything, why can't > > > this be set in message property key/value? > > > > > > > Currently it is not used by broker. However, we might leverage field > later > > on to provide advanced features like event-time based index. It is not > > covered by this proposal. > > > > Also there is another consideration - adding this field to message > metadata > > is more serialization friendly than putting it into key/value property. > > this is important for stream computing, > > because this field might be accessed when processing every message. the > > overhead in key/value property is much higher than an int64 field. > > > > Hope this make sense. > > > > - Sijie > > > > > > > > > > Andrews > > > > > > On Wed, Aug 23, 2017 at 3:03 PM, Matteo Merli <mme...@apache.org> > wrote: > > > > Proposal looks good to me. > > > > > > > > Added to wiki: > > > > https://github.com/apache/incubator-pulsar/wiki/PIP-5:-Event-time > > > > > > > > On Wed, Aug 23, 2017 at 2:52 PM Sijie Guo <guosi...@gmail.com> > wrote: > > > > > > > >> Hi pulsar folks, > > > >> > > > >> Currently Pulsar messages has fields for `publish_time`. However in > > some > > > >> use cases for example stream computing, `event_time` is required. > I'd > > > like > > > >> to start a proposal to add `event_time` support to pulsar messages. > > > >> > > > >> This proposal only covers adding `event_time` to pulsar messages and > > > make > > > >> it available to both producers and subscribers. It doesn't cover > > > advanced > > > >> features, such as event-time index or rewind subscriptions based on > > > >> event-time. > > > >> > > > >> I created the proposal here: > > > >> https://gist.github.com/sijie/60324cc892643961b923593a597109ab > > > >> > > > >> Please go over and provide your feedback/comments. > > > >> > > > >> Also, since I don't have any permissions on writing pulsar wiki > pages, > > > can > > > >> any pulsar committers help me add this PIP to the wiki pages if it > is > > > >> accepted? > > > >> > > > >> - Sijie > > > >> > > > > -- > > > > Matteo Merli > > > > <mme...@apache.org> > > > > > >
