Thank you, Travis! Working through them now.
________________________________ From: Travis Crawford <[email protected]> To: Mithun Radhakrishnan <[email protected]> Cc: "[email protected]" <[email protected]> Sent: Wednesday, November 7, 2012 11:32 AM Subject: Re: Add/delete-partition JMS message format proposal. Added a bunch of comments to the reviewboard! --travis On Wed, Nov 7, 2012 at 10:31 AM, Mithun Radhakrishnan <[email protected]> wrote: > Hey, Travis. Apologies for missing your reply on Friday. (Lost in the flood, > I'm afraid.) Thank you for reviewing the proposal. > >> How large are the partitions that they're causing issues for you? > I hadn't measured. (Thank you for doing so.) But given the large volume of > partition-related events we're slated to consume shortly in Oozie, we > figured it'd be in our best interest to shed any information that can be > queried from the metastore. The parameter-set, location-related details etc. > seemed like too much to be carrying on each event. (We're still reviewing > with Oozie that the proposed info works for them.) > >> Seems like messages would still deserialize on the other side if fields >> are added, clients just wouldn't know about the new field? > Another good question. I'd initially figured that while the message might be > received, the Partition object would not be deserialized correctly. In > hindsight, that might not be. Here're some thoughts though: > 1. We'd like to decouple the notification-format from the layout of the > Partition class, with the notification containing just enough to identify > the partition. This way, changes to the Partition-class don't change the > notifications at all. > 2. It's conceivable that we might add further information to a notification, > beyond what might be contained in a Partition object. > 3. We could have chosen a Thrift definition for the notification. JSON > seemed simpler. > > My follow-up to this change would be to add support for adding a group of > Partitions as an atomic "set". The AddPartitionEvent could conceivably > contain a collection of partition-specs, all corresponding to the same set. > (The current handling of the use-case is hacky and broken.) Adding a large > number of Partition-thrift-objects to the same notification would be > wasteful. > > The patch up on JIRA attempts to support future serialization schemes, while > providing the JSON implementation. > > Does that sound reasonable? > > Mithun > > ________________________________ > From: Travis Crawford <[email protected]> > To: [email protected]; Mithun Radhakrishnan > <[email protected]> > Sent: Friday, November 2, 2012 8:23 AM > > Subject: Re: Add/delete-partition JMS message format proposal. > > Hey Mithun - > > Cool that you're giving the oozie integration a bit more love. A few > questions inline: > > > > On Tue, Oct 30, 2012 at 2:26 PM, Mithun Radhakrishnan > <[email protected]> wrote: > > Hello, HCat-Dev. > > I'm working on modifying the HCat messages (sent over JMS/ActiveMQ, for > partition-add/delete) so that clients (such as > Oozie) would have an easier time with consumption. > Here are some limitations of what's available currently: > 1. The present implementation in HCatalog (branch-0.4/) seems to send the > entire Partition (Java) instance in serialized fashion. Since the > partition-parameters, hdfs-location etc. are all serialized, the messages > are rather, emm, garrulous. > > > Correct, the messages sent over jms include an instance of > org.apache.hadoop.hive.metastore.api.Partition. Eventually oozie needs this > info. It sounds like this object is larger than you want to publish over > jms, and would rather fetch it yourself on the other side? > > How large are the partitions that they're causing issues for you? I did a > quick test in NotificationListener.onAddPartition and see the following when > running the test: > > ByteArrayOutputStream baos = new ByteArrayOutputStream(); > ObjectOutputStream oos = new ObjectOutputStream(baos); > oos.writeObject(partition); > oos.close(); > LOG.info("Size of partition: " + baos.size()); > > [main] INFO org.apache.hcatalog.listener.NotificationListener - Size > of partition: 794 > > 794 bytes doesn't seem large enough to introduce additional complexity over. > Are you seeing drastically different object sizes? > > > > > 2. There doesn't seem to be any support for versioning either. So when new > fields are added, older clients won't work at all without update. > > > Are you referring to the message map that's actually sent over the wire? I'm > not super familiar with jms, so sorry if this is a dumb question. Seems like > messages would still deserialize on the other side if fields are added, > clients just wouldn't know about the new field? If this is the case, how > does switching to something like json help? > > I'm initially hesitant to introduce yet another serialization format to keep > track of, because of the added complexity & maintenance issues. Definitely > not opposed if its the right thing to do, but am curious to learn a bit more > about this before diving in. > > --travis > > > > > Could we consider transmitting only that info which identifies the > partitions that pertain to the operation (e.g. partition keys), and drop any > information that might be gathered from querying the metadata (e.g. storage > location, partition-parameters, etc.) > > We're also considering that the initial implementation encode the ActiveMQ > payload in JSON. Here's an example of the proposed message format for an > "add_partition" operation: > > "add_partition": { > "hcat_server" : "thrift://my.hcat.server:9080", > "hcat_service_principal" : "hcat/[email protected]", > "db": "default", > "table": "starling_jobs", > "partitions": > [ > {"grid": "AxoniteBlue", "dt": "2012_10_25"},// Sets of partition-keys. > {"grid": "AxoniteBlue", "dt": "2012_10_26"}, > {"grid": "AxoniteBlue", "dt": "2012_10_27"}, > {"grid": "AxoniteBlue", "dt": "2012_10_28"}, > ], > "timestamp": "1351534729" // In this case, interpreted as creation-time. > } > > If we continue to use JMS MapMessages, we could consider having 3 keys in > the map: > 1. version = "1" (for the first implementation. Increment as we go.) > 2. format = "json" (We could consider adding different formats if we > choose.) > 3. message = <the json message body, as above.> > > The version and format help a factory choose the right implementation to > deserialize the message. (A client-side library we supply to Oozie should > hide this and provide POJOs.) > > Since the "partitions" field is an array, and since the values corresponding > to partition-keys are all strings, we'd be able to accommodate partial > partitions-specs, or even wild-cards. This might help us add support for > "mark-set-done" later on. > > The first key ("add_partition", "drop_partition" or "alter_partition") > indicates the operation, and the value indicates the record-body. (At first > glance, the record-body doesn't change for these operations. But that might > change, so we'll keep them distinct.) > > Also note that HiveMetaStore::add_partitions_core() currently doesn't send 1 > message for the entire set of partitions being added. Instead we get one > message per partition. This could be verbose and sub-optimal. We'll tackle > this sort of thing after we've nailed the format down. > > I'm toying with the idea of adding an "other" property, an array of > key-values to accommodate stuff we hadn't considered, at "run-time" (like if > we want to introduce a hack). The need for such a property is contingent on > the behaviour of Jackson w.r.t. newly added properties in the record-body. > (I'll run experiments and keep you posted.) > > What do you think? > > Mithun > > > >
