[
https://issues.apache.org/jira/browse/HCATALOG-546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13492479#comment-13492479
]
Travis Crawford commented on HCATALOG-546:
------------------------------------------
Hey [~mithun], a few questions about this patch.
(1) The issue presented here is large message size. In a [message on the
mailing
list|http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-dev/201211.mbox/%3CCABN7xTB8HYJnKgRUqWhkdW5ftMJtF6B4FXwG0zSSfOPJoCHS2A%40mail.gmail.com%3E]
I quickly profiled the size of messages used in the unit test, and found them
to be quite small. Are your messages significantly larger for some reason?
AFAICT from the test the messages are already modest in size.
(2) Its true in your use case Oozie would need the Hive thrift structs on its
classpath to consume these messages, which I can see as being annoying.
However, the case of evolving protocols doesn't seem any different between
JSON/thrift. Thrift structs are backwards compatible, so if fields are added to
a Table/Database/Partition the existing messages will continue to deserialize.
So I'm not seeing how switching to JSON helps here.
> Rework HCatalog's JMS Notifications
> ------------------------------------
>
> Key: HCATALOG-546
> URL: https://issues.apache.org/jira/browse/HCATALOG-546
> Project: HCatalog
> Issue Type: Bug
> Components: notification
> Affects Versions: 0.4.1
> Reporter: Mithun Radhakrishnan
> Assignee: Mithun Radhakrishnan
> Fix For: 0.4.1
>
> Attachments: HCATALOG-546.patch, sample.Add.Drop.Database.json,
> sample.Add.Drop.Partition.json, sample.Add.Drop.Table.json
>
>
> In 0.4.1, the NotificationListener listens for metastore operations and emits
> JMS notifications containing the entire metastore-objects
> (Database/Table/Partitions) in Java-serialized form. The assumption at the
> time was that consumers might need access to the whole object. This policy
> poses a couple of problems:
> 1. The notifications are verbose, since it conveys a bunch of information
> that's available from querying the metastore anyway.
> 2. Consumers of these JMS notifications (e.g. Oozie) would now be dependent
> on the Java class definitions of metastore-objects. If they change, Oozie
> would also need to be restarted (with updated libs), to consume the
> notifications.
> Ideally, the notifications should convey only the minimum information that
> identifies the metastore-change unambiguously. (Everything else can be
> queried for.) They should be backward compatible. If new fields are added,
> existing consumers shouldn't break (unless they intend to consume the new
> fields). Also, the notification-format ought to be pluggable.
> For the initial rework, we're proposing to use a JSON-string to represent the
> notification-content. We're also proposing a helper-class for the likes of
> Oozie to use, that converts the strings to POJOs, in a backward-compatible
> fashion.
> I'll attach sample notifications and a tentative patch shortly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
