[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)
[
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397782#comment-16397782
]
kalyan kumar kalvagadda commented on HIVE-18942:
oh, i see. The title of the jira was misleading.
> ALTER TABLE may generate huge event (with all partitions)
> -
>
> Key: HIVE-18942
> URL: https://issues.apache.org/jira/browse/HIVE-18942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
> parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
> EventMessage.EventType.ADD_PARTITION,
> new AddPartitionEvent(newt, parts, true, handler),
> environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event
> will contain all of them. Partition object itself isn't very small either, so
> we may end up with huge events which would be stored and then transmitted
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)
[
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1639#comment-1639
]
Na Li commented on HIVE-18942:
--
[~kkalyan] The event type is "EventMessage.EventType.ADD_PARTITION", not
"EventType.ALTER_TABLE". It puts all partitions in the event.
{code:java}
In DbNotificationListener
public void onAddPartition (AddPartitionEvent partitionEvent)
throws MetaException {
Table t = partitionEvent.getTable();
NotificationEvent event = new NotificationEvent(0, now(),
HCatConstants.HCAT_ADD_PARTITION_EVENT,
msgFactory.buildAddPartitionMessage(t,
partitionEvent.getPartitions()).toString());
event.setDbName(t.getDbName());
event.setTableName(t.getTableName());
enqueue(event, partitionEvent);
}
{code}
> ALTER TABLE may generate huge event (with all partitions)
> -
>
> Key: HIVE-18942
> URL: https://issues.apache.org/jira/browse/HIVE-18942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
> parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
> EventMessage.EventType.ADD_PARTITION,
> new AddPartitionEvent(newt, parts, true, handler),
> environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event
> will contain all of them. Partition object itself isn't very small either, so
> we may end up with huge events which would be stored and then transmitted
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18942) ALTER TABLE may generate huge event (with all partitions)
[
https://issues.apache.org/jira/browse/HIVE-18942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396929#comment-16396929
]
kalyan kumar kalvagadda commented on HIVE-18942:
[~akolb] That may not be the case.
AddPartitionEvent that is generated here is different from the
NotificationEvent that is inserted in the NOTIFICATION_LOG table.
{noformat}
public void onAlterTable(AlterTableEvent tableEvent) throws MetaException {
Table before = tableEvent.getOldTable();
Table after = tableEvent.getNewTable();
NotificationEvent event =
new NotificationEvent(0, now(), EventType.ALTER_TABLE.toString(),
msgFactory
.buildAlterTableMessage(before, after,
tableEvent.getIsTruncateOp()).toString());
event.setDbName(after.getDbName());
event.setTableName(after.getTableName());
process(event, tableEvent);
}
{noformat}
This is how later table JSON is constructed as sentry is interested in below
things in the alter table notification
# Old Db Name
# New Db Name
# Old Table Name
# New Table Name
# Old Location
# New Location
{noformat}
public JSONAlterTableMessage(String server, String servicePrincipal, Table
tableObjBefore, Table tableObjAfter, Long timestamp) {
this.server = server;
this.servicePrincipal = servicePrincipal;
this.db = tableObjBefore.getDbName();
this.table = tableObjBefore.getTableName();
this.timestamp = timestamp;
try {
this.tableObjBeforeJson =
JSONMessageFactory.createTableObjJson(tableObjBefore);
this.tableObjAfterJson =
JSONMessageFactory.createTableObjJson(tableObjAfter);
} catch (TException var7) {
throw new IllegalArgumentException("Could not serialize: ", var7);
}
this.checkValid();
}
{noformat}
I think the NotificationEvent inserted in NOTIFICATION_LOG table may not have
details of the partitions.
> ALTER TABLE may generate huge event (with all partitions)
> -
>
> Key: HIVE-18942
> URL: https://issues.apache.org/jira/browse/HIVE-18942
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Priority: Major
>
> ALTER TABLE handler in HiveAlterHandler has this code:
> {code:java}
> if (isPartitionedTable) {
> parts = msdb.getPartitions(newt.getDbName(), newt.getTableName(), -1);
> MetaStoreListenerNotifier.notifyEvent(transactionalListeners,
> EventMessage.EventType.ADD_PARTITION,
> new AddPartitionEvent(newt, parts, true, handler),
> environmentContext);
> }{code}
> The problem is that table may contain huge number of partitions and the event
> will contain all of them. Partition object itself isn't very small either, so
> we may end up with huge events which would be stored and then transmitted
> over the wire to consumers.
> [~spena] [~kkalyan] [~lina.li] [~vaidyand] FYI.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
