-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69587/
-----------------------------------------------------------
(Updated Dec. 18, 2018, 11:18 p.m.)
Review request for atlas, Abhishek Kadam, Ashutosh Mestry, keval bhatt,
Kapildeo Nayak, Mehul Parikh, Nixon Rodrigues, and Sarath Subramanian.
Changes
-------
Ashutosh - thanks for reviewing and catching the cumulative counter issue in
case of import. This update addresses this issue.
Bugs: ATLAS-3002
https://issues.apache.org/jira/browse/ATLAS-3002
Repository: atlas
Description
-------
- added instrumentation to collect time taken for sub-tasks during entity
create/update - like walkEntityGraph, resolveReferences, preCreateOrUpdate,
mapAttributesAndClassifications, fullTextMapping, graphCommit,
entityNotification, entityAudit
- updated default atlas-log4j.xml to include loggers for METRICS
- optimization: updated Hive hook to avoid including quertText in
hive_column_lineage.name attribute value, as this can unnecessarily bloat the
message size for large queries
- optimziation: updated fullTextMapper to skip going through object-id
collections; this saves cycles for example while processing hive_table entities
with large number of columns
- here is the sample metrics:
```json
{
"walkEntityGraph": { "count": 1, "timeTaken": 1 },
"resolveReferences": { "count": 1, "timeTaken": 670 },
"findByUniqueAttributes": { "count": 217, "timeTaken": 668 },
"findByTypeAndPropertyName": { "count": 217, "timeTaken": 641 },
"findBySuperTypeAndPropertyName": { "count": 74, "timeTaken": 27 },
"createOrUpdate": { "count": 1, "timeTaken": 1637 },
"preCreateOrUpdate": { "count": 1, "timeTaken": 720 },
"mapAttributesAndClassifications": { "count": 1, "timeTaken": 90 },
"fullTextMapping": { "count": 1, "timeTaken": 195 },
"notification-getReferenceable": { "count": 1, "timeTaken": 37 },
"entityAudit": { "count": 1, "timeTaken": 31 },
"entityNotification": { "count": 2, "timeTaken": 8 },
"graphCommit": { "count": 75, "timeTaken": 315 }
}
```
Diffs (updated)
-----
addons/hive-bridge/src/main/java/org/apache/atlas/hive/hook/events/CreateHiveProcess.java
2ccfff4d1
common/src/main/java/org/apache/atlas/utils/AtlasPerfMetrics.java
PRE-CREATION
distro/src/conf/atlas-log4j.xml c183871eb
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java
4bec91709
repository/src/main/java/org/apache/atlas/GraphTransactionInterceptor.java
4c436779e
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListener.java
dfacb3817
repository/src/main/java/org/apache/atlas/repository/audit/EntityAuditListenerV2.java
8ca8c9a0b
repository/src/main/java/org/apache/atlas/repository/graph/FullTextMapperV2.java
08ccd9c73
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityChangeNotifier.java
a8c3363d5
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityGraphDiscoveryV2.java
6580beecd
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasEntityStoreV2.java
d97b74d9d
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/AtlasGraphUtilsV2.java
25770a334
repository/src/main/java/org/apache/atlas/repository/store/graph/v2/EntityGraphMapper.java
6c830bafe
server-api/src/main/java/org/apache/atlas/RequestContext.java 099d713f6
webapp/src/main/java/org/apache/atlas/notification/EntityNotificationListenerV2.java
e0a60a133
webapp/src/main/java/org/apache/atlas/notification/NotificationEntityChangeListener.java
b5e7ed871
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
b344c50e6
Diff: https://reviews.apache.org/r/69587/diff/2/
Changes: https://reviews.apache.org/r/69587/diff/1-2/
Testing
-------
- verified the instumentation output in metric.log
- pre-commit tests run:
https://builds.apache.org/view/A/view/Atlas/job/PreCommit-ATLAS-Build-Test/869
Thanks,
Madhan Neethiraj
