[jira] [Updated] (ATLAS-4635) Source information is missing in the Hook message of Spark entries
[
https://issues.apache.org/jira/browse/ATLAS-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dharshana M Krishnamoorthy updated ATLAS-4635:
--
Description:
Source information is missing for impala messages
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657759634684,"spooled":false,"message":...}
{code}
This is a message from ATLAS_SPARK_HOOK, the source infor of the message is
missing here
Expected:
"source":\{"source": "spark", "version":"2.1.0.7.1.8.0-723"},
but source is missing
It is available for other services:
HMS:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hive_metastore"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700529795,"spooled":false,"message":{...}
{code}
HBase:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hbase"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700897665,"spooled":false,"message":{...}
{code}
Impala:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"impala"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657695780022,"spooled":false,"message":{"...}}{code}
h4.
was:
Source information is missing for impala messages
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657759634684,"spooled":false,"message":...}
{code}
This is a message from ATLAS_SPARK_HOOK, the source infor of the message is
missing here
Expected:
"source":\{"source": "spark", "version":"2.1.0.7.1.8.0-723"},
but source is missing
It is available for other services:
HMS:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hive_metastore"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700529795,"spooled":false,"message":{
{code}
{color:#009100}...{color}
{code:java}
}{code}
HBase:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hbase"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700897665,"spooled":false,"message":{
{code}
{color:#009100}...{color}
{code:java}
}{code}
Impala:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"impala"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657695780022,"spooled":false,"message":{"...}}{code}
h4.
> Source information is missing in the Hook message of Spark entries
> --
>
> Key: ATLAS-4635
> URL: https://issues.apache.org/jira/browse/ATLAS-4635
> Project: Atlas
> Issue Type: Bug
> Components: atlas-core
>Reporter: Dharshana M Krishnamoorthy
>Priority: Major
>
> Source information is missing for impala messages
> {code:java}
> {"source":{"version":"2.1.0.7.1.8.0-723"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657759634684,"spooled":false,"message":...}
> {code}
> This is a message from ATLAS_SPARK_HOOK, the source infor of the message is
> missing here
> Expected:
> "source":\{"source": "spark", "version":"2.1.0.7.1.8.0-723"},
> but source is missing
> It is available for other services:
> HMS:
> {code:java}
>
> {"source":{"version":"2.1.0.7.1.8.0-723","source":"hive_metastore"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700529795,"spooled":false,"message":{...}
> {code}
> HBase:
> {code:java}
>
> {"source":{"version":"2.1.0.7.1.8.0-723","source":"hbase"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700897665,"spooled":false,"message":{...}
> {code}
> Impala:
> {code:java}
>
> {"source":{"version":"2.1.0.7.1.8.0-723","source":"impala"},"ver
[jira] [Updated] (ATLAS-4635) Source information is missing in the Hook message of Spark entries
[
https://issues.apache.org/jira/browse/ATLAS-4635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dharshana M Krishnamoorthy updated ATLAS-4635:
--
Description:
Source information is missing for impala messages
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657759634684,"spooled":false,"message":...}
{code}
This is a message from ATLAS_SPARK_HOOK, the source infor of the message is
missing here
Expected:
"source":\{"source": "spark", "version":"2.1.0.7.1.8.0-723"},
but source is missing
It is available for other services:
HMS:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hive_metastore"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700529795,"spooled":false,"message":{
{code}
{color:#009100}...{color}
{code:java}
}{code}
HBase:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hbase"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657700897665,"spooled":false,"message":{
{code}
{color:#009100}...{color}
{code:java}
}{code}
Impala:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"impala"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"...","msgCreatedBy":"...","msgCreationTime":1657695780022,"spooled":false,"message":{"...}}{code}
h4.
was:
Source information is missing for impala messages
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"172.27.134.1","msgCreatedBy":"hrt_qa","msgCreationTime":1657759634684,"spooled":false,"message":{"type":"ENTITY_CREATE_V2","user":"hrt...@root.hwx.site","entities":{"entities":[{"typeName":"spark_column_lineage","attributes":{"outputs":[{"typeName":"hive_column","uniqueAttributes":{"qualifiedName":"default.tab_1_2235274282_ctas.col1@cm"}}],"qualifiedName":"default.tab_1_2235274282_ctas@cm:1657759634668:col1","inputs":[{"typeName":"hive_column","uniqueAttributes":{"qualifiedName":"default.tab_1_2235274282.col1@cm"}}],"name":"default.tab_1_2235274282_ctas@cm:1657759634668:col1"},"guid":"-215346008271268","isIncomplete":false,"provenanceType":0,"version":0,"relationshipAttributes":{"process":{"typeName":"spark_process","uniqueAttributes":{"qualifiedName":"application_1657738076117_0064-execution-12"}}},"proxy":false}]}}}
{code}
This is a message from ATLAS_SPARK_HOOK, the source infor of the message is
missing here
Expected:
"source":\{"source": "spark", "version":"2.1.0.7.1.8.0-723"},
but source is missing
It is available for other services:
HMS:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hive_metastore"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"172.27.10.134","msgCreatedBy":"hive","msgCreationTime":1657700529795,"spooled":false,"message":{"type":"ENTITY_DELETE_V2","user":"hue","entities":[{"typeName":"hive_db","uniqueAttributes":{"qualifiedName":"cloudera_manager_metastore_canary_test_catalog_hive_hivemetastore_1:default@cm"}}]}}{code}
HBase:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"hbase"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"172.27.72.195","msgCreatedBy":"hbase","msgCreationTime":1657700897665,"spooled":false,"message":{"type":"ENTITY_CREATE_V2","user":"hrt_qa","entities":{"entities":[{"typeName":"hbase_namespace","attributes":{"owner":"hrt_qa","modifiedTime":1657700897665,"createTime":1657700897665,"qualifiedName":"ns_zsptr@cm","clusterName":"cm","name":"ns_zsptr","description":"ns_zsptr","parameters":null},"guid":"-3685335942958094","isIncomplete":false,"provenanceType":0,"version":0,"proxy":false}]}}}{code}
Impala:
{code:java}
{"source":{"version":"2.1.0.7.1.8.0-723","source":"impala"},"version":{"version":"1.0.0","versionParts":[1]},"msgCompressionKind":"NONE","msgSplitIdx":1,"msgSplitCount":1,"msgSourceIP":"172.27.187.129","msgCreatedBy":"impala","msgCreationTime":1657695780022,"spooled":false,"message":{"type":"ENTITY_CREATE_V2","user":"hrt...@qe-infra-ad.cloudera.com","entities":{"referredEntities":{},"entities":[{"typeName":"impala_process","attributes":{"outputs":[{"typeName":"hive_table","uniqueAttributes":{"qualifiedName":"db_mzuh.tbl_mzuh_2@cm"}}],"recentQueries":["insert
into db_mzuh.tbl_mzuh_2 select * from (select tbl_mzuh_0_col_0 from
db_mzuh.tbl_mzuh_0) as a join
