Jacek Laskowski created SPARK-27975:
---------------------------------------
Summary: ConsoleSink should display alias and options for
streaming progress
Key: SPARK-27975
URL: https://issues.apache.org/jira/browse/SPARK-27975
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.4.3
Reporter: Jacek Laskowski
{{console}} sink shows itself in progress with this internal instance
representation as follows:
{code:json}
"sink" : {
"description" :
"org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
}
{code}
That is not very user-friendly and would be much better for debugging if it
included the alias and options as {{socket}} does:
{code}
"sources" : [ {
"description" : "TextSocketV2[host: localhost, port: 8888]",
...
} ],
{code}
The entire sample progress looks as follows:
{code}
19/06/07 11:47:18 INFO MicroBatchExecution: Streaming query made progress: {
"id" : "26bedc9f-076f-4b15-8e17-f09609aaecac",
"runId" : "8c365e74-7ac9-4fad-bf1b-397eb086661e",
"name" : "socket-console",
"timestamp" : "2019-06-07T09:47:18.969Z",
"batchId" : 2,
"numInputRows" : 0,
"inputRowsPerSecond" : 0.0,
"durationMs" : {
"getEndOffset" : 0,
"setOffsetRange" : 0,
"triggerExecution" : 0
},
"stateOperators" : [ ],
"sources" : [ {
"description" : "TextSocketV2[host: localhost, port: 8888]",
"startOffset" : 0,
"endOffset" : 0,
"numInputRows" : 0,
"inputRowsPerSecond" : 0.0
} ],
"sink" : {
"description" :
"org.apache.spark.sql.execution.streaming.ConsoleSinkProvider@12fa674a"
}
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
