Re: [Structured Streaming] Query Metrics to MetricsSink

2018-03-20 Thread lucas-vsco
It actually looks like I might have the answers via these following links:

[Design] Metrics in Structured Streaming

  

JIRA - Structured Streaming - Metrics

  

Thanks.







--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org



[Structured Streaming] Query Metrics to MetricsSink

2018-03-20 Thread lucas-vsco
I am looking to take the metrics exposed in the logs from MicroBatchExecution
below and submit them as stats to implemented MetricsSinks.

2018-03-20 10:28:48 INFO  MicroBatchExecution:54 - Streaming query made
progress: {
  "id" : "42bb5c95-980d-480d-9dee-72e1baf6a5b3",
  "runId" : "d9209988-7715-47f6-8845-1ea8208ecd9e",
  "name" : null,
  "timestamp" : "2018-03-20T14:28:45.074Z",
  "batchId" : 1,
  "numInputRows" : 10,
  "inputRowsPerSecond" : 909.0909090909091,
  "processedRowsPerSecond" : 2.8019052956010086,
  "durationMs" : {
"addBatch" : 3462,
"getBatch" : 4,
"getOffset" : 2,
"queryPlanning" : 33,
"triggerExecution" : 3569,
"walCommit" : 64
  },
  "stateOperators" : [ {
"numRowsTotal" : 10,
"numRowsUpdated" : 8,
"memoryUsedBytes" : 27191
  } ],
  "sources" : [ {
"description" : "KafkaSource[Subscribe[image-enrichment.test]]",
"startOffset" : {
  "image-enrichment.test" : {
"0" : 7970
  }
},
"endOffset" : {
  "image-enrichment.test" : {
"0" : 7980
  }
},
"numInputRows" : 10,
"inputRowsPerSecond" : 909.0909090909091,
"processedRowsPerSecond" : 2.8019052956010086
  } ],
  "sink" : {
"description" :
"org.apache.spark.sql.kafka010.KafkaSourceProvider@3263da34"
  }
}

Is there a way to write a custom wrapper to handle this?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org