[Structured Streaming] Query Metrics to MetricsSink

[lucas-vsco]

I am looking to take the metrics exposed in the logs from MicroBatchExecution
below and submit them as stats to implemented MetricsSinks.

2018-03-20 10:28:48 INFO  MicroBatchExecution:54 - Streaming query made
progress: {
  "id" : "42bb5c95-980d-480d-9dee-72e1baf6a5b3",
  "runId" : "d9209988-7715-47f6-8845-1ea8208ecd9e",
  "name" : null,
  "timestamp" : "2018-03-20T14:28:45.074Z",
  "batchId" : 1,
  "numInputRows" : 10,
  "inputRowsPerSecond" : 909.0909090909091,
  "processedRowsPerSecond" : 2.8019052956010086,
  "durationMs" : {
    "addBatch" : 3462,
    "getBatch" : 4,
    "getOffset" : 2,
    "queryPlanning" : 33,
    "triggerExecution" : 3569,
    "walCommit" : 64
  },
  "stateOperators" : [ {
    "numRowsTotal" : 10,
    "numRowsUpdated" : 8,
    "memoryUsedBytes" : 27191
  } ],
  "sources" : [ {
    "description" : "KafkaSource[Subscribe[image-enrichment.test]]",
    "startOffset" : {
      "image-enrichment.test" : {
        "0" : 7970
      }
    },
    "endOffset" : {
      "image-enrichment.test" : {
        "0" : 7980
      }
    },
    "numInputRows" : 10,
    "inputRowsPerSecond" : 909.0909090909091,
    "processedRowsPerSecond" : 2.8019052956010086
  } ],
  "sink" : {
    "description" :
"org.apache.spark.sql.kafka010.KafkaSourceProvider@3263da34"
  }
}

Is there a way to write a custom wrapper to handle this?



--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org