问题描述
默认情况下,在Spark结构化流式Spark中,打印指标,但根据文档 triggerExecution ,必须是执行整个批次所花费的总时间。
triggerExecution = batch_time = (addBatch + getBatch + getEndOffset + queryPlanning + setoffsetRange + triggerExecution + walCommit)
不是这种情况,总批处理时间也不完全等于 triggerExecution 时间。 批处理的总执行时间与
batch_time = triggerExecution + addBatch
指标
"timestamp" : "2020-10-06T09:45:41.721Z","batchId" : 12609,"numInputRows" : 199800,"inputRowsPerSecond" : 651.0624570275969,"processedRowsPerSecond" : 517.4758161639968,"durationMs" : {
"addBatch" : 357041,"getBatch" : 0,"getEndOffset" : 1,"queryPlanning" : 29,"setoffsetRange" : 3012,"triggerExecution" : 386105,"walCommit" : 12949
}
参考
- https://www.waitingforcode.com/apache-spark-structured-streaming/query-metrics-apache-spark-structured-streaming/read
- https://jaceklaskowski.gitbooks.io/spark-structured-streaming/content/spark-sql-streaming-ProgressReporter.html
解决方法
暂无找到可以解决该程序问题的有效方法,小编努力寻找整理中!
如果你已经找到好的解决方法,欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@)