火花测量：无数据报告

编程问答 2022-10-25

问题描述

客观

我正在将Spark应用程序从本地计算机（客户端模式）提交到具有databricks-connect（v6.6）的Databricks集群。如Spark Measure page中所述，使用PyPi sparkmeasure==0.14.0。

问题

为什么Spark Measure不打印任何指标？可以将Spark Measure与databricks-connect一起使用吗？

代码

spark = SparkSession \
    .builder \
    .appName(app_name) \
    .config("spark.jars.packages","ch.cern.sparkmeasure:spark-measure_2.11:0.16") \
    .config("spark.driver.host","localhost") \
    .config("spark.driver.bindAddress","127.0.0.1") \
    .config("fs.azure.account.key.<my_storage>.dfs.core.windows.net",key) \
    .getorCreate()

 from sparkmeasure import StageMetrics,TaskMetrics
 df = load_data(some_path)
 StageMetrics(self.spark).runandmeasure(locals(),'df.count()'). # output 1

 df2 = load_data(some_path)
 TaskMetrics(self.spark).runandmeasure(locals(),'df2.count()'). # output 2

输出1 

Scheduling mode = FIFO
Spark Context default degree of parallelism = 8
 no data to report

输出2 

Scheduling mode = FIFO
Spark Contex default degree of parallelism = 8
Aggregated Spark task metrics:
numtasks => 0
elapsedtime => null
sum(duration) => null
sum(schedulerDelay) => null
sum(executorRunTime) => null
sum(executorcpuTime) => null
sum(executorDeserializeTime) => null
sum(executorDeserializecpuTime) => null
sum(resultSerializationTime) => null
sum(jvmGCTime) => null
sum(shuffleFetchWaitTime) => null
sum(shuffleWriteTime) => null
sum(gettingResultTime) => null
max(resultSize) => null
sum(numUpdatedBlockStatuses) => null
sum(diskBytesspilled) => null
sum(memoryBytesspilled) => null
max(peakExecutionMemory) => null
sum(recordsRead) => null
sum(bytesRead) => null
sum(recordsWritten) => null
sum(bytesWritten) => null
sum(shuffletotalBytesRead) => null
sum(shuffletotalBlocksFetched) => null
sum(shuffleLocalBlocksFetched) => null
sum(shuffleRemoteBlocksFetched) => null
sum(shuffleBytesWritten) => null
sum(shuffleRecordsWritten) => null

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

apache-sparkazure-databricksdatabricks