加载 ML PySpark 模型失败

问题描述

我有几个无法加载的回归模型。这是 Spark 初始化：

from pyspark.sql import SparkSession,sqlContext
from pyspark.ml.regression import DecisionTreeRegressor

spark = SparkSession.builder \
    .appName("Linear Regression Model") \
    .config('spark.executor.cores','2') \
    .config("spark.executor.memory","5gb") \
    .master("local[*]") \
    .getorCreate() 

sc = spark.sparkContext

这是模型拟合并成功保存：

# Decision Tree Regression
decisionTree = DecisionTreeRegressor(featuresCol = "Features",labelCol = "SalePrice",maxDepth = 15,maxBins = 32)
decisionTreeModel = decisionTree.fit(train_vector)

import os

decisionTreeModel.save(os.path.join(".",'decisionTreeModel'))

但是当我加载它时：

persistedModel = DecisionTreeRegressor.load("decisionTreeModel")

我收到此错误：

Py4JJavaError: An error occurred while calling o1201.load.
: java.lang.NoSuchMethodException: org.apache.spark.ml.regression.DecisionTreeRegressionModel.<init>(java.lang.String)
    at java.lang.class.getConstructor0(Class.java:3082)
    at java.lang.class.getConstructor(Class.java:1825)
    at org.apache.spark.ml.util.DefaultParamsReader.load(ReadWrite.scala:468)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

有人知道如何加载 PySpark 模型吗？

解决方法

错误信息不是很有帮助，但我认为加载模型的正确方法是调用模型的 load 方法，而不是估计器的方法。模型已经拟合到数据，这与估计器不同，它只包含设置/参数，但没有拟合。

所以你可以试试这个：

from pyspark.ml.regression import DecisionTreeRegressionModel

persistedModel = DecisionTreeRegressionModel.load("decisionTreeModel")

以下是加载估算器与加载模型的比较，供您参考：

from pyspark.ml.regression import DecisionTreeRegressor,DecisionTreeRegressionModel

decisionTree = DecisionTreeRegressor(featuresCol = "Features",labelCol = "SalePrice",maxDepth = 15,maxBins = 32)
decisionTree.save('tree')
persistedEstimator = DecisionTreeRegressor.load('tree')

decisionTreeModel = decisionTree.fit(df)
decisionTreeModel.save('model')
persistedModel = DecisionTreeRegressionModel.load('model')

apache-spark apache-spark-ml pyspark pyspark