我们可以用Sparktrials保存Hyperopt试用版的结果吗

问题描述

我目前正在尝试使用hyperopt库优化梯度提升方法的超参数。当我在自己的计算机上工作时，我使用了Trials类，并且可以用库腌菜来保存并重新加载结果。这使我可以保存我测试过的所有参数集。我的代码看起来像这样：

from hyperopt import SparkTrials,STATUS_OK,tpe,fmin
from LearningUtils.LearningUtils import build_train_test,get_train_test,mean_error,rmse,mae
from LearningUtils.constants import MAX_EVALS,CV,XGBOOST_OPTIM_SPACE,PARALELISM
from sklearn.model_selection import cross_val_score
import pickle as pkl

if os.path.isdir(PATH_TO_TRIALS): #we reload the past results
    with open(PATH_TO_TRIALS,'rb') as trials_file:
        trials = pkl.load(trials_file)
else : # We create the trials file
    trials = Trials()
    
# classic hyperparameters optimization  
def objective(space):
    regressor = xgb.XGBRegressor(n_estimators = space['n_estimators'],max_depth = int(space['max_depth']),learning_rate = space['learning_rate'],gamma = space['gamma'],min_child_weight = space['min_child_weight'],subsample = space['subsample'],colsample_bytree = space['colsample_bytree'],verbosity=0
                            )
    regressor.fit(X_train,Y_train)
    # Applying k-Fold Cross Validation
    accuracies = cross_val_score(estimator=regressor,x=X_train,y=Y_train,cv=5)
    CrossValMean = accuracies.mean()
    return {'loss':1-CrossValMean,'status': STATUS_OK}

best = fmin(fn=objective,space=XGBOOST_OPTIM_SPACE,algo=tpe.suggest,max_evals=MAX_EVALS,trials=trials,return_argmin=False)

# Save the trials
pkl.dump(trials,open(PATH_TO_TRIALS,"wb"))

现在，我想使此代码在具有更多CPU的远程服务器上工作，以允许并行化并获得时间。

我看到我可以使用hyperopt的SparkTrials类而不是Trials来做到这一点。但是，SparkTrials对象无法与泡菜一起保存。您是否对如何保存和重新加载存储在Sparktrials对象中的试验结果有任何想法？

解决方法

所以这可能有点晚了，但经过一番折腾之后，我找到了一种hacky解决方案：

spark_trials= SparkTrials()
pickling_trials = dict()

for k,v in spark_trials.__dict__.items():
    if not k in ['_spark_context','_spark']:
        pickling_trials[k] = v
        
pickle.dump(pickling_trials,open('pickling_trials.hyperopt','wb'))

SparkTrials 实例的 _spark_context 和 _spark 属性是无法序列化对象的罪魁祸首。事实证明，如果您想重新使用该对象，则不需要它们，因为如果您想再次重新运行优化，无论如何都会创建一个新的 spark 上下文，因此您可以将试验重新用作：

new_sparktrials = SparkTrials()

for att,v in pickling_trials.items():
    setattr(new_sparktrials,att,v)

best = fmin(loss_func,space=search_space,algo=tpe.suggest,max_evals=1000,trials=new_sparktrials)

瞧:)