使用 SparkTrials 进行 hyperopt 并行化

问题描述

我需要优化模型的超参数，并且希望并行化代码以使其更快。 运行此代码：

from hyperopt import hp,fmin,tpe,SparkTrials


def objective(space):
    loss = space['a'] + space['b']
    return loss


space = {'a': hp.quniform('a',1,10,1),'b': hp.quniform('b',1)}

spark_trials = SparkTrials()
best = fmin(objective,space,algo=tpe.suggest,max_evals=100,trials=spark_trials)
print(best)

我收到此错误：

JAVA_HOME is not set
Traceback (most recent call last):
  File "hyperparameters.py",line 13,in <module>
    spark_trials = SparkTrials()
  File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/hyperopt/spark.py",line 84,in __init__
    if spark_session is None
  File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/sql/session.py",line 228,in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/context.py",line 384,in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/context.py",line 144,in __init__
    SparkContext._ensure_initialized(self,gateway=gateway,conf=conf)
  File "/home/el/miniconda3/envs/py36/lib/python3.6/site-packages/pyspark/context.py",line 331,in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/home/el/miniconda3/envs/Xpresso/lib/python3.6/site-packages/pyspark/java_gateway.py",line 108,in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）