问题描述
我使用 Ray 1.3.0（用于 RLlib）和 SUMO 1.9.2 版 的组合来模拟多-代理场景。我已将 RLlib 配置为使用一个 PPO 网络，该网络通常被所有 N 代理更新/使用。我的评估设置如下所示：
# === Evaluation Settings ===
# Evaluate with every `evaluation_interval` training iterations.
# The evaluation stats will be reported under the "evaluation" metric key.
# Note that evaluation is currently not parallelized,and that for Ape-X
# metrics are already only reported for the lowest epsilon workers.
"evaluation_interval": 20,# Number of episodes to run per evaluation period. If using multiple
# evaluation workers,we will run at least this many episodes total.
"evaluation_num_episodes": 10,# Whether to run evaluation in parallel to a Trainer.train() call
# using threading. Default=False.
# E.g. evaluation_interval=2 -> For every other training iteration,# the Trainer.train() and Trainer.evaluate() calls run in parallel.
# Note: This is experimental. Possible pitfalls could be race conditions
# for weight synching at the beginning of the evaluation loop.
"evaluation_parallel_to_training": False,# Internal flag that is set to True for evaluation workers.
"in_evaluation": True,# Typical usage is to pass extra args to evaluation env creator
# and to disable exploration by computing deterministic actions.
# IMPORTANT NOTE: Policy gradient algorithms are able to find the optimal
# policy,even if this is a stochastic one. Setting "explore=False" here
# will result in the evaluation workers not using this optimal policy!
"evaluation_config": {
# Example: overriding env_config,exploration,etc:
"lr": 0,# To prevent any kind of learning during evaluation
"explore": True # As required by PPO (read IMPORTANT NOTE above)
},# Number of parallel workers to use for evaluation. Note that this is set
# to zero by default,which means evaluation will be run in the trainer
# process (only if evaluation_interval is not None). If you increase this,# it will increase the Ray resource usage of the trainer since evaluation
# workers are created separately from rollout workers (used to sample data
# for training).
"evaluation_num_workers": 1,# Customize the evaluation method. This must be a function of signature
# (trainer: Trainer,eval_workers: WorkerSet) -> metrics: dict. See the
# Trainer.evaluate() method to see the default implementation. The
# trainer guarantees all eval workers have the latest policy state before
# this function is called.
"custom_eval_function": None,发生的是每 20 次迭代（每次迭代收集“X”个训练样本），最少 10 集的评估运行。所有 N 代理收到的奖励总和在这些情节上相加，并设置为该特定评估运行的奖励总和。随着时间的推移，我注意到有一种模式，奖励总和在相同的评估间隔内不断重复，并且学习无处可去。
更新 (23/06/2021)
不幸的是，我没有为那次特定的运行激活 TensorBoard，但从评估期间收集的平均奖励（每 20 次迭代发生）10 集，很明显存在重复模式，如注释图如下：
场景中的 20 个智能体应该学习避免碰撞，而是继续以某种方式停滞在某个策略上并最终在评估期间显示完全相同的奖励序列？
这是我配置评估方面的特征，还是应该检查其他内容？如果有人能给我建议或指出正确的方向，我将不胜感激。
谢谢。
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）