Keras中的自定义损失功能不起作用

问题描述

我正在使用Keras实现PPO算法，但是在Keras中遇到了自定义损失函数的一些问题。

Tensorflow版本：2.3.0
Keras版本：2.4.3

这是喀拉拉邦的自定义损失功能：

def ppo_loss(old_prediction,advantage,reward,value):
    def loss(y_true,y_pred):
        # print(type(y_true),type(y_pred),type(old_prediction),type(advantage),type(reward),type(value))
        newpolicy_probs = y_pred
        ratio = K.exp(K.log(newpolicy_probs + 1e-10) - K.log(old_prediction + 1e-10))
        clip_ratio = K.clip(ratio,min_value=1 - EPSILON,max_value=1 + EPSILON)

        surrogate1 = ratio * advantage
        surrogate2 = clip_ratio * advantage
        actor_loss = -K.mean(K.minimum(surrogate1,surrogate2))

        critic_loss = K.mean(K.square(reward - value))
        entropy_loss = K.mean(-(newpolicy_probs * K.log(newpolicy_probs + 1e-10)))

        total_loss = CRITIC_disCOUNT * critic_loss + actor_loss - BETA * entropy_loss
        return total_loss
    return loss

这是actor_model：

    def actor_model(input_dims,output_dims):
        state = Input(shape=(input_dims,),name='state_input')
        old_prediction = Input(shape=(output_dims,name='old_prediction_input')
        advantage = Input(shape=(1,name='advantage_input')
        reward = Input(shape=(1,name='reward_input')
        value = Input(shape=(1,name='value_input')

        x = Dense(HIDDEN_UNITS,activation='tanh',name='fc1')(state)
        x = Dense(HIDDEN_UNITS,activation='tanh')(x)
        policy = Dense(output_dims,name='policy')(x)

        actor_network = Model(inputs=[state,old_prediction,value],outputs=[policy])
        actor_network.compile(optimizer=Adam(lr=LEARNING_RATE),loss=ppo_loss(
                old_prediction=old_prediction,advantage=advantage,reward=reward,value=value),# run_eagerly=True
        )
        # actor_network.summary()
        return actor_network

我收到此错误：

Traceback (most recent call last):
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\execute.py",line 59,in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle,device_name,op_name,TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example,the following function will fail:
  @tf.function
  def has_init_scope():
    my_constant = tf.constant(1.)
    with tf.init_scope():
      added = my_constant * 2
The graph tensor has name: old_prediction_input:0

During handling of the above exception,another exception occurred:

Traceback (most recent call last):
  File "train.py",line 426,in <module>
    agent.train()
  File "train.py",line 370,in train
    actor_loss = self.actor.fit(
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\keras\engine\training.py",line 108,in _method_wrapper
    return method(self,*args,**kwargs)
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\keras\engine\training.py",line 1098,in fit
    tmp_logs = train_function(iterator)
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\def_function.py",line 780,in __call__
    result = self._call(*args,**kwds)
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\def_function.py",line 840,in _call
    return self._stateless_fn(*args,**kwds)
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 2829,in __call__
    return graph_function._filtered_call(args,kwargs)  # pylint: disable=protected-access
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 1843,in _filtered_call
    return self._call_flat(
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 1923,in _call_flat
    return self._build_call_outputs(self._inference_function.call(
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\function.py",line 545,in call
    outputs = execute.execute(
  File "C:\Users\rahul\Desktop\Train-ml-agents\python-envs\offline_training\lib\site-packages\tensorflow\python\eager\execute.py",line 72,in quick_execute
    raise core._SymbolicException(
tensorflow.python.eager.core._SymbolicException: Inputs to eager execution function cannot be Keras symbolic tensors,but found [<tf.Tensor 'old_prediction_input:0' shape=(None,2) dtype=float32>,<tf.Tensor 'advantage_input:0' shape=(None,1) dtype=float32>,<tf.Tensor 'reward_input:0' shape=(None,<tf.Tensor 'value_input:0' shape=(None,1) dtype=float32>]

因此，在互联网上搜索后，我找到了一种解决方案，即将run_eagerly=True添加到model.compile（）方法中，如下所示：

actor_model.compile(...,run_eagerly=True)

但是添加了这个之后，我从actor_model.history['loss']得到了0损失值

有人可以指出错误，还是我执行错误？

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

keras keras loss-function python reinforcement-learning tensorflow tensorflow tensorflow