机器学习 – TensorFlow – L2丢失的正则化,如何应用于所有权重,而不仅仅是最后一个？

我正在玩一个作为Udacity DeepLearning课程一部分的ANN。

我有一个任务，涉及到使用L2丢失的一个隐藏的ReLU层向网络引入泛化。我不知道如何正确引入它，以便所有权重都受到惩罚，不仅仅是输出层的权重。

代码网络没有泛化是在底部的帖子(代码实际运行的培训是超出了问题的范围)。

引入L2的明显方法是用这样的代替损失计算(如果beta为0.01)：

loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(out_layer,tf_train_labels) + 0.01*tf.nn.l2_loss(out_weights))

但在这种情况下，它将考虑到输出层权重的值。我不确定，我们如何妥善地惩罚隐藏的ReLU层的权重。是否需要或引入惩罚的输出层将以某种方式保持隐藏的权重也在检查？

#some importing
from __future__ import print_function
import numpy as np
import tensorflow as tf
from six.moves import cPickle as pickle
from six.moves import range

#loading data
pickle_file = '/home/maxkhk/Documents/Udacity/DeepLearningCourse/SourceCode/tensorflow/examples/udacity/notMNIST.pickle'

with open(pickle_file,'rb') as f:
  save = pickle.load(f)
  train_dataset = save['train_dataset']
  train_labels = save['train_labels']
  valid_dataset = save['valid_dataset']
  valid_labels = save['valid_labels']
  test_dataset = save['test_dataset']
  test_labels = save['test_labels']
  del save  # hint to help gc free up memory
  print('Training set',train_dataset.shape,train_labels.shape)
  print('Validation set',valid_dataset.shape,valid_labels.shape)
  print('Test set',test_dataset.shape,test_labels.shape)


#prepare data to have right format for tensorflow
#i.e. data is flat matrix,labels are onehot

image_size = 28
num_labels = 10

def reformat(dataset,labels):
  dataset = dataset.reshape((-1,image_size * image_size)).astype(np.float32)
  # Map 0 to [1.0,0.0,0.0 ...],1 to [0.0,1.0,0.0 ...]
  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
  return dataset,labels
train_dataset,train_labels = reformat(train_dataset,train_labels)
valid_dataset,valid_labels = reformat(valid_dataset,valid_labels)
test_dataset,test_labels = reformat(test_dataset,test_labels)
print('Training set',train_labels.shape)
print('Validation set',valid_labels.shape)
print('Test set',test_labels.shape)


#Now is the interesting part - we are building a network with
#one hidden ReLU layer and out usual output linear layer

#we are going to use SGD so here is our size of batch
batch_size = 128

#building tensorflow graph
graph = tf.Graph()
with graph.as_default():
      # Input data. For the training data,we use a placeholder that will be fed
  # at run time with a training minibatch.
  tf_train_dataset = tf.placeholder(tf.float32,shape=(batch_size,image_size * image_size))
  tf_train_labels = tf.placeholder(tf.float32,num_labels))
  tf_valid_dataset = tf.constant(valid_dataset)
  tf_test_dataset = tf.constant(test_dataset)

  #Now let's build our new hidden layer
  #that's how many hidden neurons we want
  num_hidden_neurons = 1024
  #its weights
  hidden_weights = tf.Variable(
    tf.truncated_normal([image_size * image_size,num_hidden_neurons]))
  hidden_biases = tf.Variable(tf.zeros([num_hidden_neurons]))

  #Now the layer itself. It multiplies data by weights,adds biases
  #and takes ReLU over result
  hidden_layer = tf.nn.relu(tf.matmul(tf_train_dataset,hidden_weights) + hidden_biases)

  #time to go for output linear layer
  #out weights connect hidden neurons to output labels
  #biases are added to output labels  
  out_weights = tf.Variable(
    tf.truncated_normal([num_hidden_neurons,num_labels]))  

  out_biases = tf.Variable(tf.zeros([num_labels]))  

  #compute output  
  out_layer = tf.matmul(hidden_layer,out_weights) + out_biases
  #our real output is a softmax of prior result
  #and we also compute its cross-entropy to get our loss
  loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(out_layer,tf_train_labels))

  #Now we just minimize this loss to actually train the network
  optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(loss)

  #nice,Now let's calculate the predictions on each dataset for evaluating the
  #performance so far
  # Predictions for the training,validation,and test data.
  train_prediction = tf.nn.softmax(out_layer)
  valid_relu = tf.nn.relu(  tf.matmul(tf_valid_dataset,hidden_weights) + hidden_biases)
  valid_prediction = tf.nn.softmax( tf.matmul(valid_relu,out_weights) + out_biases) 

  test_relu = tf.nn.relu( tf.matmul( tf_test_dataset,hidden_weights) + hidden_biases)
  test_prediction = tf.nn.softmax(tf.matmul(test_relu,out_weights) + out_biases)

hidden_​​weights，hidden_​​biases，out_weights和out_biases都是您正在创建的模型参数。您可以按如下所示对所有这些参数添加L2正则化：

loss = (tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
    out_layer,tf_train_labels) +
    0.01*tf.nn.l2_loss(hidden_weights) +
    0.01*tf.nn.l2_loss(hidden_biases) +
    0.01*tf.nn.l2_loss(out_weights) +
    0.01*tf.nn.l2_loss(out_biases))