问题描述
我在 Keras 中创建了一个混合模型，为元数据和图像数据创建了权重，然后将它们组合起来进行分类。这是模型：
Model: "model_1"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_5 (InputLayer) [(None,80,120,3)] 0
__________________________________________________________________________________________________
xception (Functional) (None,3,4,2048) 20861480 input_5[0][0]
__________________________________________________________________________________________________
input_4 (InputLayer) [(None,10)] 0
__________________________________________________________________________________________________
conv2d_9 (Conv2D) (None,8) 409608 xception[0][0]
__________________________________________________________________________________________________
dense_3 (Dense) (None,4) 44 input_4[0][0]
__________________________________________________________________________________________________
global_average_pooling2d_1 (Glo (None,8) 0 conv2d_9[0][0]
__________________________________________________________________________________________________
concatenate_1 (Concatenate) (None,12) 0 dense_3[0][0]
global_average_pooling2d_1[0][0]
__________________________________________________________________________________________________
dense_4 (Dense) (None,4) 52 concatenate_1[0][0]
__________________________________________________________________________________________________
dense_5 (Dense) (None,1) 5 dense_4[0][0]
==================================================================================================
Total params: 21,271,189
Trainable params: 21,216,661
Non-trainable params: 54,528
__________________________________________________________________________________________________
由于不平衡，我决定增加图像。我使用了以下 ImageDataGenerator：
aug = ImageDataGenerator(rescale=1/255.,rotation_range=180,height_shift_range=0.2,width_shift_range=0.2,brightness_range=[0.5,1.5],channel_shift_range=100.0,horizontal_flip=True,vertical_flip=True,shear_range=45.0)
然后我编译并尝试使用 ImageDataGenerator().flow() 训练模型：
epochs = 10
BATCH_SIZE = 20
flow = aug.flow(img_train,y_train,batch_size=BATCH_SIZE)
history = model.fit([Meta_train,flow],epochs=epochs,batch_size=100,validation_data=([Meta_test,img_test],y_test),class_weight=class_weight)
ValueError: Failed to find data adapter that can handle input: (<class 'list'> containing values of
types {"<class 'pandas.core.frame.DataFrame'>","<class 'tensorflow.python.keras.preprocessing.image.NumpyArrayIterator'>"}),<class 'numpy.ndarray'>
我已经尝试了多个版本的代码，但我对后端不够熟悉，无法正确诊断问题。有人能帮我解决这个问题吗？
型号代码和 MRE
型号代码
LEARNING_RATE = 0.001
# Define inputs
Meta_inputs = Input(shape=(10,))
img_inputs = Input(shape=(80,))
# Model 1
Meta_layer1 = Dense(4,activation='relu')(Meta_inputs)
# Model 2
xception_layer = Xception(include_top=False,input_shape=(80,))(img_inputs)
img_conv_layer1 = Conv2D(8,kernel_size=(5,5),padding='same',activation='relu')(xception_layer)
img_gap_layer = GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4,activation='relu')(img_gap_layer)
# Merge models
merged_layer = Concatenate()([Meta_layer1,img_gap_layer])
merged_dense_layer = Dense(4,activation='relu')(merged_layer)
merged_output = Dense(1,activation='sigmoid')(merged_dense_layer)
# Define functional model
model = Model(inputs=[Meta_inputs,img_inputs],outputs=merged_output)
# Compile model
auc = AUC(name = 'auc')
model.compile(Adam(learning_rate=LEARNING_RATE),loss='binary_crossentropy',metrics=[auc])
model.summary()
Meta_train MRE
age_approx UnkNown female male head/neck lower extremity \
11655 45 0 0 1 0 0
24502 60 0 0 1 0 1
2524 50 0 1 0 0 1
13894 60 0 1 0 0 0
29325 45 0 1 0 0 1
oral/genital palms/soles torso upper extremity
11655 0 0 1 0
24502 0 0 0 0
2524 0 0 0 0
13894 0 0 1 0
29325 0 0 0 0
img_train MRE
y_train.shape
(23188,1)
解决方法
这不是直接的答案，但有助于解决问题。缺少解决您问题的信息。
但首先：最明显的问题是；您提供 meta_train 这是一个熊猫数据框。将其转换为数组。 请先试试这个。
第二，如果还是有问题，那么model.fit可能无法处理一个列表[meta_train,flow]，那么你可能要想办法提供两个input来处理根据您的模型。
为此，您应该提供以下内容作为 MRE 以重现您的问题。
(1) 您的模型代码，即使您可以根据提供的摘要生成相同的模型。
(2) y_train 的形状。
,首先，如果您生成以下数组，您的模型可以正常工作；
import tensorflow as tf
import numpy as np
LEARNING_RATE = 0.001
# Define inputs
meta_inputs = tf.keras.layers.Input(shape=(10,))
img_inputs = tf.keras.layers.Input(shape=(80,120,3,))
# Model 1
meta_layer1 = tf.keras.layers.Dense(4,activation='relu')(meta_inputs)
# Model 2
xception_layer = tf.keras.applications.Xception(include_top=False,input_shape=(80,))(img_inputs)
img_conv_layer1 = tf.keras.layers.Conv2D(8,kernel_size=(5,5),padding='same',activation='relu')(xception_layer)
img_gap_layer = tf.keras.layers.GlobalAveragePooling2D()(img_conv_layer1)
# img_sdense_layer = Dense(4,activation='relu')(img_gap_layer)
# Merge models
merged_layer = tf.keras.layers.Concatenate()([meta_layer1,img_gap_layer])
merged_dense_layer = tf.keras.layers.Dense(4,activation='relu')(merged_layer)
merged_output = tf.keras.layers.Dense(1,activation='sigmoid')(merged_dense_layer)
# Define functional model
model = tf.keras.models.Model(inputs=[meta_inputs,img_inputs],outputs=merged_output)
# Compile model
auc = tf.keras.metrics.AUC(name = 'auc')
model.compile(tf.keras.optimizers.Adam(learning_rate=LEARNING_RATE),loss='binary_crossentropy',metrics=[auc])
model.summary()
让我们生成如下数组；
y_array = np.zeros((23188,1))
train_array = np.zeros((23188,80,3))
meta_array = np.zeros((23188,10))
现在测试你的模型；
model.fit(x = [meta_array,train_array],y = y_array,epochs = 1)
正如你所看到的那样
>>> model.fit(x = [meta_array,epochs = 1)
2021-05-17 10:30:03.430303: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:176] None of the MLIR Optimization Passes are enabled (registered 2)
2021-05-17 10:30:03.447843: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 3093120000 Hz
47/725 [>.............................] - ETA: 6:52 - loss: 0.6818 - auc: 0.0000e+00
现在问题在于您的输入：
(1) 首先将所有元数据转换为一个数组，假设它有 1000 个批次，因此它的形状必须为 meta_train.shape 为 (1000,10)
(2) 你的 img_train.shape 是 (1000,3)
(3) 你的 y_train.shape 是 (1000,1)
这里 1000 也可以是 23188。但让我们假设您有 1000 张图片、y_train（目标）和 1000 个元数据。
当你想增加你的图像序列时，你必须在这里小心。如下使用；
import tensorflow as tf
import numpy as np
import time
这里我创建了空数组作为示例，但您的原始数据必须按照以下形状排列在数组中，除了 1000，它必须是您拥有的图像数量。
y_train = np.zeros((1000,1))
img_train = np.zeros((1000,3))
meta_train= np.zeros((1000,10))
aug = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1/255.,rotation_range=180,height_shift_range=0.2,width_shift_range=0.2,brightness_range=[0.5,1.5],channel_shift_range=100.0,horizontal_flip=True,vertical_flip=True,shear_range=45.0)
aug.fit(img_train)
# NOTE; change batch_size to 32 if your img_train is too high or your device has less memory
# set shuffle to false,you have to concatenate at each loop the meta_train and y_train for each image,so do not shuffle the images!
imagenerator = aug.flow(img_train,batch_size = img_train.shape[0],shuffle=False,sample_weight=None,seed=123,save_to_dir=None,save_prefix="",save_format="png",subset=None)
new_meta = meta_train.copy() # concatenate meta_train at each loop to new_meta
new_y = y_train.copy() # concatenate y_train at each loop to new_y
batches = 0
#let's iterate 5 times.
for x_batch in imagenerator:
print(batches,time.strftime("%Y:%m%d-%H:%M:%S"))
batches += 1
img_train = np.concatenate((img_train,x_batch),axis = 0)
new_meta = np.concatenate((new_meta,meta_train),axis = 0) # concatenate corresponding meta data
new_y = np.concatenate((new_y,y_train),axis = 0) #concatenate corresponding label/target data!
if batches >= 5:
break
model.fit(x = [new_meta,img_train],y = new_y,epochs = 1,batch_size = 32)
注意：在 model.fit
idxs = np.array([x for x in range(img_train.shape[0])])
np.random.shuffle(idxs)
img_train = img_train[idx]
new_y = new_y[idx]
new_meta = new_meta[idx]