问题描述
在机器学习的kaggle微课程中，您可以找到以下数据集和代码，以帮助您为比赛建立预测模型：https://www.kaggle.com/ [put your user name here] / exercise-categorical-variables / edit
它为您提供了两个数据集：1个训练数据集和1个测试数据集，您将使用它们进行预测并提交以查看比赛中的排名
所以在
第5步：生成测试预测并提交结果
我写了这段代码：
EDITED
# Read the data
X = pd.read_csv('../input/train.csv',index_col='Id')
X_test = pd.read_csv('../input/test.csv',index_col='Id')
# Remove rows with missing target,separate target from predictors
X.dropna(axis=0,subset=['SalePrice'],inplace=True)
y = X.SalePrice
X.drop(['SalePrice'],axis=1,inplace=True)
# To keep things simple,we'll drop columns with missing values
cols_with_missing = [col for col in X.columns if X[col].isnull().any()]
X.drop(cols_with_missing,inplace=True)
X_test.drop(cols_with_missing,inplace=True)
#print(X_test.shape,X.shape)
X_test.head()
# Break off validation set from training data
X_train,X_valid,y_train,y_valid = train_test_split(X,y,train_size=0.8,test_size=0.2,random_state=0)
X_train.head()
#Asses Viability of method ONE-HOT
# Get number of unique entries in each column with categorical data
object_nunique = list(map(lambda col: X_train[col].nunique(),object_cols))
d = dict(zip(object_cols,object_nunique))
# Print number of unique entries by column,in ascending order
sorted(d.items(),key=lambda x: x[1])
# Columns that will be one-hot encoded ####<<<<I THINK THAT THE PROBLEM STARTS HERE>>>>#####
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]
##############For X_train
# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unkNown='ignore',sparse=False)
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[low_cardinality_cols]))
# One-hot encoding removed index; put it back
OH_cols_train.index = X_train.index
OH_cols_valid.index = X_valid.index
# Remove categorical columns (will replace with one-hot encoding)
num_X_train = X_train.drop(object_cols,axis=1)
num_X_valid = X_valid.drop(object_cols,axis=1)
# Add one-hot encoded columns to numerical features
OH_X_train = pd.concat([num_X_train,OH_cols_train],axis=1)
OH_X_valid = pd.concat([num_X_valid,OH_cols_valid],axis=1)
##############For X_test
low_cardinality_cols = [col for col in object_cols if X_test[col].nunique() < 10]
# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unkNown='ignore',sparse=False)
#Se não retirar os NAs a linha abaixo dá erro
X_test.dropna(axis = 0,inplace=True)
OH_cols_test = pd.DataFrame(OH_encoder.fit_transform(X_test[low_cardinality_cols]))
#print(OH_cols_test.shape,OH_cols_train.shape)
# One-hot encoding removed index; put it back
OH_cols_test.index = X_test.index
# Remove categorical columns (will replace with one-hot encoding)
num_X_test = X_test.drop(object_cols,axis=1)
# Add one-hot encoded columns to numerical features
OH_X_test = pd.concat([num_X_test,OH_cols_test],axis=1)
#print(OH_X_test.shape,OH_X_valid.shape)
# Define and fit model
model = RandomForestRegressor(n_estimators=100,criterion='mae',random_state=0)
model.fit(OH_X_train,y_train)
# Get validation predictions and MAE
preds_test = model.predict(OH_X_test)
# Save predictions in format used for competition scoring
output = pd.DataFrame({'Id': X_test.index,'SalePrice': preds_test})
output.to_csv('submission.csv',index=False)
当我尝试预处理数据集时，我得到了用于训练数据和测试数据的不同行。然后我无法拟合模型并进行预测。
我认为我应该只拆分测试数据集以完成所有任务，但是y的行比X_test多1条，然后就无法拆分。
所以我认为我必须使用训练数据集进行拆分，然后将其用于测试数据集的预测
解决方法
我相信您的问题正在此行中发生：
low_cardinality_cols = [col for col in object_cols if X_test[col].nunique() < 10]
您要为唯一列引用X_test。在Kaggle教程之后，您应该像这样参考X_train：
# Columns that will be one-hot encoded
low_cardinality_cols = [col for col in object_cols if X_train[col].nunique() < 10]
您似乎还会在以下这一行中犯同样的错误：
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_test[low_cardinality_cols]))
您已将其标记为一键编码的培训列，但是您使用了X_test而不是X_train。您正在混合训练和测试集处理，这不是一个好主意。这行应该是：
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[low_cardinality_cols]))
我建议您再次遍历教程中的代码块，并确保所有数据集和变量正确匹配，以便您处理正确的训练和测试数据。