不平衡的多类分类管道

问题描述

在3类分类问题中，我有一个矩阵X（特征，昏暗的320x7）和一个数组y（目标，昏暗的320x1）。我为5种不同的模型实现了gridsearchcv，以检查哪种模型最好，并调整其超参数。为了尽可能严格，我创建了一个管道来对每个折叠执行数据预处理。管道包括缩放，重采样和分类器。

由于目标向量不平衡（请参见图片），因此我使用SMOTE来平衡这三个类。现在的问题是，尽管我认为整个过程的顺序是正确的，但我的训练准确度非常高（0.9-0.99），但测试准确度却很低（0.5-0.55），我无法确定这是由于过拟合造成的错误的超参数空间搜索，或者是由于我的数据质量。

如果有人能照亮它，我将不胜感激。谢谢。

这是代码

# Import modules

import pickle
import numpy as np
from imblearn.over_sampling import SMOTE
from sklearn.model_selection import  gridsearchcv
from sklearn.preprocessing import MinMaxScaler 
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier,BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from imblearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from sklearn.svm import SVC

# Load features and target
with open('data.pkl','rb') as f: 
    dataset = pickle.load(f) 
X = dataset['X']
y = dataset['y']

#%% Grid-search cross validation for different models

# Model search space
models = {
            'LogisticRegression'     : LogisticRegression(),'SVM'                    : SVC(),'RandomForestClassifier' : RandomForestClassifier(),'BaggingClassifier'      : BaggingClassifier(),'KNN'                    : KNeighborsClassifier()}

# Hyperparameter search space
hyper = {
            'LogisticRegression':{
                                        'clf__penalty'     : ['l2'],'clf__C'           : np.logspace(-4,4,10),'clf__solver'      : ['lbfgs','newton-cg','saga'],'clf__max_iter'    : [1000,1500,2000],'clf__random_state': [0]},'SVM':{
                                        'clf__C'           : [0.001,0.01,0.1,1,10,100,1000,10000],'clf__gamma'       : [0.01,0.001,0.0001],'clf__kernel'      : ['rbf','linear'],'RandomForestClassifier':{
                                        'clf__max_depth': [2,3,5,6],'clf__max_features': [2,'auto','sqrt'],'clf__n_estimators': [10,500,1000],'KNN':{
                                        'clf__n_neighbors': [3,8,16]},'BaggingClassifier':{
                                        'clf__base_estimator': [DecisionTreeClassifier()],'clf__max_samples'   : [0.05,0.2,0.5],'clf__n_estimators'  : [10,50,500],'clf__max_features'  : [0.1,0.3,0.5]}
        }

# Create variable to store CV results
grid_cv_results = dict()

# Scaler 
escaler = MinMaxScaler()

# resampling
resampler = SMOTE(sampling_strategy = 'all',random_state = 0)

# Perform cross-validation for each model
for model_name in models.keys():
    
    # Get model and hyperparameters
    clf    = models[model_name]
    params = hyper[model_name]
    
    # Pipeline (standarization + classifier)
    pipe = Pipeline([('scale',escaler),('sampling',resampler),('clf',clf)])
    
    # Gridsearch cross-validation
    grid = gridsearchcv(estimator = pipe,param_grid = params,cv = 5,return_train_score = True)
    grid.fit(X,y)
    
    # Gridsearch cross-validation results
    best_param                  = grid.best_params_
    best_param_test_score_mean  = grid.cv_results_['mean_test_score'][grid.best_index_]
    best_param_test_score_std   = grid.cv_results_['std_test_score'][grid.best_index_]
    best_param_train_score_mean = grid.cv_results_['mean_train_score'][grid.best_index_]
    best_param_train_score_std  = grid.cv_results_['std_train_score'][grid.best_index_]
    
    # Store gridsearch cross-validation results
    grid_cv_results[model_name] = {
                                      'best_param'                  : best_param,'best_param_test_score_mean'  : best_param_test_score_mean,'best_param_test_score_std'   : best_param_test_score_std,'best_param_train_score_mean' : best_param_train_score_mean,'best_param_train_score_std'  : best_param_train_score_std,

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）