如何从SuperLearner模型确定置换变量重要性？

问题描述

我的总体目标是像波士顿数据集那样，从超级学习者中确定变量的重要性。但是，当我尝试使用R中的VIP包确定变量的重要性时，出现以下错误。我怀疑是包含SuperLeaner对象的预测包装是导致错误的原因，但我不确定。

# Call:  
# SuperLearner(Y = y_train,X = x_train,family = binomial(),SL.library =  # c("SL.mean",#    "SL.glmnet","SL.ranger"),method = "method.AUC") 


#                    Risk      Coef
# SL.mean_All   0.55622189 0.3333333
# SL.glmnet_All 0.06240630 0.3333333
# SL.ranger_All 0.02745502 0.3333333
# Error in mean(actual == predicted,na.rm = FALSE): (list) object cannot be # coerced to type 'double'
# Traceback:

# 1. vi_permute(object = sl,method = "permute",feature_names = colnames,#  .     train = x_train,target = y_holdout,metric = "accuracy",#  .     type = "difference",nsim = 1,pred_wrapper = pred_wrapper)
# 2. vi_permute.default(object = sl,feature_names =    
#       colnames,pred_wrapper = pred_wrapper)
# 3. mfun(actual = train_y,predicted = pred_wrapper(object,newdata =  
#     train_x))
# 4. mean(actual == predicted,na.rm = FALSE)

我执行了以下操作：

library(MASS)
data(Boston,package = "MASS")

# Extract our outcome variable from the dataframe.
outcome = Boston$medv

# Create a dataframe to contain our explanatory variables.
data = subset(Boston,select = -medv)

set.seed(1)
# Reduce to a dataset of 150 observations to speed up model fitting.
train_obs = sample(nrow(data),150)

# X is our training sample.
x_train = data[train_obs,]

# Create a holdout set for evaluating model performance.
x_holdout = data[-train_obs,]

# Create a binary outcome variable: towns in which median home value is > 22,000.
outcome_bin = as.numeric(outcome > 22)

y_train = outcome_bin[train_obs]
y_holdout = outcome_bin[-train_obs]

library(SuperLearner)
set.seed(1)
sl = SuperLearner(Y = y_train,SL.library = c("SL.mean","SL.glmnet",method = "method.AUC")
sl

colnames <- colnames(x_train)
pred_wrapper <- function(sl,newdata) {
  predict(sl,x = as.matrix(y_holdout)) %>%
    as.vector()
}

# Plot VI scores
library(vip)
p1 <- vi_permute(object = sl,train = x_train,type = "difference",pred_wrapper = pred_wrapper) 

解决方法

对于SuperLearner对象，您可以看到它返回了概率列表

predict(sl,x_train[1:2,])
$pred
          [,1]
[1,] 0.4049966
[2,] 0.1905551

$library.predict
     SL.mean_All SL.glmnet_All SL.ranger_All
[1,]   0.3866667     0.5718232        0.2565
[2,]   0.3866667     0.1082986        0.0767

如果您阅读了引言（?predict.SuperLearner），我想您想从超级学习者那里获得预测。因此，更改函数以提取概率并将其转换为标签：

pred_wrapper <- function(sl,newdata) {
  ifelse(predict(sl,newdata)$pred>0.5,1,0)
}

我们简要检查结果：

table(pred_wrapper(sl,x_holdout),y_holdout)
   y_holdout
      0   1
  0 183  39
  1   9 125

并使用x_holdout作为火车：

p1 <- vi_permute(object = sl,method = "permute",feature_names = colnames,train = x_holdout,target = y_holdout,metric = "accuracy",type = "difference",nsim = 5,pred_wrapper = pred_wrapper) 

# A tibble: 13 x 3
   Variable Importance   StDev
   <chr>         <dbl>   <dbl>
 1 crim       0.00337  0.00126
 2 zn        -0.000562 0.00235
 3 indus      0.00337  0.00235
 4 chas       0.00674  0.00377
 5 nox        0.00225  0.00235
 6 rm         0.0315   0.0165 
 7 age        0.0213   0.0108 
 8 dis        0.0129   0.00944
 9 rad       -0.00169  0.00377
10 tax        0.00506  0.00126
11 ptratio    0.0174   0.0145 
12 black     -0.00281  0      
13 lstat      0.241    0.0204