问题描述
使用 pyspark,我创建了两个 VectorAssembler,第一个具有多个数字列('colA'、'colB'、'colC'),第二个具有多个分类列('colD'、'colE'、I在每一列上应用 OneHotEncoder)。
我可以单独创建这些 VectorAssembler。如何将输出组合成单个向量列(以便我可以将其输入 Xgboost 模型)?
我尝试了以下方法,但得到了“TypeError: can only concatenate str (not “list”) to str”
# my dataframe with all columns is df
# VectorAssembler 1: with 3 numeric columns
numeric_cols = ['colA','colB','colC']
assembler = VectorAssembler(
inputCols= numeric_cols,outputCol="numericFeatures"
)
# VectorAssembler 2: with 2 categorical columns
categ_cols = ['colD','colE']
indexers = [
StringIndexer(inputCol=c,outputCol="{0}_indexed".format(c))
for c in categ_cols
]
encoders = [
OneHotEncoder(
inputCol=indexer.getoutputCol(),outputCol="{0}_encoded".format(indexer.getoutputCol()))
for indexer in indexers
]
assemblerCateg = VectorAssembler(
inputCols = [encoder.getoutputCol() for encoder in encoders],outputCol = "categFeatures"
)
pipeline = Pipeline(stages = [assembler] + indexers + encoders + [assemblerCateg])
df2 = pipeline.fit(df).transform(df)
解决方法
解决了!只需在管道之前使用另一个 VectorAssembler(最后):
assemblerAll = VectorAssembler(inputCols= ["numericFeatures","categFeatures"],outputCol="allFeatures")
pipeline = Pipeline(stages = [assembler] + indexers + encoders + [assemblerCateg] + [assemblerAll])