scala spark rdd 错误：java.lang.ClassCastException：无法分配 java.lang.invoke.SerializedLambda 的实例

编程问答 2022-05-12

问题描述

我是 Scala 和 Spark 的初学者。

scala 版本：2.12.10

火花版本：3.0.1

我正在 Scala 中尝试一个非常简单的 spark rdd 函数。

但我收到一个错误。

(1) build.sbt


scalaVersion := "2.12.10"


name := "hello-world"
organization := "ch.epfl.scala"
version := "1.0"


libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators" % "1.1.2"
libraryDependencies +=        "org.apache.spark" %% "spark-sql" % "3.0.1" 
libraryDependencies +=        "org.apache.spark" %% "spark-core" % "3.0.1"

(2) Main.scala

import org.apache.spark.sql.SparkSession
object Main extends App {
  println("Hello,World!")

  implicit val spark = SparkSession.builder()
        .master("spark://centos-master:7077")
        // .master("local[*]")
        .appName("spark-api")
        .getorCreate()


  val inputrdd = spark.sparkContext.parallelize(Seq(("arth",10),("arth",20),("samuel",60),("jack",65)))
      println("inputrdd : ",inputrdd)
  val mapped = inputrdd.mapValues(x => (x,1))
  println("mapped : ",mapped)
  mapped.collect.foreach(println)
}

(3) 发生错误时

mapped.collect.foreach(println)部分好像出错了。

(4) 错误内容

21/04/17 20:54:19 INFO DAGScheduler: Job 0 Failed: collect at Main.scala:16,took 6.083947 s
[error] (run-main-0) org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 Failed 4 times,most recent failure: Lost task 0.3 in stage 0.0 (TID 7,192.168.0.220,executor 0):

java.lang.classCastException: cannot assign instance of java.lang.invoke.SerializedLambda 
to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in 
instance of org.apache.spark.rdd.MapPartitionsRDD

[error]         at java.io.ObjectStreamClass$FieldReflector.setobjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setobjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readobject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readobject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readobject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readobject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readobject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] 
[error] Driver stacktrace:
21/04/17 20:54:19 INFO TaskSetManager: Lost task 1.3 in stage 0.0 (TID 6) on 192.168.0.220,executor 0: java.lang.classCastException (cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD) [duplicate 7]
21/04/17 20:54:19 INFO TaskSchedulerImpl: Removed TaskSet 0.0,whose tasks have all completed,from pool 
[error] org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 Failed 4 times,executor 0): java.lang.classCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error]         at java.io.ObjectStreamClass$FieldReflector.setobjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setobjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readobject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readobject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readobject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readobject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readobject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] 
[error] Driver stacktrace:
[error]         at org.apache.spark.scheduler.DAGScheduler.failJobAndindependentStages(DAGScheduler.scala:2059)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2008)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2007)
[error]         at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
[error]         at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
[error]         at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
[error]         at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2007)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:973)
[error]         at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:973)
[error]         at scala.Option.foreach(Option.scala:407)
[error]         at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:973)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2239)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2188)
[error]         at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2177)
[error]         at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
[error]         at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:775)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2099)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2120)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2139)
[error]         at org.apache.spark.SparkContext.runJob(SparkContext.scala:2164)
[error]         at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1004)
[error]         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
[error]         at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
[error]         at org.apache.spark.rdd.RDD.withScope(RDD.scala:388)
[error]         at org.apache.spark.rdd.RDD.collect(RDD.scala:1003)
[error]         at Main$.delayedEndpoint$Main$1(Main.scala:16)
[error]         at Main$delayedInit$body.apply(Main.scala:2)
[error]         at scala.Function0.apply$mcV$sp(Function0.scala:39)
[error]         at scala.Function0.apply$mcV$sp$(Function0.scala:39)
[error]         at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
[error]         at scala.App.$anonfun$main$1$adapted(App.scala:80)
[error]         at scala.collection.immutable.List.foreach(List.scala:392)
[error]         at scala.App.main(App.scala:80)
[error]         at scala.App.main$(App.scala:78)
[error]         at Main$.main(Main.scala:2)
[error]         at Main.main(Main.scala)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error] Caused by: java.lang.classCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD
[error]         at java.io.ObjectStreamClass$FieldReflector.setobjFieldValues(ObjectStreamClass.java:2301)
[error]         at java.io.ObjectStreamClass.setobjFieldValues(ObjectStreamClass.java:1431)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2410)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readobject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2404)
[error]         at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2328)
[error]         at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2186)
[error]         at java.io.ObjectInputStream.readobject0(ObjectInputStream.java:1666)
[error]         at java.io.ObjectInputStream.readobject(ObjectInputStream.java:502)
[error]         at java.io.ObjectInputStream.readobject(ObjectInputStream.java:460)
[error]         at org.apache.spark.serializer.JavaDeserializationStream.readobject(JavaSerializer.scala:76)
[error]         at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
[error]         at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:83)
[error]         at org.apache.spark.scheduler.Task.run(Task.scala:127)
[error]         at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:446)
[error]         at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1377)
[error]         at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:449)
[error]         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[error]         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[error]         at java.lang.Thread.run(Thread.java:748)

如果我需要更多的库或者代码有误（但它在 spark-shell 中运行良好）......

如何解决？

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

apache-sparkrddscalascala