Spark 应用程序与 spark-submit 一起工作，但在 Hue 上失败

问题描述

我有一个非常简单的 spark 应用程序：

object SimpleApp {

  def main(args: Array[String]) {

    val logger = LoggerFactory.getLogger(getClass.getName)
    val spark = SparkSession.builder.appName("Simple Application").getorCreate()
    val sc = spark.sparkContext
    
    val executorNumber = sc.getConf.get("spark.executor.instances").toInt

    val coreNumber = sc.getConf.get("spark.executor.cores").toInt

    logger.info("spark.executor.instances " + executorNumber)
    logger.info("spark.executor.cores " + coreNumber)
    spark.stop()

  }
}

我使用以下 sbt 文件构建此应用程序：

name := "Simple Project"
version := "1.0"
scalaVersion := "2.11.12"
val sparkVersion = "2.4.0"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,"org.apache.spark" %% "spark-sql" % sparkVersion
)

我可以使用 spark-submit 运行这个应用程序

spark-submit --num-executors 5 --executor-cores 7  --executor-memory 4G --class "SimpleApp" --master yarn --deploy-mode cluster simple-project_2.11-1.0.jar

但是在 Hue 上，使用相同的参数，它失败并出现错误

10:43:49.253 [Driver] ERROR org.apache.spark.deploy.yarn.ApplicationMaster - User class threw exception: java.lang.NoClassDefFoundError: com/codahale/metrics/json/MetricsModule
java.lang.NoClassDefFoundError: com/codahale/metrics/json/MetricsModule
        at java.lang.class.forName0(Native Method) ~[?:1.8.0_181]
        at java.lang.class.forName(Class.java:348) ~[?:1.8.0_181]
        at org.apache.spark.util.Utils$.classForName(Utils.scala:242) ~[spark-core_2.11-2.4.0-cdh6.3.2.jar:2.4.0-cdh6.3.2]
       ...
        at scala.Option.getorElse(Option.scala:121) ~[scala-library-2.11.12.jar:?]
        at org.apache.spark.sql.SparkSession$Builder.getorCreate(SparkSession.scala:935) ~[spark-sql_2.11-2.4.0-cdh6.3.2.jar:2.4.0-cdh6.3.2]
        at SimpleApp$.main(SimpleApp.scala:14) ~[simple-project_2.11-1.0.jar:?]
        at SimpleApp.main(SimpleApp.scala) ~[simple-project_2.11-1.0.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_181]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_181]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:673) [spark-yarn_2.11-2.4.0-cdh6.3.2.jar:2.4.0-cdh6.3.2]
Caused by: java.lang.classNotFoundException: com.codahale.metrics.json.MetricsModule
        at java.net.urlclassloader.findClass(urlclassloader.java:381) ~[?:1.8.0_181]
        at java.lang.classLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_181]
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_181]
        at java.lang.classLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_181]
        ... 25 more

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

apache-spark hue scala