scala – 包含Spark Streaming Kinesis ASL库时Spark作业SBT程序集合冲突

我开始制作一个火花流媒体工作,并获得了kinesis端点的制作人.在开始工作之后,我开始制作一个消费者,但是我遇到了构建它的问题.

我正在使用程序集插件创建一个包含所有依赖项的jar.项目的依赖关系如下.

libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.4.1" % "provided","org.apache.spark" %% "spark-sql" % "1.4.1" % "provided","org.apache.spark" %% "spark-streaming" % "1.4.1" % "provided","org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.4.1","org.scalatest" %% "scalatest" % "2.2.1" % "test","c3p0" % "c3p0" % "0.9.1.+","com.amazonaws" % "aws-java-sdk" % "1.10.4.1","MysqL" % "mysql-connector-java" % "5.1.33","com.amazonaws" % "amazon-kinesis-client" % "1.5.0"

)

当我运行程序集时,文件可以编译但在合并阶段失败时会出错

[error] (streamingClicks/*:assembly) deduplicate: different file contents found in the following:
[error] /Users/adam/.ivy2/cache/org.apache.spark/spark-network-common_2.10/jars/spark-network-common_2.10-1.4.1.jar:meta-inf/maven/com.google.guava/guava/pom.properties
[error] /Users/adam/.ivy2/cache/com.google.guava/guava/bundles/guava-18.0.jar:meta-inf/maven/com.google.guava/guava/pom.properties

这是在添加spark-streaming-kinesis-asl依赖项时引起的.我该如何解决这个问题？我可以将依赖项标记为已提供,但随后将jar添加到类路径中,但这确实不是我想要做的事情.

解决方法

我不知道你的情况下的准确解决方案,但你必须使用 merge strategy来解决这些依赖关系.它可能是这样的：

lazy val strategy = assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) { (old) => {
  case "application.conf" => MergeStrategy.concat
  case Meta(_) => MergeStrategy.discard
  case x => MergeStrategy.first
}
}
lazy val core = (project in file("core")).
  settings(
    name := "core",libraryDependencies ++= Seq(
     ...
    ),strategy
  )

附：我建议还在一个“fatjar”项目中汇编所有依赖项,并在依赖项目中提供代码.然后你可以把fatjar放在hdfs上并用sbt包打包你的实际代码.要运行代码,您应该使用–jar选项提供fatjar和包.