Spark EMR 作业杰克逊错误 - java.lang.NoSuchMethodError - AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType

问题描述

我知道我们已经在这里回答了类似的问题。但出于某种原因，没有一个选项对我有用。

下面是我得到的错误：

User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType;
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._fullSerializationType(JaxbAnnotationIntrospector.java:1550)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._findContentAdapter(JaxbAnnotationIntrospector.java:1502)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findSerializationContentConverter(JaxbAnnotationIntrospector.java:914)
at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findSerializationContentConverter(AnnotationIntrospectorPair.java:388)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.findConvertingContentSerializer(StdSerializer.java:266)
at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.createContextual(AsArraySerializerBase.java:197)

以下是我添加的依赖项。我尝试删除所有 jackson 依赖项并单独添加所需版本。

dependencies {
compile (group: 'org.scala-lang',name: 'scala-library',version: '2.11.8') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'org.apache.spark',name: 'spark-core_2.11',version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}
compile (group: 'org.apache.spark',name: 'spark-sql_2.11',version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.amazonaws',name: 'aws-java-sdk',version: '1.11.633') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile group: 'org.apache.httpcomponents',name: 'httpclient',version: '4.5.2'
compile group: 'com.sun.mail',name: 'javax.mail',version: '1.6.2'

testCompile group: 'junit',name: 'junit',version: '4.12'

compile group: 'net.liftweb',name: 'lift-json_2.11',version: '2.6.3'
compile group: 'net.sf.json-lib',name: 'json-lib',version: '2.4',classifier: 'jdk15'
compile group: 'net.sf.oval',name: 'oval',version: '1.31'

compile (group: 'com.bedatadriven',name: 'jackson-datatype-jts',version: '2.4') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.fasterxml.jackson.datatype',name: 'jackson-datatype-joda',version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core',name: 'jackson-core',version: '2.11.2'
compile (group: 'com.fasterxml.jackson.core',name: 'jackson-databind',name: 'jackson-annotations',version: '2.11.2'

compile (group: 'com.fasterxml.jackson.module',name: 'jackson-module-jaxb-annotations',version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile (group: 'com.fasterxml.jackson.dataformat',name: 'jackson-dataformat-csv',version: '2.11.2') {
     exclude group: 'com.fasterxml.jackson.core'
 }

// Some custom Jars

compile (group: 'org.springframework',name: 'spring-core',version: '2.5.6') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

}

当我在本地系统中运行 spark 作业但不在 aws emr 集群中时，此配置有效。

解决方法

Spark 提供 Jackson 本身，而不是您期望的版本。该错误可能是由于类路径中有 2 个相互冲突的 Jackson 版本造成的。

您有两个选择：

• 强制使用具有某些属性的特定 Jackson 版本（例如参见 Classpath resolution between spark uber jar and spark-submit --jars when similar classes exist in both）
• 更新您的构建定义以使用 Spark 提供的相同 Jackson 版本

Spark 2.3.0 附带 Jackson 2.6.7.1（可以在此处查看，例如：https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11/2.3.0）。