spark-submit如何在群集模式下传递--driver-class-path？

问题描述

好吧，如果使用pyspark shell和driver-class-path，我可以使用docker image访问蜂巢资源：

$ pyspark --driver-class-path /etc/spark2/conf:/etc/hive/conf
Python 3.7.4 (default,Aug 13 2019,20:35:49)
Using Python version 3.7.4 (default,Aug 13 2019 20:35:49)
SparkSession available as 'spark'.
>>> from pyspark.sql import SparkSession
>>>
>>> #declaration
... appName = "test_hive_minimal"
>>> master = "yarn"
>>>
... sc = SparkSession.builder \
...     .appName(appName) \
...     .master(master) \
...     .enableHiveSupport() \
...     .config("spark.hadoop.hive.enforce.bucketing","True") \
...     .config("spark.hadoop.hive.support.quoted.identifiers","none") \
...     .config("hive.exec.dynamic.partition","True") \
...     .config("hive.exec.dynamic.partition.mode","nonstrict") \
...     .getOrCreate()
>>> sql = "show tables in user_tables"
>>> df_new = sc.sql(sql)
20/08/20 15:08:50 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>>> df_new.show()
+-----------+--------------------+-----------+
|   database|           tableName|isTemporary|
+-----------+--------------------+-----------+
|user_tables|              dummyt|      false|
|user_tables|abcdefg...dummytable|      false|

但如果通过如下所示的spark-submit使用相同的脚本，则会遇到以下错误：

spark-submit --master local --deploy-mode cluster --name test_hive --executor-memory 2g --num-executors 1 -- test_hive_minimal.py --verbose

Traceback (most recent call last):
  File "<stdin>",line 1,in <module>
  File "/opt/conda/lib/python3.7/site-packages/pyspark/sql/session.py",line 767,in sql
    return DataFrame(self._jsparkSession.sql(sqlQuery),self._wrapped)
  File "/opt/conda/lib/python3.7/site-packages/pyspark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py",line 1257,in __call__
  File "/opt/conda/lib/python3.7/site-packages/pyspark/sql/utils.py",line 71,in deco
    raise AnalysisException(s.split(': ',1)[1],stackTrace)
pyspark.sql.utils.AnalysisException: "Database 'user_tables' not found;"

test_hive_minimal.py是检查蜂巢数据库的简单脚本：

from pyspark.sql import SparkSession

appName = "test_hive_minimal"
master = "yarn"
# Creating Spark session
sc = SparkSession.builder \
    .appName(appName) \
    .master(master) \
    .enableHiveSupport() \
    .config("spark.hadoop.hive.enforce.bucketing","True") \
    .config("spark.hadoop.hive.support.quoted.identifiers","none") \
    .config("hive.exec.dynamic.partition","True") \
    .config("hive.exec.dynamic.partition.mode","nonstrict") \
    .getOrCreate()

sql = "show tables in user_tables"
df_new = sc.sql(sql)
df_new.show()
sc.stop()

我尝试了几种方法，将hive.metastore.uris，spark.sql.warehouse.dir以及xml文件作为--files进行传递。我的执行者以某种方式无法访问它的配置。有人可以帮忙吗？

更新： 我成功地将hive-site.xml作为--files传递给集群模式下的火花提交，并且日志显示其不再为metastore创建本地derby.db。但是，现在面临着另一个问题，如下所示：

20/08/21 09:59:29 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
20/08/21 09:59:31 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
20/08/21 09:59:31 INFO hive.metastore: Trying to connect to metastore with URI thrift://cluster01.cdh.com:9083
20/08/21 09:59:32 ERROR transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
        at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
        at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
        at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
        at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)

似乎是kerberos的问题，但是我已经有有效的kerberos令牌，并且能够通过终端/通过docker的spark-shell访问hdfs。在这里需要做什么？在群集上提交时，不是由yarn自动配置的吗？

解决方法

我认为您应该在spark-submit命令中传递keytab，此代码通过SSH运行？

,

更新： 在docker容器上共享vol mount并传递keytab / principal以及hive-site.xml来访问metastore之后，该问题已解决。

spark-submit --master yarn \
--deploy-mode cluster \
--jars /srv/python/ext_jars/terajdbc4.jar \
--files=/etc/hive/conf/hive-site.xml \
--keytab /home/alias/.kt/alias.keytab \ #this is mounted and kept in docker local path 
--principal [e-mail] \
--name td_to_hive_test \
--driver-cores 2 \
--driver-memory 2G \
--num-executors 44 \
--executor-cores 5 \
--executor-memory 12g \
td_to_hive_test.py