问题描述
我在 Linux (Ubuntu) 环境中创建了一个 Spark 分发集群来运行 Spark 提交作业。我有 3 台机器（m1、s1 和 s2；m1 是主机，s1 和 s2 是从机）。
在主 Spark 和从属 Spark 目录 ../spark/conf/ spark-defaults.conf 中，spark-env.sh。我已经配置好了一切。
spark-env.sh
export SPARK_MASTER_HOST=198.77.8.7
export SPARK_WORKER_HOST=198.77.8.12
export SPARK_WORKER_HOST=198.77.8.18
export SPARK_LOCAL_IP=198.77.8.7
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export SPARK_WORKER_CORES=8
spark-defaults.conf
spark.master yarn
spark.executor.memory 1g
spark.driver.memory 1g
spark.yarn.am.memory 1g
工作命令是：
bin/spark-submit --master local[*] --deploy-mode client --class com.xyApp --executor-memory 15G --executor-cores 2 --driver-memory 15G /home/user /workspace/test.jar
我正在获取输出文件，但是当我在集群模式下运行时，它给了我一个异常。命令是：
bin/spark-submit --master spark://198.47.12.8:7077 --deploy-mode cluster --class com.xyApp --executor-memory 15G --executor-cores 2 --driver-memory 15G /home/user/workspace/test.jar
异常：
21/01/08 16:51:51 INFO SecurityManager: Changing modify acls groups to:
21/01/08 16:51:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sys92); groups with view permissions: Set(); users with modify permissions: Set(sys92); groups with modify permissions: Set()
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
21/01/08 16:51:51 WARN Utils: Service 'Driver' Could not bind on a random free port. You may check whether configuring an appropriate binding address.
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' Failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'Driver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:461)
at sun.nio.ch.Net.bind(Net.java:453)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
at io.netty.channel.socket.nio.NioServerSocketChannel.dobind(NioServerSocketChannel.java:134)
at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
我检查了所有 3 个系统的 /etc/hosts 文件（我认为这不是问题）。
Master-etc/hosts
#127.0.0.1 localhost
#127.0.1.1 mater1
198.77.8.7 mater1
198.77.8.12 slave1
198.77.8.18 slave2
Salve1-etc/hosts
#127.0.0.1 localhost
#127.0.1.1 slave1
198.77.8.7 mater1
198.77.8.12 slave1
198.77.8.18 slave2
Slave2-etc/hosts
#127.0.0.1 localhost
#127.0.1.1 slave2
198.77.8.7 mater1
198.77.8.12 slave1
198.77.8.18 slave2
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）