问题描述
当我在做交替最小二乘法来执行矩阵分解时，我在 Spark 中遇到 Java Server 的错误，我不知道为什么会发生这种情况，以下是错误消息。
Exception happened during processing of request from ('127.0.0.1',61711)
Traceback (most recent call last):
File "/Library/Developer/CommandLinetools/Library/Frameworks/python3.framework/Versions/3.8/lib/python3.8/socketserver.py",line 316,in _handle_request_noblock
self.process_request(request,client_address)
File "/Library/Developer/CommandLinetools/Library/Frameworks/python3.framework/Versions/3.8/lib/python3.8/socketserver.py",line 347,in process_request
self.finish_request(request,line 360,in finish_request
self.RequestHandlerClass(request,client_address,self)
File "/Library/Developer/CommandLinetools/Library/Frameworks/python3.framework/Versions/3.8/lib/python3.8/socketserver.py",line 720,in __init__
self.handle()
File "/usr/local/opt/apache-spark/libexec/python/pyspark/accumulators.py",line 262,in handle
poll(accum_updates)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/accumulators.py",line 235,in poll
if func():
File "/usr/local/opt/apache-spark/libexec/python/pyspark/accumulators.py",line 239,in accum_updates
num_updates = read_int(self.rfile)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/serializers.py",line 564,in read_int
raise EOFError
EOFError
----------------------------------------
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:61698)
Traceback (most recent call last):
File "/Users/zhengyangzhang/Library/Python/3.8/lib/python/site-packages/IPython/core/interactiveshell.py",line 3417,in run_code
exec(code_obj,self.user_global_ns,self.user_ns)
File "<ipython-input-3-deb1baad55f8>",line 15,in <module>
MSE = ratesAndPreds.map(lambda r: (r[1][0] - r[1][1])**2).mean()
File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py",line 1386,in mean
return self.stats().mean()
File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py",line 1245,in stats
return self.mapPartitions(lambda i: [StatCounter(i)]).reduce(redFunc)
File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py",line 997,in reduce
vals = self.mapPartitions(func).collect()
File "/usr/local/opt/apache-spark/libexec/python/pyspark/rdd.py",line 949,in collect
sock_info = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",line 1304,in __call__
return_value = get_return_value(
File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/protocol.py",line 326,in get_return_value
raise Py4JJavaError(
py4j.protocol.Py4JJavaError: <unprintable Py4JJavaError object>
During handling of the above exception,another exception occurred:
Traceback (most recent call last):
File "/Users/zhengyangzhang/Library/Python/3.8/lib/python/site-packages/IPython/core/interactiveshell.py",line 2044,in showtraceback
stb = value._render_traceback_()
AttributeError: 'Py4JJavaError' object has no attribute '_render_traceback_'
During handling of the above exception,another exception occurred:
Traceback (most recent call last):
File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",line 1207,in send_command
raise Py4JNetworkError("Answer from Java side is empty")
py4j.protocol.Py4JNetworkError: Answer from Java side is empty
During handling of the above exception,line 1033,in send_command
response = connection.send_command(command)
File "/usr/local/opt/apache-spark/libexec/python/lib/py4j-0.10.9-src.zip/py4j/java_gateway.py",line 1211,in send_command
raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while receiving
以下消息在我的 Jupyter 笔记本中重复了数百次。
During handling of the above exception,line 977,in _get_connection
connection = self.deque.pop()
IndexError: pop from an empty deque
During handling of the above exception,line 1115,in start
self.socket.connect((self.address,self.port))
ConnectionRefusedError: [Errno 61] Connection refused
我发布这个问题是因为我根本不明白为什么会发生这种情况，如果有人可以向我解释为什么会发生这种情况并为我提供解决此问题的可能方法，我将不胜感激。
以下是我的导入包和设置列表：
import findspark
findspark.init()
from pyspark import SparkContext
from pyspark.mllib.recommendation import ALS,MatrixFactorizationModel,rating
我在网上看到 findspark 包可能有助于解决问题，所以我 brew apache-spark 和 pip 这个包，但仍然有问题。
解决方法
暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！
如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@）