带有张量流的 Dask 因“CRITICAL - Failed to Serialize”错误而失败

问题描述

我已经安装了 dask[complete]、tensorflow、scikeras、deplayed、dask-ml。 我在本地运行相同的示例 link。工作日志中也没有堆栈跟踪。请帮助我提供进一步调试的输入。

代码失败，错误如下：

distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py",line 70,in dumps
    frames[0] = msgpack.dumps(msg,default=_encode_default,use_bin_type=True)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/msgpack/__init__.py",line 35,in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx",line 292,in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx",line 298,line 295,line 264,in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx",line 231,line 229,line 289,in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Delayed' object
distributed.comm.utils - ERROR - can not serialize 'Delayed' object
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py",line 32,in _to_frames
    msg,serializers=serializers,on_error=on_error,context=context
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/protocol/core.py",in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Delayed' object
distributed.batched - ERROR - Error in batched write
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/batched.py",line 94,in _background_send
    payload,serializers=self.serializers,on_error="raise"
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tornado/gen.py",line 762,in run
    value = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/tcp.py",line 243,in write
    **self.handshake_options,File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py",line 51,in to_frames
    return _to_frames()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/distributed/comm/utils.py",in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Delayed' object
---------------------------------------------------------------------------

解决方法

这似乎是 dask-ml 中的一个错误：https://github.com/dask/dask/issues/7571

更新到最新版本（目前未发布，您必须从 GitHub 安装）似乎可以解决问题：

$ pip install git+https://github.com/dask/dask-ml.git