问题描述
我正在使用 pytorch,而 pytorch 又依赖于 python 本地 pickle 模块进行多处理。
我有一个不可pickle的类,因为它包含一些lambda(我不想改变它)。 由于我无法更改 pytorch 的多处理实现(以支持 dill),因此我希望我的类是 python 原生的pickleable。 我想知道我是否可以更改我的类,以便从外部可以腌制,但在内部仅指 dill?
class NonPickleableClass:
#...
def __getstate__(self):
# HACK: make it pickleable with dill :D
return dill.dumps(self.__dict__)
def __setstate__(self,state):
self.__dict__.update(dill.loads(state))
虽然它在我的单元测试中运行良好,但它在 pytorch 中超出递归限制而崩溃(不知道确切原因?)。
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 955,in _batch_appends
save(x)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 692,in save_reduce
save(args)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 691,in save_reduce
save(func)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 560,in save
f(self,obj) # Call unbound method with explicit self
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 536,in save
self.framer.commit_frame()
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/site-packages/dill/_dill.py",line 1444,in save_function
pickler.save_reduce(_create_function,(obj.__code__,File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 901,in save_tuple
save(element)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 220,in commit_frame
if f.tell() >= self._FRAME_SIZE_TARGET or force:
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 941,in save_module_dict
StockPickler.save_dict(pickler,obj)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 971,in save_dict
self._batch_setitems(obj.items())
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 1002,in _batch_setitems
save(v)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",line 931,in save_list
self._batch_appends(obj)
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",obj) # Call unbound method with explicit self
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/site-packages/dill/_dill.py",RecursionError: maximum recursion depth exceeded while calling a Python object
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",in save
self.framer.commit_frame()
File "/home/AD/muellelu/xconda3/envs/hand_seg/lib/python3.9/pickle.py",in commit_frame
if f.tell() >= self._FRAME_SIZE_TARGET or force:
RecursionError: maximum recursion depth exceeded while calling a Python object
RecursionError: maximum recursion depth exceeded while calling a Python object
解决方法
暂无找到可以解决该程序问题的有效方法,小编努力寻找整理中!
如果你已经找到好的解决方法,欢迎将解决方案带上本链接一起发送给小编。
小编邮箱:dio#foxmail.com (将#修改为@)