如何将现有对象放入共享内存？

编程问答 2022-08-16

问题描述

我设法创建了一个可以与BaseManager和NamespaceProxy共享内存中存在的对象，但是我看到的所有示例都要求我使用代理来创建对象。例如：

CREATE TABLE IF NOT EXISTS `[project].[dataset].[dest table]` (
  # Add your schema here
  tdate date,serid numeric
)
  PARTITION BY tdate
  CLUSTER BY serid;
INSERT INTO `[project].[dataset].[dest table]` SELECT * FROM `[project].[dataset].[table]`;

打印出来的

class Foo:
    def __init__(self):
        self._a = 1000
    def get_a(self):
        return self._a

class SharedFoo(NamespaceProxy):
    _exposed_ = ('__getattribute__','__getattr__','__setattr__','__init__','get_a')
    def get_a(self):
        callmethod = object.__getattribute__(self,'_callmethod')
        return callmethod('get_a',())

class FooManager(BaseManager):
    pass

def test():
    FooManager.register('Foo',Foo,SharedFoo)
    with FooManager() as manager:
        ls = []
        t = time.time()
        for i in range(100):
            ls.append(manager.Foo())
        print(time.time() - t)

由于我可能会创建数百万个0.44 (and some other numbers that i ommitted) 对象，所以这对于任务来说太慢了。我试图像这样使它更快：

Foo

这给了我这个错误：

def do_stuff(obj):
    obj.ls[4].set_a(300)
def test():
    FooManager.register('Foo',SharedFoo)
    with FooManager() as manager:
        if manager._Server != None:
            manager._Server.mutex = NoLock()
        ls = []
        t = time.time()
        for i in range(100000):
            ls.append(Foo())
        foos = manager.Foo()
        foos.ls = mp.Manager().list(ls)
        print(time.time() - t)
    processes = [Process(target=do_stuff,args = (foos,)) for _ in range(3)]
    for process in processes:
        process.start()

    for process in processes:
        process.join()
    print(foos.ls[4].get_a())

我想做的是可能的吗？如果是这样，我应该使用什么（不是寻找一个完整的解决方案，只是一些资源来做这件事）？如果相关，我正在Linux上使用Python 3.7

谢谢

编辑：这对mmap可行吗，还是我朝着完全错误的方向前进？看起来很有希望，但是文档似乎说它更多地用于文件（再次寻找完整的解决方案，即使Traceback (most recent call last): File "/path/to/lib/python3.7/multiprocessing/managers.py",line 788,in _callmethod conn = self._tls.connection AttributeError: 'ForkAwareLocal' object has no attribute 'connection' During handling of the above exception,another exception occurred: Traceback (most recent call last): File "/path/to/lib/python3.7/multiprocessing/process.py",line 297,in _bootstrap self.run() File "/path/to/lib/python3.7/multiprocessing/process.py",line 99,in run self._target(*self._args,**self._kwargs) File "/path/to/myproject/test.py",line 121,in do_stuff obj.ls[4].set_a(300) File "/path/to/lib/python3.7/multiprocessing/managers.py",line 1099,in __getattr__ return callmethod('__getattribute__',(key,)) File "/path/to/lib/python3.7/multiprocessing/managers.py",line 792,in _callmethod self._connect() File "/path/to/lib/python3.7/multiprocessing/managers.py",line 779,in _connect conn = self._Client(self._token.address,authkey=self._authkey) File "/path/to/lib/python3.7/multiprocessing/connection.py",line 492,in Client c = SocketClient(address) File "/path/to/lib/python3.7/multiprocessing/connection.py",line 619,in SocketClient s.connect(address) FileNotFoundError: [Errno 2] No such file or directory 可以与自定义对象一起使用）

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

multiprocessingpythonpython-multiprocessingshared-memory