python 3.x中的多线程和子进程问题

编程问答 2022-07-25

问题描述

我想用不同的胎面运行循环以加快过程。而且，除了创建另一个脚本并用子进程调用它并为它提供大数组作为参数外，我没有其他方法可以使用。 我在代码中加上注释以解释问题。

我要通过子流程调用的脚本：

import multiprocessing
from joblib import Parallel,delayed
import sys

num_cores = multiprocessing.cpu_count()
inputs = sys.argv[1:]
print(inputs)


def evaluate_individual(ind):
    ind.evaluate()
    ind.already_evaluate = True


if __name__ == "__main__":
    # i am trying to execute a loop with multi thread.
    # but it need to be in the "__main__" but I can in my main script,so I create an external script...
    Parallel(n_jobs=num_cores)(delayed(evaluate_individual)(i) for i in inputs)

调用其他脚本的脚本：

import subprocess,sys
from clean import IndividualExamples

# the big array
inputs = []
for i in range(200):
    ind = IndividualExamples.simple_individual((None,None),None)
    inputs.append(ind)

# and Now I need to call this code from another script...
# but as arguments I pass a big array and I don't Now if there are a better method
# AND... this don't work to call the subprocess and return no error so I don't Now who to do..
subprocess.Popen(['py','C:/Users/alexa/OneDrive/Documents/Programmes/Neronal Network/multi_thread.py',str(inputs)])

感谢您的帮助，如果您现在又要在函数中（而不是在主函数中）使用多线程运行循环的方法也告诉我。 对不起，我的英语水平差不多。

edit：我尝试了一个池，但是同样的问题，我需要将其放在“ main ”中，以便可以将其放入脚本的函数中（因为我需要使用它）

使用池修改的代码：

if __name__ == "__main__":
    tps1 = time.time()
    with multiprocessing.Pool(12) as p:
        outputs = p.map(evaluate_individual,inputs)

    tps2 = time.time()
    print(tps2 - tps1)

    New_array = outputs

其他小问题，我尝试使用一个简单的循环并使用Pool多进程，并比较两个进程的时间：

simple loop: 0.022951126098632812
multi thread: 0.9151067733764648

我去...为什么12个核上的多进程可以比简单循环更长？

解决方法

您可以使用multiprocessing.Pool 

import subprocess,sys
from multiprocessing import Pool
from clean import IndividualExamples

def evaluate_individual(ind):
    ind.evaluate()

# the big array
pool = Pool()
for i in range(200):
    ind = IndividualExamples.simple_individual((None,None),None)
    pool.apply_async(func=evaluate_individual,args=(ind,))

pool.close()
pool.join()
,

谢谢，我只是尝试，但这会引发运行时错误：

raise RuntimeError('''
RuntimeError: 
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
multithreadingpythonpython-3.xsubprocess