使用部分学习 Python 并发

问题描述

您好，我正在尝试使用 ThreadPoolExecutor 学习 Python 并发性，并需要一些帮助来理解如何将相同的数据传递给使用 Partial 和 Map 摄取相同数据的多个函数。我尝试创建一个快速示例，如下所示。我知道这可以只使用 1 个函数而不是 2 个函数来完成，但我想从概念上理解它，所以我试着想出一个简单的例子。

document.addEventListener('deviceready',onDeviceReady,false);
var nextId = 1; // initialize the starting id

function onDeviceReady() {
    // Cordova is Now initialized. Have fun!

    console.log('Running cordova-' + cordova.platformId + '@' + cordova.version);

    
}

document.addEventListener("click",add_exercise);

function add_exercise(){

    $("#add_exercise").click(function(){
        nextId++;
        var content = "<div data-role=\"collapsible\" id=\"set"+ nextId +"\" data-collapsed=\"true\">"+
        "<h3>Exercise "+ nextId +"</h3>"+
        "<p>Content</p></div>";

        $("#set").append(content).collapsibleset("refresh");
    });

}

解决方法

首先：您可以在 ThreadPoolExecutor 之前创建 open() 并且您可以（重新）使用它多次而无需（重新）一次又一次地创建它。

---

map 宁可使用不同的数据运行同一个函数，但您尝试使用相同的数据运行不同的函数。

我宁愿创建一个同时运行 count_chars 和 count_words 的函数，并将此函数与 map() 和不同的行一起使用。

def func(line):
    a = count_chars(line)
    b = count_words(line)
    return a,b

results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    with open(input_file_name) as f:    
        for piece in read_in_chunks(f,10):
            #print(piece)
            results += list(executor.map(func,list(piece)))

print(results)

它给出列表或对

[(count_chars,count_words),(count_chars,...] 

所以很容易得到一对。

最终我可以使用 zip() 创建包含所有 count_chars 的分隔列表和包含所有 count_words 的分隔列表

---

最少的工作代码。

import string
import concurrent.futures
from itertools import islice
#from functools import partial


## Function to count characters in a string
def count_chars(line):
    return len(line)


## Function to count words in a string
def count_words(line):
    return sum(word.strip(string.punctuation).isalpha() for word in line.split())


## Divide a very large file in chunks for reading
def read_in_chunks(file_object,lines_per_chunk):
    while True:
        lines = list(islice(file_object,lines_per_chunk))
        if lines:
            yield lines
        else:
            break


input_file_name = 'pool-multifunctions.py'

# --- version 1 ----

def func(line):
    a = count_chars(line)
    b = count_words(line)
    return a,list(piece)))

print('--- results ---')
print(results)

all_count_chars,all_count_words = zip(*results)

print('--- all_count_chars ---')
print(all_count_chars)
print('--- all_count_words ---')
print(all_count_words)

---

其他版本：

我创建对

 all_pairs = [(count_chars,line),(count_words,line)]

并使用

运行它们

 lambda x:x[0](x[1])

其中 x[0] 将是函数的名称，而 x[1] 将是 line

而且我不需要 partial。

它给出了平面列表

[count_chars,count_words,count_chars,...] 

所以单双不是那么容易的。

要创建包含所有 count_chars 的分隔列表和包含所有 count_words 的分隔列表，它需要 results[0::2] 和 results[1::2]

import string
import concurrent.futures
from itertools import islice
#from functools import partial


## Function to count characters in a string
def count_chars(line):
    return len(line)


## Function to count words in a string
def count_words(line):
    return sum(word.strip(string.punctuation).isalpha() for word in line.split())


## Divide a very large file in chunks for reading
def read_in_chunks(file_object,lines_per_chunk))
        if lines:
            yield lines
        else:
            break


input_file_name = 'pool-multifunctions.py'

# --- version 2 ---

results = []

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    with open(input_file_name) as f:    
        for piece in read_in_chunks(f,10):
            for line in piece:
                all_pairs = [(count_chars,line)]
                results += list(executor.map(lambda x:x[0](x[1]),all_pairs))

print('--- results ---')
print(results)

all_count_chars = results[0::2]
all_count_words = results[1::2]

print('--- all_count_chars ---')
print(all_count_chars)
print('--- all_count_words ---')
print(all_count_words)

---

编辑：

我发现它可以用作

results += list(executor.map(lambda func,data:func(data),[count_chars,count_words],[line,line]))

或

all_funcs = [count_chars,count_words]
all_data = [line] * len(all_funcs)

results += list(executor.map(lambda func,all_funcs,all_data))

---

import string
import concurrent.futures
from itertools import islice
#from functools import partial


## Function to count characters in a string
def count_chars(line):
    return len(line)


## Function to count words in a string
def count_words(line):
    return sum(word.strip(string.punctuation).isalpha() for word in line.split())


## Divide a very large file in chunks for reading
def read_in_chunks(file_object,lines_per_chunk))
        if lines:
            yield lines
        else:
            break


input_file_name = 'temp-pool-multifunctions.py'

# --- version 3 ---

results = []
all_funcs = (count_chars,count_words)

with concurrent.futures.ThreadPoolExecutor(max_workers=3) as executor:
    with open(input_file_name) as f:    
        for piece in read_in_chunks(f,10):
            for line in piece:
                #results += list(executor.map(lambda func,line]))
                all_data = [line] * len(all_funcs)
                results += list(executor.map(lambda func,all_data))

print('--- results ---')
print(results)

all_count_chars = results[0::2]
all_count_words = results[1::2]

print('--- all_count_chars ---')
print(all_count_chars)
print('--- all_count_words ---')
print(all_count_words)