使用joblib.Parallel并行化相互信息的计算

问题描述

我正在尝试计算两个数组之间的经验互信息。因此，我正在使用normalized_mutual_info_score（sklearn doc）中的函数sklearn。我使用numba尝试了各种方法来加速cuda的计算，但是这对我不起作用... 因此，我选择尝试joblib。通过以下代码，我试图并行化相互信息的计算：

from sklearn.metrics.cluster import normalized_mutual_info_score

def processInput(i):
    mi = []
    for j in range(i + 1):
        mi.append(normalized_mutual_info_score(test_matrix[:,i],test_matrix[:,j]))
    return mi

def get_mutual_@R_174_4045@ion_sklearn(data):
    mutual_inf = np.zeros((data.shape[1],data.shape[1]))
    
    for i in numba.prange(data.shape[1]):
        print('{}/{}'.format(i,data.shape[1]),end = '\r')
        for j in numba.prange(i + 1):
            mutual_inf[j,i] += normalized_mutual_info_score(data[:,data[:,j])
            
    i_lower = np.tril_indices(data.shape[1],-1)
    mutual_inf[i_lower] = mutual_inf.T[i_lower]
    return mutual_inf

test_matrix = np.random.rand(100,500)

num_cores = multiprocessing.cpu_count()

Now = time.time()
a = get_mutual_@R_174_4045@ion_sklearn(test_matrix)
print("non parallel execution took : ",time.time() - Now,"seconds")

Now = time.time()
results = Parallel(n_jobs=1)(delayed(processInput)(i) for i in range(test_matrix.shape[1]))
print("non parallel execution with 1 n_jobs took : ","seconds")

Now = time.time()
results = Parallel(n_jobs=2)(delayed(processInput)(p) for p in range(test_matrix.shape[1]))
print("parallel execution with 2 n_jobs took : ","seconds")

输出如下：

non parallel execution took :  148.19334053993225 seconds
non parallel execution with 1 n_jobs took :  146.42843222618103 seconds
parallel execution with 2 n_jobs took :  989.2411859035492 seconds

我不确定我是否做得正确。但是我很高兴能得到一些输入，以改善我的计算的并行性。变量num_cores = 14。非常感谢！

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

joblib parallel-processing