将两组向量进行置换检验和余弦相似度进行比较

编程问答 2022-08-15

问题描述

我想比较两组之间的不同向量（每个对象1个）。我想做的工作与本文中完成的工作类似。

https://www.pnas.org/content/early/2020/08/14/2003181117 图2B。

因此，我已经为每个组建立了归一化向量的列表，例如：

X = array([[0.8081178,0.1618492,1.,0.,0.52503616],[0.9155495,0.9229482,0.55023754,1.        ],[0.5497678,0.5295068,0.9580641 ],[0.8554752,0.27967405,0.43231127],[0.8771384,0.15983552,0.24160399,0.        ],[1.,0.34030336,0.8518671,0.14370875],[0.96829957,0.89825296,0.9989327,[0.19713035,0.8313886,0.69545555],0.15145707,0.62412727,0.19574052],0.6768882,0.3267132,0.53155863],[0.,0.11568664,0.06043369,0.2405336 ],0.7901962,0.55479664,0.21075204],[0.8389194,0.9723087,0.9122212,0.74783736,0.27481842,0.54764044],[0.7932238,0.78063756,0.76313186],0.28478605,0.48485696,0.5902692 ]])

Y = array([[1.,0.8730191,0.72493815,0.9373017 ],0.8563728,0.71862656,0.74088454],[0.878855,0.8799178,0.8985272 ],[0.94998175,0.924029,0.74815565,0.4086177,0.3750266,0.87822354],[0.85906726,0.37570593,0.9324212 ],[0.8055762,0.85996395,0.9541106 ],[0.96801126,0.72156,0.8689768 ],0.9446373,0.5445604,0.56854314],[0.86714363,0.6032697,0.7075365 ],0.8875634,0.8770225,0.8542803 ],0.93619907,0.8262237,0.87035996],0.8533749,0.8739984,0.97969407],0.63581806,0.7951289,0.88310444],[0.82491845,0.6478972,0.8846024 ],0.79563105,0.55089736,0.90971696]])

我想对平均组向量的空间距离（余弦相似度）进行置换测试。这样做的目的是两个确定每个组（X，Y）的向量是否可以视为不同。 我已经知道如何计算空间距离 例如：

from scipy import spatial
Cosin = spatial.distance.cosine(np.mean(X,axis=0)

但是，他们在本文中所做的是 第一：将向量随机分为两组 第二：计算空间距离 第三：用余弦值检验余弦值是否与随机值不同

如果这是经过调整的置换测试，我不知道如何将其集成到sklearn.model_selection.permutation_test_score中？

我还发现了http://rasbt.github.io/mlxtend/user_guide/evaluate/permutation_test/，但是在其功能上，X和Y不能具有不同的形状...

我可能有一个基于以下解决方案：https://stats.stackexchange.com/questions/330540/how-to-interpret-very-low-similarity-score-of-two-vectors-but-having-significant 

import sys
import math,random
from scipy import stats


similarity = lambda x1,x2: sum(xj*xk for xj,xk in zip(x1,x2))/math.sqrt(sum(xj**2 for xj in x1)*sum(xk**2 for xk in x2))

x1 = np.mean(X,axis=0)
x2 = np.mean(Y,axis=0)

s = similarity(x1,x2)

## permutation test
sr = []
for j in list(range(1,10000)):
    concat_arrays = np.concatenate((X,Y),axis=0)
    np.random.shuffle(concat_arrays)
    #put the number of indiv mac or lemur or human
    split = np.split(concat_arrays,[len(x)])
    sr.append(similarity(np.mean(split[0],axis=0),np.mean(split[1],axis=0)))
shape,loc,scale = stats.weibull_min.fit(sr)
## -log10(p)
ej = ((s-loc)/scale)**shape*math.log10(math.exp(1.))
p = 10**(-ej)

您对此主张有何看法？ 对于“ len（x）”，我不知道我是否应该具有与两组原始数组相同的形状？

解决方法

Shaeffer等人提出的余弦相似度计算似乎是基于不同余弦相似度测量的自举。 从这个意义上讲，我认为两组是堆叠在一起的，然后分成两半。引导程序可以平滑所有“个体指纹”的随机划分。

我没有测试您的代码，但没有发现任何重大问题。

您的len(x)应该等于所有单个指纹堆叠大小的一半。如果很奇怪，则忽略这两个组中的一个或两个。

cosine-similaritypermutationscikit-learnscipyscipyvectorvector