分组具有相同词但配置不正确的Python的配置文件字符串

问题描述

我有一个数据框，其中包含一列配置文件类型，如下所示：

0                                    Android Java
1                  Software Development Developer
2                            Full-stack Developer
3                      JavaScript Frontend Design
4                          Android iOS JavaScript
5                             Ruby JavaScript PHP

我已经使用NLP模糊匹配相似的配置文件，并返回了以下相似性数据帧：

left_side                       right_side                  similarity
7   JavaScript Frontend Design  Design JavaScript Frontend  0.849943
8   JavaScript Frontend Design  Frontend Design JavaScript  0.814599
9   JavaScript Frontend Design  JavaScript Frontend         0.808010
10  JavaScript Frontend Design  Frontend JavaScript Design  0.802881
12  Android iOS JavaScript      Android iOS Java            0.925126
15  Machine Learning Engineer   Machine Learning Developer  0.839165
21  Android Developer Developer Android Developer           0.872646
25  Design Marketing Testing    Design Marketing            0.817195
28  Quality Assurance           Quality Assurance Developer 0.948010

虽然这样做有所帮助，但将我从478个独特的个人资料带到了461个，但我要重点关注的是这样的个人资料：

Frontend Design JavaScript  Design Frontend JavaScript

我看到的唯一可以解决该问题的工具是difflib？我的问题是，还有哪些其他技术可以用来将由相同词组成但不按顺序编入的这些配置文件标准化为一个标准字符串。因此，所需的输出将是，将包含“ Design”，“ Frontend”和“ JavaScript”的字符串替换为“ Design Frontend JavaScript”。

现在，我将原始数据框与相似性数据框合并，以将right_side上所有出现的配置文件字符串替换为left_side，但这意味着我将下面的right_side（“ Java Python数据科学”）替换为左边的左侧（“ JavaScript Python数据科学”）。

53  JavaScript Python Data Science  Java Python Data Science

任何帮助将不胜感激！！！

编辑***我编写了以下内容来替换在words_to_keep和clean_talentpool ['profile']列中都出现的所有单词，但这似乎不起作用？有人可以指出我没有看到的内容吗？我真的很感激！

def standardize_word_order(row):
    words_to_keep = [
        "javascript frontend design","android ios javascript","android developer developer","android developer","quality assurance","quality assurance engineer","architecture developer","big data architecture developer","data architecture developer","software architecture developer","javascript python data science","frontend PHP javascript","javascript android ios","frontend design javascript","java python data science","javascript frontend android",".net javascript frontend",]
    for word in words_to_keep:
        if (sorted(word.replace(" ",""))) == sorted(
            row.replace(" ","")
        ) and word != row:
            row.replace(row,word)
    return row

clean_talentpool["profile"] = clean_talentpool["profile"].apply(
    lambda x: standardize_word_order(x)
)

解决方法

在这种情况下，我不会只关注字符串而是字符。基本上，如果两个字符串由相同的字符（排列）组成，则它们匹配。

a = "Frontend Design JavaScript"
b = "Javascript Frontend Design"

sorted(a) == sorted(b)
#prints True

您可以考虑删除空间并进行其他预处理，例如下套管。

if sorted(a.lower().replace(" ","")) == sorted(b.lower().replace(" ","")):
    # they are the same,do something

根据您的示例，一个实现可能是：

def standardize_word_order(row):
    words_to_keep = [
        "javascript frontend design","android ios javascript","android developer developer","android developer","quality assurance","quality assurance engineer","architecture developer","big data architecture developer","data architecture developer","software architecture developer","javascript python data science","frontend php javascript","javascript android ios","frontend design javascript","java python data science","javascript frontend android",".net javascript frontend",]
    for word in words_to_keep:
        if ((sorted(word.replace(" ",""))) == sorted(
            row.replace(" ","")
        ) and word != row):
            return word
    return row

clean_talentpool["profile"] = standardize_word_order(clean_talentpool["profile"])

difflib nlp python