为分层DataFrame列创建数字标识符

问题描述

我有一个Pandas DataFrame，其中包含10列以上的数据和几百万行。

三列构成具有三个不同级别的层次结构：high，medium和low。这三列包含没有丢失数据的字符串。每列在整个组合层次结构内 内按字典顺序排序，例如["A…","B…","C…"]早于["H…","A…","B…"]

我想添加三个新的整数列：high_id，medium_id，low_id。这三个X_id列中的每一个都应为每个DataFrame行都具有一个值。第一行的X_id列最初设置为1。当一列对应的X_id值与上一行不同时，X列将递增，除非更高级别的值发生变化，从而将X_id重置为1

纯Python实现示例：

rows = [
    ["high1","med1","low1"],["high1","low2"],"low3"],"low4"],"med2","low5"],"low6"],"med3","low7"],"med4","low8"],["high2","med5","low9"],"lowA"],"med6","lowB"],["high3","lowC"],"med7","lowD"],"lowE"]]

high_id,medium_id,low_id = 1,1,1
ids = [[high_id,low_id]]
previous_row = rows[0]

for row in rows[1:]:
    # Compare "high"
    if previous_row[0] != row[0]:
        high_id += 1
        medium_id = 1
        low_id = 1
    # Compare "medium"
    elif previous_row[1] != row[1]:
        medium_id += 1
        low_id = 1
    # Compare "low"
    elif previous_row[2] != row[2]:
        low_id += 1
    ids.append([high_id,low_id])
    previous_row = row

for i,v in enumerate(rows):
    print(v + ids[i])

输出：

# high,medium,low,high_id,low_id
['high1','med1','low1',1]
['high1','low2',2]
['high1','low3',3]
['high1','low4',4]
['high1','med2','low5',2,1] # medium changed; low_id reset
['high1','low6','med3','low7',3,'med4','low8',4,1] # medium changed; low_id reset
['high2','med5','low9',1] # high changed; low_id,medium_id reset
['high2','lowA',2]
['high2','med6','lowB',1] # medium changed; low_id reset
['high3','lowC',medium_id reset
['high3','med7','lowD',2]
['high3','lowE',3]

请注意，这些列实际上由地理位置名称组成：因此​​，medium和low的值原则上可以针对不同的父级序列重新出现。 （“高”值很少，我可以看到它们没有重复。）

最好通过矢量化操作添加这些列的惯用的Pandas方法是什么？

我已经阅读了许多有关“层次结构”，“计数器”，“标识符”等主题的现有问题，找不到与这种需要“重置”标识符的特定嵌套情况相匹配的内容。

解决方法

我不知道这是否是常规方法，但是我们要求提供将它们分组在一起以确定各自ID所需的信息。逻辑是将它们组合在一起，并且与列表匹配的索引是ID信息。但是，我找不到避免循环处理的方法，因此我使用了循环处理。这可能不会令您满意，但是我会以一种方法回答。

import pandas as pd
import numpy as np
import io

rows = [
    ["high1","med1","low1"],["high1","low2"],"low3"],"low4"],"med2","low5"],"low6"],"med3","low7"],"med4","low8"],["high2","med5","low9"],"lowA"],"med6","lowB"],["high3","lowC"],"med7","lowD"],"lowE"]]

df = pd.DataFrame(rows,columns=['high','medium','low'])
df['high_id'] = df['high'].str.extract(r'(\d)')
m = df.groupby('high')['medium'].unique().to_frame().reset_index()
l = df.groupby(['high','medium'])['low'].unique().to_frame().reset_index()
df = df.merge(m,on='high',how='outer')
df.rename(columns={'medium_x':'medium'},inplace=True)
df = df.merge(l,on=['high','medium'],how='outer')

df.tail()
    high    medium  low_x   high_id medium_y    low_y
16  high2   med6    lowB    2   [med5,med6]    [lowB]
17  high3   med4    lowC    3   [med4,med7]    [lowC]
18  high3   med7    low1    3   [med4,med7]    [low1,lowD,lowE]
19  high3   med7    lowD    3   [med4,lowE]
20  high3   med7    lowE    3   [med4,lowE]

df['medium_id'] = ''
for i in range(len(df)):
    con = np.where(df.loc[i,'medium'] == df.loc[i,'medium_y'])
    df.loc[i,'medium_id'] = int(con[0]) + 1

df['low_id'] = ''
for i in range(len(df)):
    con = np.where(df.loc[i,'low_x'] == df.loc[i,'low_y'])
    df.loc[i,'low_id'] = int(con[0]) + 1

df = df[['high','low_x','high_id','medium_id','low_id']]
df.columns = ['high','low','low_id']
df
    high    medium  low high_id medium_id   low_id
0   high1   med1    low1    1   1   1
1   high1   med1    low1    1   1   1
2   high1   med1    low2    1   1   2
3   high1   med1    low3    1   1   3
4   high1   med1    low3    1   1   3
5   high1   med1    low3    1   1   3
6   high1   med1    low4    1   1   4
7   high1   med2    low5    1   2   1
8   high1   med2    low6    1   2   2
9   high1   med3    low7    1   3   1
10  high1   med3    low7    1   3   1
11  high1   med3    low7    1   3   1
12  high1   med4    low8    1   4   1
13  high2   med5    low9    2   1   1
14  high2   med5    lowA    2   1   2
15  high2   med5    lowA    2   1   2
16  high2   med6    lowB    2   2   1
17  high3   med4    lowC    3   1   1
18  high3   med7    low1    3   2   1
19  high3   med7    lowD    3   2   2
20  high3   med7    lowE    3   2   3