问题描述
我有一个包含多列的时间序列数据帧，其中包含彼此独立的NaN。
每个有效元素应该至少具有的序列都有一个给定的长度“ LEN”。 （通过“顺序，我的意思是之前收集索引中的值。”
迭代的时间效率极低，但看起来与此类似：
LEN = 100
maximum_sequence_len = 0
for i in range(len(df)): # for every index
for col in df.columns: # for every column
df_ = df[col].iloc[:i].dropna()
seq_end_ix = i
seq_start_ix = get_seq_start_where_every_col_has_enough_valids(
df,seq_end,LEN)
necessary_len = len( df.loc[seq_start_ix:seq_end_ix] )
if maximum_sequence_len < necessary_len :
maximum_sequence_len = necessary_len
get_seq_start_where_every_col_has_enough_valids(df,seq_end_ix,LEN)
# determine the index where every column contains at least "LEN" valid elements
first_SEQ_LEN_Sample_start_ix = start_ix
for col in df.columns:
col_df = df[col].dropna()
temp = col_df[col_df.index <= seq_end_ix ].index[-(LEN)]
if temp < first_SEQ_LEN_Sample_start_ix:
first_SEQ_LEN_Sample_start_ix = temp
seq_start_ix = first_SEQ_LEN_Sample_start_ix
return seq_start_ix
一个例子：
LEN = 6 # in this example we have to have at least 6 valid elements in the frame of rows before
print(df)
>>>>
A B C D E F
index
0 1 1 1 1 1 1
1 1 1 1 1 1 1
2 1 1 1 1 1 | 1
3 NaN 1 1 NaN 1 | 1
4 NaN 1 1 NaN 1 | 1
5 1 1 1 1 1 | 1
6 1 1 1 1 NaN | 1
7 NaN 1 1 NaN 1 | 1
8 NaN 1 1 1 1 | 1
9 1 1 1 1 NaN | 1
10 1 1 1 1 NaN | 1
11 1 1 1 NaN NaN | 1
12 1 1 1 1 NaN | 1
13 1 1 1 1 NaN | 1
14 1 NaN 1 1 NaN |* 1
16 1 1 1 1 1 NaN
17 NaN 1 1 1 1 1
18 NaN 1 1 1 1 NaN
19 1 1 1 1 1 1
# ==> Result: 13
# *here,longest sequence necessary to get minimum 6 valids in EVERY column has a length of 13. note,that if the other columns contained more NaNs in the marked indices,then it would probably have taken more than 13.
问题是我想创建序列样本，但不知道它们必须有多长时间，以便每个样本在每一列中至少具有“ LEN”个有效元素。
解决方法
本质上，您需要维护一个矢量计数器，每一列都必须有一个计数器。
如果所有计数器至少为6，向量计数器应发出“ window-ready”信号。如果窗口（start_index，end_index）已准备好，则可以发出窗口中的所有行并将窗口的start_index，end_index重置为当前行并将所有计数器重置为零。
重复直到数据结束。
Algorithm get_windows(data[][])
counters: array of integers of length = data.cols,values initialized to 0
Begin
window_start_index = 0
window_end_index = 0
for each row in data
for each col in row
if(value(col) != NaN)
counters[index(col)]++;
end if
next // col
// check if row causes window to continue
continue_flag = false;
for each counter in counters
if(counter != 6)
continue_flag = true
exit for loop
end if
next // counter
if(continue_flag)
window_end_index++;
else
// we have a window (window_start_index,window_end_index)
// both inclusive
// do something with the window
// reset counters
for each counter in counters
counter = 0
next
end if
next // row
End Algorithm
您需要这种单程算法吗？