问题描述
为了使问题易于理解,我在下面逐步绘制了一些图。
-
名为
data
的 3D 数组,这是我想根据feature
和mask
求和的数据。 -
名为
mask
(形状与data
相同)的 3D 数组,用于对data
进行子集。 -
我有一个名为
feature
的一维 DataArray,其值是mask
的一部分。feature
的所有值都没有重复,但是time
维度有一些重复的值。
步骤:
-
通过
feature
坐标循环time
-
基于
mask
和循环feature
创建临时掩码:1
表示时间和值都等于所选特征;0
给别人 -
使用临时掩码对
data
进行掩码,对掩码数据求和,并将其保存到名为data_mask
的新数据中,该数据与feature
的形状相同。
结果如下:
我已经使用 for 循环编写了代码:
import xarray as xr
import pandas as pd
import numpy as np
# create feature example
t_feature = pd.to_datetime(['2019-07-25 00:00','2019-07-25 00:00','2019-07-25 01:00'])
feature = xr.DataArray(np.array([1,2,4]),coords=[t_feature],dims={'time': t_feature})
# create mask example
t = pd.to_datetime(['2019-07-25 00:00','2019-07-25 01:00'])
mask_t1 = np.array([[1,1,1],[2,2],[3,3,3]])
mask_t2 = mask_t1*2
mask = np.stack((mask_t1,mask_t2))
mask = xr.DataArray(mask,coords=[t,range(3),range(3)],dims=['time','x','y'])
# create data example
data = np.ones(mask.shape)
data[0,:] *= 2
data[1,...] *= 3
data = xr.DataArray(data,'y'])
data_mask = feature.copy()
for index,f in enumerate(feature):
timestamp = f.time
pair_mask = mask.sel(time=timestamp)
pair_mask = pair_mask.where(pair_mask==f,False)
data_mask[dict(time=index)] = data.sel(time=timestamp).where(pair_mask).sum()
但是,对于大型数据集来说太慢了。如果您有更好的建议,我将不胜感激!
更新
根据Oxbowerce的建议,我想出了三种方法并测试速度。
结论
pandas
方法也会导致内存错误,并且比 xarray
方法慢。
for loop
最慢,但没有内存问题,因为数据已加载。
详情
import xarray as xr
import pandas as pd
import numpy as np
len_t = int(1e3)
# create feature example
t = pd.date_range(start='1/1/2018',periods=len_t,freq='S')
feature = xr.DataArray(np.random.randint(len_t/2,size=len_t),# range(len_t),coords=[t],dims={'time': t})
# create mask example
mask = xr.DataArray(np.random.randint(len_t/2,size=(len_t,50,50)),range(50),range(50)],'y'])
# create data example
data = mask.copy()
data_mask = feature.copy()
# --- method 1: for loop --- #
for index,False)
data_mask[dict(time=index)] = data.sel(time=timestamp).where(pair_mask).sum()
# --- method 2: pandas --- #
# convert xarrays to pandas dataframes
data_df = data.to_dataframe(name="data_value").reset_index()
feature_df = feature.to_dataframe(name="feature_value")
mask_df = mask.to_dataframe(name="mask_value").reset_index()
result = (
data_df
# add mask values data
.merge(mask_df,how="left",on=["time","x","y"])
# add feature values to data,using inner join to only leave rows present in feature array
.merge(feature_df,how="inner",left_on=["time","mask_value"],right_on=["time","feature_value"])
# group rows and add up the values
.groupby("feature_value")
.sum()["data_value"]
)
# --- method 3: xarray --- #
feature_time = feature.time
merge_ds = xr.merge([data.rename('data'),mask.rename('mask')],join="left").sel(time=feature_time)
result = merge_ds['data'].where(merge_ds['mask']==feature,drop=True).sum(dim=['x','y'])
这是执行时间:
-
for 循环:
5.24 s ± 30.3 ms per loop (mean ± std. dev. of 7 runs,1 loop each)
-
熊猫方法:
1.48 s ± 27.9 ms per loop (mean ± std. dev. of 7 runs,1 loop each)
-
xarray 方法:
74.3 ms ± 2.03 ms per loop (mean ± std. dev. of 7 runs,10 loops each)
解决方法
根据您给出的示例,我首先将所有 xarray 转换为 Pandas 数据帧并使用连接来组合数据。我过滤掩码数组中的值存在于特征数组中的行,并将这些值相加。这看起来像这样:
import xarray as xr
import pandas as pd
import numpy as np
# create feature example
t_feature = pd.to_datetime(['2019-07-25 00:00','2019-07-25 00:00','2019-07-25 01:00'])
feature = xr.DataArray(np.array([1,2,4]),coords=[t_feature],dims={'time': t_feature})
# create mask example
t = pd.to_datetime(['2019-07-25 00:00','2019-07-25 01:00'])
mask_t1 = np.array([[1,1,1],[2,2],[3,3,3]])
mask_t2 = mask_t1*2
mask = np.stack((mask_t1,mask_t2))
mask = xr.DataArray(mask,coords=[t,range(3),range(3)],dims=['time','x','y'])
print(mask)
# create data example
data = np.ones(mask.shape)
data[0,:] *= 2
data[1,...] *= 3
data = xr.DataArray(data,'y'])
# convert xarrays to pandas dataframes
data_df = data.to_dataframe(name="data_value").reset_index()
feature_df = feature.to_dataframe(name="feature_value")
mask_df = mask.to_dataframe(name="mask_value").reset_index()
result = (
data_df
# add mask values data
.merge(mask_df,how="left",on=["time","x","y"])
# add feature values to data,using inner join to only leave rows present in feature array
.merge(feature_df,how="inner",left_on=["time","mask_value"],right_on=["time","feature_value"])
# group rows and add up the values
.groupby("feature_value")
.sum()["data_value"]
)
结果如下:
feature_value | data_value |
---|---|
1 | 3 |
2 | 6 |
4 | 9 |