问题描述
目标:使用来自较大FileDataset的随机采样生成向下采样的FileDataset,以用于数据标签项目。
详细信息:我有一个大型FileDataset,其中包含数百万个图像。每个文件名都包含有关其取自“节”的详细信息。一个部分可能包含数千个图像。我想随机选择特定数量的部分以及与这些部分相关的所有图像。然后将样本注册为新数据集。
请注意,下面的代码不是直接复制和粘贴,因为出于机密性原因,有些文件路径和变量等元素已被重命名。
import azureml.core
from azureml.core import Dataset,Datastore,Workspace
# Load in work space from saved config file
ws = Workspace.from_config()
# Define full dataset of interest and retrieve it
dataset_name = 'complete_2017'
data = Dataset.get_by_name(ws,dataset_name)
# Extract file references from dataset as relative paths
rel_filepaths = data.to_path()
# Stitch back in base directory path to get a list of absolute paths
src_folder = '/raw-data/2017'
abs_filepaths = [src_folder + path for path in rel_filepaths]
# Define regular expression pattern for extracting source section
import re
pattern = re.compile('\/(S.+)_image\d+.jpg')
# Create new list of all unique source sections
sections = sorted(set([m.group(1) for m in map(pattern.match,rel_filepaths) if m]))
# Randomly select sections
num_sections = 5
set_seed = 221020
random.seed(set_seed) # for repeatibility
sample_sections = random.choices(sections,k = num_sections)
# Extract images related to the selected sections
matching_images = [filename for filename in abs_filepaths if any(section in filename for section in sample_sections)]
# Define datastore of interest
datastore = Datastore.get(ws,'ml-datastore')
# Convert string paths to Azure Datapath objects and relate back to datastore
from azureml.data.datapath import DataPath
datastore_path = [DataPath(datastore,filepath) for filepath in matching_images]
# Generate new dataset using from_files() and filtered list of paths
sample = Dataset.File.from_files(datastore_path)
sample_name = 'random-section-sample'
sample_dataset = sample.register(workspace = ws,name = sample_name,description = 'Sampled sections from full dataset using set seed.')
问题:运行我在Python SDK中编写的代码并运行了新的FileDataset寄存器,但是当我尝试查看数据集详细信息或将其用于数据标签项目时,我得到了以下内容即使是所有者也会出错。
Access denied: Failed to authenticate data access with Workspace system assigned identity. Make sure to add the identity as Reader of the data service.
另外,在详细信息标签下,数据集中的文件为未知,而数据集中的文件总大小为不可用 >。
我在其他任何地方都没有遇到过这个问题。我能够以其他方式生成数据集,因此,考虑到我以非常规的方式处理数据,我怀疑这是代码问题。
附加说明:
- Azure ML版本为1.15.0
解决方法
虚拟网络背后的数据是否偶然?
,我的一位同事发现托管身份阻止了预览功能。一旦身份的这一方面被修改,我们就可以检查数据并将其用于数据标记项目。