打开远程 zarr 存储时获取 KeyError '.zmetadata'

问题描述

尝试使用 xarray 从 s3 读取 zarr 存储。得到一个关键错误。有什么想法

import fsspec
import xarray as xr


uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr"
ds = xr.open_zarr(fsspec.get_mapper(uri,anon=True),consolidated=True)

如果我先下载，我可以在本地打开它：

import s3fs
fs = s3fs.S3FileSystem(anon=True)
fs.get("s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr/*","eastward_wind_at_10_metres.zarr",recursive=True)

这是与 top 命令关联的 Traceback

Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py",line 234,in _call_s3
    return await method(**additional_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/aiobotocore/client.py",line 154,in _make_api_call
    raise error_class(parsed_response,operation_name)
botocore.errorfactory.NoSuchKey: An error occurred (NoSuchKey) when calling the Getobject operation: The specified key does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py",line 132,in __getitem__
    result = self.fs.cat(k)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/asyn.py",line 241,in cat
    raise ex
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py",line 737,in _cat_file
    resp = await self._call_s3(
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/s3fs/core.py",line 252,in _call_s3
    raise translate_boto_error(err) from err
FileNotFoundError: The specified key does not exist.

During handling of the above exception,another exception occurred:

Traceback (most recent call last):
  File "<stdin>",line 1,in <module>
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py",line 675,in open_zarr
    ds = open_dataset(
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py",line 572,in open_dataset
    store = opener(filename_or_obj,**extra_kwargs,**backend_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py",line 294,in open_group
    zarr_group = zarr.open_consolidated(store,**open_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/convenience.py",line 1178,in open_consolidated
    Meta_store = ConsolidatedMetadataStore(store,Metadata_key=Metadata_key)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/storage.py",line 2680,in __init__
    Meta = json_loads(store[Metadata_key])
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/fsspec/mapping.py",line 136,in __getitem__
    raise KeyError(key)
KeyError: '.zMetadata'

解决方法

问题是这个 zarr 数据集没有统一的元数据。该错误实际上是在告诉您这一点 (KeyError: '.zmetadata')。