AWS Sagemaker 笔记本间歇性“无法找到凭证”

编程问答 2022-04-22

问题描述

我正在尝试使用 dask 将多个文件 (JSON) 从 AWS S3 获取到 Sagemaker Jupyter Notebook 的内存中。 当我提交 10 或 20 个工人时，一切都会顺利进行。但是，当我提交 100 名工作人员时，其中 30% 到 50% 会遇到以下错误：“无法找到凭据”

最初我尝试使用 Boto3。为了尝试消除这个问题，我切换到 S3FS，但发生了同样的错误。

如果我重复实验，出现 NoCredentialError 错误的工作人员是随机的，下载失败的确切数量也是如此。

Sagemaker 通过其 IAM 角色处理所有 AWS 凭证，因此我无法访问密钥对或任何东西。 ~/.aws/config 文件仅包含默认位置 - 与凭据无关。

这似乎是 dask 的一个非常常见的用途，因此它显然能够执行这样的任务 - 我哪里出错了？

任何帮助将不胜感激！下面的代码和回溯。在此示例中，29 名工作人员因凭据而失败。 谢谢， 帕特里克

import boto3
import json
import logging
import multiprocessing
from dask.distributed import Client,LocalCluster
import s3fs
import os

THREADS_PER_dask_WORKER = 4
cpu_COUNT = multiprocessing.cpu_count()
HTTP_SUCCESSFUL_REQUEST_CODE = 200

S3_BUCKET_NAME = '-redacted-'

keys_100 = ['-redacted-']
keys_10 = ['-redacted-']

def dispatch_workers(workers):

    cluster_workers = min(len(workers),cpu_COUNT)
    cluster = LocalCluster(n_workers=cluster_workers,processes=True,threads_per_worker=THREADS_PER_dask_WORKER)
    client = Client(cluster)

    data = []
    data_futures = []

    for worker in workers:
        data_futures.append(client.submit(worker))

    for future in data_futures:
        try:
            tmp_flight_data = future.result()
            if future.status == 'finished':
                data.append(tmp_flight_data)
            else:
                logging.error(f"Future status = {future.status}")
        except Exception as err:
            logging.error(err)

    del data_futures

    cluster.close()
    client.close()

    return data

def _get_object_from_bucket(key):

    s3 = s3fs.S3FileSystem(anon=False)# uses default credentials
    with s3.open(os.path.join(S3_BUCKET_NAME,key)) as f:
        return json.loads(f.read())

def get_data(keys):

    objects = dispatch_workers(
        [lambda key=key: _get_object_from_bucket(key) for key in keys]
    )
    return objects
    
data = get_data(keys_100)

输出：

ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials
ERROR:root:Unable to locate credentials

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

amazon-sagemakerboto3dask-distributedjupyter-notebookpython-s3fs