问题描述
我正在 jupyterlab 上运行 dask。我正在尝试将一些文件保存在存储我的 python 文件的主目录中，并且它运行正常，但我无法找到我的文件的保存位置。因此，我在主目录中创建了一个名为 output 的文件夹以将文件保存在其中，但是当我将文件保存在其中时，出现以下错误：
PermissionError: [Errno 13] Permission denied: b'/home/jovyan/Output/20190101_0200'
这是我的尝试：
from dask_gateway import Gateway
gateway = Gateway(
address="http://traefik-pangeo-dask-gateway/services/dask-gateway",public_address="https://pangeo.aer-gitlab.com/services/dask-gateway",auth="jupyterhub",)
options = gateway.cluster_options()
options
cluster = gateway.new_cluster(
cluster_options=options,)
cluster.adapt(minimum=10,maximum=50)
client = cluster.get_client()
cluster
client
def get_turb(file,name):
d=[name[0:4],name[4:6],name[6:8],name[9:11],name[11:13]]
f_zip = gzip.decompress(file)
yr=d[0]
mo=d[1]
da=d[2]
hr=d[3]
mn=d[4]
fs = s3fs.S3FileSystem(anon=True)
period = pd.Period(str(yr)+str('-')+str(mo)+str('-')+str(da),freq='D')
# period.dayofyear
dy=period.dayofyear
cc=[7,8,9,10,11,12,13,14,15,16] #look at the IR channels only for Now
dat = xr.open_dataset(f_zip)
dd=dat[['recNum','trackLat','trackLon','altitude','maxEDR']]
dd=dd.to_dataframe()
dd = dd.sort_values(by=['maxEDR'])
dd = dd.dropna()
dd['num'] = np.arange(len(dd))
dd.to_csv('Output/edr.csv') <----- Saving the file
edr_files = []
for i in range(2):
print(names[i])
s3_ds = dask.delayed(get_turb)(filedata[i],names[i])
edr_files.append(s3_ds)
edr_files = dask.compute(*edr_files)
请告诉我我做错了什么或可能的解决方案。
s3.Bucket('temp').upload_file(file_name+'.zip',file_name+'.zip')
它抛出这个错误：
distributed.protocol.pickle - INFO - Failed to serialize <function get_temp at 0x7f20a9cb8550>. Exception: cannot pickle '_thread.lock' object
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/worker.py in dumps_function(func)
3319 with _cache_lock:
-> 3320 result = cache_dumps[func]
3321 except KeyError:
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py in __getitem__(self,key)
1572 def __getitem__(self,key):
-> 1573 value = super().__getitem__(key)
1574 self.data.move_to_end(key)
/srv/conda/envs/notebook/lib/python3.8/collections/__init__.py in __getitem__(self,key)
1009 return self.__class__.__missing__(self,key)
-> 1010 raise KeyError(key)
1011 def __setitem__(self,key,item): self.data[key] = item
KeyError: <function get_temp at 0x7f20a9cb8550>
During handling of the above exception,another exception occurred:
TypeError Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/protocol/pickle.py in dumps(x,buffer_callback,protocol)
52 buffers.clear()
---> 53 result = cloudpickle.dumps(x,**dump_kwargs)
54 elif not _always_use_pickle_for(x) and b"__main__" in result:
/srv/conda/envs/notebook/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dumps(obj,protocol,buffer_callback)
72 )
---> 73 cp.dump(obj)
74 return file.getvalue()
/srv/conda/envs/notebook/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dump(self,obj)
562 try:
--> 563 return Pickler.dump(self,obj)
564 except RuntimeError as e:
TypeError: cannot pickle '_thread.lock' object
解决方法
尝试使用 s3fs 而不是 boto3 在 S3 上上传文件。这可能会奏效。
,您似乎在 docker 中运行 dask 和 jupyterlab？
也许您应该添加一些标志，例如伙伴关系：
--user root \
-v new-longhorn-volume:/home/jovyan \
-e CHOWN_HOME=yes \
-e CHOWN_HOME_OPTS='-R' \