如何在R中的ncdf4并行处理中修复C函数R_nc4_get_vara_double返回的错误

问题描述

我想通过 OPENDAP 从远程存储下载 nc 数据。我将并行后端与 foreach - dopar 循环一起使用，如下所示：

# INPUTS
inputs=commandArgs(trailingOnly = T)
interimpath=as.character(inputs[1])
gcm=as.character(inputs[2])
period=as.character(inputs[3])
var=as.character(inputs[4])
source='MACAV2'

cat('\n\n EXTRACTING DATA FOR',var,gcm,period,'\n\n')

# CHANGING LIBRARY PATHS
.libPaths("/storage/home/htn5098/local_lib/R40") # local library for packages
setwd('/storage/work/h/htn5098/DataAnalysis')
source('./src/Rcodes/CWD_function_package.R') # Calling the function Rscript

# CALLING PACKAGES
library(foreach)
library(doParallel)
library(parallel)
library(filematrix)

# REGISTERING CORES FOR ParaLLEL PROCESSING
no_cores <- detectCores() 
cl <- makeCluster(no_cores)
registerDoParallel(cl)
invisible(clusterEvalQ(cl,.libPaths("/storage/home/htn5098/local_lib/R40"))) # Really have to import library paths into the workers
invisible(clusterEvalQ(cl,c(library(ncdf4))))

# EXTRACTING DATA FROM THE .NC FILES TO MATRIX FORM
url <- readLines('./data/external/MACAV2_OPENDAP_allvar_allgcm_allperiod.txt')
links <- grep(x = url,pattern = paste0('.*','.*','_.*',period),value = T) 

start=c(659,93,1) # lon,lat,time
count=c(527,307,-1)

spfile <- read.csv('./data/external/SERC_MACAV2_Elev.csv',header = T)
grids <- sort(unique(spfile$Grid))

clusterExport(cl,list('ncarray2matrix','start','count','grids')) #exporting data into clusters for parallel processing

cat('\nChecking when downloading all grids\n')

# k <- foreach(x = links,.packages = c('ncdf4')) %dopar% {
    # nc <- nc_open(x)
    # nc.var=ncvar_get(nc,varid=names(nc$var),start=start,count=count)
    # return(nc.var)
    # nc_close(nc)
# }

k <- foreach(x = links,.packages = c('ncdf4'),.errorhandling = 'pass') %dopar% {
    nc <- nc_open(x)
    print(nc)
    nc.var=ncvar_get(nc,start=c(659,1),count=c(527,-1))
    nc_close(nc)
    return(dim(nc.var))
    Sys.sleep(10)
}

# k <- parSapply(cl,links,function(x) {
    # nc <- nc_open(x)
    # nc.var=ncvar_get(nc,count=count)
    # nc_close(nc)
    # return(nc.var)
# })

print(k)

但是，我不断收到此错误：

<simpleError in ncvar_get_inner(ncid2use,varid2use,nc$var[[li]]$missval,addOffset,scaleFact,start = start,count = count,verbose = verbose,signedbyte = signedbyte,collapse_degen = collapse_degen): C function R_nc4_get_vara_double returned error>

这个问题的原因可能是什么？您能否为此推荐一种省时的解决方案（我必须对大约 20 个文件重复此操作）？

谢谢。

解决方法

我的代码中有同样的错误。问题不在于代码本身。这是我想阅读的文件之一。它有问题，所以 R 无法打开它。我识别了该文件并再次下载，同样的代码运行良好。