从Python3中的命令行和configfile参数中的字典映射变量

问题描述

上下文

我正在尝试通过命令行将一些参数传递给函数。这些参数可以直接从命令行,配置文件中读取,也可以将两者结合使用以简化某些管道。例如:

for i in { 1 .. 100 }
do
python myprogram.py -i data_${i}.nii.gz -o results_${i} -cf configfile.ini
done

我的问题

我在read_configfile()中有两个函数read_commandlinemyprogram.py可以正常工作,并以完全相同的输出格式提供字典args。如果打印出两个函数都返回的args变量,则会得到类似的信息:

{'data_path': '~/problem/data_X/','results_path': '~/problem/results_X/','base_path': '~/problem/','model_name': 'subject_X','data_name': 'data.nii.gz','image_file': None,'bvals_file': None,'bvecs_file': None,'brainmask_file': None,'header_init': None,'init_files': None,'cov_file': 'Analytic','njobs': 14,'slice_axis': None,'slice_n': None,'seed': None,'get_cov_samples': False,'num_sticks': 3,'snr': 30,'type_noise': 'Rician','framework': 'Matlab','optimizer': 'fmincon','cons_fit': True,'ard_optimization': False,'likelihood_type': 'Default','reparametrization': False,'no_run_mcmc': False,'mcmc_analysis': 'Hybrid1','burnin': 1000,'jumps': 1250,'thinning': 25,'adaptive': True,'period': 50,'duration': 'whole-run','no_ard_mcmc': False,'type_ard': 'Gaussian','ard_fudge': 1,'bingham_flag': False,'acg': False}

我想要的是映射字典args内容以创建新变量(即用variableX代替args.variableX或{{1} })。 args['variableX']是我要创建的变量的名称,而key是其各自的值。例如:

value

我想尽可能做到最简洁,可扩展,所以我不想一个个手动分配

我已经搜索并尝试了几种方法来执行此映射,例如列出的herehere(例如print(data_path) ~/problem/data_X/ print(period) 50 locals().update(args))。但是,它们全部都部分起作用。这些映射使大多数变量未定义(实际上在代码中稍后重新定义或重用的那些变量),可能是由于与命名空间发生某种冲突或某些我不了解的事情。

是否有解决方案,或者我必须一个一个地重新定义它们?

编辑

for i in list(args.keys()): exec(f'{i} = {args[i]}' 

def myprogram(): [...] var1,var2,var3,....,var50 = read_params(logger,sys.argv[1:]) [other stuff] def read_params(logger,argv): # Only CONfigFILE if isinstance(argv,str): # It is a string,i.e. a path (chain of characters) = Only configfile provided logger.info('Reading parameters from configfile....') args = read_configfile(argv) # COMMAND LINE ParaMS elif isinstance(argv,list): # Params introduced by terminal logger.info('Reading parameters from command-line....') args = read_params(argv) # One (automatic) option that should work locals().update(args) print('Variables in locals():') print(locals()) # It seems to be updated correctly to locals(). See print shown below # Re-deFinition one by one. It works but I want to avoid this # model_name = args['model_name'] # base_path = args['base_path'] # data_path = args['data_path'] # [...] # Repeat for the rest of variables # If I use these updated variables from locals().update(args) anywhere in the code,it raises an Error. if data_path is None: if (os.path.isdir(os.path.join(base_path,'data/'))): data_path = os.path.join(base_path,'data/') sys.path += [data_path] logger.info(f'Data path set in from {data_path}') else: logger.error(f'Error! Data path {data_path} does not exist.') sys.exit() [other similar stuff] return data_path,... def read_configfile(configfile): # localconfig is a wrapper on top of configparser (so fully compatible) that makes easier to import the variables in correct data types using the same configfile # https://pypi.org/project/localconfig/ from localconfig import config config.read(configfile) args = dict(list(config.args)) # returns a dict return args [...] 错误

print(locals())

如您所见,Variables in locals(): {'logger': <Logger run_mybedpostX (DEBUG)>,'argv': '~/code/config/configfile_template.py','args': {'data_path': '~/code/data/','results_path': '~/code/results/','base_path': '~/code/','image_file': 'data.nii.gz','bvals_file': 'bvals','bvecs_file': 'bvecs','brainmask_file': 'nodif_brain_mask.nii.gz','model_name': 'model_1','header_init': 'pvmfit','init_files': '~/code/data/brain/PVMFIT/','cov_file': '~/code/results/3fib/covSPD.npy','njobs': 1,'seed': 1234,'full_report': False,'framework': 'None #Matlab','optimizer': 'None #','run_mcmc': True,'ard_mcmc': True,'acg': False,'bingham_flag': False},'bingham_flag': False} Traceback (most recent call last): File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py",line 1758,in <module> main() File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py",line 1752,in main globals = debugger.run(setup['file'],None,is_module) File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py",line 1147,in run pydev_imports.execfile(file,globals,locals) # execute the script File "/Applications/PyCharm.app/Contents/helpers/pydev/_pydev_imps/_pydev_execfile.py",line 18,in execfile exec(compile(contents+"\n",file,'exec'),glob,loc) File "~/code/main.py",line 214,in <module> main(args) File "~/code/main.py",line 72,in main get_cov_samples,type_ard,ard_fudge,bingham_flag,acg = read_params(logger,args) File "~/code/utils/read_params.py",line 98,in read_params if data_path is None: UnboundLocalError: local variable 'data_path' referenced before assignment 中的所有项目也作为单独的变量出现在args中(在locals()格之后)。实际上,在Pycharm中对其进行调试,我可以从Debug Console中对其进行调用。但是,如上所示,如果您运行脚本(即从函数调用),则会引发该错误

解决方法

您尝试更新locals()或将execlocals一起使用在Python 3中不起作用。在文档中也有与此类似的警告。 locals不再作为提高性能的命令而实现,而是一个固定大小的数组。

请记住,CPython已编译为字节码,由解释器 运行。编译函数时,局部变量存储在 固定大小的数组(不是字典)和变量名被分配给 索引。这是可能的,因为您无法动态添加本地 函数的变量。然后从字面上检索局部变量 在列表中查找指针,并在PyObject上增加refcount 这是微不足道的。

将此与全局查找(LOAD_GLOBAL)进行对比,这是正确的决定 搜索涉及哈希等等。顺便说一下,这就是为什么您需要 指定全局i如果您希望它是全局的:如果您曾经分配给 范围内的变量,编译器将为其发出STORE_FAST 除非您告知不要访问。

来自:Why does Python code run faster in a function?

可能的解决方法

  1. 更新globals而不是locals,它仍然有效,因为它仍然实现为dict警告:这是一个坏主意,在使用前应三思而后行,有更好的方法。

  2. 使用namespace。您希望字典的键是可以直接使用'。'访问的适当变量。运算符,否则您将需要使用getattr来访问变量。

>>> from types import SimpleNamespace
>>> args = {"data_path": "xyx/data","base_path": "xyz"} 
>>> n = SimpleNamespace(**args)
>>> n.data_path
'xyx/data'

参考:

Why does Python code run faster in a function?

https://www.python.org/dev/peps/pep-0558/

https://stackoverflow.com/a/28345836/6741053