BigQuery的Python客户端不应该与多处理一起使用吗？

问题描述

根据Python BigQuery client documentation，看来多处理应该可以工作。但是，在尝试使用多处理从熊猫数据帧向BigQuery表进行简单加载时，我一直遇到错误，我想知道文档中的以下语句是否与此有关。

在多处理方案中，最佳实践是在multiprocessing.Pool或multiprocessing.Process调用os.fork()之后创建客户端实例。

我基于此GCP doc（google-cloud-bigquery）编写了代码，该代码仅尝试创建2个进程以在同一张表上加载两个不同的pandas数据框（我也尝试在两个不同的表并出现相同的错误）：

from google.cloud import bigquery
import pandas as pd
import numpy as np
import multiprocessing
import random

def trying():
    #This is just to create random values to create the dataframe.
    values = np.round(np.random.uniform(0,1,(6,14)),2)* random.uniform(0,1)
    df = pd.DataFrame(values,columns=list('abcdefghifklmn'))

    #I understand that I have to create the client for each process
    client = bigquery.Client()

    table_id = 'mydataset.new_table'
    job = client.load_table_from_dataframe(df,table_id)
    job.result()

if __name__ == '__main__':

    processes = []
    for i in range(2):
        p = multiprocessing.Process(target=trying)
        p.start()
        processes.append(p)
    for process in processes:
        process.join()

这是我得到的错误，我不知道发生了什么。两个进程都抛出相同的错误：

Process Process-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py",line 258,in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py",line 114,in run
    self._target(*self._args,**self._kwargs)
  File "test_multiproc_load.py",line 19,in trying
    job = client.load_table_from_dataframe(df,table_id)
  File "/home/.../sNow/local/lib/python2.7/site-packages/google/cloud/bigquery/client.py",line 1932,in load_table_from_dataframe
    parquet_compression=parquet_compression,File "/home/.../sNow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py",line 485,in dataframe_to_parquet
    arrow_table = dataframe_to_arrow(dataframe,bq_schema)
  File "/home/.../sNow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py",line 449,in dataframe_to_arrow
    bq_to_arrow_array(get_column_or_index(dataframe,bq_field.name),bq_field)
  File "/home/.../sNow/local/lib/python2.7/site-packages/google/cloud/bigquery/_pandas_helpers.py",line 224,in bq_to_arrow_array
    return pyarrow.Array.from_pandas(series,type=arrow_type)
  File "pyarrow/array.pxi",line 755,in pyarrow.lib.Array.from_pandas
    return array(obj,mask=mask,type=type,safe=safe,from_pandas=True,File "pyarrow/array.pxi",line 269,in pyarrow.lib.array
    return _sequence_to_array(obj,mask,size,type,pool,c_from_pandas)
  File "pyarrow/array.pxi",line 38,in pyarrow.lib._sequence_to_array
    check_status(ConvertPySequence(sequence,options,&out))
  File "/home/.../sNow/local/lib/python2.7/site-packages/pandas/core/frame.py",line 2927,in __getitem__
    indexer = self.columns.get_loc(key)
  File "/home/.../sNow/local/lib/python2.7/site-packages/pandas/core/indexes/base.py",line 2659,in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx",line 108,in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx",line 127,line 147,in pandas._libs.index.IndexEngine._get_loc_duplicates
KeyError: 0

这是我的数据框创建过程中的简单索引错误吗？你有什么建议吗？

解决方法

问题似乎出在columns=list('abcdefghifklmn')上。
我不确定为什么它不起作用，但是如果我这样指定columns=['a','b',...]，它就会起作用。
如果有人可以解释原因，那将是很棒的。

google-bigquery google-cloud-platform multiprocessing pandas python