问题描述
我正在使用Python API将数据导入BigQuery:
import os
from google.cloud import bigquery
from google.cloud.exceptions import BadRequest
import json
from schema import schema
target_project_name = 'project_name'
target_dataset_name = 'dataset_name'
target_table_name = 'table_name'
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = './credentials.json'
dataset_ref = bigquery.dataset.DatasetReference(target_project_name,target_dataset_name)
table_ref = bigquery.table.TableReference(dataset_ref,target_table_name)
job_config = bigquery.LoadJobConfig()
job_config.schema = schema
client = bigquery.Client()
cnt = 0
with open('./data/transformed_v2.json','r') as f:
json_list = []
for line in f.readlines():
cnt += 1
if cnt >= 10:
break
json_contents = json.loads(line)
json_list.append(json_contents)
job = client.load_table_from_json(json_list,table_ref)
try:
result = job.result()
except BadRequest as ex:
for err in ex.errors:
print(err)
print(job.errors)
上面的代码中提到的schema
包含一个有问题的字段migration_datetime
,该字段定义为bigquery.SchemaField("migration_datetime","STRING",mode="NULLABLE")
。该字段包含存储在'%Y-%m-%dT%H:%M:%s' format. The schema in the target table has the
migration_datetime field specified as
STRING`中的值。
job.result()
引发错误,提示Field migration_datetime has changed type from STRING to TIMESTAMP
。为了减轻这种情况,我在_string
列的值中添加了migration_datetime
后缀,但仍然发生错误。你可能是什么原因?
解决方法
请考虑将job_config
像here一样传递到load_table_from_json
或使用client.schema_from_json