超过 10.000 个字符的 GCP AutoML TextSnippet

问题描述

我一直在使用 GCP AUtoML Python 库 2.2.0 版进行文本提取，通常效果很好。但是有时它会给我这个错误：

    Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py",line 2447,in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py",line 1952,in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py",line 1821,in handle_user_exception
    reraise(exc_type,exc_value,tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py",line 39,in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py",line 1950,in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py",line 1936,in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/wr/api/simple_text_nlp.py",line 159,in extract_entity_from_text
    predict_data,predict_id,predict_error = extract_entity.predict(text,model_path)
  File "/usr/local/lib/python3.6/site-packages/ProfessorPatPending/ocr/textOperations.py",line 68,in predict
    response = self.__client.predict(name=model_path,payload=payload)
  File "/usr/local/lib/python3.6/site-packages/google/cloud/automl_v1/services/prediction_service/client.py",line 498,in predict
    response = rpc(request,retry=retry,timeout=timeout,Metadata=Metadata,)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py",line 145,in __call__
    return wrapped_func(*args,**kwargs)
  File "/usr/local/lib/python3.6/site-packages/google/api_core/grpc_helpers.py",line 75,in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc),exc)
  File "<string>",line 3,in raise_from
google.api_core.exceptions.InvalidArgument: 400 List of found errors:   1.Field: payload.text_snippet.content; Message: The provided string field value is longer than 10000: 10796

有问题的 TextSnippet 有超过 10.000 个字符，但是 documentation 明确指出最多可以有 250.000 个字符。有人可以向我解释这是怎么回事吗？

创建文本片段的代码是：

client = automl.PredictionServiceClient.from_service_account_file(sa_json_file)    
text_snippet = automl.TextSnippet(content=text_data,mime_type="text/plain")
payload = automl.ExamplePayload(text_snippet=text_snippet)
response = client.predict(name=model_path,payload=payload)

出于显而易见的原因，我不会在此处发布 text_data 本身。

谢谢。

解决方法

您遇到的错误发生在 client.predict() 上，因为您发送的是超过 10k 个字符的 TextSnippet。 AutoML Entity Extraction 仅限于每个预测请求 10k 个字符。

AutoML 自然语言实体提取

最多 10,000 个字符、UTF-8 NFC 编码的 TextSnippet 或大小不超过 20MB 的 .PDF、.TIF 或 .TIFF 格式的文档。

我建议您拆分 TextSnippet 并发送多个请求或将 TextSnippet 修剪为 10k 以满足字符限制。

google-cloud-automl google-cloud-platform python-3.x