不兼容的形状：[11,768]与[1,5,768]-使用保存的拥抱模型在生产中进行推断

问题描述

我从拥抱模型中保存了distilbert的预训练版本，distilbert-base-uncased-finetuned-sst-2-english，并且正在尝试通过Tensorflow Serve提供服务并进行预测。目前，所有这些都正在Colab中进行测试。

我在通过TensorFlow Serve将预测转换为模型的正确格式时遇到问题。 Tensorflow服务已经启动并且可以很好地为模型服务，但是我的预测代码不正确，我需要一些帮助来了解如何通过API上的json进行预测。

# tokenize and encode a simple positive instance
instances = tokenizer.tokenize('this is the best day of my life!')
instances = tokenizer.encode(instances)
data = json.dumps({"signature_name": "serving_default","instances": instances,})
print(data)

{“ signature_name”：“ serving_default”，“ instances”：[101、2023、2003、1996、2190、2154、1997、2026、2166、999、102]}

# setup json_response object
headers = {"content-type": "application/json"}
json_response = requests.post('http://localhost:8501/v1/models/my_model:predict',data=data,headers=headers)
predictions = json.loads(json_response.text)

预测

{'error': '{{function_node __inference__wrapped_model_52602}} {{function_node __inference__wrapped_model_52602}} Incompatible shapes: [11,768] vs. [1,5,768]\n\t [[{{node tf_distil_bert_for_sequence_classification_3/distilbert/embeddings/add}}]]\n\t [[StatefulPartitionedCall/StatefulPartitionedCall]]'}

这里的任何方向都将不胜感激。

解决方法

能够通过设置输入形状和注意遮罩的签名来找到解决方案，如下所示。这是一个简单的实现，对保存的模型使用固定的输入形状，并且需要将输入填充到预期的输入形状384。我已经看到了调用自定义签名和模型创建以匹配预期的输入形状的实现，但是下面我想通过TF Serve提供拥抱面模型来完成一个简单的案例。如果有人有更好的示例或更好地扩展此功能的方法，请发布以供将来使用。

    public bool CeilingCheck;
    public bool crouch;

    void Update()
    {
        if (Input.GetButton("Crouch"))
        {
            crouch = true;
        }
        else if (!CeilingCheck)
        {
            crouch = false;
        }
    }

通过调用get_concrete_function，我们对由两个形状为[None，384]的张量组成的输入签名进行跟踪编译模型的TensorFlow操作，第一个为输入id，第二个为注意掩码。

public PlayerMovement mainScript;

private void OnTriggerEnter2D(Collider2D collision)
{
    if (collision.tag == "Level")
    {
        mainScript.CeilingCheck = true;
    }
}
private void OnTriggerExit2D(Collider2D collision)
{
    if (collision.tag == "Level")
    {
        mainScript.CeilingCheck = false;
    }
}

保存带有签名的模型：

import os 
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
DB_PATH = os.path.join (BASE_DIR,'db_test_dir\database.db')
print(DB_PATH)

检查是否包含正确的签名：

base_url = 'https://www.alphavantage.co/query?'
params = {'function': 'OVERVIEW','symbol': 'IBM','apikey': keys}

response = requests.get(base_url,params=params)

print(response.json())
test = response.json()['Name']
print(test)

输出应如下所示：

# create callable
from transformers import TFDistilBertForQuestionAnswering
distilbert = TFDistilBertForQuestionAnswering.from_pretrained('distilbert-base-cased-distilled-squad')
callable = tf.function(distilbert.call)

测试模型：

concrete_function = callable.get_concrete_function([tf.TensorSpec([None,384],tf.int32,name="input_ids"),tf.TensorSpec([None,name="attention_mask")])

对于TF SERVE（在合作实验室中）：（这是我最初的意图）

# stored model path for TF Serve (1 = version 1) --> '/path/to/my/model/distilbert_qa/1/'
distilbert_qa_save_path = 'path_to_model'
tf.saved_model.save(distilbert,distilbert_qa_save_path,signatures=concrete_function)

saved_model_cli show --dir 'path_to_model' --tag_set serve --signature_def serving_default

The given SavedModel SignatureDef contains the following input(s):
  inputs['attention_mask'] tensor_info:
      dtype: DT_INT32
      shape: (-1,384)
      name: serving_default_attention_mask:0
  inputs['input_ids'] tensor_info:
      dtype: DT_INT32
      shape: (-1,384)
      name: serving_default_input_ids:0
The given SavedModel SignatureDef contains the following output(s):
  outputs['output_0'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1,384)
      name: StatefulPartitionedCall:0
  outputs['output_1'] tensor_info:
      dtype: DT_FLOAT
      shape: (-1,384)
      name: StatefulPartitionedCall:1
Method name is: tensorflow/serving/predict

from transformers import DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased')

question,text = "Who was Benjamin?","Benjamin was a silly dog."
input_dict = tokenizer(question,text,return_tensors='tf')

start_scores,end_scores = distilbert(input_dict)

all_tokens = tokenizer.convert_ids_to_tokens(input_dict["input_ids"].numpy()[0])
answer = ' '.join(all_tokens[tf.math.argmax(start_scores,1)[0] : tf.math.argmax(end_scores,1)[0]+1])

!echo "deb http://storage.googleapis.com/tensorflow-serving-apt stable tensorflow-model-server tensorflow-model-server-universal" | tee /etc/apt/sources.list.d/tensorflow-serving.list && \
curl https://storage.googleapis.com/tensorflow-serving-apt/tensorflow-serving.release.pub.gpg | apt-key add -
!apt update

发布请求：

!apt-get install tensorflow-model-server