执行TFMA时出现TFX管道错误：AttributeError：'NoneType'对象没有属性'ToBatchTensors'

问题描述

基本上，我只重用了iris utils和iris pipeline中的代码，但在提供输入内容上做了些微改动：

def _get_serve_tf_examples_fn(model,tf_transform_output):   
    model.tft_layer = tf_transform_output.transform_features_layer()

    feature_spec = tf_transform_output.raw_feature_spec()
    print(feature_spec)
    feature_spec.pop(_LABEL_KEY)

    @tf.function
    def serve_tf_examples_fn(*args):
        parsed_features = {}
        for arg in args:
            parsed_features[arg.name.split(":")[0]] = arg
        print(parsed_features)

        transformed_features = model.tft_layer(parsed_features)

        return model(transformed_features)


def run_fn(fn_args: TrainerFnArgs):
    ...

    feature_spec = tf_transform_output.raw_feature_spec()
    feature_spec.pop(_LABEL_KEY)

    inputs = [tf.TensorSpec(
                    shape=[None,1],dtype=feature_spec[f].dtype,name=f) for f in feature_spec]

    signatures = {
        'serving_default':
            _get_serve_tf_examples_fn(model,tf_transform_output).get_concrete_function(*inputs),}
    model.save(fn_args.serving_model_dir,save_format='tf',signatures=signatures)

来自虹膜代码的get_concrete_function（）原始输入仅是具有dtype字符串的TensorSpec。我已经尝试使用确切的输入来提供模型服务，但是当我测试REST API时，出现了解析错误。因此，我尝试更改服务输入，以便它可以接收这样的JSON输入：

{"instances": [{"feat1": 90,"feat2": 23.8,"feat3": 12}]}

当我运行管道时，培训成功了，但是随后在运行评估程序组件时发生了错误。这是最新的日志：

INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/67/serving_model_dir as candidate model.
INFO:absl:Using ./tfx/pipelines/toilet_native_keras/Trainer/model/14/serving_model_dir as baseline model.
INFO:absl:The 'example_splits' parameter is not set,using 'eval' split.
INFO:absl:Evaluating model.
INFO:absl:We decided to produce LargeList and LargeBinary types.
WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0e44560> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop,(2) passing tensors with different shapes,(3) passing Python objects instead of tensors. For (1),please define your @tf.function outside of the loop. For (2),@tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3),please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77f8a70> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop,please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
...
Traceback (most recent call last):
  File "apache_beam/runners/common.py",line 1213,in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py",line 570,in apache_beam.runners.common.SimpleInvoker.invoke_process
  File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/model_util.py",line 466,in process
    result = self._batch_reducible_process(element)
  File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/extractors/batched_predict_extractor_v2.py",line 164,in _batch_reducible_process
    self._tensor_adapter.ToBatchTensors(record_batch),input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors'

During handling of the above exception,another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",line 256,in _execute
    response = task()
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",line 313,in <lambda>
    lambda: self.create_worker().do_instruction(request),request)
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",line 483,in do_instruction
    getattr(request,request_type),request.instruction_id)
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/sdk_worker.py",line 518,in process_bundle
    bundle_processor.process_bundle(instruction_id))
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",line 983,in process_bundle
    element.data)
  File "/usr/local/lib/python3.7/site-packages/apache_beam/runners/worker/bundle_processor.py",line 219,in process_encoded
    self.output(decoded_value)
  File "apache_beam/runners/worker/operations.py",line 330,in apache_beam.runners.worker.operations.Operation.output
  ...
  File "apache_beam/runners/common.py",line 1294,in apache_beam.runners.common.DoFnRunner._reraise_augmented
  File "/usr/local/lib/python3.7/site-packages/future/utils/__init__.py",line 446,in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "apache_beam/runners/common.py",input_names)
AttributeError: 'NoneType' object has no attribute 'ToBatchTensors' [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/ExtractBatchPredictions/Predict']
...
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7f0273050> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop,please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7fa7c77fc170> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop,@tf.function has experimental_relax_shapes=True option that relaxes arg

我认为评估器组件与提供输入功能没有任何关系，因为它只是与新训练的模型和最新发布的模型进行了比较，但是我哪里出错了？

解决方法

因此，最后我还是误认为评估器组件，或者如果我改用TFMA，则更恰当。它确实使用了服务签名中定义的服务输入功能。根据{{​​3}}，TFMA EvalConfig使用的默认签名是“ serving_default”，它描述了要序列化示例的服务模型输入。这就是为什么当我更改输入签名而不是字符串时，TFMA会引发异常。

我认为此签名不是要用于通过REST API为模型提供服务的，并且因为仍然需要“ serving_default”签名，并且我不想修改EvalConfig，因此我创建了另一个签名，该签名将接收我想要的JSON输入。为了正常工作，我需要制作另一个由@ tf.function装饰的函数。就这样。希望我的回答对遇到类似问题的人们有所帮助。