MTKView 显示分辨率低于 AVCaptureVideoPreviewLayer

编程问答 2022-05-15

问题描述

我正在尝试将摄像机输入流传输到 MTKView 中，以便将一些 CI 过滤器应用于实时流。 初始化捕获会话并布局 MTKView 后，这是我设置金属的方式（金属视图是 MTKview）：

func setupMetal(){
    
    MetalDevice = MTLCreateSystemDefaultDevice()
    MetalView.device = MetalDevice
    // Write when asked
    MetalView.isPaused = true
    MetalView.enableSetNeedsdisplay = false
    // Command queue for the GPU
    MetalCommandQueue = MetalDevice.makeCommandQueue()
    // Assign the delegate
    MetalView.delegate = self
    // ???
    MetalView.framebufferOnly = false
}

然后我从 SampleBufferDelegate 抓取帧并获得 CIImage

extension ViewController: AVCaptureVideoDataOutputSampleBufferDelegate {

func captureOutput(_ output: AVCaptureOutput,didOutput sampleBuffer: CMSampleBuffer,from connection: AVCaptureConnection) {
    //try and get a CVImageBuffer out of the sample buffer
    guard let cvBuffer = CMSampleBufferGetimageBuffer(sampleBuffer) else {
        return
    }
    
    //get a CIImage out of the CVImageBuffer
    let ciImage = CIImage(cvImageBuffer: cvBuffer)
    
    self.currentCIImage = ciImage
    
    // We draw to the Metal view everytime we receive a frame
    MetalView.draw()
            
}}

然后我使用 currentCIImage 使用其委托方法在 MTKView 中绘制：

extension ViewController : MTKViewDelegate {

func mtkView(_ view: MTKView,drawableSizeWillChange size: CGSize) {
    //tells us the drawable's size has changed
}

func draw(in view: MTKView) {
    //create command buffer for ciContext to use to encode it's rendering instructions to the GPU
    guard let commandBuffer = MetalCommandQueue.makeCommandBuffer() else {
        return
    }
    
    //make sure we actually have a ciImage to work with
    guard let ciImage = currentCIImage else {
        return
    }
    
    //make sure the current drawable object for this Metal view is available (it's not in use by the prevIoUs draw cycle)
    guard let currentDrawable = view.currentDrawable else {
        return
    }
    
    //render into the Metal texture
    // Check here if we find a more elegant solution for the bounds
    self.ciContext.render(ciImage,to: currentDrawable.texture,commandBuffer: commandBuffer,bounds: CGRect(origin: .zero,size: view.drawableSize),colorSpace: CGColorSpaceCreateDeviceRGB())
    
    //register where to draw the instructions in the command buffer once it executes
    commandBuffer.present(currentDrawable)
    //commit the command to the queue so it executes
    commandBuffer.commit()
}
}

它工作正常，我能够从 MTKView 中渲染的相机中获取帧。但是，我注意到我没有获得完整的分辨率，不知何故图像在 MTKview 中被放大了。我知道这与我如何设置捕获会话无关，因为当我使用标准 AVCapturePreviewLayer 时一切正常。关于我做错了什么的任何想法？

非常感谢！

P.S 此代码主要基于此优秀教程：https://betterprogramming.pub/using-cifilters-metal-to-make-a-custom-camera-in-ios-c76134993316 但不知何故它似乎对我不起作用。

解决方法

根据捕获会话的设置，相机帧的大小将与您的 MTKView 不同。这意味着您需要在渲染之前缩放和平移它们以匹配 currentDrawable 的大小。为此，我使用以下代码（在 draw 内，就在 render 调用之前）：

    // scale to fit into view
    let drawableSize = self.drawableSize
    let scaleX = drawableSize.width / input.extent.width
    let scaleY = drawableSize.height / input.extent.height
    let scale = min(scaleX,scaleY)
    let scaledImage = input.transformed(by: CGAffineTransform(scaleX: scale,y: scale))

    // center in the view
    let originX = max(drawableSize.width - scaledImage.extent.size.width,0) / 2
    let originY = max(drawableSize.height - scaledImage.extent.size.height,0) / 2
    let centeredImage = scaledImage.transformed(by: CGAffineTransform(translationX: originX,y: originY))
avfoundationavfoundationavfoundationcore-imagemetalmetalmetalswiftswiftswiftswiftuikituikituikit