在具有透明度的 3D 图像上叠加 2D 热图

问题描述

我有一个 3D python Image 及其 2D 特征向量。我用 imshow() 显示了每一个，如这里的代码所示，我可以清楚地看到它。我现在想要的是将 2D 特征向量作为热图叠加在其 3D 图像之上。我尝试添加它们，但引发了维度问题，我通过添加第三维扩展了 2D 特征向量，但叠加的图像混乱了。维度在这里：- 特征 >> (32,96),图片 >> (32,96,3)

img_feature = np.uint8(feature22 + img_raw_np_resize)

ax0.imshow(img,extent =extent)     
ax1.imshow(feature,alpha = 0.75,interpolation = 'gaussian',cmap = plt.cm.jet,extent =extent)  
ax2.imshow(img_feature,extent =extent)

extent 是指使所有这些都以相同的大小显示，并且已经定义了。这是我通过上述代码尝试的结果

我正在寻找的显示器如下......

这是执行此操作的伪代码（我正在尝试生成最少的代码来共享，但这需要时间-因此我共享此代码以防有助于找出问题：-

def save_feature_to_img_2(self):

    feature = (
        get_feature()
    )  #  get feature of size -- > torch.Size([1,256,32,96])

    feature2 = features[
        :,:,:
    ]  #  I pick one from the 256 feature maps - [1,1,96]

    feature2 = (
        feature2.data.numpy()
    )  # convert it to numpy array for later fusion with image array - [1,96]

    features2 = features2.view(
        features2.shape[1],features2.shape[2]
    )  # reshape into 2D feature map  [32,96]

    img_raw1 = Image.open(self.img_path)  # read raw image size [128,64,3]

    img_raw_np = np.array(img_raw1)  # imge to numpy array

    newsize = (h,w)  # shape to re-size image to same size a feature
    img_raw_resize = img_raw1.resize(newsize)
    img_raw_np_resize = np.array(img_raw_resize)  # size is Now [32,3]

    # use sigmod to  normalize feature to  [0,1]
    feature2 = 1.0 / (1 + np.exp(-1 * feature2))

    # display setup
    dx,dy = 0.05,0.05

    y = np.arange(-8,8,dx)  # Y - axis range
    x = np.arange(-4.0,4.0,dy)  # X -axis range
    X,Y = np.meshgrid(x,y)  # meshgrid to enclose my display into
    extent = (
        np.min(x),np.max(x),np.min(y),np.max(y),)  # extent - the X and Y range  - just to make unifrom display

    feature2 = (255.5 * feature2 / np.amax(feature2)).astype(
        np.uint8
    )  # put feature into [0-255] range for colour image

    # to display img,feature and img_feature
    fig = plt.figure()
    ax0 = fig.add_subplot(131,title="Image")
    ax1 = fig.add_subplot(132,title="Heatmap")
    ax2 = fig.add_subplot(133,title="overlayed")

    # hereunder I used this code share in the answer to fuse
    alpha = 0.5

    img_heatmap = (
        feature2[:,None].astype(np.float64) * alpha
        + img_raw_np_resize * (1 - alpha)
    ).astype(np.uint8)

    ax0.imshow(img_raw1,alpha=1.0,interpolation="gaussian",cmap=plt.cm.jet)
    ax1.imshow(feature2,cmap=plt.cm.jet)
    ax2.imshow(
        img_heatmap,alpha=0.7,cmap=plt.cm.jet
    )

    cv2.imwrite("./img_heatmap.jpg",img_heatmap)

这是我得到的新显示器

我使用以下内容来融合图像和特征......

alpha = 0.5         
img_feature = ((plt.cm.jet(feature2)[:,:3] * 255) * alpha + (1-alpha)*img_raw_np_resize).astype(np.uint8)  and displaying it with ax2.imshow(img_feature,alpha = 0.7,cmap = plt.cm.jet)

解决方法

变体 1

如果您只想显示图像的加热区域，则必须乘以热图而不是相加。

适合您的公式为 img_feature = (img * (feature[:,:,None].astype(np.float64) / np.amax(feature))).astype(np.uint8)。

完整示例代码（带有我自己的图像和自动生成的示例热图）：

Try it online!

import requests,PIL.Image,io,numpy as np,matplotlib.pyplot as plt
# load some image
img = np.array(PIL.Image.open(io.BytesIO(requests.get('https://i.stack.imgur.com/vPlCG.jpg').content)))
# load or compute some features
h,w,_ = img.shape
mg = np.mgrid[:h,:w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
img_feature = (img * (feature[:,None].astype(np.float64) / np.amax(feature))).astype(np.uint8)
# show images
fig,(ax0,ax1,ax2) = plt.subplots(1,3)
ax0.imshow(img)
ax1.imshow(feature,alpha = 1.,interpolation = 'gaussian',cmap = plt.cm.jet)
ax2.imshow(img_feature,cmap = plt.cm.jet)
plt.show()

输出：

变体 2

如果您只想使加热的区域变亮（变白）并使未加热的区域变暗（变暗），您只需通过公式 alpha = 0.5; img_feature = (feature[:,None].astype(np.float64) * alpha + img * (1 - alpha)).astype(np.uint8) 进行 alpha 混合。

Try it online!

import requests,:w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
alpha = 0.5; img_feature = (feature[:,None].astype(np.float64) * alpha + img * (1 - alpha)).astype(np.uint8)
# show images
fig,cmap = plt.cm.jet)
plt.show()

输出：

变体 3

与 Variant-2 相同（带有 alpha 混合），但使用基于 matplotlib.pyplot.cm.jet 配色方案的 RGB 特征阵列代替黑白特征阵列。

正如您从代码中看到的，您可以在表达式 plt.cm.jet(feature) 中使用 any coloring scheme 而不是 plt.cm.jet 颜色。

Try it online!

import requests,:w]
feature = mg[0].astype(np.float64) * mg[1].astype(np.float64)
feature = (255.5 * feature / np.amax(feature)).astype(np.uint8)
# compute heated image
alpha = 0.5; img_feature = ((plt.cm.jet(feature)[:,:3] * 255) * alpha + img * (1 - alpha)).astype(np.uint8)
# show images
fig,axs = plt.subplots(2,2)
axs[0,0].imshow(img)
axs[0,1].imshow(feature,cmap = plt.cm.jet)
axs[1,0].imshow(img_feature,1].remove()
plt.show()

输出：

附注。我只是注意到 Matplotlib 在绘制热图时做了标准化 (feature - min) / (max - min) 因此我决定在叠加公式中做同样的事情，最终公式变成这样：

alpha = 0.5; img_feature = ((
    plt.cm.jet(
        (feature - np.amin(feature)).astype(np.float32)
        / (np.amax(feature) - np.amin(feature)).astype(np.float32)
    )[:,:3] * 255
) * alpha + img * (1 - alpha)).astype(np.uint8)

使用上面is here（或here和for your images）公式的示例代码。使用上面is here（和for your images）公式的结果图像。最终覆盖您的图像：

heatmap overlay python superimpose