返回NUMA节点0 Tensorflow

问题描述

我最近在笔记本电脑上安装了ubuntu 18.04,具有8GB内存,128GB SSD,1TB硬盘和NVIDIA MX-110。

我安装了带有CUDA和CUDNN的Tensorflow 2.2(分别为10.1和7.6.5)。

输出
$ python3 -c "import tensorflow as tf; print(tf.test.is_gpu_available())"

WARNING:tensorflow:From <string>:1: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2020-08-22 16:49:00.218791: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your cpu supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-08-22 16:49:00.243455: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] cpu Frequency: 1800000000 Hz
2020-08-22 16:49:00.243773: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f48bc000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-22 16:49:00.243798: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host,Default Version
2020-08-22 16:49:00.245631: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-08-22 16:49:00.272044: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),but there must be at least one NUMA node,so returning NUMA node zero
2020-08-22 16:49:00.272327: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56105b3aa1e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-22 16:49:00.272346: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce MX110,Compute Capability 5.0
2020-08-22 16:49:00.272476: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:49:00.272656: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce MX110 computeCapability: 5.0
coreClock: 1.006GHz coreCount: 2 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-22 16:49:00.272812: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-08-22 16:49:00.274015: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-08-22 16:49:00.275222: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-08-22 16:49:00.275443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-08-22 16:49:00.276756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-08-22 16:49:00.277520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-08-22 16:49:00.280247: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-22 16:49:00.280378: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:49:00.280622: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:49:00.280782: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-22 16:49:00.280816: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-08-22 16:49:00.281477: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-22 16:49:00.281489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-08-22 16:49:00.281494: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-08-22 16:49:00.281574: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:49:00.281770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:49:00.281952: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/device:GPU:0 with 61 MB memory) -> physical GPU (device: 0,name: GeForce MX110,pci bus id: 0000:01:00.0,compute capability: 5.0)
True

现在,当我运行Jupyter笔记本并制作基本模型时,

#JUPYTER NOTEBOOK CELL NUMBER 1
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data
y = iris.target
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import to_categorical
from tensorflow.keras.layers import Dense
y = to_categorical(y)
from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test = train_test_split(X,y)

"""
JUPYTER NOTEBOOK CELL # 2
This code below takes almost 6,7 mins to run
"""

model1 = Sequential([
    Dense(512,activation='tanh',input_shape = X_train[0].shape),Dense(512//2,activation='tanh'),Dense(512//4,Dense(512//8,Dense(32,activation='relu'),Dense(3,activation='softmax')
])

我在终端上看到的主要内容

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero

并且完整的终端输出

[I 16:37:13.657 NotebookApp] Starting buffering for 427db0cf-ee0b-49fa-b075-ce24cdc456bb:6564a2664692489a8473d40138e004c7
2020-08-22 16:38:17.412089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-08-22 16:38:17.465067: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.465381: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce MX110 computeCapability: 5.0
coreClock: 1.006GHz coreCount: 2 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-22 16:38:17.465580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-08-22 16:38:17.467132: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-08-22 16:38:17.468592: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-08-22 16:38:17.468928: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-08-22 16:38:17.470530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-08-22 16:38:17.471467: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-08-22 16:38:17.474846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-22 16:38:17.475036: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.475556: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.475947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-22 16:38:17.476205: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your cpu supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-08-22 16:38:17.482259: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] cpu Frequency: 1800000000 Hz
2020-08-22 16:38:17.482562: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fa670000b20 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-22 16:38:17.482589: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host,Default Version
2020-08-22 16:38:17.510525: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.510905: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55b17c39ce70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-22 16:38:17.510933: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce MX110,Compute Capability 5.0
2020-08-22 16:38:17.511104: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.511357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce MX110 computeCapability: 5.0
coreClock: 1.006GHz coreCount: 2 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-22 16:38:17.511403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-08-22 16:38:17.511423: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-08-22 16:38:17.511441: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-08-22 16:38:17.511458: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-08-22 16:38:17.511475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-08-22 16:38:17.511493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-08-22 16:38:17.511512: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-08-22 16:38:17.511562: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.511828: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.512056: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-22 16:38:17.512092: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-08-22 16:38:17.513132: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-22 16:38:17.513144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]      0 
2020-08-22 16:38:17.513150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0:   N 
2020-08-22 16:38:17.513236: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.513508: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1),so returning NUMA node zero
2020-08-22 16:38:17.513756: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1461 MB memory) -> physical GPU (device: 0,compute capability: 5.0)

我已经调查了this,但我什么都找不到。

有人可以告诉一些简单的解决方案来解决此问题。谢谢。

编辑:

NVIDIA-SMI的输出

Sat Aug 22 17:03:25 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce MX110       On   | 00000000:01:00.0 Off |                  N/A |
| N/A   55C    P5    N/A /  N/A |   1977MiB /  2004MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1069      G   /usr/lib/xorg/Xorg                 15MiB |
|    0   N/A  N/A      1200      G   /usr/bin/gnome-shell               46MiB |
|    0   N/A  N/A      1481      G   /usr/lib/xorg/Xorg                 91MiB |
|    0   N/A  N/A      1655      G   /usr/bin/gnome-shell               77MiB |
|    0   N/A  N/A      2042      G   ...AAAAAAAAA= --shared-files       80MiB |
|    0   N/A  N/A      3620      G   ...token=4935481349596337741       32MiB |
|    0   N/A  N/A      8276      C   ...nda3/envs/deep/bin/python     1624MiB |
+-----------------------------------------------------------------------------+

输出 nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools,release 10.1,V10.1.243

解决方法

暂无找到可以解决该程序问题的有效方法,小编努力寻找整理中!

如果你已经找到好的解决方法,欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@)