nvidia-smi GPU卡信息获取
1.GPU Initialization & Info
| # nvidia-smi Tue Jan ?9 14:39:59 2024 ?????? +---------------------------------------------------------------------------------------+ | NVIDIA-SMI 535.146.02 ????????????Driver Version: 535.146.02 ??CUDA Version: 12.2 ????| |-----------------------------------------+----------------------+----------------------+ | GPU ?Name ????????????????Persistence-M | Bus-Id ???????Disp.A | Volatile Uncorr. ECC | | Fan ?Temp ??Perf ?????????Pwr:Usage/Cap | ????????Memory-Usage | GPU-Util ?Compute M. | | ????????????????????????????????????????| ?????????????????????| ??????????????MIG M. | |=========================================+======================+======================| | ??0 ?NVIDIA GeForce RTX 3060 ???????Off | 00000000:01:00.0 Off | ?????????????????N/A | | ?0% ??53C ???P8 ?????????????18W / 170W | ??2785MiB / 12288MiB | ?????0% ?????Default | | ????????????????????????????????????????| ?????????????????????| ?????????????????N/A | +-----------------------------------------+----------------------+----------------------+ ????????????????????????????????????????????????????????????????????????????????????????? +---------------------------------------------------------------------------------------+ | Processes: ???????????????????????????????????????????????????????????????????????????| | ?GPU ??GI ??CI ???????PID ??Type ??Process name ???????????????????????????GPU Memory | | ???????ID ??ID ????????????????????????????????????????????????????????????Usage ?????| |=======================================================================================| | ???0 ??N/A ?N/A ????40200 ?????C ??python3 ???????????????????????????????????2780MiB | +---------------------------------------------------------------------------------------+ |
2.GPU Status Query
| # nvidia-smi -L GPU 0: NVIDIA GeForce RTX 3060 (UUID: GPU-543d8aa8-ce2f-5b19-a98a-6a1e7f111dee) |
3.GPU Details
| # nvidia-smi --query-gpu=index,name,uuid,serial --format=csv index, name, uuid, serial 0, NVIDIA GeForce RTX 3060, GPU-543d8aa8-ce2f-5b19-a98a-6a1e7f111dee, [N/A] |
4.Monitor GPU Usage
| # nvidia-smi dmon # gpu ???pwr ?gtemp ?mtemp ????sm ???mem ???enc ???dec ???jpg ???ofa ??mclk ??pclk # Idx ?????W ?????C ?????C ?????% ?????% ?????% ?????% ?????% ?????% ???MHz ???MHz ????0 ????19 ????56 ?????- ?????0 ?????0 ?????0 ?????0 ?????0 ?????0 ???405 ???210 ????0 ????19 ????56 ?????- ?????0 ?????0 ?????0 ?????0 ?????0 ?????0 ???405 ???210 ????0 ????19 ????56 ?????- ?????0 ?????0 ?????0 ?????0 ?????0 ?????0 ???405 ???210 ^C ???0 ????19 ????56 ?????- ?????0 ?????0 ?????0 ?????0 ?????0 ?????0 ???405 ???210 |
5.Monitor GPU Processes
| # nvidia-smi pmon # gpu ????????pid ?type ???sm ???mem ???enc ???dec ???command # Idx ??????????# ??C/G ????% ?????% ?????% ?????% ???name ????0 ?????40200 ????C ?????- ?????- ?????- ?????- ???python3 ??????? ????0 ?????40200 ????C ?????- ?????- ?????- ?????- ???python3 ??????? ????0 ?????40200 ????C ?????- ?????- ?????- ?????- ???python3 ?? |
6.List of Available Clocks
| # nvidia-smi -q -d SUPPORTED_CLOCKS ==============NVSMI LOG============== Timestamp ????????????????????????????????: Tue Jan ?9 14:44:31 2024 Driver Version ???????????????????????????: 535.146.02 CUDA Version ?????????????????????????????: 12.2 Attached GPUs ????????????????????????????: 1 GPU 00000000:01:00.0 ????Supported Clocks ????????Memory ???????????????????????????: 7501 MHz ????????????Graphics ?????????????????????: 2100 MHz ????????????Graphics ?????????????????????: 2092 MHz ????????????Graphics ?????????????????????: 2085 MHz ????????????Graphics ?????????????????????: 2077 MHz ????????????Graphics ?????????????????????: 2070 MHz ????????????Graphics ?????????????????????: 2062 MHz ????????????Graphics ?????????????????????: 2055 MHz ????????????Graphics ?????????????????????: 2047 MHz ????????????Graphics ?????????????????????: 2040 MHz ????????????Graphics ?????????????????????: 2032 MHz ????????????Graphics ?????????????????????: 2025 MHz ????????????Graphics ?????????????????????: 2017 MHz ????????????Graphics ?????????????????????: 2010 MHz ....... |
7.Current GPU Clock Speed
| # nvidia-smi -q -d CLOCK ==============NVSMI LOG============== Timestamp ????????????????????????????????: Tue Jan ?9 14:46:20 2024 Driver Version ???????????????????????????: 535.146.02 CUDA Version ?????????????????????????????: 12.2 Attached GPUs ????????????????????????????: 1 GPU 00000000:01:00.0 ????Clocks ????????Graphics ?????????????????????????: 210 MHz ????????SM ???????????????????????????????: 210 MHz ????????Memory ???????????????????????????: 405 MHz ????????Video ????????????????????????????: 555 MHz ????Applications Clocks ????????Graphics ?????????????????????????: N/A ????????Memory ???????????????????????????: N/A ????Default Applications Clocks ????????Graphics ?????????????????????????: N/A ????????Memory ???????????????????????????: N/A ????Deferred Clocks ????????Memory ???????????????????????????: N/A ????Max Clocks ????????Graphics ?????????????????????????: 2100 MHz ????????SM ???????????????????????????????: 2100 MHz ????????Memory ???????????????????????????: 7501 MHz ????????Video ????????????????????????????: 1950 MHz ????Max Customer Boost Clocks ????????Graphics ?????????????????????????: N/A ????SM Clock Samples ????????Duration ?????????????????????????: Not Found ????????Number of Samples ????????????????: Not Found ????????Max ??????????????????????????????: Not Found ????????Min ??????????????????????????????: Not Found ????????Avg ??????????????????????????????: Not Found ????Memory Clock Samples ????????Duration ?????????????????????????: Not Found ????????Number of Samples ????????????????: Not Found ????????Max ??????????????????????????????: Not Found ????????Min ??????????????????????????????: Not Found ????????Avg ??????????????????????????????: Not Found ????Clock Policy ????????Auto Boost ???????????????????????: N/A ????????Auto Boost Default ???????????????: N/A |
8.GPU Performance
| # nvidia-smi -q -d PERFORMANCE ==============NVSMI LOG============== Timestamp ????????????????????????????????: Tue Jan ?9 14:47:08 2024 Driver Version ???????????????????????????: 535.146.02 CUDA Version ?????????????????????????????: 12.2 Attached GPUs ????????????????????????????: 1 GPU 00000000:01:00.0 ????Performance State ????????????????????: P8 ????Clocks Event Reasons ????????Idle ?????????????????????????????: Active ????????Applications Clocks Setting ??????: Not Active ????????SW Power Cap ?????????????????????: Not Active ????????HW Slowdown ??????????????????????: Not Active ????????????HW Thermal Slowdown ??????????: Not Active ????????????HW Power Brake Slowdown ??????: Not Active ????????Sync Boost ???????????????????????: Not Active ????????SW Thermal Slowdown ??????????????: Not Active ????????Display Clock Setting ????????????: Not Active |
9.GPU Topology
| # nvidia-smi topo --matrix ????????GPU0 ???CPU Affinity ???NUMA Affinity ??GPU NUMA ID GPU0 ????X ?????0-7 ????0 ??????????????N/A Legend: ??X ???= Self ??SYS ?= Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) ??NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node ??PHB ?= Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) ??PXB ?= Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) ??PIX ?= Connection traversing at most a single PCIe bridge ??NV# ?= Connection traversing a bonded set of # NVLinks |
10.NVLink Status
| #?nvidia-smi nvlink --status GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-84ccface-663f-f5fd-8e8e-109d0f78bd2f) Link 0: <inactive> |
11.Display GPU Details
| # nvidia-smi -i 0 -q ==============NVSMI LOG============== Timestamp ????????????????????????????????: Tue Jan ?9 14:49:16 2024 Driver Version ???????????????????????????: 535.146.02 CUDA Version ?????????????????????????????: 12.2 Attached GPUs ????????????????????????????: 1 GPU 00000000:01:00.0 ????Product Name ?????????????????????????: NVIDIA GeForce RTX 3060 ????Product Brand ????????????????????????: GeForce ????Product Architecture ?????????????????: Ampere ????Display Mode ?????????????????????????: Disabled ????Display Active ???????????????????????: Disabled ????Persistence Mode ?????????????????????: Disabled ????Addressing Mode ??????????????????????: None ????MIG Mode ????????Current ??????????????????????????: N/A ????????Pending ??????????????????????????: N/A ????Accounting Mode ??????????????????????: Disabled ...... |
12.GPU App Details
| # nvidia-smi -i 0 -q -d MEMORY,UTILIZATION,POWER,CLOCK,COMPUTE ==============NVSMI LOG============== Timestamp ????????????????????????????????: Tue Jan ?9 14:50:08 2024 Driver Version ???????????????????????????: 535.146.02 CUDA Version ?????????????????????????????: 12.2 Attached GPUs ????????????????????????????: 1 GPU 00000000:01:00.0 ????FB Memory Usage ????????Total ????????????????????????????: 12288 MiB ????????Reserved ?????????????????????????: 243 MiB ????????Used ?????????????????????????????: 2785 MiB ????????Free ?????????????????????????????: 9259 MiB ????BAR1 Memory Usage ????????Total ????????????????????????????: 256 MiB ????????Used ?????????????????????????????: 4 MiB ????????Free ?????????????????????????????: 252 MiB ????Conf Compute Protected Memory Usage ????????Total ????????????????????????????: 0 MiB ????????Used ?????????????????????????????: 0 MiB ????????Free ?????????????????????????????: 0 MiB ????Compute Mode ?????????????????????????: Default ????Utilization ????????Gpu ??????????????????????????????: 0 % ????????Memory ???????????????????????????: 0 % ????????Encoder ??????????????????????????: 0 % ????????Decoder ??????????????????????????: 0 % ????????JPEG ?????????????????????????????: 0 % ????????OFA ??????????????????????????????: 0 % ????GPU Utilization Samples ????????Duration ?????????????????????????: 14.00 sec ????????Number of Samples ????????????????: 71 ????????Max ??????????????????????????????: 0 % ????????Min ??????????????????????????????: 0 % ????????Avg ??????????????????????????????: 0 % ????Memory Utilization Samples ????????Duration ?????????????????????????: 14.00 sec ????????Number of Samples ????????????????: 71 ????????Max ??????????????????????????????: 0 % ????????Min ??????????????????????????????: 0 % ????????Avg ??????????????????????????????: 0 % ????ENC Utilization Samples ????????Duration ?????????????????????????: 14.00 sec ????????Number of Samples ????????????????: 71 ????????Max ??????????????????????????????: 0 % ????????Min ??????????????????????????????: 0 % ????????Avg ??????????????????????????????: 0 % |
13.Change GPU Performance Level
| # nvidia-smi -q -d PERFORMANCE ==============NVSMI LOG============== Timestamp ????????????????????????????????: Tue Jan ?9 14:50:58 2024 Driver Version ???????????????????????????: 535.146.02 CUDA Version ?????????????????????????????: 12.2 Attached GPUs ????????????????????????????: 1 GPU 00000000:01:00.0 ????Performance State ????????????????????: P8 ????Clocks Event Reasons ????????Idle ?????????????????????????????: Active ????????Applications Clocks Setting ??????: Not Active ????????SW Power Cap ?????????????????????: Not Active ????????HW Slowdown ??????????????????????: Not Active ????????????HW Thermal Slowdown ??????????: Not Active ????????????HW Power Brake Slowdown ??????: Not Active ????????Sync Boost ???????????????????????: Not Active ????????SW Thermal Slowdown ??????????????: Not Active ????????Display Clock Setting ????????????: Not Active |