Linux Kernel KDB KGDB 实体机内核调试

发布时间:2023年12月20日

内核编译配置

CONFIG_DEBUG_INFO=y                     # 编译debug info

CONFIG_MAGIC_SYSRQ=y                    # 功能编译开关
CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1   # 默认开启项 默认存在 /proc/sysrq-trigger
CONFIG_MAGIC_SYSRQ_SERIAL=y             # 串行sysrq (许多嵌入式主板都具有断开连接的 TTL 电平串行,这可能会产生一些无效的信号。导致检测到虚假sysrq )

CONFIG_KGDB=y                           # 需要开启 gdb 远程调试内核
CONFIG_KGDB_KDB=y                       # KDB调试前端
CONFIG_KDB_DEFAULT_ENABLE=0x0           # 默认不启动KDB 通过模块拔插方式开启
CONFIG_KDB_KEYBOARD=y                   # 需要开启 可以使用键盘
CONFIG_KDB_CONTINUE_CATASTROPHIC=0      # KDB调试内核引发的Oops等容错,=0 总是继续 =1 尝试继续 =2 强制重新启动

# CONFIG_SERIAL_KGDB_NMI is not set     # 此特殊驱动程序允许您临时使用 NMI 调试器端口作为普通控制台(假设该端口已连接到 KGDB)
CONFIG_HAVE_ARCH_KGDB=y                 # 依赖项
CONFIG_KGDB_SERIAL_CONSOLE=y            # KGDB 串口调试终端配置,与 kgdb 共享串行控制台。初始化必须使用 Sysrq -g 进入
CONFIG_KGDB_TESTS=y                     # 这是一个 kgdb I/O 模块,专门用于测试kgdb 的内部功能。回归测试使用
# CONFIG_KGDB_TESTS_ON_BOOT is not set  # 启动时进行模块内部测试
CONFIG_KGDB_LOW_LEVEL_TRAP=y            # 这将为断点异常处理程序添加对 kgdb 的额外回调,这将允许 kgdb 单步执行。

内核运行时设置

  • 关闭随机化

    grub 行添加 nokaslr
    

    注:不设置会报错 -22 Permisson denied

  • kdb on conlose 设置 (对应CONFIG_KDB_DEFAULT_ENABLE=0x0)

    echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc # ttyS0 tty0 ttyAMA0 需要确认串口输出设备
    
  • 开启 kdb 命令行

    echo 1 > /sys/module/kdb/parameters/cmd_enable 
    

    注:不设置会报错 -22 Permisson denied

  • 进入命令行

    • 方法1:panic 自动进入
    • 方法2:手动进入
      echo -g > /proc/sysrq-trigger
      
  • kdb 开机即起

    grub 行添加 nokaslr kgdbwait kgdboc=ttyS0,115200n8 kdb.cmd_enable=1 
    

    注:kgdb属于模块且依赖串口,cpu 初始化 调度算法初始化等流程执行已过

kdb调试

  • 栈 bt

    [0]kdb> bt 
    Stack traceback for pid 1092
    0xffff88823568c680     1092     1080  1    2   R  0xffff88823568d9c0  bash
    
  • 所有CPU运行任务 -btc

    [0]kdb> btc
    btc: cpu status: Currently on cpu 0
    Available cpus: 0-1(I), 2, 3-7(I)
    Stack traceback for pid 0
    0xffffffff82412780        0        0  1    0   I  0xffffffff82413ac0 *swapper/0
    Stack traceback for pid 0
    0xffff888106e05e00        0        0  1    1   I  0xffff888106e07140  swapper/1
    Stack traceback for pid 1092
    0xffff88823568c680     1092     1080  1    2   R  0xffff88823568d9c0  bash
    Stack traceback for pid 0
    0xffff888106e04680        0        0  1    3   I  0xffff888106e059c0  swapper/3
    Stack traceback for pid 0
    0xffff888106e2af00        0        0  1    4   I  0xffff888106e2c240  swapper/4
    Stack traceback for pid 0
    0xffff888106e2c680        0        0  1    5   I  0xffff888106e2d9c0  swapper/5
    Stack traceback for pid 0
    0xffff888106e29780        0        0  1    6   I  0xffff888106e2aac0  swapper/6
    Stack traceback for pid 0
    0xffff888106e28000        0        0  1    7   I  0xffff888106e29340  swapper/7
    
  • 模块状态 lsmod

    [2]kdb> lsmod
    Module                  Size  modstruct     Used by
    sunrpc                364544  0xffffffffc04d4e00    1  (Live) 0xffffffffc0488000 [ ]
    kvm_intel             237568  0xffffffffc052d900    0  (Live) 0xffffffffc04fb000 [ ]
    snd_hda_codec_generic   86016  0xffffffffc0623580    1  (Live) 0xffffffffc0611000 [ ]
    kvm                   679936  0xffffffffc05fd100    1  (Live) 0xffffffffc056a000 [ kvm ]
    irqbypass              16384  0xffffffffc03bc040    1  (Live) 0xffffffffc03ba000 [ irqbypass ]
    crct10dif_pclmul       16384  0xffffffffc0377200    0  (Live) 0xffffffffc0375000 [ ]
    crc32_pclmul           16384  0xffffffffc034f200    0  (Live) 0xffffffffc034d000 [ ]
    qxl                    73728  0xffffffffc04f76c0    1  (Live) 0xffffffffc04e8000 [ ]
    snd_hda_intel          45056  0xffffffffc03b7000    0  (Live) 0xffffffffc03ae000 [ ]
    ghash_clmulni_intel    16384  0xffffffffc00b3400    0  (Live) 0xffffffffc00b1000 [ ]
    snd_hda_codec         131072  0xffffffffc0482600    2  (Live) 0xffffffffc0467000 [ snd_hda_codec snd_hda_codec ]
    ttm                   118784  0xffffffffc0427e80    1  (Live) 0xffffffffc0411000 [ ttm ]
    drm_kms_helper        176128  0xffffffffc0563600    1  (Live) 0xffffffffc053e000 [ drm_kms_helper ]
    snd_hda_core           77824  0xffffffffc0328380    3  (Live) 0xffffffffc0318000 [ snd_hda_core snd_hda_core snd_hda_core ]
    snd_hwdep              20480  0xffffffffc03eb080    1  (Live) 0xffffffffc03e8000 [ snd_hwdep ]
    syscopyarea            16384  0xffffffffc03c5000    1  (Live) 0xffffffffc03c3000 [ syscopyarea ]
    snd_pcm                98304  0xffffffffc03a9640    3  (Live) 0xffffffffc0395000 [ snd_pcm snd_pcm snd_pcm ]
    intel_rapl_perf        16384  0xffffffffc0383740    0  (Live) 0xffffffffc0381000 [ ]
    sysfillrect            16384  0xffffffffc037c040    1  (Live) 0xffffffffc037a000 [ sysfillrect ]
    sysimgblt              16384  0xffffffffc0365000    1  (Live) 0xffffffffc0363000 [ sysimgblt ]
    fb_sys_fops            16384  0xffffffffc035e000    1  (Live) 0xffffffffc035c000 [ fb_sys_fops ]
    snd_timer              32768  0xffffffffc0372100    1  (Live) 0xffffffffc036c000 [ snd_timer ]
    virtio_balloon         20480  0xffffffffc0355240    0  (Live) 0xffffffffc0352000 [ ]
    sg                     40960  0xffffffffc034a0c0    0  (Live) 0xffffffffc0342000 [ ]
    drm                   475136  0xffffffffc02f5940    4  (Live) 0xffffffffc028d000 [ drm drm drm ]
    snd                    86016  0xffffffffc03123c0    6  (Live) 0xffffffffc0302000 [ snd snd snd snd snd snd ]
    psmouse               159744  0xffffffffc027bec0    0  (Live) 0xffffffffc0259000 [ ]
    soundcore              16384  0xffffffffc0252040    1  (Live) 0xffffffffc0250000 [ soundcore ]
    i2c_piix4              24576  0xffffffffc0249540    0  (Live) 0xffffffffc0245000 [ ]
    pcspkr                 16384  0xffffffffc023f100    0  (Live) 0xffffffffc023d000 [ ]
    ip_tables              28672  0xffffffffc0031280    0  (Live) 0xffffffffc002c000 [ ]
    ext4                  675840  0xffffffffc01f43c0    2  (Live) 0xffffffffc015c000 [ ]
    mbcache                16384  0xffffffffc0097080    1  (Live) 0xffffffffc0095000 [ mbcache ]
    jbd2                  110592  0xffffffffc0139b00    1  (Live) 0xffffffffc0122000 [ jbd2 ]
    sr_mod                 28672  0xffffffffc0109140    0  (Live) 0xffffffffc0104000 [ ]
    cdrom                  57344  0xffffffffc011f280    1  (Live) 0xffffffffc0113000 [ cdrom ]
    sd_mod                 49152  0xffffffffc01014c0    3  (Live) 0xffffffffc00f7000 [ ]
    ata_generic            16384  0xffffffffc006c700    0  (Live) 0xffffffffc006a000 [ ]
    ata_piix               36864  0xffffffffc0159240    2  (Live) 0xffffffffc0152000 [ ]
    crc32c_intel           24576  0xffffffffc010f200    4  (Live) 0xffffffffc010c000 [ ]
    libata                237568  0xffffffffc00ef9c0    2  (Live) 0xffffffffc00bc000 [ libata libata ]
    serio_raw              16384  0xffffffffc00ae200    0  (Live) 0xffffffffc00ac000 [ ]
    virtio_console         32768  0xffffffffc00a4400    0  (Live) 0xffffffffc009e000 [ ]
    e1000                 143360  0xffffffffc0091140    0  (Live) 0xffffffffc0071000 [ ]
    floppy                 81920  0xffffffffc0063c40    0  (Live) 0xffffffffc0055000 [ ]
    
  • 进程状态 ps

    [2]kdb> ps
    7 idle processes (state I) and 
    39 sleeping system daemon (state M) processes suppressed,
    use 'ps A' to see all.
    Task Addr               Pid   Parent [*] cpu State Thread             Command
    0xffff88823568c680     1092     1080  1    2   R  0xffff88823568d9c0 *bash
    
    0xffff888106db4680        1        0  0    0   S  0xffff888106db59c0  systemd
    0xffff888106db0000        3        2  0    0   D  0xffff888106db1340  rcu_gp
    0xffff888106db5e00        4        2  0    0   D  0xffff888106db7140  rcu_par_gp
    0xffff888106db2f00        5        2  0    0   D  0xffff888106db4240  kworker/0:0
    0xffff888106deaf00        6        2  0    0   D  0xffff888106dec240  kworker/0:0H
    0xffff888106dec680        7        2  0    2   D  0xffff888106ded9c0  kworker/u16:0
    0xffff888106de9780        8        2  0    0   D  0xffff888106deaac0  mm_percpu_wq
    0xffff888106dede00       10        2  0    7   D  0xffff888106def140  rcu_sched
    0xffff888106e01780       11        2  0    0   D  0xffff888106e02ac0  rcu_bh
    0xffff888106e49780       17        2  0    1   D  0xffff888106e4aac0  kworker/1:0
    0xffff888106e48000       18        2  0    1   D  0xffff888106e49340  kworker/1:0H
    0xffff888106e75e00       22        2  0    2   D  0xffff888106e77140  kworker/2:0
    0xffff888106e72f00       23        2  0    2   D  0xffff888106e74240  kworker/2:0H
    0xffff888106e90000       27        2  0    3   D  0xffff888106e91340  kworker/3:0
    0xffff888106e95e00       28        2  0    3   D  0xffff888106e97140  kworker/3:0H
    0xffff888106ecaf00       32        2  0    4   D  0xffff888106ecc240  kworker/4:0
    0xffff888106ecc680       33        2  0    4   D  0xffff888106ecd9c0  kworker/4:0H
    0xffff888106ef0000       37        2  0    5   D  0xffff888106ef1340  kworker/5:0
    0xffff888106ef5e00       38        2  0    5   D  0xffff888106ef7140  kworker/5:0H
    0xffff888106f1de00       42        2  0    6   D  0xffff888106f1f140  kworker/6:0
    0xffff888106f1af00       43        2  0    6   D  0xffff888106f1c240  kworker/6:0H
    0xffff888106f41780       47        2  0    7   D  0xffff888106f42ac0  kworker/7:0
    0xffff888106f40000       48        2  0    7   D  0xffff888106f41340  kworker/7:0H
    0xffff888106f75e00       51        2  0    1   D  0xffff888106f77140  netns
    0xffff888106f74680       53        2  0    0   D  0xffff888106f759c0  kworker/0:1
    0xffff888237811780       56        2  0    2   D  0xffff888237812ac0  writeback
    0xffff888237858000       60        2  0    3   D  0xffff888237859340  crypto
    0xffff88823785de00       61        2  0    4   D  0xffff88823785f140  kintegrityd
    0xffff88823785af00       62        2  0    6   D  0xffff88823785c240  kblockd
    0xffff888237922f00       63        2  0    5   D  0xffff888237924240  md
    0xffff888237924680       64        2  0    7   D  0xffff8882379259c0  edac-poller
    0xffff888237920000       66        2  0    4   D  0xffff888237921340  kworker/4:1
    0xffff888237925e00       67        2  0    2   R  0xffff888237927140  kworker/2:1
    0xffff888236c61780       68        2  0    1   D  0xffff888236c62ac0  kworker/1:1
    0xffff888236c60000       69        2  0    3   D  0xffff888236c61340  kworker/3:1
    0xffff8882337a9780       96        2  0    6   D  0xffff8882337aaac0  kworker/6:1
    0xffff88823306af00      101        2  0    7   D  0xffff88823306c240  kworker/u16:1
    0xffff888233069780      103        2  0    7   D  0xffff88823306aac0  kworker/7:1
    0xffff888233002f00      134        2  0    0   D  0xffff888233004240  kthrotld
    0xffff888233004680      135        2  0    2   D  0xffff8882330059c0  acpi_thermal_pm
    0xffff888233005e00      136        2  0    4   D  0xffff888233007140  kmpath_rdacd
    0xffff888233000000      137        2  0    1   D  0xffff888233001340  kaluad
    0xffff8882337ac680      138        2  0    5   D  0xffff8882337ad9c0  ipv6_addrconf
    0xffff8882337aaf00      139        2  0    3   D  0xffff8882337ac240  kworker/3:2
    0xffff8882337ade00      140        2  0    7   D  0xffff8882337af140  kstrp
    0xffff8882337a8000      141        2  0    5   D  0xffff8882337a9340  kworker/5:1
    0xffff8882330dc680      229        2  0    6   D  0xffff8882330dd9c0  kworker/6:2
    0xffff888233601780      238        2  0    5   D  0xffff888233602ac0  kworker/5:2
    0xffff888233734680      243        2  0    4   D  0xffff8882337359c0  kworker/4:2
    0xffff888233731780      244        2  0    4   D  0xffff888233732ac0  kworker/4:3
    0xffff888233735e00      391        2  0    4   D  0xffff888233737140  ata_sff
    0xffff888233020000      398        2  0    4   D  0xffff888233021340  scsi_tmf_0
    0xffff888233704680      400        2  0    4   D  0xffff8882337059c0  scsi_tmf_1
    0xffff888233605e00      401        2  0    3   D  0xffff888233607140  kworker/u16:2
    0xffff888233600000      403        2  0    1   D  0xffff888233601340  kworker/u16:3
    0xffff88823306de00      420        2  0    0   D  0xffff88823306f140  kworker/0:2
    0xffff888233068000      423        2  0    2   D  0xffff888233069340  kworker/2:1H
    0xffff888233021780      424        2  0    5   D  0xffff888233022ac0  kworker/5:1H
    0xffff888236c65e00      425        2  0    2   D  0xffff888236c67140  kworker/2:2
    0xffff88823089c680      426        2  0    4   D  0xffff88823089d9c0  kworker/4:1H
    0xffff888230899780      430        2  0    3   D  0xffff88823089aac0  kworker/3:1H
    0xffff888230898000      431        2  0    1   D  0xffff888230899340  kworker/1:1H
    0xffff88823089de00      432        2  0    1   D  0xffff88823089f140  kworker/1:2
    0xffff88823089af00      433        2  0    7   D  0xffff88823089c240  kworker/7:1H
    0xffff888230929780      435        2  0    6   D  0xffff88823092aac0  kworker/6:1H
    0xffff88823092de00      446        2  0    0   D  0xffff88823092f140  kworker/0:1H
    0xffff88823092c680      452        2  0    4   D  0xffff88823092d9c0  ext4-rsv-conver
    0xffff88823092af00      511        2  0    6   D  0xffff88823092c240  kworker/6:3
    0xffff888233634680      513        2  0    7   D  0xffff8882336359c0  kworker/7:2
    0xffff888233632f00      519        2  0    1   D  0xffff888233634240  kworker/1:3
    0xffff8882330daf00      553        1  0    5   S  0xffff8882330dc240  systemd-journal
    0xffff8882356eaf00      567        1  0    0   S  0xffff8882356ec240  systemd-udevd
    0xffff888234b0de00      688        2  0    0   D  0xffff888234b0f140  ttm_swap
    0xffff888234b08000      702        2  0    4   D  0xffff888234b09340  kworker/4:4
    0xffff888233630000      705        2  0    3   D  0xffff888233631340  ext4-rsv-conver
    0xffff888233038000      715        1  0    1   S  0xffff888233039340  mdadm
    0xffff88823303de00      716        1  0    1   S  0xffff88823303f140  auditd
    0xffff888232a10000      717        1  0    4   S  0xffff888232a11340  auditd
    0xffff888233039780      719        2  0    4   D  0xffff88823303aac0  kworker/4:5
    0xffff8882330f5e00      735        1  0    0   S  0xffff8882330f7140  dbus-daemon
    0xffff888235688000      742        1  0    5   S  0xffff888235689340  chronyd
    0xffff8882361ade00      745        1  0    7   S  0xffff8882361af140  irqbalance
    0xffff88823314af00      746        1  0    2   S  0xffff88823314c240  gmain
    0xffff8882330dde00      753        1  0    5   S  0xffff8882330df140  NetworkManager
    0xffff888232a15e00      758        1  0    4   S  0xffff888232a17140  gmain
    0xffff888232a11780      759        1  0    2   S  0xffff888232a12ac0  gdbus
    0xffff8882330d9780      755        1  0    5   S  0xffff8882330daac0  systemd-logind
    0xffff888233148000      756        2  0    3   D  0xffff888233149340  kworker/3:3
    0xffff88823314de00      757        2  0    3   D  0xffff88823314f140  kworker/3:4
    0xffff8882329dde00      763        2  0    5   D  0xffff8882329df140  rpciod
    0xffff8882329daf00      764        2  0    0   D  0xffff8882329dc240  kworker/u17:0
    0xffff8882329d9780      765        2  0    2   D  0xffff8882329daac0  xprtiod
    0xffff888233602f00      766        1  0    0   S  0xffff888233604240  sshd
    0xffff88823568de00      783      753  0    4   S  0xffff88823568f140  dhclient
    0xffff888234b1de00      794        1  0    0   S  0xffff888234b1f140  gssproxy
    0xffff8882361a8000      801        1  0    4   S  0xffff8882361a9340  gssproxy
    0xffff8882361a9780      802        1  0    3   S  0xffff8882361aaac0  gssproxy
    0xffff8882361aaf00      803        1  0    6   S  0xffff8882361ac240  gssproxy
    0xffff8882361ac680      804        1  0    7   S  0xffff8882361ad9c0  gssproxy
    0xffff888233631780      805        1  0    3   S  0xffff888233632ac0  gssproxy
    0xffff888233701780      816        1  0    7   S  0xffff888233702ac0  crond
    0xffff888233705e00      817        1  0    2   S  0xffff888233707140  agetty
    0xffff888233022f00     1072        2  0    0   D  0xffff888233024240  kworker/0:3
    0xffff888233025e00     1080        1  0    1   S  0xffff888233027140  login
    0xffff8882330d8000     1084        1  0    6   S  0xffff8882330d9340  systemd
    0xffff888230a6af00     1085     1084  0    3   S  0xffff888230a6c240  (sd-pam)
    0xffff88823568c680     1092     1080  1    2   R  0xffff88823568d9c0 *bash
    
  • 设置断点 bp

  • 查看断点 bl

  • 清除断点 bc

  • 开启断点 be

  • 关闭断点 bd

  • 单步执行 ss

  • 帮助 help / ?

     [3]kdb> ? 
      Command         Usage                Description
      ----------------------------------------------------------
      md              <vaddr>             Display Memory Contents, also mdWcN, e.g. md8c1
      mdr             <vaddr> <bytes>     Display Raw Memory
      mdp             <paddr> <bytes>     Display Physical Memory
      mds             <vaddr>             Display Memory Symbolically
      mm              <vaddr> <contents>  Modify Memory Contents
      go              [<vaddr>]           Continue Execution
      rd                                  Display Registers
      rm              <reg> <contents>    Modify Registers
      ef              <vaddr>             Display exception frame
      bt              [<vaddr>]           Stack traceback
      btp             <pid>               Display stack for process <pid>
      bta             [D|R|S|T|C|Z|E|U|I|M|A]
                                          Backtrace all processes matching state flag
      btc                                 Backtrace current process on each cpu
      btt             <vaddr>             Backtrace process given its struct task address
      env                                 Show environment variables
      set                                 Set environment variables
      help                                Display Help Message
      ?                                   Display Help Message
      cpu             <cpunum>            Switch to new cpu
      kgdb                                Enter kgdb mode
      ps              [<flags>|A]         Display active task list
      pid             <pidnum>            Switch to another task
      reboot                              Reboot the machine immediately
      lsmod                               List loaded kernel modules
      sr              <key>               Magic SysRq key
      dmesg           [lines]             Display syslog buffer
      defcmd          name "usage" "help" Define a set of commands, down to endefcmd
      kill            <-signal> <pid>     Send a signal to a process
      summary                             Summarize the system
      per_cpu         <sym> [<bytes>] [<cpu>]
                                          Display per_cpu variables
      grephelp                            Display help on | grep
      bp              [<vaddr>]           Set/Display breakpoints
      bl              [<vaddr>]           Display breakpoints
      bph             [<vaddr>]           [datar [length]|dataw [length]]   Set hw brk
      bc              <bpnum>             Clear Breakpoint
      be              <bpnum>             Enable Breakpoint
      bd              <bpnum>             Disable Breakpoint
      ss                                  Single Step
      dumpcommon                          Common kdb debugging
      dumpall                             First line debugging
      dumpcpu                             Same as dumpall but only tasks on cpus
      ftdump          [skip_#lines] [cpu] Dump ftrace log
    
文章来源:https://blog.csdn.net/qq_24423085/article/details/135086454
本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。