kk Blog —— 通用基础

date [-d @int|str] [+%s|"+%F %T"]

KGDB配置

Host机:一个装有Ubuntu12.04-x86-64的主机
Target机:运行在vmware上的 Ubuntu12.04-server-x86-64 的Linux.

Target机器配置

  1. 配置好VMware对外串口, 详情见:http://my.oschina.net/u/139611/blog/110052
  2. 下载源码到/usr/src/linux-source-3.2.0下, 解压.
  3. make menuconfig
  4. 进入General setup,把Local version设置一下(-kgdb)
  5. 进入Kernel hacking,选"Compile the kernel with debug info"为*
  6. 选"KGDB: kernel debugging with remote gdb"为*
  7. 选"Write protect kernel read-only data structures"为空 (否则在断下来继续执行的时候可能会报错:Cannot remove breakpoints because program is no longer writable)
  8. 进入"KGDB: … “ 选"KGDB: use KGDB over the serial console"为*,选"KGDB: internal test suite“为空,否则kgdboc会注册不了
  9. 保存,编译: make -j4 && make modules install && make install
  10. 把vmliunux和System.map拷贝到host机器上
  11. 修改/boot/grub/grub.cfg中menuentry为kgdb的项,在kernel后面添加参数: kgdboc=ttyS1,115200 kgdbwait
  12. 重启,系统进入等待状态。

Host机:

  1. 安装好GDB,配好串口等。 2.运行 socat TCP-LISTEN:5555,fork /tmp/ttyS1 & , 链接到vmware对外的串口文件
  2. gdb vmlinux
  3. 在GDB中: (gdb) target remote 0:5555 就可以进入调试状态了
  4. (gdb) c ,则target进入Linux系统

KGDB--Cannot insert breakpoint

原因:

内核编译选项CONFIG_DEBUG_RODATA,会对kernel text做write protect。 那么kgdb就不能设置断点了。

解决方法是:

编辑kernel source目录下生成的.config文件, 禁用CONFIG_DEBUG_RODATA=n (read only data)重新编译即可


http://www.mail-archive.com/kgdb-bugreport@lists.sourceforge.net/msg03464.html

Hi Folks,

I’m wondering if anyone has had issues with setting breakpoints. I’m
able to break into the kernel, access data, do a backtrace, etc, but
when I attempt to set a breakpoint, then continue, I get the following error:

Cannot insert breakpoint 1.
Error accessing memory address 0xffffffff81310931: Unknown error 4294967295.

I’m attaching a sample session, I had set remote debug to 1

Thanks!
Pat Thomson

Hi Thomson,

It seems that your problem is the CONFIG_DEBUG_RODATA option was
enabled, It is recommend to turn CONFIG_DEBUG_RODATA off when using kgdb.

From the kgdb document(DocBook/kgdb.tmpl):

If the architecture that you are using supports the kernel option
CONFIG_DEBUG_RODATA, you should consider turning it off.  This
option will prevent the use of software breakpoints because it
marks certain regions of the kernel's memory space as read-only.
If kgdb supports it for the architecture you are using, you can
use hardware breakpoints if you desire to run with the
CONFIG_DEBUG_RODATA option turned on, else you need to turn off
this option.

Thanks, Dongdong

Connecting Two Virtual Machines

You can set up the virtual serial ports in two virtual machines to connect to each other. This is useful, for example, if you want to use an application in one virtual machine to capture debugging information sent from the other virtual machine’s serial port.

To install a direct serial connection between two virtual machines (a server and a client), take the following steps:

Windows Host In the server virtual machine

  1. Open the virtual machine settings editor (VM > Settings).
  2. Click Add to start the Add Hardware Wizard.
  3. Select Serial Port, then click Next.
  4. Select Output to named pipe, then click Next.
  5. Use the default pipe name, or enter another pipe name of your choice. The pipe name must follow the form \.\pipe\ — that is, it must begin with \.\pipe.
  6. Select This end is the server.
  7. Select The other end is a virtual machine.
  8. By default, the device status setting is Connect at power on. You may deselect this setting if you wish. Click Advanced if you want to configure this serial port to use polled mode. This option is of interest primarily to developers who are using debugging tools that communicate over a serial connection. For more information, see Special Configuration Options for Advanced Users.
  9. Click Finish, then click OK to close the virtual machine settings editor.

In the client virtual machine

  1. Open the virtual machine settings editor (VM > Settings).
  2. Click Add to start the Add Hardware Wizard.
  3. Select Serial Port, then click Next.
  4. Select Use named pipe.
  5. Use the default name, or enter another pipe name of your choice. The pipe name must follow the form \.\pipe\ — that is, it must begin with \.\pipe. The pipe name must be the same on both server and client.
  6. Select This end is the client.
  7. Select The other end is a virtual machine.
  8. By default, the device status setting is Connect at power on. You may deselect this setting if you wish. Click Advanced if you want to configure this serial port to use polled mode. This option is of interest primarily to developers who are using debugging tools that communicate over a serial connection. For more information, see Special Configuration Options for Advanced Users.
  9. Click Finish, then click OK to close the virtual machine settings editor.

Linux Host In the server virtual machine

  1. Open the virtual machine settings editor (VM > Settings).
  2. Click Add to start the Add Hardware Wizard.
  3. Select Serial Port, then click Next.
  4. Select Output to named pipe, then click Next.
  5. In the Path field, enter /tmp/ or another Unix socket name of your choice.
  6. Select This end is the server.
  7. Select The other end is a virtual machine.
  8. By default, the device status setting is Connect at power on. You may deselect this setting if you wish. Click Advanced if you want to configure this serial port to use polled mode. This option is of interest primarily to developers who are using debugging tools that communicate over a serial connection. For more information, see Special Configuration Options for Advanced Users.
  9. Click Finish, then click OK to save your configuration and close the virtual machine settings editor.

In the client virtual machine

  1. Open the virtual machine settings editor (VM > Settings).
  2. Click Add to start the Add Hardware Wizard.
  3. Select Serial Port, then click Next.
  4. Select Output to named pipe, then click Next.
  5. In the Path field, enter /tmp/ or another Unix socket name of your choice. The pipe name must be the same on both server and client.
  6. Select This end is the client.
  7. Select The other end is a virtual machine.
  8. By default, the device status setting is Connect at power on. You may deselect this setting if you wish. Click Advanced if you want to configure this serial port to use polled mode. This option is of interest primarily to developers who are using debugging tools that communicate over a serial connection. For more information, see Special Configuration Options for Advanced Users.
  9. Click Finish, then click OK to save your configuration and close the virtual machine settings editor.

vmware 串口调试

在系统内核开发中,经常会用到串口调试,利用VMware的Virtual Machine更是为调试系统内核如虎添翼。那么怎么搭建串口调试环境呢?

Virtual Machine 主要有三种串口调试技术,可以在serial port的配置界面找到:

  1. Use physical serial port 即使用物理机串口,当用串口线盒另一台电脑连接时,就用这种方式
  2. Use output file 即把串口数据输出到宿主机某文件中,当只需要看输出结果的时候可以用这种方式,简单方便
  3. Use named pipe 把串口输出到命名管道,命名管道可读可写,也就可以交互,进行一些debug工作,而不是简单的看结果

因为前两种相对简单易用就不具体介绍了,这里主要说第三种用命名管道调试方法。命名管道,在Linux中是进程间通信(IPC)的一种方式,两个进 程可以通过读写管道来交换数据。这里就是运用了这种技术,通过把串口绑定一个有名管道,对有名管道的读写交换串口数据。也有两种方式:1. 宿主机与虚拟机之间, 2. 在同一宿主机上,两虚拟机间通过绑定同一个宿主机上的有名管道。问题的关键在于如何把虚拟机串口绑定到宿主机的某一有名管道,而第一种方式则需要找到一种 方式使得主机如何读写有名管道来交互,经过一阵Google终于找到分别在Linux和Windows分别试验成功的工具。

在Windows中有名管道式通过路径//./pipe/namedpipe来创建的,当然你可以指定到其他机子如//192.168.1.10 /pipe/namedpipe,而在Linux中,/tmp/mypipe就可以了。创建好有名管道后,就是如何和管道交互了。目前,无论是 Windows还是Linux,似乎都没有一款工具可以直接读写有名管道的,而我找到的两个工具都是通过把有名管道和Socket绑定,通过读写 Socket来间接读写管道。

下面我就简要介绍一下在Windows和Linux下如何配置:

Linux Host:
Host ~ Virtual Machine
1
2
3
4
5
6
7
8
9
10
11
1. configure VM
a. add hardware -> Serial port
b. using named pipe
c. /tmp/isocket
d. this end is server & far end is application
e. check Yield CPU on poll
f. start Virtual Machine

2. socat /tmp/isocket tcp4-listen:9001 &
/tmp/socket: VMware call it 'named piped', actually it is Unix Daemon Socket, so you shouldn't use pipe:/tmp/socket
3. telnet 127.0.0.1 9001

Trouble Shoot: 有时候会遇到错误Connection closed by foreign host,或者telnet一开,socat就能退出,很可能是你没power on虚拟机,有名管道还没创建,你就socat,这样也会创建一个名为isocket的文件但只是普通文件。具体的细节请看socat help
start Virtual Machine first, than run the socat, and telnet
(Note you must have permission to all resource, /tmp/socket, VM and so on)

Vritual Machine ~ Virtual Machine
1
2
3
4
5
6
7
8
9
10
11
12
13
1. configure VM
a. add hardware --> serial port
b. Using named pipe, configure /tmp/isocket 
c.  this end is server & far end is Virtual Machine
d. check Yield CPU on poll
e. start VM

2. Another VM
a. add hardware  -->  Serial Port
b. Using named pipe, configure /tmp/isocket 
c.  this end is client & far end is Virtual Machine 
d. check Yield CPU on poll
e. start VMs
Windows Host:
Host ~ Virtual Machine
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
1. configure VM
a. add hardware --> serial port
b. using named pipe
c. //./pipe/vmwaredebug
d. this end is client & far end is application
e. check Yield CPU on poll

2. using 3rd-party tool to communicate with named pipe
a. down the tool 
b. install service
cmd>vmwaregateway.exe /r
c. start service
c:/> net start vmwaregateway
d. telnet 127.0.0.1 567
3. start Virtual Machine

如果你使用的是vmwaregateway.exe这个小工具,这里的管道名就必须是vmwaredebug,除非你把它的源代码download下来自己改改。

Vritual Machine ~ Virtual Machine
1
2
3
4
5
6
7
8
9
10
11
12
13
1. configure VM
a. add hardware --> serial port
b. Using named pipe, //./pipe/vmwaredebug
c.  this end is server & far end is Virtual Machine
d. check Yield CPU on poll
e. start VM

2. Another VM
a. add hardware  -->  Serial Port
b. Using named pipe, //./pipe/vmwaredebug
c.  this end is client & far end is Virtual Machine 
d. check Yield CPU on poll
e. start VMs

独立的调试符号文件

这种将可执行程序与调试符号分离的方案好处多多。一方面,缩减了可执行程序的文件大小,在一定程度上提高了程序的执行性能,另一方面,对应的调试符号文件也方便了一些不时之需。本文就来看一下与此相关的两个问题。

一 如何给应用程序创建对应的调试符号文件?

这很简单,看个演示实例。有代码如下:

1
2
3
4
5
6
7
[root@lenky gdb]# cat t.c
#include <stdio.h> 
int main(int argc, char *argv[])
{
  printf("Hello world!\n");
  return 0;
}

依次执行命令如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@lenky gdb]# ls -l
total 4
-rw-r--r--. 1 root root 103 Mar 20 07:52 t.c
[root@lenky gdb]# gcc -g t.c -o t
[root@lenky gdb]# ls -l
total 12
-rwxr-xr-x. 1 root root 7717 Mar 20 07:58 t
-rw-r--r--. 1 root root  103 Mar 20 07:52 t.c
[root@lenky gdb]# objcopy --only-keep-debug t t.debuginfo
[root@lenky gdb]# objcopy --strip-debug t
[root@lenky gdb]# objcopy --add-gnu-debuglink=t.debuginfo t
[root@lenky gdb]# ls -l
total 20
-rwxr-xr-x. 1 root root 6470 Mar 20 07:58 t
-rw-r--r--. 1 root root  103 Mar 20 07:52 t.c
-rwxr-xr-x. 1 root root 6109 Mar 20 07:58 t.debuginfo

OK,可执行程序t和对应的调试符号文件t.debuginfo就生成了。

几条命令,给以分别解释一下:
1. gcc -g t.c -o t
这个无需多说,值得注意的是,-g和-O2可以同时使用。
2. objcopy –only-keep-debug t t.debuginfo
将可执行程序文件t内的与调试相关的信息拷贝到文件t.debuginfo内。也可以这样:
cp t t.debuginfo
strip –only-keep-debug t.debuginfo
3. objcopy –strip-debug t
删除可执行程序文件t内的调试相关信息。也可以直接使用strip命令,不过strip命令会把symtab也删除掉,导致在没有debuginfo文件的情况下,打印堆栈信息会受到影响,比如变得不那么清晰。
4. objcopy –add-gnu-debuglink=t.debuginfo t
在可执行程序文件t内添加一个名为.gnu_debuglink的section段,该段内包含有debuginfo文件的name名称和checksum校验和,以确保后续在进行实际调试时,可执行程序和对应的调试符号文件是一致的。

二 如何使用gdb调试带有调试符号文件的应用程序?

其实想想也知道,这只需解决一个问题,即如何把应用程序与调试符号文件关联起来。
gdb会按照一定的规则去搜索对应路径,找寻应用程序的调试符号文件,比如gdb会自动查找可执行程序所在目录下的.debug文件夹:

1
2
3
4
5
6
[root@lenky ~]# pwd
/root
[root@lenky ~]# gdb /home/work/gdb/t -q
Reading symbols from /home/work/gdb/t...Reading symbols from /home/work/gdb/.debug/t.debug...done.
done.
(gdb)

把调试符号文件放到同一个目录也可以:

1
2
3
4
5
6
[root@lenky ~]# rm -fr /tmp/.debug/
[root@lenky ~]# cp /home/work/gdb/.debug/t.debug /tmp/
[root@lenky ~]# gdb /tmp/t -q
Reading symbols from /tmp/t...Reading symbols from /tmp/t.debug...done.
done.
(gdb)

下面再介绍另外几种主动设置方法:

1,通过gdb启动参数-s指定:
1
2
3
[root@lenky ~]# gdb -s /home/work/gdb/.debug/t.debug -e /tmp/t -q
Reading symbols from /home/work/gdb/.debug/t.debug...done.
(gdb)

注意:可执行程序必须通过-e指定,否则貌似gdb会拿它覆盖-s参数,比如如下:

1
2
3
4
5
[root@lenky ~]# gdb -s /home/work/gdb/.debug/t.debug /tmp/t -q
Reading symbols from /tmp/t...Missing separate debuginfo for /tmp/t
Try: yum --disablerepo='*' --enablerepo='*-debuginfo' install /usr/lib/debug/.build-id/01/f1df7f4971caacd934aca9523c4e4b5ae95332.debug
(no debugging symbols found)...done.
(gdb)

可以看到,gdb直接尝试从文件/tmp/t内读取符号了,而不是文件t.debug。

2,利用gdb的命令设置搜索路径:set debug-file-directory directories

这是gdb官方文档提到的,可以设置搜索路径的命令,但是貌似并没有起作用,或者是我漏掉了什么。具体不说了,请看参考5。

参考:

  1. How to generate gcc debug symbol outside the build target?
    http://stackoverflow.com/questions/866721/how-to-generate-gcc-debug-symbol-outside-the-build-target
  2. Creating separate debug info
    https://blogs.oracle.com/dbx/entry/creating_separate_debug_info
  3. man objcopy
  4. .gnu_debuglink or Debugging system libraries with source code
    https://blogs.oracle.com/dbx/entry/gnu_debuglink_or_debugging_system
  5. 18.2 Debugging Information in Separate Files
    http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html
  6. http://www.technovelty.org/code/separate-debug-info.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
#!/bin/sh
# 可以改成:将要提取的文件加到参数中

#set -x

objcopyout()
{
#       if [ -f `file -N $1 | sed -n -e 's/^\(.*\):[   ]*.*ELF.*, not stripped/\1/p'` ]; then
	    val=`objdump -s -j .gnu_debuglink "$1" | grep 'Contents of section .gnu_debuglink:'`
	    if [ -z "$val" ]; then
	            #debug_out=".debug/$1.debug".`date +%s`
	            tmp_name=`echo "$1" | awk -F/ '{ print $NF}'`
	            debug_out=".debug/$tmp_name.debug".`date +%s`
	            objcopy --only-keep-debug "$1" "$debug_out"
	            objcopy --strip-debug "$1"
	            objcopy --add-gnu-debuglink="$debug_out" "$1"
	    fi
#       objdump -s -j .gnu_debuglink "$1"
}

mkdir -p .debug/

while [ $# -gt 0 ]; do
	    if [ -f "$1" ]; then
	            objcopyout "$1"
	    fi
	    shift
done

#find . -name *.ko -print |
#while read f
#do
#       objcopyout "$f"
#done