kk Blog —— 通用基础

• ‘?’ means that the information about this stack entry is probably not reliable.

The stack output mechanism (see the implementation of dump_trace() function) was unable to prove that the address it has found is a valid return address in the call stack.

‘?’ itself is output by printk_stack_address().

The stack entry may be valid or not. Sometimes one may simply skip it. It may be helpful to investigate the disassembly of the involved module to see which function is called at ClearFunctionName+0x88 (or, on x86, immediately before that position).

Concerning reliability

On x86, when dump_stack() is called, the function that actually examines the stack is print_context_stack() defined in arch/x86/kernel/dumpstack.c. Take a look at its code, I’ll try to explain it below.

I assume DWARF2 stack unwind facilities are not available in your Linux system (most likely, they are not, if it is not OpenSUSE or SLES). In this case, print_context_stack() seems to do the following.

It starts from an address (‘stack’ variable in the code) that is guaranteed to be an address of a stack location. It is actually the address of a local variable in dump_stack().

The function repeatedly increments that address (while (valid_stack_ptr …) { … stack++}) and checks if what it points to could also be an address in the kernel code (if (__kernel_text_address(addr)) …). This way it attempts to find the functions' return addresses pushed on stack when these functions were called.

Of course, not every unsigned long value that looks like a return address is actually a return address. So the function tries to check it. If frame pointers are used in the code of the kernel (%ebp/%rbp registers are employed for that if CONFIG_FRAME_POINTER is set), they can be used to traverse the stack frames of the functions. The return address for a function lies just above the frame pointer (i.e. at %ebp/%rbp + sizeof(unsigned long)). print_context_stack checks exactly that.

If there is a stack frame for which the value ‘stack’ points to is the return address, the value is considered a reliable stack entry. ops->address will be called for it with reliable == 1, it will eventually call printk_stack_address() and the value will be output as a reliable call stack entry. Otherwise the address will be considered unreliable. It will be output anyway but with ‘?’ prepended.

[NB] If frame pointer information is not available (e.g. like it was in Debian 6 by default), all call stack entries will be marked as unreliable for this reason.

The systems with DWARF2 unwinding support (and with CONFIG_STACK_UNWIND set) is a whole another story.

1.强制不内联

一个函数，如果代码量比较少的话，用 -O3优化开关的话，gcc有可能将这个函数强制内联(inline)即使，你在函数前没有写inline助记符。
如果是一个手写汇编的函数，那样的话很有可能破坏参数。gcc里有强制不内联的，用法如下

1

void foo() __attribute__((noinline));

但是有的gcc可能会忽略 noinline。
那么你可以将你实现的这个函数写到调用函数之后，就不会被inline了。这是因为编译器gcc只内联当前函数之前可见(实现代码在前)的函数。

2.优化时无法识别inline函数中的ASM汇编

当GCC尝试内联一个函数时，如果该函数中存在内联汇编，则该汇编语句块可能被丢弃；

1
2
3
4
5
6
7
8
9
10
11
12
13

__inline__ __attribute__((always_inline))int Increment(int volatile *add, int inc)
{
    int res;
    __asm__
    (
    "lock \n\t"
    "xaddl %0,(%1)\n\t"
    :"=r"(res)
    :"r"(add),"0"(inc)
    :"memory"
    );
    return res;
}

原因：

http://vault.centos.org/5.11/os/SRPMS/kexec-tools-1.102pre-165.el5.src.rpm
这个包的一个patch(kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch)是为了得到dmesg的，
但是它判断dmesg的结束是用logged_chars(看kernel/printk.c)，logged_chars应该是输出的结束，所以不对。
改成log_end就行，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

diff --git a/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch b/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch
index 3938280..76c402a 100644
--- a/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch
+++ b/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch
@@ -68,7 +68,7 @@ diff -up kexec-tools-testing-20070330/makedumpfile/makedumpfile.c.orig kexec-too
 +dump_dmesg()
 +{
 +      int log_buf_len, length_log, length_oldlog, ret = FALSE;
-+      unsigned long log_addr, logged_chars, index;
++      unsigned long log_addr, logged_chars, log_end, index;
 +      char *log_buffer = NULL;
 +
 +      if (!open_files_for_creating_dumpfile())
@@ -101,10 +101,15 @@ diff -up kexec-tools-testing-20070330/makedumpfile/makedumpfile.c.orig kexec-too
 +               printf("Failed to get logged_chars.\n");
 +               return FALSE;
 +      }
++      if (!readmem(VADDR, SYMBOL(log_end), &log_end, sizeof(log_end))) {
++               printf("Failed to get log_end.\n");
++               return FALSE;
++      }
 +      DEBUG_MSG("\n");
 +      DEBUG_MSG("log_addr     : %lx\n", log_addr);
 +      DEBUG_MSG("log_buf_len  : %d\n", log_buf_len);
 +      DEBUG_MSG("logged_chars : %ld\n", logged_chars);
++      DEBUG_MSG("log_end      : %ld\n", log_end);
 +
 +      if ((log_buffer = malloc(log_buf_len)) == NULL) {
 +               ERRMSG("Can't allocate memory for log_buf. %s\n",
@@ -112,21 +117,16 @@ diff -up kexec-tools-testing-20070330/makedumpfile/makedumpfile.c.orig kexec-too
 +               return FALSE;
 +       }
 +
-+      if (logged_chars < log_buf_len) {
++      if (log_end < log_buf_len) {
 +               index = 0;
-+               length_log = logged_chars;
++               length_log = log_end;
 +
 +               if(!readmem(VADDR, log_addr, log_buffer, length_log)) {
 +                        printf("Failed to read dmesg log.\n");
 +                        goto out;
 +               }
 +      } else {
-+               if (!readmem(VADDR, SYMBOL(log_end), &index, sizeof(index))) {
-+                        printf("Failed to get log_end.\n");
-+                        goto out;
-+               }
-+               DEBUG_MSG("log_end      : %lx\n", index);
-+               index &= log_buf_len - 1;
++               index = log_end & (log_buf_len - 1);
 +               length_log = log_buf_len;
 +               length_oldlog = log_buf_len - index;
 +

如果不修改上面bug，kdump得到vmcore后用 makedumpfile –dump-dmesg 无法解得dmesg，补救办法如下：

kdump得到vmcore后
1、vmlinux没有debuginfo，crash不能运行
2、makedumpfile -F –dump-dmesg vmcore > dmesg 只能显示开头一下部分dmesg （不懂为什么）
解决：

方法一、

通过/boot/System.map 或者 /proc/kallsyms 找到 log_buf 地址，例如 0xffffffff81a9ac30

1
2
3
4
5
6
7
8
9
10
11
12
13
14

gdb vmlinux vmcore

set print repeats 100
set print elements 0
set logging file XXX
set pagination off
set logging on
p {char*} 0xffffffff81a9ac30
quit

---

vi XXX
:%s/\\n/\r/g

方法二、

是另一命令，但不好用

1
2
3
4
5
6
7
8
9
10
11

cat /proc/kallsyms | grep log_end
ffffffff81e30de0 b log_end

x/1dw 0xffffffff81e30de0
0xffffffff81e30de0:     85689

x/1xg 0xffffffff81a9ac30
0xffffffff81a9ac30:     0xffffffff81e30ee0

显示最后4000字符
x/5s 0xffffffff81e30ee0+85689-4000

一、如果有函数调用，他会进入该函数

1

step <count>

单步跟踪，如果有函数调用，他会进入该函数。进入函数的前提是，此函数被编译有 debug信息。很像 VC等工具中的 step in。后面可以加 count也可以不加，不加表示一条条地执行，加表示执行后面的 count条指令，然后再停住。

二、如果有函数调用，他不会进入该函数

1

next <count>

同样单步跟踪，如果有函数调用，他不会进入该函数。很像 VC等工具中的 step over。后面可以加 count也可以不加，不加表示一条条地执行，加表示执行后面的 count条指令，然后再停住。

三、无debuginfo时需要set step-mode [on/off]

1
2
3
4
5
6

set step-mode [on/off]
set step-mode on
#打开 step-mode模式，于是，在进行单步跟踪时，程序不会因为没有 debug信息而不停住。这个参数有很利于查看机器码。

set step-mode off
关闭 step-mode模式。

四、跟踪子进程

使用gdb调试的时候，gdb只能跟踪一个进程。

可以在fork函数调用之前，通过指令设置gdb调试工具跟踪父进程或子进程。

默认情况下gdb是跟踪父进程的。

1
2

set follow-fork-mode child   # 命令设置gdb在fork之后跟踪子进程。
set follow-fork-mode parent  # 设置跟踪父进程。

五、修改值

1
2
3
4
5
6

set $rax=0   # 寄存器

set m=6       # 变量，需要debuginfo

p $rsp+0x714  # 内存
set *0x7f962c1fa564 = 0x6

可以使用examine命令(简写是x)来查看内存地址中的值。x命 令的语 法如下所示：

1

x/<n/f/u> <addr>  

n、f、u是可选的参数。
n是一个正整数，表示需要显示的内存单元的个数， 也就是说从当前地址向后显示几个 内存单元的内容，一个内存单元的大小由后面的u定义。
f 表示显示的格式，参见下面。如果地址所指的是字符串，那么格式可以是s，如果地十是指令地址，那么格式可以是i。
u 表示从当前地址往后请求的字节数，如果不指定的话，GDB默认是4个bytes。u参数可以用下面的字符来代替，b表示单字节，h表示双字节，w表示四字 节，g表示八字节。当我们指定了字节长度后，GDB会从指内存定的内存地址开始，读写指定字节，并把其当作一个值取出来。
表示一个内存地址。

注意：严格区分n 和u的关系，n表示单元个数，u表示每个单元的大小。

n/f/u三个参数可以一起使用。例如：
命令：x/3uh 0x54320 表示，从内存地址0x54320读取内容，h表示以双字节为一个单位，3表示输出三个单位，u表示按十六进制显示。

输出格式

一般来说，GDB会根据变量的类型输出变量的值。但你也可以自定义GDB的输出的格式。例如，你想输出一个整数的十六进制，或是二进制来查看这个整型变量 的中的位的情况。要做到这样，你可以使用GDB的数据显示格式：

1
2
3
4
5
6
7
8

x 按十六进制格式显示变量。
d 按十进制格式显示变量。
u 按十六进制格式显示无符号整型。
o 按八进制格式显示变量。
t 按二进制格式显示变量。
a 按十六进制格式显示变量。
c 按字符格式显示变量。
f 按浮点数格式显示变量。

(gdb) help x Examine memory: x/FMT ADDRESS. ADDRESS is an expression for the memory address to examine. FMT is a repeat count followed by a format letter and a size letter. Format letters are o(octal), x(hex), d(decimal), u(unsigned decimal), t(binary), f(float), a(address), i(instruction), c(char) and s(string). Size letters are b(byte), h(halfword), w(word), g(giant, 8 bytes). The specified number of objects of the specified size are printed according to the format.

Defaults for format and size letters are those previously used. Default count is 1. Default address is following last thing printed with this command or “print”.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

(gdb) p f1
$4 = 8.25
(gdb) p f2
$5 = 125.5
(gdb) x/x &f1
0xbffff380:   0x41040000
(gdb) x/xw &f1
0xbffff380:   0x41040000
(gdb) x/xw &f2
0xbffff384:   0x42fb0000
(gdb) x/2xw &f1
0xbffff380:   0x41040000   0x42fb0000
(gdb) x/4xw &f1
0xbffff380:   0x41040000   0x42fb0000   0xbffff408   0x00bcba66
(gdb) x/tw &f1
0xbffff380:   01000001000001000000000000000000
(gdb) x/2tw &f1
0xbffff380:   01000001000001000000000000000000   01000010111110110000000000000000
(gdb) p record
$10 = {12, 76, 48, 62, 94, 17, 32, 37, 52, 69}
(gdb) p &record
$11 = (int (*)[10]) 0x8049740
(gdb) x/4uw 0x8049740
0x8049740 <record>:   12   76   48   62
(gdb) x/6ow 0x8049740
0x8049740 <record>:   014   0114   060   076
0x8049750 <record+16>:   0136   021