kk Blog —— 通用基础

• ‘?’ means that the information about this stack entry is probably not reliable.

The stack output mechanism (see the implementation of dump_trace() function) was unable to prove that the address it has found is a valid return address in the call stack.

‘?’ itself is output by printk_stack_address().

The stack entry may be valid or not. Sometimes one may simply skip it. It may be helpful to investigate the disassembly of the involved module to see which function is called at ClearFunctionName+0x88 (or, on x86, immediately before that position).

Concerning reliability

On x86, when dump_stack() is called, the function that actually examines the stack is print_context_stack() defined in arch/x86/kernel/dumpstack.c. Take a look at its code, I’ll try to explain it below.

I assume DWARF2 stack unwind facilities are not available in your Linux system (most likely, they are not, if it is not OpenSUSE or SLES). In this case, print_context_stack() seems to do the following.

It starts from an address (‘stack’ variable in the code) that is guaranteed to be an address of a stack location. It is actually the address of a local variable in dump_stack().

The function repeatedly increments that address (while (valid_stack_ptr …) { … stack++}) and checks if what it points to could also be an address in the kernel code (if (__kernel_text_address(addr)) …). This way it attempts to find the functions' return addresses pushed on stack when these functions were called.

Of course, not every unsigned long value that looks like a return address is actually a return address. So the function tries to check it. If frame pointers are used in the code of the kernel (%ebp/%rbp registers are employed for that if CONFIG_FRAME_POINTER is set), they can be used to traverse the stack frames of the functions. The return address for a function lies just above the frame pointer (i.e. at %ebp/%rbp + sizeof(unsigned long)). print_context_stack checks exactly that.

If there is a stack frame for which the value ‘stack’ points to is the return address, the value is considered a reliable stack entry. ops->address will be called for it with reliable == 1, it will eventually call printk_stack_address() and the value will be output as a reliable call stack entry. Otherwise the address will be considered unreliable. It will be output anyway but with ‘?’ prepended.

[NB] If frame pointer information is not available (e.g. like it was in Debian 6 by default), all call stack entries will be marked as unreliable for this reason.

The systems with DWARF2 unwinding support (and with CONFIG_STACK_UNWIND set) is a whole another story.

1.强制不内联

一个函数，如果代码量比较少的话，用 -O3优化开关的话，gcc有可能将这个函数强制内联(inline)即使，你在函数前没有写inline助记符。
如果是一个手写汇编的函数，那样的话很有可能破坏参数。gcc里有强制不内联的，用法如下

1

void foo() __attribute__((noinline));

但是有的gcc可能会忽略 noinline。
那么你可以将你实现的这个函数写到调用函数之后，就不会被inline了。这是因为编译器gcc只内联当前函数之前可见(实现代码在前)的函数。

2.优化时无法识别inline函数中的ASM汇编

当GCC尝试内联一个函数时，如果该函数中存在内联汇编，则该汇编语句块可能被丢弃；

1
2
3
4
5
6
7
8
9
10
11
12
13

__inline__ __attribute__((always_inline))int Increment(int volatile *add, int inc)
{
    int res;
    __asm__
    (
    "lock \n\t"
    "xaddl %0,(%1)\n\t"
    :"=r"(res)
    :"r"(add),"0"(inc)
    :"memory"
    );
    return res;
}

原因：

http://vault.centos.org/5.11/os/SRPMS/kexec-tools-1.102pre-165.el5.src.rpm
这个包的一个patch(kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch)是为了得到dmesg的，
但是它判断dmesg的结束是用logged_chars(看kernel/printk.c)，logged_chars应该是输出的结束，所以不对。
改成log_end就行，

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

diff --git a/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch b/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch
index 3938280..76c402a 100644
--- a/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch
+++ b/kexec-tools-1.102pre-makedumpfile-dump-dmesg.patch
@@ -68,7 +68,7 @@ diff -up kexec-tools-testing-20070330/makedumpfile/makedumpfile.c.orig kexec-too
 +dump_dmesg()
 +{
 +      int log_buf_len, length_log, length_oldlog, ret = FALSE;
-+      unsigned long log_addr, logged_chars, index;
++      unsigned long log_addr, logged_chars, log_end, index;
 +      char *log_buffer = NULL;
 +
 +      if (!open_files_for_creating_dumpfile())
@@ -101,10 +101,15 @@ diff -up kexec-tools-testing-20070330/makedumpfile/makedumpfile.c.orig kexec-too
 +               printf("Failed to get logged_chars.\n");
 +               return FALSE;
 +      }
++      if (!readmem(VADDR, SYMBOL(log_end), &log_end, sizeof(log_end))) {
++               printf("Failed to get log_end.\n");
++               return FALSE;
++      }
 +      DEBUG_MSG("\n");
 +      DEBUG_MSG("log_addr     : %lx\n", log_addr);
 +      DEBUG_MSG("log_buf_len  : %d\n", log_buf_len);
 +      DEBUG_MSG("logged_chars : %ld\n", logged_chars);
++      DEBUG_MSG("log_end      : %ld\n", log_end);
 +
 +      if ((log_buffer = malloc(log_buf_len)) == NULL) {
 +               ERRMSG("Can't allocate memory for log_buf. %s\n",
@@ -112,21 +117,16 @@ diff -up kexec-tools-testing-20070330/makedumpfile/makedumpfile.c.orig kexec-too
 +               return FALSE;
 +       }
 +
-+      if (logged_chars < log_buf_len) {
++      if (log_end < log_buf_len) {
 +               index = 0;
-+               length_log = logged_chars;
++               length_log = log_end;
 +
 +               if(!readmem(VADDR, log_addr, log_buffer, length_log)) {
 +                        printf("Failed to read dmesg log.\n");
 +                        goto out;
 +               }
 +      } else {
-+               if (!readmem(VADDR, SYMBOL(log_end), &index, sizeof(index))) {
-+                        printf("Failed to get log_end.\n");
-+                        goto out;
-+               }
-+               DEBUG_MSG("log_end      : %lx\n", index);
-+               index &= log_buf_len - 1;
++               index = log_end & (log_buf_len - 1);
 +               length_log = log_buf_len;
 +               length_oldlog = log_buf_len - index;
 +

如果不修改上面bug，kdump得到vmcore后用 makedumpfile –dump-dmesg 无法解得dmesg，补救办法如下：

kdump得到vmcore后
1、vmlinux没有debuginfo，crash不能运行
2、makedumpfile -F –dump-dmesg vmcore > dmesg 只能显示开头一下部分dmesg （不懂为什么）
解决：

方法一、

通过/boot/System.map 或者 /proc/kallsyms 找到 log_buf 地址，例如 0xffffffff81a9ac30

1
2
3
4
5
6
7
8
9
10
11
12
13
14

gdb vmlinux vmcore

set print repeats 100
set print elements 0
set logging file XXX
set pagination off
set logging on
p {char*} 0xffffffff81a9ac30
quit

---

vi XXX
:%s/\\n/\r/g

方法二、

是另一命令，但不好用

1
2
3
4
5
6
7
8
9
10
11

cat /proc/kallsyms | grep log_end
ffffffff81e30de0 b log_end

x/1dw 0xffffffff81e30de0
0xffffffff81e30de0:     85689

x/1xg 0xffffffff81a9ac30
0xffffffff81a9ac30:     0xffffffff81e30ee0

显示最后4000字符
x/5s 0xffffffff81e30ee0+85689-4000