How to use LLDB to debug the multi-thread progarm

Posted on

Debug is very difficult when you develop a multi-thread program.

This article will show you how to use LLDB to do it.(the example is TinyWebServer)

about the project

TinyWebServer will create A acceptor thread,2 worker thread(or you can set the number of worker in web.json) and a ThreadPool which contains some threads(the number of threads is equal to the number of CPUs)

1. Set the breakpoint

First, you should set the breakpoint

the command of lldb to set breakpoint is:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
1.use line

breakpoint set -file [filename] -line [line number]
br s -f [filename] -l [line number]

2.use name

breakpoint set -file [filename] -name [method/function name]
br s -f [filename] -n [method/function name]

3.set pointer for this file

break [line number]
break [method/function name]

First, we can set a breakpoint for acceptor, just like this

1
br s -f kqueue_net_event.cpp -n KqueueAcceptEvent::Handle

and then, run the Server and connect to localhost:10086

the debugger will stop on the breakpoint, now the TCP request is waitting for handling

2.Get the information of threads

Now, we input continue to make the Server to continue with running

when we debug the multi-thread program, we want to know the status of threads

so, we can use thread list to get the information of threads

1
2
3
4
5
6
7
8
9
10
(lldb) thread list
Process 87572 stopped
* thread #1: tid = 0x45163f, 0x00007fff592f99de libsystem_kernel.dylib`__ulock_wait + 10, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  thread #2: tid = 0x451654, 0x00007fff592fd78e libsystem_kernel.dylib`kevent + 10
  thread #3: tid = 0x451652, 0x00007fff592fd78e libsystem_kernel.dylib`kevent + 10
  thread #4: tid = 0x451653, 0x00007fff592fd78e libsystem_kernel.dylib`kevent + 10
  thread #5: tid = 0x451655, 0x00007fff592fa86a libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #6: tid = 0x451656, 0x00007fff592fa86a libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #7: tid = 0x451657, 0x00007fff592fa86a libsystem_kernel.dylib`__psynch_cvwait + 10
  thread #8: tid = 0x451658, 0x00007fff592fa86a libsystem_kernel.dylib`__psynch_cvwait + 10

we can see the #1 is main thread, and #2,#3,#4 is calling the kevent function,#5,6,7,8 is waitting

so, we can guess #2,#3,#4 are a acceptor thread and 2 worker threads, #5,6,7,8 are the threads which in ThreadPool

3.Get the information of bug

When we upload a big file to the server, server will crash(This is a bug)

1
2
3
4
5
6
7
8
9
Process 87773 stopped
* thread #6, stop reason = EXC_BAD_ACCESS (code=2, address=0x7000006ea000)
    frame #0: 0x000000010005652a Server`FastCgi::makeNameValueBody(this=0x00007000006e7b80, name="file", nameLen=4, value="��\x11ࡱ\x1a�, valueLen=37378, bodyBuffPtr="", bodyLenPtr=0x00007000006e7464) at fcgi.cpp:115:24
   113 	    for(auto ch : value)
   114 	    {
-> 115 	        *bodyBuffPtr++ = ch;
   116 	    }
   117
Target 0: (Server) stopped.

now, we can see the thread#6 has crashed, from information we know

  • the code is runned by ThreadPool
  • error: EXC_BAD_ACCESS