gdb macros

gdb provides an amazingly versatile debugging platform supporting mechanisms to debug local and remote software.

One of gdb's less known features is its rich macro language. Although I had used it before working at Apple, that's where I was really exposed to how gdb can be used to the max.

OSX can be debugged from a remote machine via gdb remote. Navigating around the guts of a running OS is challenging. You need to locate the task structures etc that you are debugging by finding them in lists of running tasks. Doing this manually takes time and is error prone. So to make this easy, Apple provides a file of gdb macros that will do this for you.

While these macros won't work on your embedded system, they do provide examples for how you can construct complex macros containing ifs, while loops etc.

To give you a taster, here's a very simple example using a linked list of structures.

#include
#include

struct enu{
float x;
float y;
float z;
struct enu *next;
};

struct enu *addpoint(float x, float y, float z, struct enu *next)
{
struct enu * np;
np = malloc(sizeof(struct enu));
if(np){
np->x = x;
np->y = y;
np->z = z;
np->next = next;
}
return np;
}

main()
{
struct enu *list = NULL;
struct enu *i;

list = addpoint(1,2,3,list);
list = addpoint(1,2,4,list);
list = addpoint(1,2,5,list);
list = addpoint(1,2,6,list);

for(i = list; i; i=i->next)
printf("%f %f %f\n",i->x,i->y,i->z);
}

Here are a few macros

define dump_entry
set $l=(struct enu *)($arg0)
printf "Entry is %f %f %f\n", $l->x, $l->y, $l->z
end
document dump_entry
Print out a list entry
Usage: dump_entry pointer
end

define dump_list
set $list = $arg0
if(!$list)
printf "List is empty\n"
else
while($list)
dump_entry $list
set $list=$list->next
end
printf "End of list\n"
end
end
document dump_list
Print out a whole list
Usage: dump_list pointer
end

And here is a session using them:

gdb-demo$ gdb gdb-demo
...
Reading symbols from /home/charles/gdb-demo/gdb-demo...done.
(gdb) source demo.gdb <---- loading up the macros
(gdb) list main
23 }
24 return np;
25 }
26
27 main()
28 {
29 struct enu *list = NULL;
30 struct enu *i;
31
32 list = addpoint(1,2,3,list);
(gdb) break 32 Breakpoint 1 at 0x8048468: file gdb-demo.c, line 32.
(gdb) run Starting program: /home/charles/gdb-demo/gdb-demo
Breakpoint 1, main () at gdb-demo.c:32
32 list = addpoint(1,2,3,list);
(gdb) dump_list list
List is empty
(gdb) n
33 list = addpoint(1,2,4,list);
(gdb) n
34 list = addpoint(1,2,5,list);
(gdb) n
35 list = addpoint(1,2,6,list);
(gdb) dump_list list
Entry is 1.000000 2.000000 5.000000
Entry is 1.000000 2.000000 4.000000
Entry is 1.000000 2.000000 3.000000
End of list
(gdb)

In embedded systems, macros are really handy as a way to explore the state of RTOS tasks, semaphores etc as well as peripherals etc. Careful though... reading some peripheral registers can change their state.

Building infrastructure like this takes some time but pays off in the long run.

Simple realtime plotting

A picture is worth a thousand words, they say. Well that's certainly the case when you're trying to view the impact of some code changes over a period of time.

One thing I needed to do when writing the new background garbage collection feature for the yaffs file system was to see how the background garbage collection heuristics were working. In particular I needed to trace how the erased space compares to the free space (the garbage collector changes dirty free space into erased free space).... and I wanted a graphic trace in real time.

This really helped to monitor and debug some interesting problems where the background garbage collector should have stopped working but did not.

There are quite a few ways to do this for static logs (gnuplot, spreadsheets, etc) and there are also quite a few logging database tools (rrdtool etc) but those just all seemed too much effort to set up or docs to read. Why use a chainsaw to cut butter?

I was able to crank out something simple using bash and gnuplot. The bash script has three parallel processes: data gathering and driving gnuplot and gnuplot itself.

#!/bin/sh
# Script that gathers data erased vs free data from /proc/yaffs_stats and simultaneously \
# plots it using gnuplot.


#Gather settings
log_file=data
gather_delay=1

# Plot settings
trunc_file=trunc_data
plot_samples=1000
plot_delay=2



# Gathering task

gather_data() {
i=0;
rm -f $log_file

while true; do
str=$(cat /proc/yaffs_debug)
echo "$i, $str"
echo "$i, $str" >> $log_file
let i=$i+1
sleep $gather_delay
done
}


# Plotting task
# Periodically creates a truncated version of the log file and
# outputs commands into gnuplot, thus driving gnuplot

drive_gnuplot(){
sleep 5
tail -$plot_samples $log_file > $trunc_file

plot_str=" plot '$trunc_file' using 1:3 with linespoints title 'free', '' using 1:4 with linespoints title 'erased'"

echo "set title 'yaffs free space and erased space'"

echo $plot_str

while true; do
sleep $plot_delay
tail -$plot_samples $log_file > $trunc_file
echo replot
done
}



echo "Start gathering task in background"
gather_data &
echo "Run plotting task"
drive_gnuplot | gnuplot

The data gatherer periodically reads from a Linux kernel procfs entry and dumps a new data record onto the end of a log file.

The gnu-plot driver process uses tail to truncate the last n lines of the log into a truncated log file, then sends commands to gnuplot to either start a plot (for the first time around) or replot (on subsequent iterations). The nifty bit here is the pipe. Since the process is piping into gnuplot, output from this process is treated as if a user typed the command in gnuplot.

The above script gets run in a terminal and we get a nice gnuplot picture that automatically updates every few seconds.

The same basic idea is readily modified for all sorts of applications. For example if your embedded system spits out stats periodically on a serial port that info can get logged and run through a similar plotting task.