gdb macros

gdb provides an amazingly versatile debugging platform supporting mechanisms to debug local and remote software.

One of gdb's less known features is its rich macro language. Although I had used it before working at Apple, that's where I was really exposed to how gdb can be used to the max.

OSX can be debugged from a remote machine via gdb remote. Navigating around the guts of a running OS is challenging. You need to locate the task structures etc that you are debugging by finding them in lists of running tasks. Doing this manually takes time and is error prone. So to make this easy, Apple provides a file of gdb macros that will do this for you.

While these macros won't work on your embedded system, they do provide examples for how you can construct complex macros containing ifs, while loops etc.

To give you a taster, here's a very simple example using a linked list of structures.

#include
#include

struct enu{
float x;
float y;
float z;
struct enu *next;
};

struct enu *addpoint(float x, float y, float z, struct enu *next)
{
struct enu * np;
np = malloc(sizeof(struct enu));
if(np){
np->x = x;
np->y = y;
np->z = z;
np->next = next;
}
return np;
}

main()
{
struct enu *list = NULL;
struct enu *i;

list = addpoint(1,2,3,list);
list = addpoint(1,2,4,list);
list = addpoint(1,2,5,list);
list = addpoint(1,2,6,list);

for(i = list; i; i=i->next)
printf("%f %f %f\n",i->x,i->y,i->z);
}

Here are a few macros

define dump_entry
set $l=(struct enu *)($arg0)
printf "Entry is %f %f %f\n", $l->x, $l->y, $l->z
end
document dump_entry
Print out a list entry
Usage: dump_entry pointer
end

define dump_list
set $list = $arg0
if(!$list)
printf "List is empty\n"
else
while($list)
dump_entry $list
set $list=$list->next
end
printf "End of list\n"
end
end
document dump_list
Print out a whole list
Usage: dump_list pointer
end

And here is a session using them:

gdb-demo$ gdb gdb-demo
...
Reading symbols from /home/charles/gdb-demo/gdb-demo...done.
(gdb) source demo.gdb <---- loading up the macros
(gdb) list main
23 }
24 return np;
25 }
26
27 main()
28 {
29 struct enu *list = NULL;
30 struct enu *i;
31
32 list = addpoint(1,2,3,list);
(gdb) break 32 Breakpoint 1 at 0x8048468: file gdb-demo.c, line 32.
(gdb) run Starting program: /home/charles/gdb-demo/gdb-demo
Breakpoint 1, main () at gdb-demo.c:32
32 list = addpoint(1,2,3,list);
(gdb) dump_list list
List is empty
(gdb) n
33 list = addpoint(1,2,4,list);
(gdb) n
34 list = addpoint(1,2,5,list);
(gdb) n
35 list = addpoint(1,2,6,list);
(gdb) dump_list list
Entry is 1.000000 2.000000 5.000000
Entry is 1.000000 2.000000 4.000000
Entry is 1.000000 2.000000 3.000000
End of list
(gdb)

In embedded systems, macros are really handy as a way to explore the state of RTOS tasks, semaphores etc as well as peripherals etc. Careful though... reading some peripheral registers can change their state.

Building infrastructure like this takes some time but pays off in the long run.

Simple realtime plotting

A picture is worth a thousand words, they say. Well that's certainly the case when you're trying to view the impact of some code changes over a period of time.

One thing I needed to do when writing the new background garbage collection feature for the yaffs file system was to see how the background garbage collection heuristics were working. In particular I needed to trace how the erased space compares to the free space (the garbage collector changes dirty free space into erased free space).... and I wanted a graphic trace in real time.

This really helped to monitor and debug some interesting problems where the background garbage collector should have stopped working but did not.

There are quite a few ways to do this for static logs (gnuplot, spreadsheets, etc) and there are also quite a few logging database tools (rrdtool etc) but those just all seemed too much effort to set up or docs to read. Why use a chainsaw to cut butter?

I was able to crank out something simple using bash and gnuplot. The bash script has three parallel processes: data gathering and driving gnuplot and gnuplot itself.

#!/bin/sh
# Script that gathers data erased vs free data from /proc/yaffs_stats and simultaneously \
# plots it using gnuplot.


#Gather settings
log_file=data
gather_delay=1

# Plot settings
trunc_file=trunc_data
plot_samples=1000
plot_delay=2



# Gathering task

gather_data() {
i=0;
rm -f $log_file

while true; do
str=$(cat /proc/yaffs_debug)
echo "$i, $str"
echo "$i, $str" >> $log_file
let i=$i+1
sleep $gather_delay
done
}


# Plotting task
# Periodically creates a truncated version of the log file and
# outputs commands into gnuplot, thus driving gnuplot

drive_gnuplot(){
sleep 5
tail -$plot_samples $log_file > $trunc_file

plot_str=" plot '$trunc_file' using 1:3 with linespoints title 'free', '' using 1:4 with linespoints title 'erased'"

echo "set title 'yaffs free space and erased space'"

echo $plot_str

while true; do
sleep $plot_delay
tail -$plot_samples $log_file > $trunc_file
echo replot
done
}



echo "Start gathering task in background"
gather_data &
echo "Run plotting task"
drive_gnuplot | gnuplot

The data gatherer periodically reads from a Linux kernel procfs entry and dumps a new data record onto the end of a log file.

The gnu-plot driver process uses tail to truncate the last n lines of the log into a truncated log file, then sends commands to gnuplot to either start a plot (for the first time around) or replot (on subsequent iterations). The nifty bit here is the pipe. Since the process is piping into gnuplot, output from this process is treated as if a user typed the command in gnuplot.

The above script gets run in a terminal and we get a nice gnuplot picture that automatically updates every few seconds.

The same basic idea is readily modified for all sorts of applications. For example if your embedded system spits out stats periodically on a serial port that info can get logged and run through a similar plotting task.

Using python + ctypes + gdb for testing yaffs

I'm doing a short lightning talk at KiwiPycon this week end. You can't put much info into a lightning talk so this blog posting provides some further info.

You can get the yaffs code with python ctypes stuff here . Grab the tarball. Inside that go to yaffs2/direct/python.

All up, yaffs is over 20k lines of C and is used all sorts of applications, particularly cell phones. yaffs is under [almost] daily modification and needs a reasonable test environment. Regression testing is handy to prevent things taking a step backward, but a more dynamic test framework is needed when testing new features/mods. I call this "walk testing".

I used to do walk testing using C test routines + gdb but am switching over much of that to python + ctypes + gdb.

The problem with doing this in C is that the slightest change to the "walk" needs a whole coding/compile/restart cycle.

In-kernel walk testing using bash gives a dynamic environment but no gdb. An error can crash the kernel which forces a reboot.

Using python, rather than C, as the test harness environment provides both a programming environment + a command line. This provides a far superior "walk testing" enviroment.

Why not SWIG?
I tried SWIG first, but soon gave up because it just seemed far more cumbersome.... and I really didn't want to have to learn the SWIG command syntax. Further, SWIG requires compilation against the correct python headers which makes building a pain if you use different python versions.

ctypes
ctypes provides a way to wrap C code, in this case the YAFFS direct library, so that it can be accessed from python. The actual ctypes wrapping is in the yaffsfs.py module and examples.py shows some calling.

Ctypes wrapping is pretty simple for the most part but a few things are tricky, like handling pointers to structures. The yaffsfs cases do a few interesting things so this can be a useful example, particularly with handling the stat and directory entry structures:

def yaffs_ls(dname):
if dname[-1] != "/": dname = dname + "/"
dc = yaffs_opendir(dname)
if dc != 0 :
    sep = yaffs_readdir(dc)
    while bool(sep):
        se = sep.contents
        fullname = dname + se.d_name
        #print fullname, " ", se.d_ino," ",ord(se.d_type)
        st = yaffs_stat_struct()
        result = yaffs_stat(fullname,byref(st))
        perms = st.st_mode & 0777
        isFile = True if st.st_mode & 0x8000 else False
        isDir  = True if st.st_mode & 0x4000 else False

        if isFile :
            print "File ",se.d_ino, hex(perms), st.st_size, fullname
        if isDir :
            print "Dir  ",se.d_ino, hex(perms), fullname
            yaffs_ls(fullname)
        
        sep = yaffs_readdir(dc)
    yaffs_closedir(dc)
    return 0
else:
    print "Could not open directory"
    return -1

What is appealing about using python is that it provides a programming environment as well as a command shell to immediately execute code. This makes it really handy for testing a library. I like to do this with gdb when "walk testing" individual fixes and features.

Using gdb

Firstly you'll need a debug version of python with symbols if you don't already have that.

sudo apt-get install python-dbg

You use two terminal windows: one for python and the other for gdb You could use one window, but this is neater and easier.

In the python window start python and load up the modules:

yaffs2/direct/python$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaffsfs import *
>>> from examples import *
yaffs: Mounting /yaffs2
...
>>>

Now in the gdb window you need to find out the python process id, start gdb and attach to the python process. Attaching to a process halts that process. You will have to tell gdb to continue. But we're first going to set a breakpoint on yaffs_open().

yaffs2/direct/python$ ps -ef | grep python
charles 3791 3597 0 Nov03 ? 00:00:00 python /usr/share/system-config-printer/applet.py
charles 4070 18972 0 15:21 pts/4 00:00:00 python
charles 4100 4071 0 15:22 pts/9 00:00:00 grep python

yaffs2/direct/python$ gdb /usr/bin/python
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(gdb) attach 4070
Attaching to program: /usr/bin/python, process 4070
Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7e226c0 (LWP 4070)]
Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
Reading symbols from /lib/tls/i686/cmov/libutil.so.1...done.
Loaded symbols for /lib/tls/i686/cmov/libutil.so.1
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/tls/i686/cmov/libm.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libm.so.6
Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/python2.6/lib-dynload/readline.so...Reading symbols from /usr/lib/debug/usr/lib/python2.6/lib-dynload/readline.so...done.
done.
Loaded symbols for /usr/lib/python2.6/lib-dynload/readline.so
Reading symbols from /lib/libreadline.so.5...done.
Loaded symbols for /lib/libreadline.so.5
Reading symbols from /lib/libncursesw.so.5...done.
Loaded symbols for /lib/libncursesw.so.5
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /usr/lib/python2.6/lib-dynload/_ctypes.so...Reading symbols from /usr/lib/debug/usr/lib/python2.6/lib-dynload/_ctypes.so...done.
done.
Loaded symbols for /usr/lib/python2.6/lib-dynload/_ctypes.so
Reading symbols from /opt/y/dev/yaffs2/direct/python/libyaffsfs.so...done.
Loaded symbols for ./libyaffsfs.so
0xb7ff9430 in __kernel_vsyscall ()
(gdb) b yaffs_open
Breakpoint 1 at 0xb745779e: file yaffsfs.c, line 402.
(gdb) c
Continuing.

The python session is now running as per normal, but as soon as you call yaffs_open, the breakpoint will trigger and you're back in the debugger

python session

>>> yaffs_open("/yaffs2/xx",66,0666)

gdb session

[Switching to Thread 0xb7e226c0 (LWP 4070)]

Breakpoint 1, yaffs_open (path=0x95ac714 "/yaffs2/xx", oflag=66, mode=438) at yaffsfs.c:402
402 yaffs_Object *obj = NULL;
(gdb)

Conclusion

Using python + ctypes + gdb give me a much simpler and faster way to do this sort of testing than the approach I was using before.

Setting up Debian chroot under Ubuntu

Wanting to build for the balloon board using emdebian, I started out with the Debian instructions. However this caused my Ubuntu-based system to pick up a whole lot of changes from Debian which broke my system.

So on some advise from Wookey I tried chroot.

I ended up using a variant of the instructions from here and here.

Some important differences...

Started with:

 sudo debootstrap --arch i386 lenny /debian http://ftp.uk.debian.org/debian/

That bootstraps the basic chroot debian environment in a directory called /debian

From there you can follow the instructions from here, but:
* Skip over the first buch of stuff about cdebootstrap.
* using /debian instead of /path/to/chroot
* Don't copy in the /etc/apt/sources from the host distro.

Ubuntu does not use /etc/inittab so to set vt8 up as a debian terminal go to /etc/event.d and
create tty8 with the following:

# tty8 - debian chroot tty
#
# This service maintains a getty on tty8 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3

stop on runlevel 0
stop on runlevel 1
stop on runlevel 4
stop on runlevel 5
stop on runlevel 6

respawn
exec /usr/sbin/chroot /debian /sbin/getty 38400 tty8

Then execute
  sudo initctl start tty8

If you F8 you should now have a debian terminal that will login with no root.

Code on github

git is a really interesting way to do source control and github.com provides a free git server.

I've moved my AVR code to github to make it more accessable. Find it here.

Phototransistor speed

I tried a phototransistor instead of an LDR. Look at the speed difference! The scale here is different (far faster sampling) which means that the phototransistor is at least 50x the speed of the LDR.

I'm designing a multi-channel colour sensor which will have 4 colour sensors controlled by a single AVR ATMEGA48.
* Each sensor has 3 LEDs red, green, blue.
* Each of these LEDs is turned on in turn, the ADC is then sampled approx 4 times and some filtering is done on the returned signal. The signal is then corrected for differences in phototransistor sensitivity etc by applying a simple gamma correction to give reasonably matched RGB values.

Using phototransistors should make this all fast enough to operate 4 channels at 100Hz or so.

100Hz sampling should be fast enough for an NXT robot.