Using python + ctypes + gdb for testing yaffs

I'm doing a short lightning talk at KiwiPycon this week end. You can't put much info into a lightning talk so this blog posting provides some further info.

You can get the yaffs code with python ctypes stuff here . Grab the tarball. Inside that go to yaffs2/direct/python.

All up, yaffs is over 20k lines of C and is used all sorts of applications, particularly cell phones. yaffs is under [almost] daily modification and needs a reasonable test environment. Regression testing is handy to prevent things taking a step backward, but a more dynamic test framework is needed when testing new features/mods. I call this "walk testing".

I used to do walk testing using C test routines + gdb but am switching over much of that to python + ctypes + gdb.

The problem with doing this in C is that the slightest change to the "walk" needs a whole coding/compile/restart cycle.

In-kernel walk testing using bash gives a dynamic environment but no gdb. An error can crash the kernel which forces a reboot.

Using python, rather than C, as the test harness environment provides both a programming environment + a command line. This provides a far superior "walk testing" enviroment.

Why not SWIG?
I tried SWIG first, but soon gave up because it just seemed far more cumbersome.... and I really didn't want to have to learn the SWIG command syntax. Further, SWIG requires compilation against the correct python headers which makes building a pain if you use different python versions.

ctypes
ctypes provides a way to wrap C code, in this case the YAFFS direct library, so that it can be accessed from python. The actual ctypes wrapping is in the yaffsfs.py module and examples.py shows some calling.

Ctypes wrapping is pretty simple for the most part but a few things are tricky, like handling pointers to structures. The yaffsfs cases do a few interesting things so this can be a useful example, particularly with handling the stat and directory entry structures:

def yaffs_ls(dname):
if dname[-1] != "/": dname = dname + "/"
dc = yaffs_opendir(dname)
if dc != 0 :
    sep = yaffs_readdir(dc)
    while bool(sep):
        se = sep.contents
        fullname = dname + se.d_name
        #print fullname, " ", se.d_ino," ",ord(se.d_type)
        st = yaffs_stat_struct()
        result = yaffs_stat(fullname,byref(st))
        perms = st.st_mode & 0777
        isFile = True if st.st_mode & 0x8000 else False
        isDir  = True if st.st_mode & 0x4000 else False

        if isFile :
            print "File ",se.d_ino, hex(perms), st.st_size, fullname
        if isDir :
            print "Dir  ",se.d_ino, hex(perms), fullname
            yaffs_ls(fullname)
        
        sep = yaffs_readdir(dc)
    yaffs_closedir(dc)
    return 0
else:
    print "Could not open directory"
    return -1

What is appealing about using python is that it provides a programming environment as well as a command shell to immediately execute code. This makes it really handy for testing a library. I like to do this with gdb when "walk testing" individual fixes and features.

Using gdb

Firstly you'll need a debug version of python with symbols if you don't already have that.

sudo apt-get install python-dbg

You use two terminal windows: one for python and the other for gdb You could use one window, but this is neater and easier.

In the python window start python and load up the modules:

yaffs2/direct/python$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaffsfs import *
>>> from examples import *
yaffs: Mounting /yaffs2
...
>>>

Now in the gdb window you need to find out the python process id, start gdb and attach to the python process. Attaching to a process halts that process. You will have to tell gdb to continue. But we're first going to set a breakpoint on yaffs_open().

yaffs2/direct/python$ ps -ef | grep python
charles 3791 3597 0 Nov03 ? 00:00:00 python /usr/share/system-config-printer/applet.py
charles 4070 18972 0 15:21 pts/4 00:00:00 python
charles 4100 4071 0 15:22 pts/9 00:00:00 grep python

yaffs2/direct/python$ gdb /usr/bin/python
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(gdb) attach 4070
Attaching to program: /usr/bin/python, process 4070
Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7e226c0 (LWP 4070)]
Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
Reading symbols from /lib/tls/i686/cmov/libutil.so.1...done.
Loaded symbols for /lib/tls/i686/cmov/libutil.so.1
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/tls/i686/cmov/libm.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libm.so.6
Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/python2.6/lib-dynload/readline.so...Reading symbols from /usr/lib/debug/usr/lib/python2.6/lib-dynload/readline.so...done.
done.
Loaded symbols for /usr/lib/python2.6/lib-dynload/readline.so
Reading symbols from /lib/libreadline.so.5...done.
Loaded symbols for /lib/libreadline.so.5
Reading symbols from /lib/libncursesw.so.5...done.
Loaded symbols for /lib/libncursesw.so.5
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /usr/lib/python2.6/lib-dynload/_ctypes.so...Reading symbols from /usr/lib/debug/usr/lib/python2.6/lib-dynload/_ctypes.so...done.
done.
Loaded symbols for /usr/lib/python2.6/lib-dynload/_ctypes.so
Reading symbols from /opt/y/dev/yaffs2/direct/python/libyaffsfs.so...done.
Loaded symbols for ./libyaffsfs.so
0xb7ff9430 in __kernel_vsyscall ()
(gdb) b yaffs_open
Breakpoint 1 at 0xb745779e: file yaffsfs.c, line 402.
(gdb) c
Continuing.

The python session is now running as per normal, but as soon as you call yaffs_open, the breakpoint will trigger and you're back in the debugger

python session

>>> yaffs_open("/yaffs2/xx",66,0666)

gdb session

[Switching to Thread 0xb7e226c0 (LWP 4070)]

Breakpoint 1, yaffs_open (path=0x95ac714 "/yaffs2/xx", oflag=66, mode=438) at yaffsfs.c:402
402 yaffs_Object *obj = NULL;
(gdb)

Conclusion

Using python + ctypes + gdb give me a much simpler and faster way to do this sort of testing than the approach I was using before.