Using python + ctypes + gdb for testing yaffs

I'm doing a short lightning talk at KiwiPycon this week end. You can't put much info into a lightning talk so this blog posting provides some further info.

You can get the yaffs code with python ctypes stuff here . Grab the tarball. Inside that go to yaffs2/direct/python.

All up, yaffs is over 20k lines of C and is used all sorts of applications, particularly cell phones. yaffs is under [almost] daily modification and needs a reasonable test environment. Regression testing is handy to prevent things taking a step backward, but a more dynamic test framework is needed when testing new features/mods. I call this "walk testing".

I used to do walk testing using C test routines + gdb but am switching over much of that to python + ctypes + gdb.

The problem with doing this in C is that the slightest change to the "walk" needs a whole coding/compile/restart cycle.

In-kernel walk testing using bash gives a dynamic environment but no gdb. An error can crash the kernel which forces a reboot.

Using python, rather than C, as the test harness environment provides both a programming environment + a command line. This provides a far superior "walk testing" enviroment.

Why not SWIG?
I tried SWIG first, but soon gave up because it just seemed far more cumbersome.... and I really didn't want to have to learn the SWIG command syntax. Further, SWIG requires compilation against the correct python headers which makes building a pain if you use different python versions.

ctypes
ctypes provides a way to wrap C code, in this case the YAFFS direct library, so that it can be accessed from python. The actual ctypes wrapping is in the yaffsfs.py module and examples.py shows some calling.

Ctypes wrapping is pretty simple for the most part but a few things are tricky, like handling pointers to structures. The yaffsfs cases do a few interesting things so this can be a useful example, particularly with handling the stat and directory entry structures:

def yaffs_ls(dname):
if dname[-1] != "/": dname = dname + "/"
dc = yaffs_opendir(dname)
if dc != 0 :
    sep = yaffs_readdir(dc)
    while bool(sep):
        se = sep.contents
        fullname = dname + se.d_name
        #print fullname, " ", se.d_ino," ",ord(se.d_type)
        st = yaffs_stat_struct()
        result = yaffs_stat(fullname,byref(st))
        perms = st.st_mode & 0777
        isFile = True if st.st_mode & 0x8000 else False
        isDir  = True if st.st_mode & 0x4000 else False

        if isFile :
            print "File ",se.d_ino, hex(perms), st.st_size, fullname
        if isDir :
            print "Dir  ",se.d_ino, hex(perms), fullname
            yaffs_ls(fullname)
        
        sep = yaffs_readdir(dc)
    yaffs_closedir(dc)
    return 0
else:
    print "Could not open directory"
    return -1

What is appealing about using python is that it provides a programming environment as well as a command shell to immediately execute code. This makes it really handy for testing a library. I like to do this with gdb when "walk testing" individual fixes and features.

Using gdb

Firstly you'll need a debug version of python with symbols if you don't already have that.

sudo apt-get install python-dbg

You use two terminal windows: one for python and the other for gdb You could use one window, but this is neater and easier.

In the python window start python and load up the modules:

yaffs2/direct/python$ python
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
[GCC 4.3.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from yaffsfs import *
>>> from examples import *
yaffs: Mounting /yaffs2
...
>>>

Now in the gdb window you need to find out the python process id, start gdb and attach to the python process. Attaching to a process halts that process. You will have to tell gdb to continue. But we're first going to set a breakpoint on yaffs_open().

yaffs2/direct/python$ ps -ef | grep python
charles 3791 3597 0 Nov03 ? 00:00:00 python /usr/share/system-config-printer/applet.py
charles 4070 18972 0 15:21 pts/4 00:00:00 python
charles 4100 4071 0 15:22 pts/9 00:00:00 grep python

yaffs2/direct/python$ gdb /usr/bin/python
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu"...
(gdb) attach 4070
Attaching to program: /usr/bin/python, process 4070
Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb7e226c0 (LWP 4070)]
Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0
Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/tls/i686/cmov/libdl.so.2
Reading symbols from /lib/tls/i686/cmov/libutil.so.1...done.
Loaded symbols for /lib/tls/i686/cmov/libutil.so.1
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/tls/i686/cmov/libm.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libm.so.6
Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/tls/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/python2.6/lib-dynload/readline.so...Reading symbols from /usr/lib/debug/usr/lib/python2.6/lib-dynload/readline.so...done.
done.
Loaded symbols for /usr/lib/python2.6/lib-dynload/readline.so
Reading symbols from /lib/libreadline.so.5...done.
Loaded symbols for /lib/libreadline.so.5
Reading symbols from /lib/libncursesw.so.5...done.
Loaded symbols for /lib/libncursesw.so.5
Reading symbols from /lib/libncurses.so.5...done.
Loaded symbols for /lib/libncurses.so.5
Reading symbols from /usr/lib/python2.6/lib-dynload/_ctypes.so...Reading symbols from /usr/lib/debug/usr/lib/python2.6/lib-dynload/_ctypes.so...done.
done.
Loaded symbols for /usr/lib/python2.6/lib-dynload/_ctypes.so
Reading symbols from /opt/y/dev/yaffs2/direct/python/libyaffsfs.so...done.
Loaded symbols for ./libyaffsfs.so
0xb7ff9430 in __kernel_vsyscall ()
(gdb) b yaffs_open
Breakpoint 1 at 0xb745779e: file yaffsfs.c, line 402.
(gdb) c
Continuing.

The python session is now running as per normal, but as soon as you call yaffs_open, the breakpoint will trigger and you're back in the debugger

python session

>>> yaffs_open("/yaffs2/xx",66,0666)

gdb session

[Switching to Thread 0xb7e226c0 (LWP 4070)]

Breakpoint 1, yaffs_open (path=0x95ac714 "/yaffs2/xx", oflag=66, mode=438) at yaffsfs.c:402
402 yaffs_Object *obj = NULL;
(gdb)

Conclusion

Using python + ctypes + gdb give me a much simpler and faster way to do this sort of testing than the approach I was using before.

Setting up Debian chroot under Ubuntu

Wanting to build for the balloon board using emdebian, I started out with the Debian instructions. However this caused my Ubuntu-based system to pick up a whole lot of changes from Debian which broke my system.

So on some advise from Wookey I tried chroot.

I ended up using a variant of the instructions from here and here.

Some important differences...

Started with:

 sudo debootstrap --arch i386 lenny /debian http://ftp.uk.debian.org/debian/

That bootstraps the basic chroot debian environment in a directory called /debian

From there you can follow the instructions from here, but:
* Skip over the first buch of stuff about cdebootstrap.
* using /debian instead of /path/to/chroot
* Don't copy in the /etc/apt/sources from the host distro.

Ubuntu does not use /etc/inittab so to set vt8 up as a debian terminal go to /etc/event.d and
create tty8 with the following:

# tty8 - debian chroot tty
#
# This service maintains a getty on tty8 from the point the system is
# started until it is shut down again.

start on runlevel 2
start on runlevel 3

stop on runlevel 0
stop on runlevel 1
stop on runlevel 4
stop on runlevel 5
stop on runlevel 6

respawn
exec /usr/sbin/chroot /debian /sbin/getty 38400 tty8

Then execute
  sudo initctl start tty8

If you F8 you should now have a debian terminal that will login with no root.

Code on github

git is a really interesting way to do source control and github.com provides a free git server.

I've moved my AVR code to github to make it more accessable. Find it here.

Phototransistor speed

I tried a phototransistor instead of an LDR. Look at the speed difference! The scale here is different (far faster sampling) which means that the phototransistor is at least 50x the speed of the LDR.

I'm designing a multi-channel colour sensor which will have 4 colour sensors controlled by a single AVR ATMEGA48.
* Each sensor has 3 LEDs red, green, blue.
* Each of these LEDs is turned on in turn, the ADC is then sampled approx 4 times and some filtering is done on the returned signal. The signal is then corrected for differences in phototransistor sensitivity etc by applying a simple gamma correction to give reasonably matched RGB values.

Using phototransistors should make this all fast enough to operate 4 channels at 100Hz or so.

100Hz sampling should be fast enough for an NXT robot.

LDR Speed

Light dependent resistors are cheap and relatively wide-band light sensors, but they are a bit slow relative to photodiodes and phototransistors.

I used an AVR-based board to measure this, as follows:

* Put an LDR and 6.8k resistor in series between 5V and ground as a voltage divider.

* Attach the AVR's ADC pin to the connection between the resistor and LDR.
* Set another pin up to toggle an LED every 50msec.
* AVR samples ADConce each millisecond and spits out the value as a string on the RS232 port.

* PC logs serial port into file
*Use gnuplot to plot results.

As you can see, after approx 15msec the ADC value has settled quite well.


I don't quite know why the formatting got hosed.... but so be it.

Make your own Lego NXT sensors

Just finished putting together a presentation for the Christchurch Robotics Group on making custom sensors for the Lego NXT.

Making resistive/switch sensors is very simple, but one of the sensors is an AVR-based I2C sensor which puts all the flexibility of a microcontroller in the sensor. More info on that to be posted later.

Presentation pdf and code for the micro here.

Update: Code is now on github.