Fantasm: Master Class - Memory Protection

This article is designed to demonstrate how you can use Fantasm's more powerful features to achieve a goal that at first seems either impossible or expensive. It explains how you can use Fantasm to automatically verify memory writes at run time from your code and stop if the write would write outside of your application heap. This will catch many bugs that can be extremely difficult to find.

Over the last couple of weeks I have had spurious crashes in various applications, including things like the Finder, whilst running some development software written with Fantasm. I finally realized that my App must be writing out of it's heap and
corrupting other applications.

The Mac (currently) does not have "protected memory", which would have helped me find the fault. There are utilities available for about 200 bucks that will try to spot writes out of your allocated space, or failing that I could write a "padded cell" for my app that would run the instructions via emulation, checking where a given instruction was about to write to. Neither of these appealed to me very much, but the second was looking more and more likely.

A few days back, I was working on something completely different (as you do when faced with a big problem) when it suddenly dawned on me how to find my fault and provide some excellent future debugging aids into the bargain. It was going to cost me zero bucks and about an hour of my time. How?

Two of Fantasm's many directives, macs_first and macs_last, allow one to tell Fantasm how to search for a given item. For example, when faced with "stw", it would normally search the directives first, then the instruction list and finally the macros (there are good reasons for this seemingly non-efficient search order that I won't go into here). Now, macs_first and last allow you change this behaviour; specifically macs_first causes Fantasm to search in the defined macros list first. This means we can override instructions with macros.

Back to my writing out of my heap problem. All I did was defined a macro called "stw" and used the macs_first directive at the top of my source files(see note at end). This causes Fantasm to execute the "stw" macro rather than assemble "stw" instructions, which was the most likely candidate that was writing out of my heap with a bad address.

My stw macro read something like this:
if the address we are about to store to is not in our heap, then cause an
illegal instruction.

Simple and incredibly effective 'cause we get to stop right before the instruction that would cause our Mac to go all wobbly. This found the fault within 3 seconds of booting the reassembled app up! What had happened was I had erronously used a volatile register to hold a pointer over an OS call, which was trashing the volatile register.

I could have found it by eyeballing the code, but 110,000 lines of code is a lot to eyeball!

OK, so this mod added on about 200K to my executable size, and cut execution speed by 20%, but it's only debugging code that can be switched off simply by setting a global definition.

Here's the details.
1. Add two global variables to your project:

**test.globs
heap_low:	rs.w	1
heap_high:	rs.w	1	*range of valid addresses for our app

2. Execute this code immediately your app starts up:

	import	GetZone
	Xcall	GetZone	*bottom of heap
	stw	r3,heap_low(`bss)
	stw	sp,heap_high(`bss)

3. The macro to check the address looks like this:

check_addr:	macro	*needs an offset in \1 and a register in \2
**E.G.  check_addr 32,r3
		addi	`temp_reg1,\2,\1	*temp_reg1=the address
		lwz	`temp_reg2,heap_low(`bss)
		cmpw	`temp_reg1,`temp_reg2
		bgt	low_ok\@
		illegal
low_ok\@:
		lwz	`temp_reg2,heap_high(`bss)
		cmpw	`temp_reg1,`temp_reg2
		ble	high_ok\@
		illegal
high_ok\@:
		endm

The actual macro that replaces stw receives two parameters separated by a comma, as per a normal stw instruction. The first parameter is rS, which is the general purpose register (GPR) that contains the data we are storing, the second has the form of d(rA), where d is an optional offset, and rA is the GPR containing an address.
What we want to do is check the address held in rA+d against our heap bounds. Hence we need to decode two items - d and rA and pass them to check_addr. This will involve some string manipulation, but Fantasm is reasonably good at that.
1. Find the open brackets. If there is no open brackets, it's a syntax error.
2. The text to the left of the open brackets is the offset.
3. Find the close brackets. If there is no close brackets it's a syntax error.
4. The text between the open and close brackets is rA.

safe_store:	macro
open_p:	fndc	\2,"("	*the position in \2 of the open brackets
		iflt	open_p	*if fndc fails, it returns -1
			fail	Syntax 
		endif
length:	len	\2
**get offset
[o$]:		left$	\2,open_p
**get register
[r$]:		right$	\2,length-open_p+1	*get text between open parenthesis and end of string
close_p:	fndc	[r$],")"
		iflt	close_p	*if fndc fails, it returns -1
			fail	Syntax
		endif
[r$]:	left$	[r$],close_p			*get register
		check_addr	[o$],[r$]		*generate code to check the address
		endm

This macro is called from a macro called stw which reads as follows:

**stw rs,d(ra)
stw:	macro	*Ensure macs_first is in-force to ensure this macro overrides stw.
	safe_store	\1,\2
	macs_last	*without this we would recurse in the stw instruction
	stw	\1,\2	*The actual instruction
	macs_first
	endm

Here's a complete program that demonstrates this:

The program starts at main. The first thing it does is load up the bss pointer and heap_low and heap_high variables. Then it overrides normal assembly with the macs_first directive. For this example we will purposly make the mistake of storing the data in var1(r3) rather than var1(`bss). We load up r4 with the data "1" and the address 0 in r3. Then we make the (deliberate) mistake of storing r4 in var1(r3) rather than var1(`bss). When we run the program we end up in Macsbug:

PowerPC illegal instruction at 0497F7E4 main+00054
 Disassembling PowerPC code from 0497F7BC
  main
     +0003C 0497F7CC   li         r3,0x0000                               | 38600000
     +00040 0497F7D0   li         r4,0x0001                               | 38800001
     +00044 0497F7D4   addi       r20,r3,0x0008                           | 3A830008
     +00048 0497F7D8   lwz        r21,0x0000(r30)                         | 82BE0000
     +0004C 0497F7DC   cmpw       r20,r21                                 | 7C14A800
     +00050 0497F7E0   bgt        main+00058                 ; 0x0497F7E8 | 41810008
     +00054 0497F7E4  *dc.l       0x7C00012A                              | 7C00012A
     +00058 0497F7E8   lwz        r21,0x0004(r30)                         | 82BE0004
     +0005C 0497F7EC   cmpw       r20,r21                                 | 7C14A800
     +00060 0497F7F0   ble        main+00068                 ; 0x0497F7F8 | 40810008
     +00064 0497F7F4   dc.l       0x7C00012A                              | 7C00012A
     +00068 0497F7F8   stw        r4,0x0008(r3)                           | 90830008
  Closing log

Which is exactly what we wanted. We have stopped before the stw r4,8(r3) instruction which would have made our machine unsafe. For comprehensive checking you'd need macros for other stores and possibly reads as well; you may also want to allow writing to the System heap which would mean modifications to check_addr.

We always enjoy feedback from articles, so if you have any constructive criticism, corrections or wish to clarify something, please email us or post to the Fantasm mailing list.

*Actually, what I did was include the macs_first directive in one of my globally included files.

Other articles in this series: Fantasm and C