Hamcrest was born in the Java world as the matcher framework in jMock. It was quickly extracted into its own framework and has become somewhat of a monster in the testing world. It’s now included directly in JUnit (since v4.4), and has been ported to many languages (OCHamcrest in Objective-C, Hamcrest-AS3 in Actionscript, PyHamcrest in Python, etc.). Additionally, the matcher concept is generally useful, and Hamcrest is used is lots of different places (my favorite is collection filtering with Hamcrest in LambdaJ).

When writing unit tests, OCHamcrest offers lots of advantages over the vanilla SenTest assertions. First, there’s a ton of matchers that really make life easy, especially when testing collections like NSArray. Second, OCHamcrest matchers are very readable in code, almost self-documenting. Lastly, OCHamcrest automatically provides excellent failure messages when actual is not equal to expected.

Matching Strings

Some string matching examples:

• is – match the complete string
• startsWith – match the beginning of a string
• endsWith – match the end of a string
• containsString – match part of the string
• equalTo – match the complete string
• equalToIgnoringCase – match the complete string but ignore case
• equalToIgnoringWhiteSpace – match the complete string but ignore extra whitespace (new line, tab, or double spaces)

NSString *s = @"FooBar";
 
assertThat(s, is(@"FooBar"));
 
assertThat(s, startsWith(@"Foo"));
assertThat(s, endsWith(@"Bar"));
assertThat(s, containsString(@"oo"));
assertThat(s, equalToIgnoringCase(@"foobar"));
 
assertThat(@" X \n  Y \t\t  Z \n", equalToIgnoringWhiteSpace(@"X Y Z"));

NOTE: Technically, is isn’t really a matcher, it’s a matcher decorator that implicity converts to the equalTo matcher. [thanks Jon!]

Combining Matchers

You can combine multiple matchers with:

• allOf – AND together all matchers
• anyOf – OR togehter all matches

NSString *s = @"FooBar";
 
assertThat(s, allOf(startsWith(@"Foo"), endsWith(@"Bar"), nil));
assertThat(s, anyOf(startsWith(@"Foo"), startsWith(@"Bar"), nil));
assertThat(s, anyOf(endsWith(@"Foo"), endsWith(@"Bar"), nil));

NOTE: The list of matchers must be nil terminated.

You can invert a matcher, or multiple matchers, with:

• isNot – negate the matcher

NSString *s = @"FooBar";
 
assertThat(s, isNot(@"foo"));
assertThat(s, isNot(endsWith(@"Baz")));
assertThat(s, isNot(allOf(startsWith(@"Baz"), endsWith(@"Baz"), nil)));
assertThat(s, isNot(anyOf(startsWith(@"Baz"), startsWith(@"Baz"), nil)));

Matching nil

You can match nil with:

• nilValue() – stands in for nil
• notNilValue() – stands in for !nil

NSObject *o = nil;
assertThat(o, nilValue());
 
NSString *s = @"FooBar";
assertThat(s, notNilValue());

Matching Classes

You can match an instance’s class with:

• instanceOf – match the class

NSString *s = @"FooBar";
assertThat(s, instanceOf([NSString class]));

Matching Numbers

One of the great pains of Objective-C is typing numbers from primitive types to objects and back again. OCHamcrest has a variety of matchers the help make life easy.

• assertThatInt – typed assert that expects an int (other types too: assertThatFloat, assertThatDouble, etc.)
• equalToInt – typed equals that takes an int (other types too: equalToFloat, equalToDouble, equalToBool, etc.)
• closeTo – match a number with a target number plus or minus a delta (both params are double)
• lessThan – match a number less than the given number (param is NSNumber), also lessThanOrEqualTo
• greaterThan – match a number greater than the given number (param is NSNumber), also greaterThanOrEqualTo

assertThatInt(5, equalToInt(5));
assertThatFloat(3.14, equalToFloat(3.14f));
assertThatBool( false, equalToBool(NO) );
 
NSNumber *i = [NSNumber numberWithInt:5];
assertThat(i, equalToInt(5));
assertThat(i, is([NSNumber numberWithInt:5]));
 
NSNumber *f = [NSNumber numberWithFloat:3.14f];
assertThat(f, equalToFloat(3.14f));
assertThat(f, is([NSNumber numberWithFloat:3.14f]));

The easiest cleanest approach is to use assertThatInt with equalToInt, the next best option is to use the vanilla assertThat with equalToInt, the most verbose option is to use NSNumber everywhere.

It’s easy to make rough number comparisons too:

NSNumber *f = [NSNumber numberWithFloat:3.14f];
assertThat(f, closeTo(3.0f, 0.25f));
 
assertThat(f, lessThan([NSNumber numberWithInt:4]));
assertThat(f, greaterThan([NSNumber numberWithInt:3]));

NOTE: It is a little weird, but closeTo takes double params, but everything else expects NSNumber params.

Numeric comparisons also work great on dates too:

NSDate *now = [NSDate date];
 
//now minus 1000 seconds
NSDate *beforeNow = [NSDate dateWithTimeIntervalSinceNow:-1000]; 
 
assertThat(now, greaterThan(beforeNow));

Matching Arrays

Easily the best part of OCHamcrest is its ability to match lists of objects. Array matchers are every powerful, but don’t forget to add the terminating nil to all lists.

• hasItem – match if given item appears in the list
• hasItems – match if all given items appear in the list (in any order)
• contains – match exactly the entire array
• containsInAnyOrder – match entire array, but in any order
• hasCountOf – match the size of the array
• empty – match an empty array

Here some basic array examples:

NSArray *a = [NSArray array];
 
assertThat(a, is(empty()));
assertThat(a, hasCountOf(0));

Here some hasItem examples:

NSArray *a = [NSArray arrayWithObjects:@"a", @"b", @"c", nil];
 
assertThat(a, hasItem(@"a"));
assertThat(a, isNot(hasItem(@"X")));
assertThat(a, hasItem(equalToIgnoringCase(@"A")));

The last matcher may look a little weird, but remember matchers expect a matcher as their input param, and only default to equalTo if none is given. Thus, the first matcher hasItem(@"a") can be rewritten as hasItem(equalTo(@"a")).

We repeat the above example, but this time using numbers in our NSArray. As you can see below, all the number matchers require us to explicitly use equalToInt everywhere:

NSArray *a = [NSArray arrayWithObjects:
    [NSNumber numberWithInt:2],
    [NSNumber numberWithInt:3],
    [NSNumber numberWithInt:5],
    nil];
 
assertThat(a, hasItem(equalToInt(2)));
assertThat(a, isNot(hasItem(equalToInt(13))));
 
assertThat(a, contains(equalToInt(2), equalToInt(3), equalToInt(5), nil));

Here are some more complex array matchers:

NSArray *a = [NSArray arrayWithObjects:@"a", @"b", @"c", nil];
 
assertThat(a, hasItems(@"b", @"a", nil));
 
assertThat(a, contains(@"a", @"b", @"c", nil));
assertThat(a, containsInAnyOrder(@"c", @"b", @"a", nil));
 
assertThat([a componentsJoinedByString:@","], is(@"a,b,c"));

And as I show in the last matcher, you can always dump to a string, and just match strings.

Matching Dictionaries

The dictionary matchers build on the array matchers:

• hasKey – match a key
• hasValue – match a value
• hasEntry – match a key-value pair
• hasEntries – match a list of k-v pairs

NSDictionary *d = [NSDictionary dictionaryWithObjectsAndKeys:
    @"valA", @"keyA", @"valB", @"keyB", @"valC", @"keyC", nil];
 
assertThat(d, hasCountOf(3));
assertThat(d, isNot(empty()));
 
assertThat(d, hasKey(@"keyA"));
assertThat(d, isNot(hasKey(@"keyX")));
 
assertThat(d, hasValue(@"valA"));
 
assertThat(d, hasEntry(@"keyA", @"valA"));
 
assertThat(d, hasEntries(@"keyA", @"valA", @"keyC", @"valC", nil));

Matcher Error Messages

When a matcher fails, you get a standardize error message of: Expected "foo", but was "bar". This default message is easy to modify by using the describedAs() matcher in place of the typical is() matcher.

NSString *s = @"bar";
 
assertThat(s, is(@"foo"));
//Expected "foo", but was "bar"
 
assertThat(s, describedAs(@"doh! this should be 'foo'", equalTo(@"foo"), nil));
//Expected doh! this should be 'foo', but was "bar"
 
assertThat(s, describedAs(@"doh! this should be foo, %0, %1", equalTo(@"foo"), @"baz", [NSNumber numberWithInt:42], nil));
//Expected doh! this should be foo, "baz", <42>, but was "bar"

NOTE: The argument list for describedAs() MUST end with nil or your tests will crash instantly with no useful error message.

Building a Custom Matcher

Writing your own custom matchers is relatively easy. Here’s an example of a matcher that matches the value of some property on an object:

#import <OCHamcrestIOS/OCHamcrestIOS.h>
#import <objc/objc-api.h>
 
@interface HasProperty : HCBaseMatcher {
    NSString *property;
    id<HCMatcher> valueMatcher;
}
 
+ (id) hasProperty:(NSString *)aProperty value:(id<HCMatcher>)aValueMatcher;
- (id) initWithProperty:(NSString *)aProperty value:(id<HCMatcher>)aValueMatcher;
 
@end
 
OBJC_EXPORT id<HCMatcher> hasProperty(NSString *property, id valueMatcher);

We extends HCBaseMatcher with our custom HasProperty class. We store the name of the property and a value matcher.

And the implementation:

#import "HasProperty.h"
#import <OCHamcrestIOS/HCDescription.h>
#import <OCHamcrestIOS/HCWrapInMatcher.h>
 
@implementation HasProperty
 
+ (id) hasProperty:(NSString *)aProperty value:(id<HCMatcher>)aValueMatcher {
    return [[[self alloc] initWithProperty:aProperty value:aValueMatcher] autorelease];
}
 
- (id) initWithProperty:(NSString *)aProperty value:(id<HCMatcher>)aValueMatcher {
    self = [super init];
    if (self != nil) {
        property = [aProperty copy];
        valueMatcher = [aValueMatcher retain];
    }
    return self;
}
 
- (void) dealloc {
    [property release];
    [valueMatcher release];
    [super dealloc];
}
 
- (BOOL)matches:(id)item {
    SEL propertyGetter = NSSelectorFromString(property);
 
    if ([item respondsToSelector:propertyGetter]) {
        if ([valueMatcher matches:[item performSelector:propertyGetter]])
            return YES;
    }
    return NO;
}
 
- (void) describeTo:(id<HCDescription>)description {
    [[[description appendText:[NSString stringWithFormat:@"an object with a property named '%@' with a value of {", property]]
        appendDescriptionOf:valueMatcher]
        appendText:@"}"];
}
@end
 
OBJC_EXPORT id<HCMatcher> hasProperty(NSString *property, id valueMatcher) {
    return [HasProperty hasProperty:property value:HCWrapInMatcher(valueMatcher)];
}

When we write a custom matcher, we must implement two methods, matches: to do the matching and describeTo: to provide feedback in case of match failure. In the above code, we first construct a selector from the given property name, then call the selector to get the actual property value, and finally check if actual matches the expected value (given by the valueMatcher).

Usage looks like this:

Person *p = [Person personWithFirstName:@"Joe" andLastname:@"Doe"];
 
assertThat(p, hasProperty(@"firstName", @"Joe"));

Or more importantly, we can now use our custom hasProperty matcher to match an arrays of objects:

NSArray *a = [NSArray arrayWithObjects:
    [Person personWithFirstName:@"Joe" andLastname:@"Doe"],
    [Person personWithFirstName:@"Joe" andLastname:@"Smith"],
    [Person personWithFirstName:@"Jane" andLastname:@"Allen"],
    nil];
 
assertThat(a, contains(
    hasProperty(@"firstName", @"Joe"),
    hasProperty(@"firstName", @"Joe"),
    hasProperty(@"firstName", @"Jane"),
    nil));

That’s it. Go forth and match.

UPDATE: I put the above hasProperty matcher into a pull request, and Jon Reid accepted it into OCHamcrest v1.6. He even wrote a nice post about it. Get the lastest OCHamcrest from github.

Links

• OCHamcrest – github repo
• OCHamcrest Docs – docs
• Hamcrest – on Java

Suddenly, I realized that I really needed to test some private methods. So, a quick google for “testing private methods java” brings up a good article by Bill Venners. He lists all possible options to test private methods:

1. Don’t test private methods
2. Give the methods package access
3. Use a nested test class
4. Use reflection

Basically, the only real one is #4, use reflection. Bill didn’t give me the exact code I needed, so lots of googling later I realized that the world is filled with opinionated people (like me), and boy do they love to talk about #1. I just wanted some code, not a lecture, so I had to write my own code. Here is that code for anyone else that just wants to test private methods.

Imagine you have a class MyClass and it has a private method, myMethod(). Sorta like this:

public class MyClass {
    private String myMethod(String s) {
        return s;
    }
}

Then you could use reflection to invoke the method like this:

MyClass myClass = new MyClass();
Method method = MyClass.class.getDeclaredMethod("myMethod", String.class);
method.setAccessible(true);
String output = (String) method.invoke(myClass, "some input");

The real magic is setAccessible(true) which allows the private method to be called outside the class. And shazam, I can now test all my private methods. I was really hoping JUnit 4 would provide some additional facilities specifically for testing private methods, but not luck.

A Better Example

Here’s a more complete example. Suppose we have a NovelWriter class that in a feat of API cleanliness only exposes the writeNovel() method. It happens to have a few private utility methods that we’d like to test:

public class NovelWriter {
    public String writeNovel() {
        //...the magic goes here...
        return null;
    }
 
    private String shout(String s) {
        return s.toUpperCase().replaceAll("\\.", "!");
    }
 
    private List<Integer> countLetters(List<String> words) {
        List<Integer> out = new ArrayList<Integer>();
        for (String word : words) {
            out.add( word.replaceAll("[^A-Za-z]+","").length() );
        }
        return out;
    }
}

I won’t get into all the details, but it seems easy to imagine a clean API that has private helper methods. Furthermore, it seems very logical to me to want to bring all methods, both public and private, so I can be sure they are being exercised to the fullest.

Our JUnit 4 + Hamcrest test class:

public class NovelWriterTest {
    public static NovelWriter novelWriter;
 
    @BeforeClass
    public static void beforeClass() {
        novelWriter = new NovelWriter();
    }
 
    @Test
    public void privateShout() throws NoSuchMethodException,
            InvocationTargetException, IllegalAccessException {
 
        String input = "This is magic.";
 
        Method method = NovelWriter.class.getDeclaredMethod("shout", String.class);
        method.setAccessible(true);
        String output = (String) method.invoke(novelWriter, input);
 
        assertThat(output, notNullValue());
        assertThat(output, is("THIS IS MAGIC!"));
    }
 
    @Test
    @SuppressWarnings("unchecked")
    public void privateCountLetters() throws NoSuchMethodException,
            InvocationTargetException, IllegalAccessException {
 
        List<String> input = Arrays.asList("Foo", "Foobar123", "Foo Bar Baz");
 
        Method method = NovelWriter.class.getDeclaredMethod("countLetters", List.class);
        method.setAccessible(true);
        List<Integer> output = (List<Integer>) method.invoke(novelWriter, input);
 
        assertThat(output, notNullValue());
        assertThat(output.size(), is(3));
        assertThat(output, hasItems(3, 6, 9));
    }
}

The nice thing about using the Reflection API like this is that it really doesn’t get too messy. I’m not inspecting anything at runtime, because I know exactly the return type and the types of all the parameters. I’m just invoking the method with known inputs, followed by a simple cast on the output type. And as you can see in the second test above, privateCountLetters(), it’s not a problem to use generics because we’re not doing any inspection only invocation.

Happy testing.

Files

• novel.tgz – download the complete Eclipse project

The new hotness in Flex testing is, of course, FlexMonkey, developed and open-sourced by my company, Gorilla Logic. The next best Flex testing platform, and the new hotness in its own right, is FlexUnit 4, developed and open-sourced by our partners at Digital Primates. FlexUnit 4 is the Flex 4 unit testing framework. Because of its awesome async testing support, along with many other great features, it is ideally suited to test client-side service layers. In this post, I’m going to explore async testing with FlexUnit 4 to better understand how I can help mitigate the pain of asynchronous backend services that are ever-present in enterprise Flex applications.

Test: Create Team

Let’s imagine I have my favorite data model of teams and players, with a one-to-many relationship between the two entities. Next, let’s assume that I wrote a beautiful client-side service layer that has both basic CRUD operations like create team and delete team, and more complex operations like trade player. I’d like to cover everything with a set of tests so I can spend my time worrying about other things.

Here’s a simple async test case for the basic create team operation:

[Test(async)]
public function createTeam():void {
    var token:AsyncToken = service.createTeam('Los Angeles Lakers');
    token.addResponder(Async.asyncResponder(this, new TestResponder(createTeam2, fault), TIMEOUT));
}
public function createTeam2(event:ResultEvent, passThroughData:Object):void {
    var token:AsyncToken = service.getAllTeams();
    token.addResponder(Async.asyncResponder(this, new TestResponder(createTeam3, fault), TIMEOUT));
}
public function createTeam3(event:ResultEvent, passThroughData:Object):void {
    var teams:ArrayCollection = event.result as ArrayCollection;
    assertThat('Team not created', 'Los Angeles Lakers', inArray(teams.toArray()));
}

First, we create a new team, then we load all the teams, and lastly, we verify that the newly created team is in the list. There are two important things to note: async stuff is everywhere ([Test(async)] metadata, AsyncToken, AsyncResponder, etc.), and there is a chain of functions (createTeam() chains to createTeam2() which chains to createTeam3()). In particular, the chain pattern is characteristic of any async testing. Every single non-trivial async test involves a chain of function calls to do the work of testing an asynchronous backend.

Here’s a simple diagram of the chain for the create team test:

Each diagram box is just a logic operation in our test, and they also happen to correspond exactly to the functions that make up the test chain.

Test: Trade Player

When testing the more complex client-side service layer operations, or simply writing more complex tests, the chain pattern often develops branches and sub-chains as various pieces of state are verified asynchronously.

A good example is the trade player operation, which we might test using a chain with two branches: one to verify the player was removed from old team, and one to verify the player was added to the new team. Here’s the diagram:

We don’t really care what goes on inside the client-side service layer to achieve this, or even what happens on the backend (it’s probably just as simple as changing the team_id column on the players table to the new team’s id). We only care that the test passes.

And the accompanying test code:

[Test(async)]
public function tradePlayer():void {
    var token:AsyncToken = service.tradePlayer('Carmelo Anthony', 'Denver Nuggets', 'Cleveland Cavaliers');
    token.addResponder(Async.asyncResponder(this, new TestResponder(tradePlayer2, fault), TIMEOUT));
}
public function tradePlayer2(event:ResultEvent, passThroughData:Object):void {
    var token:AsyncToken = service.getPlayersByTeam('Denver Nuggets');
    token.addResponder(Async.asyncResponder(this, new TestResponder(tradePlayer3, fault), TIMEOUT));
 
    var token2:AsyncToken = service.getPlayersByTeam('Cleveland Cavaliers');
    token2.addResponder(Async.asyncResponder(this, new TestResponder(tradePlayer4, fault), TIMEOUT));
}
public function tradePlayer3(event:ResultEvent, passThroughData:Object):void {
    var players:ArrayCollection = event.result as ArrayCollection;
    assertThat('Traded player not removed from old team', 'Carmelo Anthony', not(inArray(players.toArray())));
}
public function tradePlayer4(event:ResultEvent, passThroughData:Object):void {
    var players:ArrayCollection = event.result as ArrayCollection;
    assertThat('Traded player not added to new team', 'Carmelo Anthony', inArray(players.toArray()));
}

The interesting part occurs in the second step in the chain, tradePlayer2(). In this function, we use a pair of AsyncTokens, to fork the chain into two sub-chains. One sub-chain gets all the players on the old team and verifies that the traded player has been removed. And the other sub-chain gets all the players on the new team and verifies that the trade player has been added.

A Better Approach

Right now, the chained function approach is the only approach for testing an asynchronous client-side service layer. As another example, you can see the chained function approach appears again when I tested an LCDS-powered backend in my Getting Real with LCDS, Part 1 article at InsideRIA.com.

There has got to be something better, right? Chained functions work fine, but boy are they ugly looking in code. I’ve been having a discussion on the FlexUnit forums about better async testing. The general wisdom is that one could use the Sequence interfaces to build an async action and have the SequenceRunner manage the chain. Currently, the best documentation on Sequences is the old Fluint wiki doc. In enterprise Flex development, async backends tend to swarm like locusts, so I hope to have some code to show soon to streamline the testing process.

Files

• TestBasketball (run tests, view source, download)

Service Layer

First, assume we have a nice service layer in Flex that talks asynchronously to our backend. Just something simple to start:

public class MyService {
    public function getSomething(result:Function, fault:Function):AsyncToken {
        //call the backend
        var token:AsyncToken = backend.getSomething();
 
        //wire the callbacks to the result
        token.addResponder(new AsyncResponder(result, fault, token));
 
        return token;
    }
}

In this example, our service only has one method, getSomething() that takes two callback functions. It simply calls the backend method, wires up the callbacks (which get called when the backend method returns a result), and returns the token. It is absolutely critical that our callback-powered service method return the AsyncToken. The reason for this will become apparent.

We might use our service like this:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application
        xmlns:mx="http://www.adobe.com/2006/mxml"
        creationComplete="complete()">
 
    <mx:Script>
    <![CDATA[
        import com.saturnboy.services.MyService;
        private var service:MyService;
 
        private function complete():void {
            service = new MyService();
            service.getSomething(resultHandler, faultHandler);
        }
 
        private function resultHandler(result:Object, token:Object=null):void {
            lbl.text = result.result.name;
        }
 
        public function faultHandler(error:Object, token:Object=null):void {
            lbl.text = 'fault';
        }
    ]]>
    </mx:Script>
 
    <mx:Label id="lbl" text="initial" />
</mx:Application>

We make a call our service, and then use the callbacks to alter the UI however we want depending on the result. In common usage, the fact that our service returns an AsyncToken is worthless, it might as well return void. So, why did I say this is critical? Throw Fluint testing into the mix and it’s “Show em what’s behind door number 2, Johnny!”

Fluint Testing

Fluint provides two different async wrapper methods: asyncHandler and asyncResponder. The first allows a test to be wired to an async method by events, the second allows a test to be wired to an async method by a responder. Since the service method we’re trying to test doesn’t throw any events, we’ll need to use the latter. So inside a Fluint test case, we have our test method:

public function testGetSomething():void {
    //call service with dummy callback
    var token:AsyncToken = service.getSomething(dummyResult, dummyFault);
 
    //create async test responder
    var responder:IResponder = asyncResponder(
            new TestResponder(testHandler, faultHandler), 1000, token);
 
    //wire test responder as 2nd callback
    token.addResponder(responder);
}
private function testHandler(result:Object, passThroughData:Object):void {
    assertEquals('something', result.result.name);
}

The trick is to wire a second callback via Fluint’s asyncResponder helper that actually does the testing, and just give the original service call some dummy callbacks. Note that if the service method didn’t return its AsyncToken there would be no way to wire a second callback. The Fluint async helper do two import operations: they handle the event or call the callback AND they mark the test method as an async method so the result is correctly reported by the test harness. You can read more about Async Testing in Fluint’s wiki. The rest of Fluint is your standard chain of crap borrowed from JUnit: test runner, test suites, and test cases.

Digging Deeper: It is equally critical to use dummy callbacks in the original service method call because in a failure situation they will cause Flash Player to error out instead of being caught by Fluint and reported as a test failure.

Files

The complete code is up on GitHub here: test_fluint_async. The code is MIT licensed and includes a working fluint.swc (see below) plus a mock async backend (so timeouts and faults are easily testable).

Alas, Fluint v1.1.0 was built incorrectly and is missing the TestResponder class (see issue 35). So if you want to try out Fluint in your project, I recommend you grab it from svn and build the swc yourself. Hopefully, this will all be fixed in the next release.

UPDATE: Fluint v1.1.1 was release on May 1, 2009 and fixes this issues and a few others. Download it here.