I needed a very simple Twitter cache for a project I’m working on. And I was very happy to trade off some realtime accuracy for reliability. In addition to caching the tweets, I also needed to pre-process them into css-able html with clickable links, usernames, and hashtags. The web had a few nice examples of how to use regular expressions to parse the raw tweet text, but I decided to take what I liked and do the rest myself.

Links

Here’s the PHP code for parsing links out of the raw tweet text:

$text = preg_replace(
    '@(https?://([-\w\.]+)+(/([\w/_\.]*(\?\S+)?(#\S+)?)?)?)@',
     '<a href="$1">$1</a>',
    $text);

I only wanted http and https links, with an optional query part (\?\S+)? and an option anchor part (#\S+)?. The conversion of a text link into an html link is done using back references, which in PHP is $1, $2, etc. In the expression above, I use $1 twice to put the matched link into both the href attribute and the link text.

Users

Here’s the PHP code for parsing Twitter usernames:

$text = preg_replace(
    '/@(\w+)/',
    '<a href="http://twitter.com/$1">@$1</a>',
    $text);

Nothing special, just take the @ and all following word characters (letters, digits, and underscores), and turn it into a user link.

Hashtags

Here’s the PHP code for parsing Twitter hashtags:

$text = preg_replace(
    '/\s+#(\w+)/',
    ' <a href="http://search.twitter.com/search?q=%23$1">#$1</a>',
    $text);

Getting the hashtags right was the most tricky of the three. I decided to only grab hashtags that were proceeded by one or more spaces. The real magic is the %23 in the query string, which forces a search on the complete hashtag, including the # part. For example, compare a search for #flex to a search for flex.

The Cache

The cache is just a simple cron job that periodically queries Twitter and retrieves the latest tweets. Most importantly, the cache fails gracefully if Twitter is inaccessible, which it does by doing exactly nothing if Twitter is down. This guarantees that my app always has valid data (when my server is up, the cache is up too), but with the possibility that the data is a little old.

Here’s the notable function in the cache:

function getTweets($user, $num = 3) {
    //first, get the user's timeline
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_URL, "http://twitter.com/statuses/user_timeline/$user.json?count=$num");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    $json = curl_exec($ch);
    curl_close($ch);
 
    if ($json === false) { return false; } //abort on error
 
    //second, convert the resulting json into PHP
    $result = json_decode($json);
 
    //third, build up the html output
    $s = '';
    foreach ($result as $item) {
        //handle any special characters
        $text = htmlentities($item->text, ENT_QUOTES, 'utf-8');
 
        //build the metadata part
        $meta = date('g:ia M jS', strtotime($item->created_at)) . ' from ' . $item->source;
 
        //parse the tweet text into html
        $text = preg_replace('@(https?://([-\w\.]+)+(/([\w/_\.]*(\?\S+)?(#\S+)?)?)?)@', '<a href="$1">$1</a>', $text);
        $text = preg_replace('/@(\w+)/', '<a href="http://twitter.com/$1">@$1</a>', $text);
        $text = preg_replace('/\s#(\w+)/', ' <a href="http://search.twitter.com/search?q=%23$1">#$1</a>', $text);
 
        //assemble everything
        $s .= '<p class="tweet">' . $text . "<br />\n" . '<span class="tweet-meta">' . $meta . "</span></p>\n";
    }
 
    return $s;
}

First, we query the user’s JSON timeline using cURL. Second, we use PHP’s awesome json_decode function to convert the JSON into objects. And lastly, we iterate over the tweets and parse everything into our desired HTML output.

Here some sample output from my twitter feed:

<p class="tweet">Been reading Programming Goggle App Engine. Actually feeling dumber now than before I started. Too much to learn.<br /> 
<span class="tweet-meta">2:58pm Feb 14th from <a href="http://www.tweetdeck.com/" rel="nofollow">TweetDeck</a></span></p>
 
<p class="tweet">Blog Post :: Async Testing with FlexUnit 4 :: <a href="http://bit.ly/cGLnaI">http://bit.ly/cGLnaI</a><br /> 
<span class="tweet-meta">3:33pm Feb 11th from <a href="http://www.tweetdeck.com/" rel="nofollow">TweetDeck</a></span></p>
 
<p class="tweet">Blog Post :: A Better HTML Template for Flex 4 :: <a href="http://bit.ly/70DLsj">http://bit.ly/70DLsj</a><br /> 
<span class="tweet-meta">12:55pm Jan 25th from <a href="http://www.tweetdeck.com/" rel="nofollow">TweetDeck</a></span></p>

Once I have the output, I can do whatever I want with it: save to disk, stick it in the database, keep it in memory, cache it in memcache, etc. In my case, I wanted the simplest possible option, so I chose to write it out as a static html file.

The end. The rest of the app’s not ready yet…

Alas, LCDS 3 doesn’t support the notion of transaction (in the sense of a Database Transaction). I assume that somewhere deep within the system transactions are used to ensure correctness of data in the db, but none of this is exposed to the user. Fortunately, it is possible to bundle up a bunch of updates on the client and push them all at once to the server. Fake transactions, as I like to call them, are quite useful. But they can also be dangerous because they are not really transactions (hence the fake part), and don’t always work as you would hope.

Fake It

Faking a tranaction is done using the autoCommit property on a generated service. According to the docs, setting autoCommit to false blocks Data Management from pushing any changes to the server until commit() is called manually.

Here’s a direct quote:

“…set a DataService component autoCommit property to false to allow only manual calls to the commit() method. It is important to set autoCommit to false when you are going to make more than one change…so that the DataService component can batch those changes and send them in one batch to the destination.”

So to make life easy, I made a simple static function that does the autoCommit gymnastics:

public static function fakeIt(service:DataService, func:Function):AsyncToken {
    if (!service.autoCommit) {
        throw new Error("ERROR: autoCommit is already off.");
    }
    if (service.commitRequired) {
        throw new Error("ERROR: another transaction is already open.");
    }
 
    service.autoCommit = false;
    func();
    var token:AsyncToken = service.commit();
    service.autoCommit = true;
    return token;
}

After some error handling, we toggle autoCommit to false, call our function, then commit any updates, toggle autoCommit back on, and return the token.

Usage

Using our fake transaction function is straight forward, and typically involves passing in and inline anonymous function.

Imagine our favorite example of teams and players, where each team has one-to-many players. We might choose to perform a sequence of operations to add a new player to an existing team, like this:

var token:AsyncToken = FakeTransaction.fakeIt(
    teamService.serviceControl,
    function ():void {
        //rename the team
        team.name = 'Denver Nuggetz';
 
        //create player
        var p:Player = new Player();
        p.name = 'Carmelo Anthony';
 
        //wire both side of relationship
        p.team = team;
        team.players.addItem(p);
 
        playerService.createPlayer(p);
    });
 
token.addResponder(new AsyncResponder(successHandler, faultHandler));

In this case, the fake transaction is used to ensure the team is renamed and the player is added. Both operations occur together, so it should not be possible to have the new player on the old team, or have the renamed team without the new player.

Warning! Danger!

Experience has shown me that fake transactions don’t always work as one would expect if they were real transactions. Sometimes, particularly when you are manipulating entities already under Data Management, it seems like operations are not really batched but instead executed one at a time. Thankfully, it appears that LCDS is order preserving, and by that I mean it doesn’t magically re-order things. LCDS always executes operations in the order they come in. I can only recommend you test your application vigorously to ensure what you expect to happen actually happens (FlexUnit4 is pretty awesome for testing async backends).

Files

• FakeTransaction.as

I just wrote about how to handle special characters in Flex 4 when written as HTML entities in MXML. Now I’ve moved my data with the special characters out of MXML and down into a MySQL database. Data access is provided by a vanilla LCDS 3 backend. I now have a very different problem than what I had before: How do I get UTF-8 data out of the database with LCDS and onto the display?

MySQL and UTF-8

In theory, LCDS is perfectly happy with special characters and foreign languages (here’s a link to supported characters sets in LiveCycle ES2). So this time around, our problem has nothing to do with Flex 4 or LCDS, instead it’s all about the database. For our example, we’ll skip the pure model driven development route and just start with a simple database with a single players table.

Create the database:

CREATE DATABASE ballerz DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;
CREATE USER 'baller'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON ballerz.* TO baller@localhost;

It is possible to configure MySQL to default to UTF-8 friendly behavior but the CREATE DATABASE command guarantees that the newly created db will be happy.

Create the players table:

CREATE TABLE players (
  id int(11) NOT NULL AUTO_INCREMENT,
  name varchar(255),
  PRIMARY KEY (id)
);

Nothing special here, just use VARCHAR for the text fields. In this case, we only have the player’s name.

Insert some sample data:

INSERT INTO players (id,name) VALUES (1, "Carmelo Anthony");
INSERT INTO players (id,name) VALUES (2, "Chaunçey Billups");
INSERT INTO players (id,name) VALUES (3, "Nenê");
INSERT INTO players (id,name) VALUES (4, "Këñÿõn Martin");
INSERT INTO players (id,name) VALUES (5, "LeBrøn James");
INSERT INTO players (id,name) VALUES (6, "Mo Williams");
INSERT INTO players (id,name) VALUES (7, "Shaquille O†Neal");
INSERT INTO players (id,name) VALUES (8, "Ænderson Varejao");
INSERT INTO players (id,name) VALUES (9, "Zýdrunãs Ílgauskãs");

I put some extra special characters into the INSERT statements just for fun.

Digging Deeper: When connecting to MySQL from the commandline you can use the --default-character-set=utf8 option to force your terminal to show UTF-8 characters correctly.

LCDS and UTF-8

Now that the database is correctly setup to handle UTF-8, the rest of the LCDS setup is straight forward (see my getting started part 1 and part 2 posts). Create a new LCDS webapp via copy-and-paste from the template app, then fire up Flash Builder 4 and get to work. In the Modeler plugin, configure a new RDS connection and just drag-and-drop the players table into the model.

Here’s a screenshot of our LCDS model:

And here’s a screenshot of the running app (remember this is backed by LCDS, so no running demo):

Lastly, the frontend code showing just the highlights:

<?xml version="1.0" encoding="utf-8"?>
<s:Application ...
        creationComplete="complete()">
 
    <fx:Script>
        <![CDATA[
           private function complete():void {
                getPlayers.token = playerService.getAll();
            }
        ]]>
    </fx:Script>
 
    <fx:Declarations>
        <s:CallResponder id="getPlayers" />
        <Ballerz:PlayerService id="playerService" />
    </fx:Declarations>
 
    <s:List 
            dataProvider="{getPlayers.lastResult}"
            labelField="name" />
</s:Application>

Again, no magic here, I use a simple getAll() query to retrieve the entire players table then feed it into a List via the CallResponder’s lastResult property.

Conclusion

So the moral of our story is: if you correctly configure your database to support UTF-8 and you correctly get UTF-8 data into your tables, then everything just works. LCDS will transparently get data out of the db and Flex will transparently get it onto the screen.

Files

• Ballerz.tgz
• create_db.sql
• create_tables.sql
• data.sql

I tend to spend a large portion of my development time worrying about the various interfaces across the application. I like to worry about UX (aka the interface between the user and my app). I like to worry about the ORM (aka the interface between the database and my code). And I especially like to worry about the client-side service layer (aka the interface between the backend and the frontend). When I worry, I very quickly find myself writing tests.

The new hotness in Flex testing is, of course, FlexMonkey, developed and open-sourced by my company, Gorilla Logic. The next best Flex testing platform, and the new hotness in its own right, is FlexUnit 4, developed and open-sourced by our partners at Digital Primates. FlexUnit 4 is the Flex 4 unit testing framework. Because of its awesome async testing support, along with many other great features, it is ideally suited to test client-side service layers. In this post, I’m going to explore async testing with FlexUnit 4 to better understand how I can help mitigate the pain of asynchronous backend services that are ever-present in enterprise Flex applications.

Test: Create Team

Let’s imagine I have my favorite data model of teams and players, with a one-to-many relationship between the two entities. Next, let’s assume that I wrote a beautiful client-side service layer that has both basic CRUD operations like create team and delete team, and more complex operations like trade player. I’d like to cover everything with a set of tests so I can spend my time worrying about other things.

Here’s a simple async test case for the basic create team operation:

[Test(async)]
public function createTeam():void {
    var token:AsyncToken = service.createTeam('Los Angeles Lakers');
    token.addResponder(Async.asyncResponder(this, new TestResponder(createTeam2, fault), TIMEOUT));
}
public function createTeam2(event:ResultEvent, passThroughData:Object):void {
    var token:AsyncToken = service.getAllTeams();
    token.addResponder(Async.asyncResponder(this, new TestResponder(createTeam3, fault), TIMEOUT));
}
public function createTeam3(event:ResultEvent, passThroughData:Object):void {
    var teams:ArrayCollection = event.result as ArrayCollection;
    assertThat('Team not created', 'Los Angeles Lakers', inArray(teams.toArray()));
}

First, we create a new team, then we load all the teams, and lastly, we verify that the newly created team is in the list. There are two important things to note: async stuff is everywhere ([Test(async)] metadata, AsyncToken, AsyncResponder, etc.), and there is a chain of functions (createTeam() chains to createTeam2() which chains to createTeam3()). In particular, the chain pattern is characteristic of any async testing. Every single non-trivial async test involves a chain of function calls to do the work of testing an asynchronous backend.

Here’s a simple diagram of the chain for the create team test:

Each diagram box is just a logic operation in our test, and they also happen to correspond exactly to the functions that make up the test chain.

Test: Trade Player

When testing the more complex client-side service layer operations, or simply writing more complex tests, the chain pattern often develops branches and sub-chains as various pieces of state are verified asynchronously.

A good example is the trade player operation, which we might test using a chain with two branches: one to verify the player was removed from old team, and one to verify the player was added to the new team. Here’s the diagram:

We don’t really care what goes on inside the client-side service layer to achieve this, or even what happens on the backend (it’s probably just as simple as changing the team_id column on the players table to the new team’s id). We only care that the test passes.

And the accompanying test code:

[Test(async)]
public function tradePlayer():void {
    var token:AsyncToken = service.tradePlayer('Carmelo Anthony', 'Denver Nuggets', 'Cleveland Cavaliers');
    token.addResponder(Async.asyncResponder(this, new TestResponder(tradePlayer2, fault), TIMEOUT));
}
public function tradePlayer2(event:ResultEvent, passThroughData:Object):void {
    var token:AsyncToken = service.getPlayersByTeam('Denver Nuggets');
    token.addResponder(Async.asyncResponder(this, new TestResponder(tradePlayer3, fault), TIMEOUT));
 
    var token2:AsyncToken = service.getPlayersByTeam('Cleveland Cavaliers');
    token2.addResponder(Async.asyncResponder(this, new TestResponder(tradePlayer4, fault), TIMEOUT));
}
public function tradePlayer3(event:ResultEvent, passThroughData:Object):void {
    var players:ArrayCollection = event.result as ArrayCollection;
    assertThat('Traded player not removed from old team', 'Carmelo Anthony', not(inArray(players.toArray())));
}
public function tradePlayer4(event:ResultEvent, passThroughData:Object):void {
    var players:ArrayCollection = event.result as ArrayCollection;
    assertThat('Traded player not added to new team', 'Carmelo Anthony', inArray(players.toArray()));
}

The interesting part occurs in the second step in the chain, tradePlayer2(). In this function, we use a pair of AsyncTokens, to fork the chain into two sub-chains. One sub-chain gets all the players on the old team and verifies that the traded player has been removed. And the other sub-chain gets all the players on the new team and verifies that the trade player has been added.

A Better Approach

Right now, the chained function approach is the only approach for testing an asynchronous client-side service layer. As another example, you can see the chained function approach appears again when I tested an LCDS-powered backend in my Getting Real with LCDS, Part 1 article at InsideRIA.com.

There has got to be something better, right? Chained functions work fine, but boy are they ugly looking in code. I’ve been having a discussion on the FlexUnit forums about better async testing. The general wisdom is that one could use the Sequence interfaces to build an async action and have the SequenceRunner manage the chain. Currently, the best documentation on Sequences is the old Fluint wiki doc. In enterprise Flex development, async backends tend to swarm like locusts, so I hope to have some code to show soon to streamline the testing process.

Files

• TestBasketball (run tests, view source, download)

Adobe took a big step forward using swfobject.js to handle swf embedding in Flex 4’s index.template.html, but in my opinion they didn’t go quite far enough.

Here are my changes:

1. Short and sweet, so I can understand what the hell is going on.
2. Use Google AJAX Libraries API to include swfobject.js.
3. No IE6 crap.
4. No Browser History crap.

Here’s my current version:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>${application}</title>
 
<style type="text/css" media="screen">
html, body, #flashcontent { height:${height}; }
body { margin:0; padding:0; overflow:hidden; }
</style>
 
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/swfobject/2.2/swfobject.js"></script>
<script type="text/javascript">
swfobject.embedSWF(
    '${swf}.swf',
    'flashcontent',
    '${width}', '${height}',
    '${version_major}.${version_minor}.${version_revision}',
    '${expressInstallSwf}',
    false,
    { 'bgColor':'${bgcolor}' });
</script>
</head>
 
<body>
<div id="flashcontent">
Flash is required.  <a href="http://www.adobe.com/go/getflashplayer">Get it here!</a>
</div>
</body>
</html>

Flashvars

Using flashvars is easy with SWFObject (see the embedSWF() method in the docs), just put them in as a Javascript hash like this:

<script type="text/javascript">
swfobject.embedSWF(
    '${swf}.swf',
    'flashcontent',
    '${width}', '${height}',
    '${version_major}.${version_minor}.${version_revision}',
    '${expressInstallSwf}',
    { key1:'val1', key2:'val2', key3:'val3' },
    { 'bgColor':'${bgcolor}' });
</script>

View Source

Lastly, here’s a little trick that I use to embed flash into a post on this blog (or anywhere else) that enables view source to work with a relative url, so I can leave the viewSourceURL attribute alone (defaults to viewSourceURL="srcview/index.html").

<script type="text/javascript">
swfobject.embedSWF(
    '${swf}.swf',
    'flashcontent',
    '${width}', '${height}',
    '${version_major}.${version_minor}.${version_revision}',
    '${expressInstallSwf}',
    false,
    { 'bgColor':'${bgcolor}', base:'.' });
</script>

Adding the base parameter, and setting it to '.', tells the embedded SWF to lookup urls relative to itself. The net effect is that view source just works.

Files

• index.template.html