Silverlight Multi-Touch In the Real World (once you go touch you never go back)

John Papa asked me to help do a 10 minute segment for the Silverlight TV show for Channel 9. This got me thinking about multi touch from a demo standpoint and made me really think about what we have learned so far.

In the general since we (as an industry or society) are really just starting to see touch become a major part of our day to do human machine interaction. On the Microsoft stack using Surface, Windows 7 and WPF/Silverlight Multi touch support it is really becoming standard stuff. At Wirestone we have really been on the fore front of doing user experience testing and real world development over the past year primarily in the retail space. Now the specific projects I can't talk about yet but let me see if I can come up with some usability rules that might help in your development. If you first just trying to figure out how to do it in Silverlight (starting in at least Silverlight3 or better) there is one main API you need to care about.

take a look at this line of code:

Touch.FrameReported += new TouchFrameEventHandler(Touch_FrameReported);

this associates our one event handler that we care about. now lets take a look at using the event handler:

void Touch_FrameReported(object sender, TouchFrameEventArgs e)
{
foreach (TouchPoint tp in e.GetTouchPoints(Positions))
{
if (tp.Action == TouchAction.Down)
{
// DO SOMETHING
}
else if (tp.Action == TouchAction.Up)
{
// DO SOMETHING ELSE
}
else if (tp.Action == TouchAction.Move)
{
// OR DO SOMETHING MORE ELSE
}
}
}

you'll note that basically we are looping through the touch points, we get touch id's and based on the touch action we can do whatever it is we want. I encourage you to take a look at Tim's post on this and download his sample as he goes through this in detail and built a great multi touch app for testing how touch works using this method.

check it out here and download Tims app. (it also converts to SL4 seamlessly):

http://timheuer.com/blog/archive/2009/07/30/silverlight-3-multi-touch-introduction-fundamentals-basics.aspx

so now that we talked out how to do basic multi touch lets get back to my experience in user experience development with multi touch.

In our work we have been doing allot of users studies with Retail systems including kiosks, touch walls, surface and the like. But my favorite lesson learned has todo with the UX professionals... One day have a long day of debugging and unit testing I went home and wanted to check email on our home computer. I forget what it was exactly but I went to touch the screen and for several minutes I couldn't, I think I almost broke it before I realized that it wasn't touch. And the next week I realized that everyone on the team was having the same problem. Worse still once you realized that was a problem I found I can't stop... really once I went touch... I can't go back. don't get my wrong 'touch' will not replace the mouse in its current form/technology but it extends our experience and allows technology to be better integrated into our lives.

So lets talk about another story we saw in users studies. There are lots of different kinds of hardware, apps and technologies. One good thing to watch out for is the 'kinds' of things that register as a 'touch'. So in Silverlight on my multitouch laptop its not a big deal but I have seen hardware that the light conditions effect touches and even worse some of the hard where is so sensitive that the brush of a sleeve can cause problems. Granted this is more a surface thing then the typical Silverlight retail kiosk but still. When building experiences you need to get they system, hardware and any other bits in place so that you can see how users interact with the system so I can identify things like this 'before' they go live.

When considering design some good things to think about include How the user will tend to interact with the system. On thing you notice is that you loose those the mouse. no mouse overs etc. We have found that user's need and expect feedback. and many times when users 'touch' elements in the UI and nothing happens the user then tends to think that they did something wrong. Help the users by providing feed back even if its just a subtle 'bonk' noise that lets the users know the touch was received. When the user does touch something that does something make sure that 'something' shows that on the screen IE drop shadow forms and element moves slightly etc.

Also remember that we have trained our users to do and respond to computer UI a certain way. Consider that frequently we see users 'double' touch where the users expects there to be something like a double click. Users also tend to click the bottom of images or things below the image before things above. I can't express enough how much better a system is that has gone through extensive user testing and user centric design can be from a User Experience standpoint.

Another big topic in now days is gestures, in Silverlight this can mean 'more' code :) but focusing on the user we found that users tend to do only a few things 90% of the time or more... Touch, move and pinch... everything else that users use once they get past the fact they can touch everything is secondary at least currently. Although subtle things like letting users resize via pinch and the use of touch in the UI that has been carefully thought out regarding interaction really cranks up the end product.

But given considerations like that... touch is where its at and once you go touch, you never go back. :)