Archive for the 'Evan Lang' Category

August 3, 2011

Simple Tips: Application Image Processing

A lot of people are familiar with the WriteableBitmapEx project on Codeplex. I got an opportunity to use it in a recent project of mine, and although many of its functions are very helpful, I noticed a couple common image processing functions it was missing. It supports resizable blits (with bilinear filtering), flips, and rotates, which are nice. More impressively, it supports arbitrary convolutions, which are very powerful and versatile in the image processing world. What it was missing, however, was the ability to perform a linear colorspace transform, and apply a color ramp lookup table.

Linear transforms in an image’s color space (RGB in the case of this implementation) can perform a variety of functions. They work by treating each pixel as a vector of color components, <R, G, B, A>. That vector is transformed by a 4×4 matrix, and the output vector represents the new color. To use this function, call

WriteableBitmapColorExtensions.TransformColors(Matrix3D matrix);

The matrix you supply determines how the color is transformed.  The alpha component in most images is usually 1, making the matrix usually affine, though the method does not give it special treatment, so you may get some… interesting… behavior with translucent images, depending on how you put your matrix together.

The WriteableBitmapColorExtensions class defines a few common ones:

   NegativeTransform – Produces an image’s film negative

$latex \left( \begin{matrix}1&0&0&-1 \\ 0&1&0&-1 \\ 0&0&1&-1 \\ 0&0&0&1 \end{matrix} \right)$

   BlackAndWhiteTransform – Transforms the image into black and white

$latex \left( \begin{matrix}0.30&0.30&0.30&0 \\ 0.59&0.59&0.59&0 \\ 0.11&0.11&0.11&0 \\ 0&0&0&1 \end{matrix} \right)$

   SepiaTransform – Transform an image into sepia

$latex \left( \begin{matrix}0.393&0.349&0.272&0 \\ 0.769&0.686&0.534&0 \\ 0.189&0.168&0.131&0 \\ 0&0&0&1 \end{matrix} \right)$

 

What else can you do with it? You can try brightness changes:

$latex Brightness(B) = \left( \begin{matrix}B&0&0&0 \\ 0&B&0&0 \\ 0&0&B&0 \\ 0&0&0&1 \end{matrix} \right) $

 

Note you can vary the brightness of independent color components, if you like.

You can also desaturate an image by interpolating between the identity matrix and the black and white matrix. If your saturation level goes from 0 (black and white) to 1 (normal):

$latex Saturation(S)= Brightness(S) + BlackAndWhiteTransform * (1 – S) $

You can even do a hue shift. A hue shift basically rotates a color vector around the <1, 1, 1> axis:

HueShift(Degrees) = Matrix3D.Identity.Rotate(Degrees, new Vector3D(1,1,1))

The purists among us will note that this isn’t exactly right… to do a true hue shift you need to normalize the brightness of the color components first (you can use the Brightness(S) function above), but this gets the idea across. The sample image here uses a subtle rotation of 15 degrees.

Those are the ‘ordinary’ functions that a color transform can do, but you can stylize things as well.  For example, you can swap around or even turn off color components:

There are other interesting things you can do with TransformColors as well, such as skewing colors or projecting them onto a plane (or a line, creating a nice two-tone effect—this is essentially what the black and white matrix does). And remember the most powerful feature of linear transformations: you can take any or all of these effects, combine them into a single matrix, and apply all of them to an image at once without any additional performance hit.

The other function introduced into the library, which I will elaborate more on in my next post, is WriteableBitmapColorExtensions. RampColors. This method replaces all the colors in an image using a lookup table. This is useful for nonlinear things like gamma correction.  In fact, gamma correction is by far the most common application of this function, so I included a special method just for it, WriteableBitmapColorExtensions.AdjustGamma.

 

June 30, 2011

Kinect Code Camp – Evan Lang

Microsoft recently had a special Channel 9 Live event for the launch of Kinect. Before this event Microsoft held a 24 hour Code Camp to allow developers to play with the Kinect SDK for 24 hours to see what they could come up with.
One of our UX developers, Evan Lang, led a one man team to create an app that turns you into a Cylon from Battlestar Gallactica as well as developing a gesture-driven PowerPoint app. Check out the video below

View the original post here

May 9, 2011

Using Kinect for NUI Part 5 – Multiuser Scenarios

IdentityMine has been experimenting with the best approaches to produce a good User Experience (UX) on Kinect – specifically ways that users might interact with a software application GUI, as opposed to playing games. The software development industry is enthusiastically grappling with this issue.

This is the fifth post of a 5-post series that will dive into User Interface considerations when developing a software applications using gesture and Kinect sensors.

Multiuser Scenarios

The Kinect can reasonably support up to 4 users simultaneously, if personal boundary issues aren’t a big concern. Of course, most of the issues regarding multiuser applications will be specific to the application’s design and intended purpose – as far as general motion is concerned, not much changes. The application will need to keep track of and support a cursor and focus model for each user independently. Users can be easily distinguished from each other by assigning each a color and displaying it prominently in their cursor and in highlights displayed when a user focuses on a specific control. Continue reading ‘Using Kinect for NUI Part 5 – Multiuser Scenarios’

April 27, 2011

Using Kinect for NUI Part 4 – Buttons

IdentityMine has been experimenting with the best approaches to produce a good User Experience (UX) on Kinect – specifically ways that users might interact with a software application GUI, as opposed to playing games. The software development industry is enthusiastically grappling with this issue.

This is the fourth post of a 5-post series that will dive into User Interface considerations when developing a software applications using gesture and Kinect sensors.

Buttons are ubiquitous throughout computing. In many ways, they are the fundamental GUI control. The metaphor works with a mouse: clicking a mouse button translates directly to clicking the virtual button. The metaphor works with a touchscreen: pressing the virtual button works like pressing any real button.

The metaphor doesn’t work with the Kinect; in fact, it completely breaks. There is no machinery in front of the user to press anything with. It is possible to recognize a gesture that looks sort of like the user is pressing an invisible, midair button, but users have different ideas of just what that action is supposed to look like. Some users press ‘down’ (moving their hand towards the screen) and then quickly back, like tapping someone on the shoulder. Others are more deliberate, and move their hand in, wait, then move it back out. Still others will only move their hand towards the screen, and leave it there before letting their hand fall to rest (like they’re punching something). Finally, many are familiar with the way Xbox UI works, and will simply hold still over the button expecting it to eventually activate. What’s clear is that the metaphor is too far gone to be of much use.

This represents a major departure from traditional UX design. Simply put, the use of buttons needs to be greatly reduced when we talk about motion interaction UI. Instead, special controls with specific purposes will replace many of the common uses for buttons. Time and experience will reveal a standard set of controls that can cover their traditional functionality. Buttons will still be valuable, using the Xbox’s hover-over approach, but must always be weighed against the usability impact of busy-waiting. For games it’s not really an issue, since the UI is inherently simple; but in applications that call for a complex UI, busy-waiting for buttons to activate will quickly become frustrating.

EXAMPLE

Here are some simple ideas for controls that would work in common scenarios where you might otherwise use a button.

A common scenario asks the user to either confirm or deny something. Usually this is represented by two Yes/No or OK/Cancel buttons. Instead, the user could interact with a Kinect control that—while it’s in focus—includes a slider-like visual that tracks with their hand. Slide it to the left to deny, to the right to confirm.

Another common use for a button is to navigate in a dialogue-type scenario. With the Kinect, you could use something that looks like a joystick; while focused, swipe to the left, right, up, down, or whatever direction makes sense to go forward/back/finish/cancel.

Checkboxes are just buttons with a special style, so they don’t work as-is. But if you change the metaphor to an on/off switch, the rest solves itself.

In the next installment of this series, we will focus on Multiuser Scenarios.  You can also follow my personal blog here!

Part 1: Introduction

Part 2: Gestures

Part 3: Cursor

Part 5: Multiuser Scenarios

April 25, 2011

Using Kinect for NUI Part 3 – Cursor

IdentityMine has been experimenting with the best approaches to produce a good User Experience (UX) on Kinect – specifically ways that users might interact with a software application GUI, as opposed to playing games. The software development industry is enthusiastically grappling with this issue.

This is the third post of a 5-post series that will dive into User Interface considerations when developing a software applications using gesture and Kinect sensors.

Gesturing with the Kinect is similar to using a keyboard; signals are sent to an application. But how does the application know what to do with them? If a user has two text boxes, keyboard input is sent to the control with focus. The same is true with gesturing. To focus on a particular control, just move the cursor over it; to help a user identify which controls they can interact with, the cursor should be ‘attracted’ to such controls (like a magnet) so they are easier to identify and navigate to. The control should animate or otherwise indicate that it does indeed have focus.

We’ve been experimenting with two ways to use the cursor with gesture. In the first experiment we physically pointed at a control, arm fully extended. Unfortunately, pointing is too subjective (more so than you might think), and the Kinect sensor is too imprecise for simple pointing to be practical. So we modified the experiment to function more like a Wii controller, where the direction of the point is a suggestion rather than a requirement.  The Wii cursor is rarely positioned precisely where the controller is pointing, but it doesn’t really matter; as long as the user can see the cursor they can move it by moving the controller, and that’s what matters. The same applies with the Kinect and pointing.

Pointing is effective because it doesn’t get confused with other gestures. While the user’s arm is fully extended, the application recognizes that the only thing the user is doing is positioning the cursor. The user is not going to accidentally turn a page on an e-reader, for example, by pointing too fast. Pointing does have a drawback, however. In the world of touchscreen-based NUI, the term ‘gorilla arm’ refers to the pain and fatigue users experience after using a vertical touchscreen for more than a few minutes. Inflicting pain on users is bad UX, and pointing risks causing a lot of gorilla arm.

So how can we move a cursor, independent of gesturing, without hurting people? One idea comes from the realization that users have two hands (let’s set aside issues of accessibility for this discussion, since Kinect’s motion is actually antithetical to accessibility.) A user could use one hand for cursor movement without pointing, and the other for gesturing. There are several questions about this approach; Is it possible to accurately recognize when the user is done positioning the cursor and wants to rest their arm without the cursor moving? Is it necessary to hold the cursor stationary during that time? Will cursor movement and gesturing conflict with each other or require users to bang their arms together? Can lefties and righties have the same dominant-hand experience? We’re looking into all of these issues, and hope to have answers soon (stay tuned.)

In the next installment of this series, we will focus on Buttons.  You can also follow my personal blog here!

Part 1: Introduction

Part 2: Gestures

Part 4: Buttons

Part 5: Multiuser Scenarios