Conference Presentation: Customer Researchers as Health Professionals

A colleague and I will be in Las Vegas next week for the yearly AMA Marketing Research Conference. We will be presenting a session on user experience research entitled Customer Researchers as Health Professionals - How eBay Uses Research to Improve Product Health (view a slightly shortened version online here, or below).

I'll try to summarize briefly what we will be talking about. We start with a little context about how research works at eBay and where it sits in the organization, and then we go over the research strategy I head up at eBay, called Product Health. We use a variety of quantitative and qualitative user experience research methods to track the health of different eBay site areas (or key flows) over time. Below is a schematic of the different research components of Product Health, which, as you can see, aims to cover a broad and holistic view of the Product:

It is our point of view that there is no single research methodology that can tell a complete story. If you want to have a holistic view of your product and how users feel about it and interact with it, it is essential to combine a variety of methodologies together, and keep doing the research over time so that you are able to (1) accurately assess how you're doing, and also (2) identify the areas that need to be improved. The quantitative components help us understand what is happening, and the qualitative components help us understand why we're seeing what we're seeing, and how to fix it.

We also spend some time talking about a synthesis project we did, where we pulled together the results from all the different Product Health research components, and used the insights to come up with 5 guiding principles for product development, which we believe translates pretty well to any e-commerce organization:

• Build Trust. Be responsive to customer needs for security and service. Ensure the quality and accuracy of the information available on the site. Build responsibility and accountability in the online community.
• Simplify. Streamline and clarify processes, navigation, site performance issues and fees that can make the customer’s experience unnecessarily complex and impact commerce.
• Be Relevant. Design an experience around how people naturally explore, evaluate and purchase items. Provide relevant, quality information that supports this experience.
• Provide value. Align buyers and sellers with a common means of determining item value fees. Treat sellers as “paying customers” who deserve value-added service for the fees they pay.
• Connect people through commerce. Leverage areas on the site to connect people with common interests through commerce.

We feel that this is a great example of how user-generated insights gathered in a methodologically robust way can drive product strategy and resource allocation effectively within an online organization. So, anyway, if you happen to be in Vegas next week, come check it out!

Usability testing on Halo 3

Wired Magazine just published an article about usability testing on Halo 3, the much-anticipated next installment in the Halo video game franchise, and the first Halo game for Xbox 360.  Video game usability has always fascinated me (and I try to get hands-on experience in this area as much as I can...), and this is the first time I've seen a mainstream magazine cover it in such detail.

Halo is a genre-changing first person shooter that brought gaming to a new level with its intricate story-line, cinematic feel and epic soundtrack.  And the creators got there through endless hours of testing...  Here are some excerpts from the article showing how they left no stone unturned:

The room we're monitoring is wired with video cameras that Pagulayan can swivel around to record the player's expressions or see which buttons they're pressing on the controller. Every moment of onscreen action is being digitally recorded.

Midway through the first level, his test subject stumbles into an area cluttered with boxes, where aliens — chattering little Grunts and howling, towering Brutes — quickly surround her. She's butchered in about 15 seconds. She keeps plowing back into the same battle but gets killed over and over again.

"Here's the problem," Pagulayan mutters, motioning to a computer monitor that shows us the game from the player's perspective. He points to a bunch of grenades lying on the ground. She ought to be picking those up and using them, he says, but the grenades aren't visible enough. "There's a million of them, but she just missed them. She charged right in." He shakes his head. "That's not acceptable."

After each session Pagulayan analyzes the data for patterns that he can report to Bungie. For example, he produces snapshots of where players are located in the game at various points in time — five minutes in, one hour in, eight hours in — to show how they are advancing. If they're going too fast, the game might be too easy; too slow, and it might be too hard. He can also generate a map showing where people are dying, to identify any topographical features that might be making a battle onerous. And he can produce charts that detail how players died, which might indicate that a particular alien or gun is proving unexpectedly lethal or impotent.

Pagulayan and his team have now analyzed more than 3,000 hours of Halo 3 played by some 600 everyday gamers, tracking everything from favored weapons to how and where — down to the square foot — players most frequently get killed.

The article goes into many more interesting examples of how they solved user issues with clever design.  Be sure to check it out.  And if you haven't seen the Halo 3 trailer yet, here it is for your viewing pleasure...

Google ads unethical or just clever design?

Jakob Nielsen's latest Alertbox is sure to get some defensive responses from the folks at Google...  He publishes results from a recent eye tracking study that clearly shows that users do not look at banner ads on web sites at all when they are looking for information or engrossed in the content on the page.  This isn't particularly new information, we've known this for a long time, but he does take it a step further.  First, he explains that there are three main design elements that are effective at attracting eyeballs to online ads (Plain text,  Faces and of course Cleavage and other "private" body parts).  Then he goes on to explain a fourth design element:

In addition to the three main design elements that occasionally attract fixations in online ads, we discovered a fourth approach that breaks one of publishing's main ethical principles by making the ad look like content:

• The more an ad looks like a native site component, the more users will look at it.
• Not only should the ad look like the site's other design elements, it should appear to be part of the specific page section in which it's displayed.

This overtly violates publishing's principle of separating "church and state" -- that is, the distinction between editorial content and paid advertisements should always be clear. Reputable newspapers don't allow advertisers to mimic their branded typefaces or other layout elements. But, to maximize fixations, that's exactly what you should do in a Web ad.

A specific ad may or may not be ethical, depending on how closely it masquerades as content. I caution against going too far, because it can backfire and mislead users. Unethical ads will get you more fixations, but ethical business practices will attract more loyal customers in the long run.

It doesn't take a genius to figure out that he's taking a shot at Google here, because they're obviously really good at making ads look like native site components on their search results pages.  My question is if it's really unethical or just clever design?  How will we know if users are annoyed by these ads, or if the relevance of the ads makes it ok in their minds?  I would be interested to know what kind of answers follow-up qualitative research might uncover.  Eye tracking by itself won't show you how people are feeling about these ads.  My guess is that users wouldn't care as long as the ads are relevant...

Making the case for ethnographic research to inform design

As a big fan of ethnographic research, two recent articles caught my attention.  Though pretty basic and meant for audiences who are not familiar with this type of research, the articles do make a good case for using this methodology in a business context as part of the design process.  I wanted to quote a few paragraphs that showcase insights that can be gained from ethnographic research that you can not get as effectively from other methodologies.  Bold emphasis added by me.

From a United Airlines publication entitled Executive Secrets - Greenhouse Effect:

Though ethnography is indeed a social science, a number of companies use it to gain a greater understanding of their customers. Their objective is to garner information to help create and develop products and services that better meet customers’ needs — especially those that customers haven’t yet articulated.

Jan Chipchase is an ethnographic specialist with the communications firm Nokia. Last summer, he and a team of designers and other ethnographers spent several weeks in Uganda. They traveled to multiple villages and lived with and observed the residents going about their activities. “We’re charged with bringing the experiences of the local culture into the company,” Chipchase says.

While in Uganda, Chipchase’s team noticed local entrepreneurs who had purchased their own cell phones and then sold minutes to other residents. Because customers paid in advance for their calls, they kept close track of their allotted minutes. Drawing from these observations, Nokia designed phones for use globally so that callers could easily see on the screen the number of minutes used per call.

To determine which observations are significant, the researchers focus not on the sensational but on the patterns that appear. Their goal is to find the actions that are common across many participants and discern their meaning. The insight that results can be compelling. “When it’s done right,” says Chipchase, “ethnography can inform and inspire the design process.”

From a Business Week article entitled Nokia's Global Design Sense:

Our process starts with a team of anthropologists and psychologists working in our design group. They spend time with specific types of people around the world to understand how they behave and communicate. This helps us to understand better and to spot early signals of new patterns of behavior that could be harnessed into mobile communication. Our designers often go out into the field to understand the world they are designing for. All of these observations are brought into the design process to inspire and inform our ideas.

One thing both articles don't mention is the fact that for ethnographic research to be truly effective, it should never be done on its own.  Exactly where in the research process ethnography fits in depends on the specific situation:

• Ethnographic research is usually done at the beginning of the design process, as is the case in most of the examples in the articles I just referenced.  This research is used to uncover user needs and help designers come up with high-level concepts.  This should be followed up by quantitative work (desirability surveys, needs & attitude surveys) as well us additional usability testing to further flesh out the concepts.
• Ethnography can also be extremely useful as a follow-up methodology to quantitative research like segmentation studies.  Once a market segmentation has taken place, ethnography can help companies understand each segment better by "living a day in their shoes" and understanding how customers use their products/web site within the context of the rest of their lives.

Ethnography is not an easy methodology to get right -- observing people is easy; knowing what to look for and how to uncover unmet needs and desires is not.  But it can yield extremely valuable insights that all levels of the business can utilize.

Social Networking approaches for e-commerce Web sites

I came across this pretty cool ethnographic research study on cell phone usage that is also relevant to how we think about community on the Web.  It’s interesting both from a methodological and a findings point of view.  It’s a pretty short deck, so check it out:
http://sfaapodcasts.files.wordpress.com/2007/05/sfaa-2007-metcalf.pdf

This from the authors:

When we talk about the "user experience" the main emphasis is often on an individual's experience with a particular technology. Even with a purported social technology, for example a social networking site, we still tend to create for the individual's interaction with the site (how does someone find their friend, how do they access this site easily from a mobile device).

However, designing for sociability means thinking about how people experience each other through the technological medium, not just thinking about how they experience the technology. The emphasis is on the human-to-human relationship, not the human-to-technology relationship. This is a crucial difference in design focus. It means designing for an experience between people.

Of course designing for an experience between people doesn't mean ignoring the interaction with the device, but it calls for taking something else into account. That "something else" is often another person or people. How do we, as developers of communication technologies, make the communications more interesting, more exciting and more stimulating for the receiver? How do we help our users meet the needs of the other people in their social network? How do we create a shared experience that is equally compelling for all participating parties? When we begin to think like this, we truly start to think of designing social software, social applications, social media.

A lot of e-commerce web sites seem to be scrambling to figure out how to deal with the social networking phenomenon, and in my opinion there are a lot of knee-jerk reactions going on.  E-commerce sites shouldn't try to become social networking sites.  They should leverage their commerce platform to connect people to each other.

A great example is Facebook apps.  eBay has a brand new Facebook application that allows users to connect their eBay profiles to their Facebook accounts.  You can see others' watch lists and even add items to their list if you think they might be interested.  It's also another avenue for sellers to showcase the items they have for sale.  By using its commerce platform to integrate into an existing social networking site, eBay is building on its strength as an online retailer and plugging into an enormous network without re-inventing the wheel by trying to become a social networking site unto itself.