© 2020 by Beijia Wang. All Rights Reserved

HearU

Timeframe

Sep. 2018 - May. 2019

Team

Mengxiao Song

Yuting Han

Role/Skills

Research Lead

Survey

Focus Group

Ideation

Concept Testing

Visual Design

VUI Design​

Recognition

Overview

HearU is a six-month self-initiated research and design side project where I worked in a group of three between school quarters. Hearing loss is a prevalent public health crisis that often goes unnoticed in the public eye due to its invisibility. People who are hard of hearing (HH) often experience social isolation caused by the lack of understanding from the hearing community. Our research revealed that there is a communication gap between the hearing and HH communities. We designed HearU - an integrated, multichannel system that includes public interaction. Through the creation of an immersive experience that increases people’s awareness of hearing loss, we aim to use HearU as a medium to weave these two communities together as a whole. HearU is selected as the Runners Up (top 2%) of 2019 CHI Student Design Competition and the team presented in the 2019 CHI Conference at Glasgow, UK.

 

I would like to take you through the entire design process and use this story as an example to demonstrate my problem-solving process.

Raising public awareness through the simulation of the experience of hearing loss

 
1
Interactive Display

A voice-controlled interactive display that visualizes and simulates the conversation experience of hard-of-hearing people to raise the attention on the invisibility characteristics of hearing loss as well as what people can do to improve the quality of a conversation.

* This video has sound. Make sure to watch it with the speaker on for best VUI experience.

2
HearU Website

HearU website is designed to provide the public more information about hearing loss and what people can do to facilitate the conversation better.

3
Information Card

Information card are printed in standard business card sized to allow people to carry around easily and prompt them to check more information on the website later.

Background 

Two years ago, my grandpa’s hearing got worse. This has been a challenge for my family. He started to isolate himself from family conversations and I always wonder how to communicate with him better. All of our team members know someone who is hard of hearing, they could be our close families, friends or even strangers on the street passing by. So we decided to look into this topic. We started by talking to an audiologist as a way to understand the space.

Design Process
design process.png
 
 
How Might We - 
bridge the communication gap between hearing and hard of hearing community by increasing public awareness of hearing loss?
Design Research

If you are interested in skipping the research and moving right into the design and reasoning,

click to jump to Concept Development.

Secondary Research: Understanding the problem space

To understand the current state of hearing loss and existing solutions to communication barriers, we reviewed over 20 literature and conducted a

competitive analysis. 

38.1% of people aged >18
report having trouble hearing 

Hearing loss is an

invisible disability

Yes, hearing aids will benefit you...but you need to understand there are many situations where you might feel that they're not doing enough for you, because you're still not catching everything. You're not understanding everything.

- Audiologist at UW Speech and Hearing Clinic

Hearing Loss is a Major Disability in the United States

Hearing loss is one of the most prevalent public health crises given the significant physical and psychological repercussions on adults. 

People who are hard-of-hearing (HH) not only experience emotional frustration but can also be socially isolated due to its invisibility

Hearing loss is often referred as an invisible disability, going unnoticed in the public eye. It has great impacts on every aspect of a person's daily life.

Existing products focus on using technology to help HH people’s hearing but not effective enough

The approach to hearing loss management mostly focuses on facilitating HH people participation in conversations. for example, by providing hearing aids to amplify sounds. However, these products are not always accurate or effective enough to give HH people expected autonomy.

Primary Research: Learning about current communication challenges and frustrations from perspectives of both people who are hearing
and hard-of-hearing

Based on the secondary research, we were surprised by the dearth of studies on the discomfort and the role of hearing people, especially since communication is a two-way phenomenon. We had the hypothesis that the hearing community lacks understanding about hearing loss when interacting with hard-of-hearing people. 

We developed an online survey that focused on individuals’ awareness of hearing loss and their interactions with the HH people. The survey was distributed in college forums and social network platforms. 

No

Expert Interview

We had multiple interview sessions with an audiologist at the University of Washington Speech and Hearing Clinic to further understand the impact hearing loss has on people, how audiologists help HH people overcome the communication barriers as well as the underlying reason behind these barriers.

Two Focus Groups with Hard-of-Hearing People and their Communication Partners

In order to gain insights and pain points from both the hard-of-hearing and the hearing communities, we had one focus group session with 5 hard-of-hearing people as well as another session with their communication partners.

Survey Summary: From 60+ hearing participants

18-22

23-27

>42

28-32

33-37

38-42

Other

26%

34%

20%

11%

Age

No

Other

Yes

36%

52%

12%

No

Learn sign

language

Yes

21%

21%

58%

Was your behavior changed after learning he/she has hearing loss?

 

(58 out of 60 responses)

Do you know any existing resources to understand the hearing loss population?

(45 out of 60 responses) 

Key Insight #1: Only 10% of participants have the understanding of how to facilitate effective communication - by speaking slower and clearer, or facing people directly during the conversation.

Speak Slower and Clear

37%

8%

13%

34%

Speak Louder

Use Sign Language

8%

Face Directly

Others

What would you do to better communicate with hard-of-hearing people?


(60 out of 60 responses 

Key Insight #2: People who are hard of hearing feel uncomfortable and frustrated when interacting with the hearing community for the following two reasons:

1) A hard-of-hearing person’s personality, especially their willingness to share information about their hearing loss.

What happens is, the person will often complain, I can hear, because they're hearing the lower frequency sounds of speech, but I can't understand. But normal hearing people don't know this. I mean, it's hard for them to even understand it.

- Audiologist at UW Speech and Hearing Clinic

2) The hearing community’s lack of understanding about hearing loss

when interacting with HH.

With hearing loss, it doesn't just impact the person that has it, it impacts everyone around them...people don't necessarily know you have hearing loss, right? Hearing loss is not always visible...

- Audiologist at UW Speech and Hearing Clinic

Key Insight #3: People from both hearing and hard-of-hearing communities
are unsure about how to communicate around hearing loss. Unsuccessful communication frustrates people from both communities.

It’s a struggle. It takes certain kinds of personalities to do it [inform other people] naturally.

- Participant from Hard-of-Hearing Communication Partner Focus Group

Key Insight #4: In a conversation, hearing people don’t know there are things they can do while hard-of-hearing people tend to take more responsibility

Communication is a two-way thing, but I still take too much responsibility for it.   
                                        
-Participant from Hard-of-Hearing Focus Group

 
Concept Development

Ideation

With these findings, we realized that there’s a significant communication gap that exists between Hard of Hearing Community and Hearing Community. Therefore, we brainstormed 30 ideas with a focus on bridging the communication gap. Our ideas can be categorized into three categories:
 

  1. Design to let other people understand hearing loss better

  2. Design to improve communication between hard of hearing people and their communication partners

  3. Design to help hard of hearing people in the restaurant because a restaurant is one of the most common place to have social events

Downselection

We developed three design principles based on research to better evaluate these ideas and repeatedly ask ourselves the following key question to make sure the design response is realistic enough to bring the desired impact.

 
Downselection Key Question
What motivates people to care about hearing loss and use our design?
Design Principle
01

Make the design simple to allow users to grasp the meaning of the design quickly

02

Capture users’ attention rapidly to provoke immediate interactions with

the design

03

Create an uplifting
and immersive experience to have a substantial impact on users’ understanding of hearing loss

The idea "Fill in the blanks" was chosen for its effectiveness to capture people's attention, the unique format for introducing the knowledge of hearing loss as well as motivating people to learn about hearing loss. 

Design Decision

I would like to highlight three key design decisions for you.

Design Decision #1: Designing for public participation 

We decided to use an interactive display as a trigger to initiate effective interaction and encourage public participation. This is inspired by “Climate on the Wall”, an interactive projection on a city center building wall that encouraged people’s participation in forming climate statements. It is found that such displays have significant potential in promoting awareness to others outside the community and sharing content that people have in common within the community. This aligned with our project goal and motivated us to move in the direction of designing an interactive public display.

© Peter Dalsgaard, Kim Halskov 

Design Decision #2: Using Vocal Interaction to simulate high-frequency hearing loss experience

To provoke an immersive interaction, the instructions on the display are written in the first-person view. We chose vocal interaction because it simulates what a conversation is like with Hard of Hearing people.

By simulating the high-frequency hearing loss experience, we want to help the public build the knowledge of hearing loss through meaningful interactions. Our design for empathy was inspired by a communication partner in our focus group session.

I have no idea what the hearing loss was... finally, at one time, I put on a pair of hearing aids,

and I immediately understood when my wife would tell me I don't need to get louder, I need

to be clearer.

- Participant from Hard-of-Hearing Communication Partner Focus Group

Design Decision #3: Evolving into an integrated system: display + companion Website

As a tool for awareness, the public display would only capture people’s attention for a short amount of time, and provide limited knowledge about hearing loss to users.
So we decided to expand our project from an interactive display into an integrated system. The system contains a display, a companion website, as an extension of
the display.

Here's a quick glimpse of our final concept. Now that you have got an idea of how the HearU System works, I would love to share with you the stories of how we got there. 

Design & Iterations

From the initial concept development, we built up a lo-fi prototype and conducted 7 rounds of Wizard of Oz testing to validate whether our design was understandable and usable. These tests also led to fast design iterations from both the conceptual level as well as detailed UI improvements.

As much as I want to talk about every single detail of how we crafted the UX of HearU, in order to make the design process more straightforward, I would like to share two design iterations stories with you.

Story #1: Crafting the immersive learning experience of the interactive public display

Initial Design: Visualizing the hearing fatigue experience 

We gleaned insight from people’s experiences as a foundation for the interactive display. From focus groups, we found that high-frequency hearing loss is one of 
the most common hearing loss types. Hard-of-hearing tend to experience hearing fatigue. This causes them to take in incomplete sounds, it feels like seeing a sentence with blanks you need to fill in. Thus, we decided to visualize this by showing a complete sentence breaking into pieces. 

"Fill in the Blanks" Task

  • Prompt the user to speak out the sentence

  • Shows the broken sentence

  • Playing back the user’s voice that has been filtered with muffled sounds to match the missing letters in the sentence. 

 

We created a mid-fidelity digital prototype and conducted several rounds of Wizard of Oz study to test our initial design concept. Intrigued by the display, all participants were able to complete the task and figure out the purpose of the design. The initial tests also revealed space for improvement:

  1. Using real-time sound wave animation to emphasize the vocal interaction

    Some participants intuitively wanted to touch the screen at the beginning of the task before realizing it was not a multi-touch interaction, and even with the circle horn icon on the display, the vocal interaction was not clear. 
     

  2. Making the "Fill in the blank" task more meaningful to create impactful interactions about hearing loss

                 I want to try again to see how I can improve my speech pattern...

                - Participant from initial WoZ Testing​ 

Design Iteration A: Adding learning elements that help users improve their speech pattern

Users wanted more interaction with our design, therefore, we decided to utilize the initial hearing fatigue visualization to help users improve their speech pattern. We added three modes on top of the “fill in the blanks” task: speak slower, speak louder, and change position. After completing the initial task, users will be prompt to try again with instructions randomly selected from the existing three modes.

Mode #1: Slow Down

Mode #2: Speak Louder

Mode #3: Change Position

Initial vocal interaction for hearing fatigue visualization 

Make the interaction meaningful by prompting the user to learn more about hearing loss

Demonstrate the effectiveness of the user’s behavior change

Provide the
user with a concrete action

Based on the first round iteration, we built several web-based application as hi-fi functional prototypes to validate if our design is understandable and usable. Several usability issues were uncovered during testing:
 

  1. Make the vocal interaction more conversational to increase engagement and smooth the experience  

    Overall, even though all users successfully completed the prototype testing, they appeared to struggle following the vocal interactions, as some screens flew by too fast, too much information was showing on one single page and the interactions didn’t strike to them as natural and intuitive enough.

     

  2. Replay the user's original vocal input first to make the contrast more clear

    More than half of the users were puzzled by the filtered vocal output of their own voice, so they thought something was wrong with the audio before realizing this is part of the experience.

     

 

We also sought expert advice from Richard Ladner, a Computer Science Professor at the University of Washington who specializes in accessible computing. He challenged the current design user flow and encourage us to reverse the user flow and test it out. He challenged the current design user flow and encourage us to reverse the user flow and test it out.


             Currently, you are [the design] asking the user to read the sentence first before playing the
             distorted audio, so the user already knows what the sentence is.... however, for people with
            hard of hearing, they don't always know what the person is going to say...


            - Richard Ladner, professor emeritus in the Paul G. Allen School of Computer Science &
               Engineering, University of Washington​ 

Design Iteration B: Exploring alternative user flows for the best representation of the hearing loss experience

We decided to proceed with the expert's advice and built out the reverse user flow. We tested the two user flows interchangeably with both hearing and hard of hearing participants seek the best way to represent the hearing loss experience.

Alternative Flow: Display Initiate Conversation 

* This video has sound. Make sure to watch it with the speaker on for best VUI experience.

Invite a user to come close and listen

Play the filtered high-frequency hearing loss sound 

Put the user in the shoe of the hard-of-hearing people

Reshow the letters disappearing
to reinforce the experience

Although the alternative user flow received some positive feedback from both hard of hearing and hearing participants.  The majority of the hearing participants were frustrated and taken aback by the initial experience of not being able to understand the sentence immediately. 


             I can’t hear it well… it took me sometime to get what this [design] is about… but I’m not sure I
            have the patience in real life…    


            - Hearing Participant from Functional Prototype Testing
 

Therefore, although the alternative design in some way simulates the hard-of-hearing experience more authentically, it led to a negative experience for the user and decreased their interest in further interacting with the design. In the end, we decided to go back to our initial design concept. 

Final Design: User Initiate Conversation for Engaging and Meaningful Interaction

Even though we went back to our initial concept, we revised the user flow based on the previous testing results and enhanced the interaction details to make the vocal interaction more intuitive and natural. From the design perspective, we changed the typography hierarchy to make the instruction more actionable.

          It’s really impactful for me to hear how my own voice sounds like in hearing loss people’s ear. 

          - Hearing Participant from Functional Prototype Testing

Make the conversation process more clear

Invite the
user to start a conversation

Playback the
user’s recorded sound to show the display captured their voice clearly

Explain to the
user why hearing loss matters

Raise
awareness with specific actions

Story #2: Extending the meaningful interaction for further learning: the HearU System evolves

The initial HearU system contains an interactive display and a companion website.
The website is designed to encourage users to learn more about hearing loss by themselves after the initial interaction with the display.

 

Initial HearU Website Design

The website includes four main functional areas:

  1. Quotes from the Hard-of-Hearing community;

  2. A message box that enables people from two communities to share thoughts with each other;

  3. Communication tactics about hearing loss to further enhance people’s awareness;

  4. A hearing loss simulator that extends the interaction of the interactive display.

* This video has sound. Make sure to watch it with the speaker on for the best experience.

However, through our testing with users, we found the integrated system didn't quite reach our intention of motivating the user to learn more about hearing loss:

Only one (out of 10+) hearing participant took out the phone and went to the HearU website

This revealed that showing the website address/QR alone at the end of the task screen cannot motivate participants to take out their phone and check out the website. More incentives are needed as well as making the address more accessible.

I can see if I'm not in a rush for time... taking a picture or looking at it later... if I'm at all in a rush, whenever I see a website I go: 'Oh, I hope I remember that'...

- Hearing Participant from Functional Prototype Testing

I haven't really got used to... just pulling my phone out and taking a picture of the website or a QR code...I just don't do that... it doesn't seem natural to me...

- Hearing Participant from Functional Prototype Testing

Information Cards

We brainstormed on the incentives for motivating users to visit the website on their phone. As a result, physical information cards with the website address were added to the HearU system. This added layer of the physical product experience on top of the vocal-visual interaction experience heightened/improved portability and convenience.

We also improved the website based on user's feedback from the tests.

  1. Reorganized the website flow
    We moved the mini hearing loss simulator to the top of the page became this is the feature that all the users are interested in. We replaced the quote board with resources for hard-of-hearing communities as well as ways to improve communication skills.
     

  2. Make the design more visually accessible
    We reconsidered the color palette to increase the readability of the content because the previous design was low on color contrast.
     

  3. Add live hearing loss simulator
    We enabled the web-based live hearing loss simulator on the website that provides the same experience as the interactive display. In this way, users can revisit this experience anytime they want and share the link freely with their
    close ones.

I found this design very interesting… I want to try again and share this with my father-in-law [who has hearing loss] later…

- Hearing Participant from Functional Prototype Testing

Final HearU Website Design

* This video has sound. Make sure to watch it with the speaker on for the best experience.

Mini simulator with link to the live simulator prototype

Screen Shot 2019-06-25 at 1.47.20 AM.png

Actionable tips for improving speech patterns

Screen Shot 2019-06-25 at 1.47.36 AM.png

Reinforce why hearing loss matters

More knowledge about hearing loss

Screen Shot 2019-06-25 at 1.50.20 AM.png

Resources for both communities to encourage more
self-exploration 

HearU system is a bridge that weaves the hearing and hard-of-hearing community together by encouraging the hearing community to learn more about hearing loss. Here's how the system works.

hearu system

Interactive Display

Information Cards

HearU Website

Outcome & Impacts

Accuracy rating (out of 5) of the simulation of the experience given by six hard-of-hearing participants

That's [design] pretty accurate and I think is like a very good example... 

 

- Hard-of-Hearing Participant from Functional Prototype Testing

4.7

*As the average rating

1 - Extremely inaccurate 

5 - Extremely accurate

100%

Participants from Final Hi-Fi Functional Prototype Testing

Hearing participants expressed that they would
want to change
their speech patterns for more
effective communication.

Oh, I didn’t realize this [communication] is a problem, so it was more of a realization of my part that I could adjust the way I talk... in order to be more accessible to everyone.

- Hearing Participant from Functional Prototype Testing

97.5%

Participants found the experience to be positive
or uplifting.

Participants w/o. hearing loss from

40 testing sessions

This [design] makes me

happy actually.

- Hard-of-Hearing Participant from Functional Prototype Testing

This [design] is interesting, I want to try again.

- Hearing Participant from Functional Prototype Testing

Currently, we are working with the Hearing Loss Association of America and UW Speech and hearing clinic on how to bring bigger impacts to the society. We believe our design could make a difference.

 
 
Reflection

This project started off as a self-initiated passion project that later evolved into so much more. I am grateful for starting this amazing journey with my two intelligent and beautiful teammates. I am also thankful for all the helps I've got along the way! As a process-oriented designer who likes to ground design decision on user research, it's such a satisfaction to have the freedom to apply the complete human-center design process from problem discovery to the end of design implementation and testing for proof of concept.

 

One of the most important lessons I learned is: design is not only a process - it's both a process and a result. The nine-month time frame enabled us to revisit the participants from focus groups and the smile on their face when they understood how our design works were the most meaningful and enjoyable moment of this journey. That was the moment I knew, as a designer, I could use the power of design to bring joy to people's lives and to make an impact! Looking back, this journey was full of dramatic ups and downs but I'm proud that I never gave up in face of hardships. I am a designer who can wear multiple hats and problem solvers who always look at the silver linings behind dark clouds.