Usability testing can make the difference between success and utter failure for a firm, a manufacturer or a service provider, between standing out in the market and sinking into obscurity.
Below, we have prepared a comprehensive guide to website usability testing, covering everything you need to know about it.
Usability testing is the process of evaluating a product or service through testing by representative users.
The objective of testing is to spot and assess usability issues, collect qualitative and quantitative data and measure users satisfaction with the performance and effectiveness of the product/ service, as well as the extent to which it meets their expectations.
Naturally, usability testing is often used by designers and service providers that wish to improve their products or services. Everything from commonplace objects, like smartphones, ovens, and cables to complex machines like cars and drones, is repeatedly tested by representative users. Experience designers frequently conduct usability tests to improve the feeling a customer gets while unboxing newly purchased products, and software designers to identify problems or areas requiring improvement in their solution packages.
If there is one product or service out there in need of constant usability testing, it is the online experience. Nowadays, web designers pass all their websites, e-shops, and applications through usability testing to make sure everything works correctly.
A representative user is the person assessing the quality of a given service or product, chosen because they have all or most of the basic characteristics of expected users (consumers of the product/service). Such a person is in the best position to identify possible issues before launching the product to the market.
Remember: usability testing is all about specified users with specified goals in a specified context of use. Of course, any kind of testing is better than no testing, but having your product examined by people who belong to your target audience provides the best feedback.
Picking out representative users for usability testing is critical to overall success. Factors such as age, gender, educational level, occupation, location, nationality, financial and social status, and prior exposure to the product/service should be taken into account and potentially used as selection criteria.
For instance, having a member of the team that created a website evaluate its functionality wouldn’t exactly serve the usability study purpose, since the person would already know the fastest or easiest way to do things.
People with too much prior knowledge or exposure to the tested product/service are usually not the best choice for representative users. You cannot have a father judge his own child, can you?
Hence, selecting targeted users – especially when testing the functionality of a specific feature on a website – is of the essence. Testing with randomly selected users could lead to seriously distorted or flawed results and ultimately lead to a launch disaster.
We can help with that. Use our specially designed feature that allows you to pick targeted users who meet your demographic criteria, thus ensuring your participants will yield truly valuable feedback.
There are two basic types of usability testing, namely formal and informal. They can both prove valuable at certain stages of the development process. Let us see how.
This one usually involves early stage prototypes – that is, prototypes that are actually far from being realistic and sophisticated. At such an early stage, web designers wish to test their central concept and see which ideas and features meet their client’s needs - and which don't – thus saving a tremendous amount of time and energy.
In short, informal usability testing verifies the viability of your main ideas.
After repeated informal usability testing, the time will come for the real thing. Remember, at this stage, you have already decided which ideas to work on and now wish to see to what extent they fulfill their purpose.
A formal usability study should aim to add the finishing touches, not change the main concept of your team’s work. So, before planning a formal usability study, make sure your work has successfully undergone several informal ones and has integrated the resulting feedback.
Now, let us turn to formal usability testing. The prototype should already seem quite ready to launch, and just needs some… refining or polishing The study should unveil all flaws and oversights and help the team concentrate their efforts on areas that could disappoint or even annoy your future clients.
Needless to say, it's important for a company or team to use both types of usability testing repeatedly before launching, since each offers valuable data at different stages of your work.
Many companies have unhappily found themselves in the position of practically starting from scratch, just because they neglected to perform informal usability testing, while others have damaged their reputation by launching products without conducting formal usability testing, and saw thousands of their customers bid them farewell.
Take our word for it – both types of usability testing are critical to your success.
Typically, one or more representative users are asked to complete certain tasks while being watched by a moderator and a few observers, who listen, ask questions and take notes, but never take part or offer help.
However, things are much less spontaneous than they may seem. Usability testing requires careful planning and preparation in advance, including:
Identifying the target users and specifying their needs
Selecting proper representative users;
Designing a tailored study script usually referred to as the “experiment protocol.” It’s essential to assign this job to a skilled person, since this document is the core of the whole testing session. It should combine profound knowledge on qualitative research and domain expertise
Website usability testing usually involves a live one-on-one session between a moderator and a study participant. The moderator asks the participant to complete specific tasks that correspond to the ones the actual users are expected to perform in the future. Quite often, the user is also required to answer specific questions, both before and after the testing session.
The moderator’s role, aside from guiding the participant and reading the tasks to be performed, is to collect as much usability data and feedback as possible during the session, thus enabling the development team to get a deep understanding of user behavior and consequently draw a clearer picture of the user’s needs. Such extensive understanding can only improve the product and boost its market success.
In most cases, the think-aloud protocol is preferred for website usability testing, since it enables the participant to share their thoughts and observations in real time while completing the requested tasks.
Remote usability testing utilizes an online software program that records the user’s actions while they test the product. This type of usability testing allows the participant to complete the testing session from the comfort of their own home or office.
The first and most important benefit of remote usability testing is the vast widening of the pool of potential representative users, since location and travel are no longer issues. Moreover, the moderator can conduct more testing sessions in a shorter amount of time, saving time, energy, and staff expenses.
Another significant advantage of remote usability testing is the user’s overall state of mind. They are more relaxed in their own environment, and their reactions, thoughts, and observations are much more spontaneous – hence all the genuine!
Remote usability testing can be performed in two different ways moderated or unmoderated. Which is more suitable for you depends on your goals and development stage. Let’s take a closer look.
In remote moderated testing, the moderator and user are in different locations, but they share screens and communicate directly through an online conference tool. In other words, they are in the same “virtual” working space, interacting in real time.
Moderated usability testing offers two main advantages.
First, making the most of real-time interaction, the moderator can ask the participant to amplify or better explain their answers, or even ask additional questions.
This is often necessary since some user comments are far too vague or generic to serve the test’s purpose. To illustrate: When a participant comments something like, “that’s not what I was expecting” or “this feature is a bit disappointing,” the moderator can ask them to explain what exactly they mean.
Second, remote moderated usability testing is far more flexible and easily controlled, since the moderator can offer guidance or help the user better understand what is expected of them during a task, or make changes to the tasks, depending on how the session goes.
There are some cons though. First, the moderator is obliged to stay through the whole session, which is more time-consuming and tiring and can often result in a smaller sample size and less feedback.
Second, the quality of the testing session greatly depends on the skill and experience of the moderator. Too much or too little guidance can have a negative impact on the results of the session, or even ruin it.
In addition, moderated usability testing requires excellent conference tools, an impeccable connection, and a strict schedule – which translates into more time-consuming preparation. Such demands considerably increase the risk of technical problems. For it to work, everything in a remote moderated usability testing must run clockwork.
In a remote unmoderated usability test, the testing session is automated by using a software program to record the user’s actions to. The user receives the list of tasks that must be performed while their screen (and sometimes their voice) are recorded.
The place and time of the testing session are decided by the user, and all observations, comments, and feedback are sent to the moderator afterward, in the form of an audiovisual recording. In turn, the moderator can review and analyze the results of the testing session at their leisure, whenever and wherever it suits them best.
Naturally, such a session does not include any live interaction, and follow-up questions must be asked and answered sometime later.
Remote unmoderated usability testing makes preparations for testing sessions far easier and allows for a higher number of participants to provide their valuable feedback. Everyone works when and where it suits them best, which can only affect the quality of their work positively.
On the other hand, the moderator has much less control over the session, and live interaction and following-up are evidently out of the question. Users can easily get confused and have their attention diverted to the wrong features. This also means that the tasks and overall test plan cannot be too complicated on challenging since the user won’t have any help to understand tasks.
It is true that moderated usability testing is a far preferable option in testing sessions that complex or tricky tasks. In such cases, the actual online presence of a moderator simplifies things and ensures that the feedback obtained has the depth and quality necessary to be of any help.
Additionally, testing sessions that require a chained sequence of task completion can only be properly performed with a moderator present, since failure to conclude a task immediately terminates the session.
Generally speaking, the more experienced or skilled the user must be to conclude the session, the smaller the sample size is – hence, moderated usability testing turns its con into an advantage when used in this case.
Finally, when there are safety, security or technical issues, moderated usability testing is often the only option.
Then again, unmoderated usability testing is better when the session includes very straightforward and specific tasks that do not require profound knowledge or extensive experience. Any product or service in need of a large sample size in a relatively small amount of time can only turn to unmoderated usability testing.
We are sure most of the benefits are quite clear from what we have already said, but summing it up a bit will help you understand better how important usability testing is for your business.
Usability testing does require some expenses on your part, but the last thing you should see it as an unnecessary cost. Usability testing is an investment as well as an insurance against failure. Let us explain.
Usability testing enables your company to discover and foresee timely pain points, possible complaints, and embarrassments and tackle them before launching. Hence, besides saving you tons of time, energy, operational cost, and frustration, it lays the foundation for the further expansion of your business, extending your reach, amplifying your brand awareness and reputation, and getting new customers – which automatically means higher profits for the future.
In the meantime, you get the chance to pinpoint problematic procedures within the company, weak links in your staff chain, and unnecessary expenses, thus allowing you to improve your business’ efficiency from top to bottom.
Usability testing affords valuable insight into your users’ way of thinking, needs and wishes, and ever-changing whims.
Such live interaction simplifies things and stimulates the mind, making ideas and solutions flow faster – a genuine treasure for any development team.
Note also that usability testing has the advantage of highlighting user behavior, not preferences. The majority of users hardly know exactly what they want – much less what they need! Still, by observing and studying their behavior, you will be able to understand which solutions best suit their real goals and desires.
Undoubtedly, the better you get to know your users, the more effective the solutions you offer them will be.
Web development is an art, and artists are often obstinate and ready to fight for their ideas to the end! Such moods seldom contribute to a calm working spirit within the team.
Usability testing can help quell the passions before they become a problem, for a very simple reason: it moves decision-making from the developer’s mindset and opinions to the client’s feedback and real needs. Ideas that do not appeal to the user are quickly dispensed without any complaint from their creators. All their energy and passion can now be funneled into making your product better, instead of having heated debates with their colleagues.
Finally, getting your users’ feedback and guidance right from the beginning means you will eventually achieve the best possible outcome, first for your customers and then for the reputation of your business.
All in all, usability testing can do something truly wonderful for your business: it can guarantee your success. Without it, you just work and wish for the best, like looking for a place by choosing random turns. You may find it, you may not. Having a map, however, will ensure you reach your destination, no matter how many mistakes you make on the way. This map is usability testing.
The truth is that the creator can never be truly objective with its creation. Being too close to it blinds the web developer, making it difficult to perceive what matters to the user. Usability testing affords a fresh, genuinely objective look at things.
Moreover, of all professionals, web developers should never underestimate the impact minor flaws and nuances may have on the user’s experience. Modern-day web users are more demanding and less patient than ever, ready to reject an elaborate project that took months of hard work on a whim, just because something did not run as smoothly as they expected.
it's official: based on a survey by the Nielsen Norman Group, website usability testing has doubled conversion rates for all participating agencies and companies, and skyrocketed user satisfaction. Considering the benefits mentioned above, this is hardly a surprise.
Usability testing also minimizes support costs, since it prevents unnecessary development and redesign work.
Usability testing, after benefiting your business in so many ways, gives you the upper hand in the competition and helps you stand out in your industry. Suffice it to say that brand awareness and visibility, as well as reviews, are your best advertisers out there. Usability testing guarantees both.
Now we are ready to get to the essence of things. Setting up a usability test can be divided into a few stages.
Naturally, there is some preparation to do before scheduling the testing sessions. First of all, you must determine your goals. The questions below can help a lot. All key team members must sit down and ask themselves:
Which are we trying to achieve here? Test our basic concept, a few core elements, or identify weak points?
What are the specific questions we need to answer through this usability study? Is there any chance we already know the answers to some of them?
At which stage of the development are we?
When is the best time to conduct usability testing?
Who is our audience? Can we write down the criteria we are going to use to select our users?
As soon as you know the basics, you can start shaping the program and character of the testing sessions themselves. Ask yourselves:
Which type of usability testing best suits our purpose?
How many representative users do we need?
How long will the sessions have to be?
How much will the participant be compensated?
The answers to these questions will all depend on your goals and type of product. As a general rule of thumb, most website usability studies involve around 5-6 participants, but the higher the number, the surer you can be of spotting all the pain points.
As for the length of the session, 45 minutes to an hour usually does the trick, but again, your needs may vary.
Again, much of this stage depends upon your goals and intentions, but you must do a little work to find the proper user sample, regarding both size and characteristics.
You need to specify the level of experience of the participants, as well as their degree of exposure to prior versions or editions of your product. Other factors, such as their age, gender, educational level, etc., depend on your target audience.
Finding proper participants who meet all or most of your criteria can prove challenging. Possible pools of users are friends, family, and colleagues (provided they fit your profile), but always consider the danger of allowing politeness or fear of displeasing you to affect their responses!
Other possible sources are your social networks, primarily LinkedIn, Facebook, and Twitter, as well as forums and chat rooms of colleagues and people in your industry.
As soon as you have a list of the selection criteria for your participants, you must assign roles to the members of your team that will prepare the list of tasks, the conference tools and questionnaires to be used, the contact details of the participants, the session schedule, etc.
Only when we are positive we have covered all the above can we start framing testing sessions.
The first thing to do is to structure the session. Most studies include three parts:
The first part consists of some questions the participants must answer before getting down to the actual tasks. Such questions usually aim to identify the participant’s opinion of your brand, expectations of your product, and view of previous versions or editions of your work.
This is the most important part and will require some time and effort to be properly prepped. You will need to come up with a list of tasks, usually with escalating difficulty. They should be written in an unbiased, simple language and described in a clear-cut way.
Defining the tasks shouldn’t be too difficult. Simply pinpoint the features and ideas you wish to check that are critical to your product’s success.
While it is vital to use simple terms and wording, you don’t want to be too helpful or leading. For example, avoid using the same terms that appear in your interface as much as possible.
During the sessions, you must be consistent: ensure that all participants are asked to perform the same number and types of tasks, in the same wording and order.
At the end of the task, ask the participant to express their opinion on the overall quality and feel of their experience, and optionally rate some parameters like satisfaction, ease of use, effectiveness, efficiency, value, etc.
Ask them to state plainly what they liked or disliked during the session, and what they would replace or add to make the experience easier or more effective.
Try your best to make your script and questions as unbiased as possible. This is not as easy as it sounds. For example, while asking participants to rate your product, do not include a “no problem at all” option. It's best to opt for a numbered scale, from 1 to 10, with 1 being the lowest score and 10 the highest.
Another example: “How many times did you use the online tutoring feature last year? ______ times.” Such wording takes for granted that the participant did use this feature, and may encourage them to be inaccurate if they actually didn’t.
It is better to use the format:
“Have you ever used the online tutoring feature?”
“If yes, how many times did you use it last year? Pick one of the following options
While planning the test, try to not only spot possible weak points or problems, but also measure their impact and frequency. It is one thing to have half of your participants complain about an issue, and another to have four of them unable to perform a task altogether because of it. Noting frequency only gives you half of the story and does not indicate stress its priority.
Therefore, try to discover all aspects of a given problem: frequency, severity, extent, impact etc.
First of all, greet and thank the participant for their role and cooperation, and stress the importance of following your directions carefully throughout the session. Explain how things will unfold during the testing session and make the participant feel as relaxed as possible. Make it clear that the session aims to evaluate the product, not them!
State the session’s length, ask them to be thorough, frank, and as elaborative in their answers and observations as possible, and help them understand that you will not take any negative comment or criticism they may express during their evaluation of the product personally.
Again, be direct and clearly state what you wish them to do; for example, “Please think and describe aloud your actions and steps.”
Remind them which of their actions are recorded and how, and explain to them how you plan to use their feedback and comments, especially if you wish to post key feedback from the session online. Ask them directly for their consent to record the session and possibly publish some of their thoughts online.
Moderation is an art, and while it requires skill and knowledge, it is something that can be learned through repetition. Just remember the following principles:
Be polite, patient and considerate
Be ready to step in when needed
Remain unbiased and objective. Never comment, verbally or with your body language, on the observations the participant makes
Resist the temptation to tell the user how to perform a task best
Speak as little as possible, minimizing the risk of inadvertently revealing information you shouldn’t
Make sure participants feel relaxed
Check their reactions, facial expressions, and body language. Some participants may exhibit signs of frustration through their body instead of their comments
Encourage participants to express themselves freely. Listen carefully and take good notes
This last one, taking notes, is crucial. Note everything the user does – their navigation around the interface, their comments, quotes and observations. Also write down how long it took the user to perform each task.
Again, facial expressions and body reactions are as important as comments.
Now that the testing sessions are done, your team must get down to analyzing all the data you have assembled. Your goal here is to take the details and shape a precise picture of the strong and weak points of your product. Analyzing this information properly will allow you to make the most of your usability study and improve your product.
There are several ways of analyzing usability test data. The first thing to do is review your notes, perhaps gathering them in a single spreadsheet and grouping them into different categories such as pros, cons, positive and negative comments, miscellaneous, etc.
Next, isolate data for a specific aspect, feature or angle of your product. When possible, translate things into numbers. How many times has there been an issue with this particular feature? Which type of problem was the most frequent?
Look for patterns, grouping participants’ comments when possible. For instance, how many of them characterized this feature as ineffective or frustrating?
Try to be accurate while you categorize your data. You must link each given problem with a particular frequency, impact, and severity.
Lastly, you need to report the findings and share them with your team members in a way that will help everyone understand and rectify the problems that were discovered.
Concentrate your attention on presenting all issues and associated findings categorized by level of severity. Make sure you attach all the related data from the test plan and provide additional details to present a clear picture of the nature, severity, and impact of each issue.
Do your best to visualize the results and findings, making them as transparent and clear-cut as possible. Use concise sections, tables, screenshots, video clips, video annotations, highlights videos and graphics for your presentation.
Let us break down these points into stages:
The summary should include a short description of the features that were tested, details about the type of usability testing and the way it was conducted, the number of participants, the tools used, the names of the moderators and testing team members, as well as a brief account of all the positive points and the issues and weaknesses that came up.
Sharing the key points of the method you used to obtain and analyze the data will help the rest of the team to understand the gravity and reliability of your findings. Make sure to mention the:
Number of test sessions
Number of participants
Criteria used for their selection
Tasks the participants were asked to perform
In this section, you present the results of the study. Don’t rush into conclusions yet. Just state the facts in a straightforward way, using the visual aids we mentioned above (charts, video annotations, highlights videos, graphics etc.). Make sure to mention:
The number of participants that completed and failed to complete each task
The average time required for the completion of each task
The issues users encountered and all pertinent information (frequency, variations, impact, severity)
Comments, observations, and recommendations of users
Other useful information moderators noted about the overall attitude and behavior of participants
Now is the proper time to state your conclusions and general findings, as well as making some initial recommendations.
Each of your recommendations should be firmly based on specific data you have presented in the previous section.
Remember to mention the positive findings of the study. Strong points and encouraging comments by users should definitely be communicated to team members, thus strengthening their resolve to create a user-friendly website.
Creating a highlights video to summarize the most critical findings is a great way to motivate your team and spread the notion that your website should be improved, without requiring all team members to watch entire test videos or read a full report. Userfeel helps you create such a video in a matter of minutes. Here is an example:
A great way to measure the perceived usability of a website or app is to use the System Usability Scale, invented by John Brooke in 1986. SUS has gained wide recognition and repute for delivering highly reliable results with relatively small user samples.
SUS is a consistent way to measure the perceived usability of a website or application, mobile phone, computing system, software program, call center, etc. All the user has to do is rate their level of agreement with 10 different statements on a scale of one to five, with one meaning “I completely disagree” and five meaning “I fully agree.”
The SUS rating questions are:
I think that I would like to use this system frequently.
I found the system unnecessarily complex.
I thought the system was easy to use.
I think that I would need the support of a technical person to be able to use this system.
I found the various functions in this system were well integrated.
I thought there was too much inconsistency in this system.
I would imagine that most people would learn to use this system very quickly.
I found the system very cumbersome to use.
I felt very confident using the system.
I needed to learn a lot of things before I could get going with this system.
The overall score runs from 0 to 100, indicating how friendly the feature or website seems to the users. SUS scores have been proven accurate and dependable over the decades by several disinterested parties, despite being much simpler and more economical than other similar evaluation methods.
Needless to say, none of this has any point if you neglect to integrate these valuable insights into your development efforts. Set your priorities and try to implement as many as possible of the recommendations that arose from analyzing the usability testing data.
Accurate usability testing requires time, effort and certain expenses – but, when done correctly, it can pave the way for the success of your project and further growth of your business.
Create user tests using a better tool. No monthly fees. No credit card required to start.
Start testing
Discover how powerful Userfeel is for doing user testing more effectively and ask us anything.
Book a demo call
