3-F usability test for software developers and startup founders
This post describes a short and simple usability test.
There are many possibilities to spend time with a potential user. I describe a usability test I would recommend on base of my personal preferences and my personal experience. I call it the 3-F Test. You can perform it even if you don't know anything about usability.
I describe a test of a web portal but the same script may be easily adapted for PC applications, smartphone apps and other kinds of software. It is possible to perform a test without software with simple prototypes, screenshoot images and even with paper, pens and scissors. Unfortunately you must be an experienced usability expert to get plausible results. Of course any test produces a lot of results but in this case it is difficult to get the information which is correct and relevant for a dynamic application.
A note for permanent blog readers:
Традиционная просьба посмотреть английский до того, как это ушло на другие ресурсы. Если попадутся совсем бредовые ошибки, удалять комментарии всё равно не надо, я лучше их скрою. Да, если кто-нибудь надумает переводить на русский, просьба всё-таки связаться со мной. А то в прошлый раз вышло что-то странное. Хотя, там английский текст был довольно мутный.
To perform a usability test you also need a user. The best case would be someone from your target audience. However in your first tests this is not important because your goal is to find most obvious problems. The only criterion is the absence of expertise. You can ask what a software developer thinks about your program or what a designer thinks about composition and colors on your web pages but please test your design on a programmer and the clearness of your menus on a designer.
Let's say there are 2 test persons. A hipster (23 years, male, freelancer and gamer) and a financial analyst (32 years, female, MBA, two kids. No chances). Maybe I will use this in the next version of this article. Yet I would randomly reference a test person with "a user", "your user", "he", "she" or "it".
The last important condition for a correct test I would mention is the silence. The source of information is observation. A dialog with a test person would supply opinions not the facts.
There are many usability test kinds when a test manager says something to a test person, when a test person speaks about hers or his impressions and thoughts. There are tests when they both speak with each other. I would recommend to forget about them except you have a lot of knowledge and experience. It is important to remember that your real user would browse your site without comments and questions.
People are very sensitive. They react to your questions and even to the tone of your voice. If you control is a bit weak, a test person becomes an actor and you become a director instead of an observer. This inserts aberrations in your results. Also your guidance will keep the user away from critical problems you could discover otherwise.
3-F Test consist of 3 simple tasks: First Impression, Information Search and Happy Path. It mirrors 3 important flags you must raise to have success in web:
- A potential user sees your site for the first time and decides whether to stay on it or to leave it for another site
- A user searches for important information
- And a user performs the process which is the main goal of your site and which brings you profits: buys an item, writes a post, connects to another member, finds a restaurant nearby, places a delivery order, etc.
Simply speaking 3-F Test consist of 3 independent tests. I refer them below as "steps", "test stages" and simple "tests".
Let's say you have planned 30 minutes for your test session. I recommend to divide this time in the following way:
- 10 minutes - buffer time reserve
- 5 minutes - First Impression test
- 5 minutes - Information Search test
- 10 minutes - Happy Path test
This includes test instructions and a discussion after each test stage.
If you think you need more time for one of the steps, you already have a serious problem. In real life a user quickly gets bored or irritated. If you hope she or he will invest more efforts in your wonderful site, you are wrong. If your potential users are stopped by obstacles, most of them simply leave and never return. The goal of a usability test is to prevent this flow of events.
This means: do not break your test, if a test person is not able to reach one of the goals. It is more important to search for a solution than to improve the later processes your potential users never reach. Try to dig a bit deeper.
It is usually recommended to make excuses before a test and to say that you do not test your user, that he must not blame himself for the errors, that your site is not perfect, etc.
I would skip this part. A test person will be frustrated when something goes wrong, she will have bad feelings by errors and you cannot prevent with words user's reactions on your astonishment, disbelief, joy or surprise, if your face or your voice reveal them.
If your test person did not read this post, you can simply say: "We will test how a user can use our site. Please complete our tasks, we will observe you and then ask you some questions."
It is important to prevent discussions and guidance. A real user will never call your help desk to ask questions. At least this concerns problems that you can discover in this test. I usually say: "Our goal is to find errors and misleading elements on our site. If you are blocked please try to find a solution yourself. Please ask questions only, if you have feeling, you cannot do it without help."
At least half of the users will do this. Others will try to ask for your help or test your reactions. It is important to try to block this attempts with shortest possible answers. If you'll be manipulated by test participants, the results of your tests will be not relevant.
It is important to give a short guidance, if a test person is blocked by a bug or an absent functionality. It is wise to do not frustrate people more than necessary. Simply say "This is an error in the software. It does not work yet." If a user is in the middle of a process and there is a way further, simply add "Please click XXX". Write -100 in your karma, if you say something more.
If there is no solution, say "Thank you. Unfortunately we must stop here. This is a software bug that must be fixed."
If your user is blocked, there are two kinds of problems: "How?" and "What?"
A question "How can I do X?" points on a minor usability problem. Note it, give your user a short guidance and continue the test.
A question "What must I do next?" is a sign for a critical blocker. Note it and draw a big exclamation mark. This is the moment when your real users will leave your site. I recommend to stop a test and to examine the causes. There is no standard solution how to do this, use your creativity. Please do not forget about your test person. If you can discover something by asking questions, do this. If questions cannot give you more information or you do not know what to ask, postpone this to after test analysis and do something else.
If you store video of your user, record voice, or track user actions, do not forget to ask permissions before the test.
First Impression
To start your test simple say: "This is a web site we will to test. Please look at it. Here is a PC (or a tablet) you can use. You can read or click whatever you what. You should not study our site in depth but try to get a big picture in about 1-2 minutes. Please stop as soon as you have impression you understand the goals of this site and find what services this site can offer you."
Before you give your user full control do not forget to ask: "Do you have any questions before you start?"
If your user does not understand something, clarify and make a note in your test script to check, if it must be corrected. If he tries to start a discussion, block it with short answers and suggest to speak about this after the test.
When your user is ready, start a timer. I usually mark all notes and records with the current time or the time after test start. If you use professional software such as Morae, this will be done automatically.
There are 3 major characteristics for observation: what a user does, how she reacts, which ideas cross your mind.
Ideas are most important. To note them you can ignore other things. It is not needed to write full sentences. Usually several keywords are enough to recall the full idea afterwards. But write your ideas properly as soon as you have time.
User's reactions may mark critical problems. Note moments when she searches for something, becomes confused, looks irritated or chooses a wrong path. I usually draw a symbol or write a verb and add the full description after the test.
Do not try to guess user's mood. Do not hope that a test person smiles because she is happy. She may be simply conscious that you look at her. (Yes, there are several gender and cultural differences. For instance males more frequently try to fake competence. However you can forget about this now and simply ignore psychology.)
User's reactions are more important than user's actions because a real time observation is more precise. Do not believe in video. Firstly any technical process may fall. Secondly you can observe more than may be captured by a camera. People are accustomed to understand body language. If you feel something about your user, write this. However do not forget to place a big question mark after it.
User's actions are the most simple part. There are a lot of tools that track mouse, keyboard and even user's eyes. They can deliver you a lot of numbers that may be converted in pretty diagrams and tables with statistics. I recommend to forget about these wonderful things. Shortest times do not necessary mean better understanding, user's frustration has no direct connection with number of clicks, and scrolling may be sometimes bad and sometimes good.
Do not forget that technical tools are limited. Note when your user reads something, searches for control elements, clicks in wrong places or simply plays with a mouse because of anxiety. I usually prepare printed screenshots where I place marks and words with colored pens. Good tools enable you to write comments and to synchronize them with mouse tracking and recorded video, but old analog technologies are also effective. If you have several observers, it is good to divide responsibilities. For instance one writes down user's reactions and the other marks which menu items the user has chosen and what information is he searching for.
It is wise to prepare protocols before tests. Reserve one column for time and one column for the pages your user sees at the moment. Chaotic comments on a blank paper sheet consume much more time for understanding and analysis as the time you must invest in the most scrupulous preparation. Additionally test scripts and protocols may be used many times.
As you remember we are in the First Impression test. At least after 60 and 90 seconds ask: "OK! Are you ready?" At least after 2 minutes say: "OK! Let's stop here. Please answer our questions."
You are sitting near the test person and making your notes in a protocol. Your "OK!" signals that you are ready for input. If you have professional setup and perform your observation remotely think about more polite words.
Take 10 seconds to review your notes and then ask the following questions:
Discoverability. Will a visitor remember your site, come back and tell about it his or her friends.
- What is the name of this site?
- What does it mean?
- What is the url of this site?
Goals. Will a visitor become a user.
- What can a user do on this site?
- For which users is it intended? Age, gender, occupation, education etc.
- Are there real life situations when you may use this site?
- What needs can this site fulfill for its main audience?
- What needs can this site fulfill for you?
- What can you do, if you start using this site?
Credibility. This is a frequently forgotten aspect but it is critical, if you will get user's money, user's data or even user's time. Simply speaking credibility means that your user believe in you and the services you offer. You must think out the questions that are appropriate for the functionality of your site. Below is a sample that may be relevant.
- Is this site secure?
- Do you believe the data you leave on this site will be used properly?
- If you register on this site, would you use your real name or rather a fake name or a nickname?
- Do you think, you would register on this site with your main e-mail address or will you prefer to use an e-mail address you rarely use?
- Do you think, you would register with your real address and your personal information?
- If you find a teddy bear you like on this site would you buy it or would you prefer first to search it in the web shop you usually use?
- If you write a post on our site, will it be stored permanently?
- Do you think, you can meet in this social network people that are nice and interesting for you?
- To whom may you recommend this site?
Note, your goal is not to fill the questionnaire, but to understand your user. Ask "why?", if there may be any further information to discover. Change your questions and ask anew, if you suppose the user did not understood them correctly. Ask for comparisons and differences with the sites your user prefers.
Of course this must not become an interrogation. If you are disappointed with the user's answers and this impression is shown on your face, immediately ask what could be done better. It is not probably that a test person suggests a perfect solution but you do not last an impression that he had harmed you. Ask even if your user knows about this.
After you are done with questions look at your notes and ask about moments that are important. For instance, what your user had searched on the main page, why could she not find the link, why did he use search input, what had he awaited below the image he has clicked. If your user does not understand, show the place in question. I prefer to use printed screenshots because they are simpler and quicker to show.
Do not ask about feelings. The memory about them is vague and the reasons are unconscious.
If you have to spend the whole time for the first test only, spend it. This script is about 3 flags. You may have a whole page full of security certificates, your goal audience may match the test person perfectly, you may offer an excellent service and the users of your social network may be all friendly amazing extraverts. This does not matter because first impression wins. Your chances to survive are bad, if you fail to raise the first flag.
Before you start the next test stage review your notes and correct them if needed. It is always frustrating, when you are unable to decipher your discoveries and are forced to strike through notes that may be important.
Information Search
To start this test simply say: "Please find X, read about it and after that I'll ask you some questions. Please stop, when you are ready."
"X" must be something practical. Do not ask "Which great services offers this site?", "How much positive comments are posted under this post?" or "How can this site save the world?" Depending of the purpose of the site you may ask "Please find a pair of running shoes for a 12 years old boy.", "Please learn how you can insert a video clip in your blog post" or "Please find the delivery conditions and how much would a 4 kg package cost for a customer from UK?"
The results of the First Impression test may suggest good candidates for "X". Feel free to change the prepared script.
It is not needed to open the main page before Information Search test. Stories like "a friend had sent you a link to a blog post..." also cannot add valuable information. If the navigation and the information structure of your site is good, the results would be not affected by the place where you start. If some paths to the requested information are simple and others are difficult and tricky, do not hope you have luck and the real users will always start from the right place. However it is reasonable to avoid errors that you have already found.
The access to the information must be quick and texts on your pages must be easy to read. It is good to learn how users search through you site but stop the test, if it takes more than 3 minutes or you feel your user is bored.
Again shortly review you notes then take prepared questionnaire.
The questions to ask after this test depend of the purpose of your site and the task you had given the test person.
Firstly check, if your user had understood the information. For instance "What must you do to insert a video clip? Can you embed a video from one other site? Can you upload a file from your computer?" or "How much do this shoes cost? Which colors are available? What is the meaning of this 3 stars below the photo?"
The page your user had found must be opened and he must be free to read it or to click anything to get an answer. You test your site not the user's memory. Even if your user had found a wrong page you can ask the questions and to suggest to try to search for the information again.
If user's answers are not correct, simply say "No. The price would be 5 $ because..." and then explain why. It is important to check, if the user had understood your explanation. "O, yes!" means that the problem is much smaller than by "What?" or "Eh?"
After that take your notes and ask "Why?" questions. Why had the user followed this link? Why had he skipped the introduction part? Why had she not scrolled to the end of the page?
You may also ask, if the menu structure is clear, if the images are good to understand, if your user is happy to read the texts you had written. However I doubt the answers would be useful to improve your site. The question "What do you think is wrong?" also delivers usually information that is more polite than relevant.
If the test goes quick and you have enough time, you can ask to find one other "X". You can also postpone this to the time after Happy Path test.
You may ask different test persons the same question. You also may set different goals. Information Search must find problems in navigation. It is usually more important to cover more paths than to find that some users can solve the same task better than others. On the other hand it is better to collect statistics, if you compare two different kinds of information representation. It is also reasonable to repeat the same test with a different user, if you have discovered a problem and went to understand it better.
In contrast to the First Impression test it is not important to dig into problems you have found. Yes, if the second flag is not raised, many users will leave your site. But a bad navigation is a bad navigation and an unclear text is an unclear text. Correct them and then test the new versions.
Happy Path
Before the start of the Happy Path test make a short pitch and describe the goals of your site, why it is good and what services does it offer. It is OK to explain things your test person had not understood and even to discuss how your site may make the world better. However do not forget about time.
The Happy Path test is about the main process of your site. Set the proper start page before the test start. Select the shoes your user must buy, open the blog in which your user must make a post, open the list of restaurants, from which your user must select the most appropriate place for a party on Sunday.
Also prepare all information your user must enter: surname, password, address, nickname, place of birth, name of the dog he had, promocode, title of the blog post, text your user must enter, etc. If the process includes file upload or a confirmation per e-mail, this must also be ready and easy to access.
Some users may have difficulties by inventing nicknames or hesitate before they enter their credit card data in a payment form. This are not the processes you went test now. Simple say your user: "Please buy this shoes. Here is the customer name and the delivery address. Please register when the site asks you, if you went to do this. Here are password and e-mail address. Please open this inbox to confirm the link. Please enter this credit card data by payment." Ask, if your user understands the task and start the timer.
The third test is the simplest however your observation must be most careful and your eyes most critical. Here small usability errors may easily turn your Happy Path into Frustration Valley and your customers into passers-by.
This test must be completed. If your user is blocked, give hints. If this does not help, say what must be done to process further. Of course you must mark such points in your protocol with big red exclamation marks.
The questions to ask after test depend on the tested process. Again, do not forget to review your notes in test protocol and to ask "Why?" questions.
After all tests are completed ask about user's experience: "How often you use Internet?", "Do you use Internet for your professional work?", "How often do you order food online?", "Can you remember similar sites and have you used them?", etc. Do not ask about gender, age or marital status except you are testing a dating site.
If you do not pay your test persons, it is good to give them something: a gift certificate, a discount code, a promotional item or a business card.
If you are an experienced software developer and you make a usability test the first time, you will be shocked with what you see. Please keep your voice down. It is also better to draw an angry face in your protocol than to show it yourself. If you do not discover critical usability problems in your first test, you are a usability genius, your test is not correct or the persons you test do not represent your real users. You can choose what you like.
Experts
Usability tests are simple. I would claim that you can find major usability problems with this 3-F Test without professional laboratory setup, expensive software, formal training and deep knowledge. Of course it is necessary to write your test script, to prepare questionnaires and protocols and to perform several times the whole test without users to train your moderation skills and to check, if your software functions correctly. I do not say that this tasks are easy but you can do them.
Because usability people always argue about ROI and marketing of usability and managers always ask how to test user experience designers I would mention what can you get for money. If you have bad previous experience with professional services, you must know that there are other cases.
Firstly, there are professionals and there are experts. Unfortunately this are not always the same people. Professionals charge you money. Experts save your money. Professionals know many wise words and prepare thick reports with nice diagrams and many tables. Experts improve your business and make your users happy. Or at least they offer you such possibility.
Experts can do many wonderful things:
- Experts have seen many real users. They have seen how people of different ages, genders, cultures and education levels interact with computers. They know how users really are, and they do not imagine how users could be in a perfect world. This is the experts' most important advantage.
- Experts know human psychology. They do not simply observe what a user does but also can understand what is inside user's head. They can also predict user's reactions.
- There are a lot of usability patterns, usability guidelines and "best examples". But one size does not fit all. Experts know what is good for your goals. I have seen many samples when a "bad usability" gave more profits.
- Experts know much more than usability, user experience and even human-computer interaction (HCI). They can say that your yellow-rosa background do not match with photos on your site or that your user will probably abort the purchasing process when he sees a question about his annual income, even if the answer is optional.
- Experts know many ways to solve a problem. They can estimate advantages and disadvantages of different solutions and can explain them.
- Experts know the process of software development. It is simple to think out a theoretical solution that corrects defined usability problems. Experts do not explain one week before a deadline that you must redesign your site completely. They suggest quick and easy hacks that do not make your site perfect but help to avoid most severe usability damage. (However it is a bad idea to develop a software and then to try to add usability to it.)
- Experts can test with screenshot images and even with paper, pens and scissors thing that you need many man months to develop.
- Experts say not what you like to hear but what they think. And they can prove their words.
I can continue this list further however I would mention only one thing: Experts do not blindly believe in experts' opinion and prove their ideas with real users.
Honestly speaking I have seen only several usability professionals who may match this description completely however many people are good or very good. At least the services they offer are without doubt worth the money.
(CC) Vit R, 2014
This text may be updated or moved in the future. For more information visit my site http://vit-r.com/hci
There are many possibilities to spend time with a potential user. I describe a usability test I would recommend on base of my personal preferences and my personal experience. I call it the 3-F Test. You can perform it even if you don't know anything about usability.
I describe a test of a web portal but the same script may be easily adapted for PC applications, smartphone apps and other kinds of software. It is possible to perform a test without software with simple prototypes, screenshoot images and even with paper, pens and scissors. Unfortunately you must be an experienced usability expert to get plausible results. Of course any test produces a lot of results but in this case it is difficult to get the information which is correct and relevant for a dynamic application.
A note for permanent blog readers:
Традиционная просьба посмотреть английский до того, как это ушло на другие ресурсы. Если попадутся совсем бредовые ошибки, удалять комментарии всё равно не надо, я лучше их скрою. Да, если кто-нибудь надумает переводить на русский, просьба всё-таки связаться со мной. А то в прошлый раз вышло что-то странное. Хотя, там английский текст был довольно мутный.
To perform a usability test you also need a user. The best case would be someone from your target audience. However in your first tests this is not important because your goal is to find most obvious problems. The only criterion is the absence of expertise. You can ask what a software developer thinks about your program or what a designer thinks about composition and colors on your web pages but please test your design on a programmer and the clearness of your menus on a designer.
Let's say there are 2 test persons. A hipster (23 years, male, freelancer and gamer) and a financial analyst (32 years, female, MBA, two kids. No chances). Maybe I will use this in the next version of this article. Yet I would randomly reference a test person with "a user", "your user", "he", "she" or "it".
The last important condition for a correct test I would mention is the silence. The source of information is observation. A dialog with a test person would supply opinions not the facts.
There are many usability test kinds when a test manager says something to a test person, when a test person speaks about hers or his impressions and thoughts. There are tests when they both speak with each other. I would recommend to forget about them except you have a lot of knowledge and experience. It is important to remember that your real user would browse your site without comments and questions.
People are very sensitive. They react to your questions and even to the tone of your voice. If you control is a bit weak, a test person becomes an actor and you become a director instead of an observer. This inserts aberrations in your results. Also your guidance will keep the user away from critical problems you could discover otherwise.
3-F Test consist of 3 simple tasks: First Impression, Information Search and Happy Path. It mirrors 3 important flags you must raise to have success in web:
- A potential user sees your site for the first time and decides whether to stay on it or to leave it for another site
- A user searches for important information
- And a user performs the process which is the main goal of your site and which brings you profits: buys an item, writes a post, connects to another member, finds a restaurant nearby, places a delivery order, etc.
Simply speaking 3-F Test consist of 3 independent tests. I refer them below as "steps", "test stages" and simple "tests".
Let's say you have planned 30 minutes for your test session. I recommend to divide this time in the following way:
- 10 minutes - buffer time reserve
- 5 minutes - First Impression test
- 5 minutes - Information Search test
- 10 minutes - Happy Path test
This includes test instructions and a discussion after each test stage.
If you think you need more time for one of the steps, you already have a serious problem. In real life a user quickly gets bored or irritated. If you hope she or he will invest more efforts in your wonderful site, you are wrong. If your potential users are stopped by obstacles, most of them simply leave and never return. The goal of a usability test is to prevent this flow of events.
This means: do not break your test, if a test person is not able to reach one of the goals. It is more important to search for a solution than to improve the later processes your potential users never reach. Try to dig a bit deeper.
It is usually recommended to make excuses before a test and to say that you do not test your user, that he must not blame himself for the errors, that your site is not perfect, etc.
I would skip this part. A test person will be frustrated when something goes wrong, she will have bad feelings by errors and you cannot prevent with words user's reactions on your astonishment, disbelief, joy or surprise, if your face or your voice reveal them.
If your test person did not read this post, you can simply say: "We will test how a user can use our site. Please complete our tasks, we will observe you and then ask you some questions."
It is important to prevent discussions and guidance. A real user will never call your help desk to ask questions. At least this concerns problems that you can discover in this test. I usually say: "Our goal is to find errors and misleading elements on our site. If you are blocked please try to find a solution yourself. Please ask questions only, if you have feeling, you cannot do it without help."
At least half of the users will do this. Others will try to ask for your help or test your reactions. It is important to try to block this attempts with shortest possible answers. If you'll be manipulated by test participants, the results of your tests will be not relevant.
It is important to give a short guidance, if a test person is blocked by a bug or an absent functionality. It is wise to do not frustrate people more than necessary. Simply say "This is an error in the software. It does not work yet." If a user is in the middle of a process and there is a way further, simply add "Please click XXX". Write -100 in your karma, if you say something more.
If there is no solution, say "Thank you. Unfortunately we must stop here. This is a software bug that must be fixed."
If your user is blocked, there are two kinds of problems: "How?" and "What?"
A question "How can I do X?" points on a minor usability problem. Note it, give your user a short guidance and continue the test.
A question "What must I do next?" is a sign for a critical blocker. Note it and draw a big exclamation mark. This is the moment when your real users will leave your site. I recommend to stop a test and to examine the causes. There is no standard solution how to do this, use your creativity. Please do not forget about your test person. If you can discover something by asking questions, do this. If questions cannot give you more information or you do not know what to ask, postpone this to after test analysis and do something else.
If you store video of your user, record voice, or track user actions, do not forget to ask permissions before the test.
First Impression
To start your test simple say: "This is a web site we will to test. Please look at it. Here is a PC (or a tablet) you can use. You can read or click whatever you what. You should not study our site in depth but try to get a big picture in about 1-2 minutes. Please stop as soon as you have impression you understand the goals of this site and find what services this site can offer you."
Before you give your user full control do not forget to ask: "Do you have any questions before you start?"
If your user does not understand something, clarify and make a note in your test script to check, if it must be corrected. If he tries to start a discussion, block it with short answers and suggest to speak about this after the test.
When your user is ready, start a timer. I usually mark all notes and records with the current time or the time after test start. If you use professional software such as Morae, this will be done automatically.
There are 3 major characteristics for observation: what a user does, how she reacts, which ideas cross your mind.
Ideas are most important. To note them you can ignore other things. It is not needed to write full sentences. Usually several keywords are enough to recall the full idea afterwards. But write your ideas properly as soon as you have time.
User's reactions may mark critical problems. Note moments when she searches for something, becomes confused, looks irritated or chooses a wrong path. I usually draw a symbol or write a verb and add the full description after the test.
Do not try to guess user's mood. Do not hope that a test person smiles because she is happy. She may be simply conscious that you look at her. (Yes, there are several gender and cultural differences. For instance males more frequently try to fake competence. However you can forget about this now and simply ignore psychology.)
User's reactions are more important than user's actions because a real time observation is more precise. Do not believe in video. Firstly any technical process may fall. Secondly you can observe more than may be captured by a camera. People are accustomed to understand body language. If you feel something about your user, write this. However do not forget to place a big question mark after it.
User's actions are the most simple part. There are a lot of tools that track mouse, keyboard and even user's eyes. They can deliver you a lot of numbers that may be converted in pretty diagrams and tables with statistics. I recommend to forget about these wonderful things. Shortest times do not necessary mean better understanding, user's frustration has no direct connection with number of clicks, and scrolling may be sometimes bad and sometimes good.
Do not forget that technical tools are limited. Note when your user reads something, searches for control elements, clicks in wrong places or simply plays with a mouse because of anxiety. I usually prepare printed screenshots where I place marks and words with colored pens. Good tools enable you to write comments and to synchronize them with mouse tracking and recorded video, but old analog technologies are also effective. If you have several observers, it is good to divide responsibilities. For instance one writes down user's reactions and the other marks which menu items the user has chosen and what information is he searching for.
It is wise to prepare protocols before tests. Reserve one column for time and one column for the pages your user sees at the moment. Chaotic comments on a blank paper sheet consume much more time for understanding and analysis as the time you must invest in the most scrupulous preparation. Additionally test scripts and protocols may be used many times.
As you remember we are in the First Impression test. At least after 60 and 90 seconds ask: "OK! Are you ready?" At least after 2 minutes say: "OK! Let's stop here. Please answer our questions."
You are sitting near the test person and making your notes in a protocol. Your "OK!" signals that you are ready for input. If you have professional setup and perform your observation remotely think about more polite words.
Take 10 seconds to review your notes and then ask the following questions:
Discoverability. Will a visitor remember your site, come back and tell about it his or her friends.
- What is the name of this site?
- What does it mean?
- What is the url of this site?
Goals. Will a visitor become a user.
- What can a user do on this site?
- For which users is it intended? Age, gender, occupation, education etc.
- Are there real life situations when you may use this site?
- What needs can this site fulfill for its main audience?
- What needs can this site fulfill for you?
- What can you do, if you start using this site?
Credibility. This is a frequently forgotten aspect but it is critical, if you will get user's money, user's data or even user's time. Simply speaking credibility means that your user believe in you and the services you offer. You must think out the questions that are appropriate for the functionality of your site. Below is a sample that may be relevant.
- Is this site secure?
- Do you believe the data you leave on this site will be used properly?
- If you register on this site, would you use your real name or rather a fake name or a nickname?
- Do you think, you would register on this site with your main e-mail address or will you prefer to use an e-mail address you rarely use?
- Do you think, you would register with your real address and your personal information?
- If you find a teddy bear you like on this site would you buy it or would you prefer first to search it in the web shop you usually use?
- If you write a post on our site, will it be stored permanently?
- Do you think, you can meet in this social network people that are nice and interesting for you?
- To whom may you recommend this site?
Note, your goal is not to fill the questionnaire, but to understand your user. Ask "why?", if there may be any further information to discover. Change your questions and ask anew, if you suppose the user did not understood them correctly. Ask for comparisons and differences with the sites your user prefers.
Of course this must not become an interrogation. If you are disappointed with the user's answers and this impression is shown on your face, immediately ask what could be done better. It is not probably that a test person suggests a perfect solution but you do not last an impression that he had harmed you. Ask even if your user knows about this.
After you are done with questions look at your notes and ask about moments that are important. For instance, what your user had searched on the main page, why could she not find the link, why did he use search input, what had he awaited below the image he has clicked. If your user does not understand, show the place in question. I prefer to use printed screenshots because they are simpler and quicker to show.
Do not ask about feelings. The memory about them is vague and the reasons are unconscious.
If you have to spend the whole time for the first test only, spend it. This script is about 3 flags. You may have a whole page full of security certificates, your goal audience may match the test person perfectly, you may offer an excellent service and the users of your social network may be all friendly amazing extraverts. This does not matter because first impression wins. Your chances to survive are bad, if you fail to raise the first flag.
Before you start the next test stage review your notes and correct them if needed. It is always frustrating, when you are unable to decipher your discoveries and are forced to strike through notes that may be important.
Information Search
To start this test simply say: "Please find X, read about it and after that I'll ask you some questions. Please stop, when you are ready."
"X" must be something practical. Do not ask "Which great services offers this site?", "How much positive comments are posted under this post?" or "How can this site save the world?" Depending of the purpose of the site you may ask "Please find a pair of running shoes for a 12 years old boy.", "Please learn how you can insert a video clip in your blog post" or "Please find the delivery conditions and how much would a 4 kg package cost for a customer from UK?"
The results of the First Impression test may suggest good candidates for "X". Feel free to change the prepared script.
It is not needed to open the main page before Information Search test. Stories like "a friend had sent you a link to a blog post..." also cannot add valuable information. If the navigation and the information structure of your site is good, the results would be not affected by the place where you start. If some paths to the requested information are simple and others are difficult and tricky, do not hope you have luck and the real users will always start from the right place. However it is reasonable to avoid errors that you have already found.
The access to the information must be quick and texts on your pages must be easy to read. It is good to learn how users search through you site but stop the test, if it takes more than 3 minutes or you feel your user is bored.
Again shortly review you notes then take prepared questionnaire.
The questions to ask after this test depend of the purpose of your site and the task you had given the test person.
Firstly check, if your user had understood the information. For instance "What must you do to insert a video clip? Can you embed a video from one other site? Can you upload a file from your computer?" or "How much do this shoes cost? Which colors are available? What is the meaning of this 3 stars below the photo?"
The page your user had found must be opened and he must be free to read it or to click anything to get an answer. You test your site not the user's memory. Even if your user had found a wrong page you can ask the questions and to suggest to try to search for the information again.
If user's answers are not correct, simply say "No. The price would be 5 $ because..." and then explain why. It is important to check, if the user had understood your explanation. "O, yes!" means that the problem is much smaller than by "What?" or "Eh?"
After that take your notes and ask "Why?" questions. Why had the user followed this link? Why had he skipped the introduction part? Why had she not scrolled to the end of the page?
You may also ask, if the menu structure is clear, if the images are good to understand, if your user is happy to read the texts you had written. However I doubt the answers would be useful to improve your site. The question "What do you think is wrong?" also delivers usually information that is more polite than relevant.
If the test goes quick and you have enough time, you can ask to find one other "X". You can also postpone this to the time after Happy Path test.
You may ask different test persons the same question. You also may set different goals. Information Search must find problems in navigation. It is usually more important to cover more paths than to find that some users can solve the same task better than others. On the other hand it is better to collect statistics, if you compare two different kinds of information representation. It is also reasonable to repeat the same test with a different user, if you have discovered a problem and went to understand it better.
In contrast to the First Impression test it is not important to dig into problems you have found. Yes, if the second flag is not raised, many users will leave your site. But a bad navigation is a bad navigation and an unclear text is an unclear text. Correct them and then test the new versions.
Happy Path
Before the start of the Happy Path test make a short pitch and describe the goals of your site, why it is good and what services does it offer. It is OK to explain things your test person had not understood and even to discuss how your site may make the world better. However do not forget about time.
The Happy Path test is about the main process of your site. Set the proper start page before the test start. Select the shoes your user must buy, open the blog in which your user must make a post, open the list of restaurants, from which your user must select the most appropriate place for a party on Sunday.
Also prepare all information your user must enter: surname, password, address, nickname, place of birth, name of the dog he had, promocode, title of the blog post, text your user must enter, etc. If the process includes file upload or a confirmation per e-mail, this must also be ready and easy to access.
Some users may have difficulties by inventing nicknames or hesitate before they enter their credit card data in a payment form. This are not the processes you went test now. Simple say your user: "Please buy this shoes. Here is the customer name and the delivery address. Please register when the site asks you, if you went to do this. Here are password and e-mail address. Please open this inbox to confirm the link. Please enter this credit card data by payment." Ask, if your user understands the task and start the timer.
The third test is the simplest however your observation must be most careful and your eyes most critical. Here small usability errors may easily turn your Happy Path into Frustration Valley and your customers into passers-by.
This test must be completed. If your user is blocked, give hints. If this does not help, say what must be done to process further. Of course you must mark such points in your protocol with big red exclamation marks.
The questions to ask after test depend on the tested process. Again, do not forget to review your notes in test protocol and to ask "Why?" questions.
After all tests are completed ask about user's experience: "How often you use Internet?", "Do you use Internet for your professional work?", "How often do you order food online?", "Can you remember similar sites and have you used them?", etc. Do not ask about gender, age or marital status except you are testing a dating site.
If you do not pay your test persons, it is good to give them something: a gift certificate, a discount code, a promotional item or a business card.
If you are an experienced software developer and you make a usability test the first time, you will be shocked with what you see. Please keep your voice down. It is also better to draw an angry face in your protocol than to show it yourself. If you do not discover critical usability problems in your first test, you are a usability genius, your test is not correct or the persons you test do not represent your real users. You can choose what you like.
Experts
Usability tests are simple. I would claim that you can find major usability problems with this 3-F Test without professional laboratory setup, expensive software, formal training and deep knowledge. Of course it is necessary to write your test script, to prepare questionnaires and protocols and to perform several times the whole test without users to train your moderation skills and to check, if your software functions correctly. I do not say that this tasks are easy but you can do them.
Because usability people always argue about ROI and marketing of usability and managers always ask how to test user experience designers I would mention what can you get for money. If you have bad previous experience with professional services, you must know that there are other cases.
Firstly, there are professionals and there are experts. Unfortunately this are not always the same people. Professionals charge you money. Experts save your money. Professionals know many wise words and prepare thick reports with nice diagrams and many tables. Experts improve your business and make your users happy. Or at least they offer you such possibility.
Experts can do many wonderful things:
- Experts have seen many real users. They have seen how people of different ages, genders, cultures and education levels interact with computers. They know how users really are, and they do not imagine how users could be in a perfect world. This is the experts' most important advantage.
- Experts know human psychology. They do not simply observe what a user does but also can understand what is inside user's head. They can also predict user's reactions.
- There are a lot of usability patterns, usability guidelines and "best examples". But one size does not fit all. Experts know what is good for your goals. I have seen many samples when a "bad usability" gave more profits.
- Experts know much more than usability, user experience and even human-computer interaction (HCI). They can say that your yellow-rosa background do not match with photos on your site or that your user will probably abort the purchasing process when he sees a question about his annual income, even if the answer is optional.
- Experts know many ways to solve a problem. They can estimate advantages and disadvantages of different solutions and can explain them.
- Experts know the process of software development. It is simple to think out a theoretical solution that corrects defined usability problems. Experts do not explain one week before a deadline that you must redesign your site completely. They suggest quick and easy hacks that do not make your site perfect but help to avoid most severe usability damage. (However it is a bad idea to develop a software and then to try to add usability to it.)
- Experts can test with screenshot images and even with paper, pens and scissors thing that you need many man months to develop.
- Experts say not what you like to hear but what they think. And they can prove their words.
I can continue this list further however I would mention only one thing: Experts do not blindly believe in experts' opinion and prove their ideas with real users.
Honestly speaking I have seen only several usability professionals who may match this description completely however many people are good or very good. At least the services they offer are without doubt worth the money.
(CC) Vit R, 2014
This text may be updated or moved in the future. For more information visit my site http://vit-r.com/hci