Final round

This last phase of the project explores evaluation of interaction with the high-fidelity prototypes we developed in phase III.

Read more keyboard_arrow_down

Evaluation strategy

Our user evaluation will be conducted in our university lab session with other students who are not part of our group as the users. They will be evaluating two variations of our hi-fidelity prototype for the mobile app for the FitConnect project, which is built using Adobe XD. This is a controlled setting where we will give the participants tasks to complete, using the two different variations, and we will observe them using the application.

We will collect both quantitative and qualitative data, using direct observation, screen capture, and a short questionnaire for the participants.

The constraints we have are that we are performing the evaluation during the SENG 310 weekly lab, in the ELW 220 computer lab, with no funding, out of a population sample of university students in SENG 310.

The purpose of this evaluation is to gather ideas and data to inform and hopefully invent the future design of our app. We hope to learn about bugs in our current design, and user effectiveness (and preference...) between our designs and our app.

Evaluation plan

Why:
We are trying to gauge how effective the core aspects of the app are, and their ease of use. Participant and volunteer sign up will be tested, as it’s required that users are able to sign up easily. Another test will be on the appointment system, and whether or not it is too complicated. This is a key requirement of the app as it allows for UVHS to screen and train volunteers. Participant/volunteer matching will be tested for ease of use and impressions on the features’ functionality since participants need to be able to easily find volunteers to work out with. Finally, the report functionality will be tested to ensure users can feel safe using the app.

Who:
6 people total: 2 participants, 2 volunteers, 2 admins

How:
Users will be required to sign a short consent form. For the participant study, users who don’t have much gym experience will be selected. For the volunteers, users who have gym experience will be selected. Admins will be chosen at random, to try and get a representative sample, as most people have experience with a mobile device.

Where:
Done in groups of 3, 1 participant, volunteer, and admin in an ECS lab.

What:
Participant and volunteer sign up, matching, and reporting volunteers. Admins will be tested approving users, including setting appointments, and acting upon reported users. We will obtain metrics about how easy users find the signup and appointment process. We will also find out how easy participants find reporting volunteers, as safety if essential for the app. Admins will be observed acting upon a volunteer report, so we will obtain data on how easy it is for admins to manage users.

When:
Participants will be evaluated in stages. We will evaluate a pair of 1 participant and 1 volunteer signing up and when they are waiting for approval, have an admin go and approve them. Then the participant and volunteer will proceed with using the app.

Download consent form get_app

Task description

We’ll have them perform the tasks on one version of the prototype and then on the other.

The evaluation process involves 3 different groups of possible users of the application: the participants, volunteers and admins, each performing a different set of tasks and some of the common tasks are applicable to all.

We asked the user to perform 6 simple tasks as a volunteer/participant:

  1. Login to the application.
  2. Assume you would like to match with a user who is on the home screen. “Like” the users profile from the home screen and start a new conversation with them.
  3. Assume you would like to message “Layne” from the messages screen. Go to Layne’s chat and message her. Assume you do not want to be matched with Layne anymore and unmatch with her.
  4. Assume you would like to talk to Alex. Alex has said something inappropriate to you at one of your meetings. Report Alex.
  5. Assume you would like to edit your profile. Upload a new picture of yourself and update your “About me” section.
  6. Assume you are not liking the FitConnect app and you would like to delete your account. Please delete your account.

We asked the users to perform 4 simple tasks as an admin:

  1. Login as an admin.
  2. Assume you would like to “approve” the user Makaila. Approve Makaila so she can use the application and schedule an appointment with her. Start a new chat with Makaila to check in on her.
  3. Assume you received a new notification about a reported user. Check the reported users and start a new chat with the user to ask what went wrong. Decide whether you want to “Forgive” or “Ban” the user.
  4. Assume you would like to see only the volunteers who have a “pending approval” status.

Evaluation script

The evaluation of the high-fidelity FitConnect prototype, will be done by choosing 6 candidates from the lab. Three groups will be made, each with two candidates, wherein each group will have the roles assigned as: volunteer, participant and administrator.

After forming the groups, roles of each candidate will be explained and how they could connect with the application making the understanding more clear. Moreover, for better understanding the individual roles, personas from phase I will be shown to each candidate pertaining to the role they are assigned. The next process would be getting consent from all the candidates. This would be done by briefly reading out the consent form and addressing the possible concerns/questions they might have.

Each candidate would be asked to be seated in front of a lab monitor and questions would be asked to each as written in Part 4 of this phase. This process entails asking the candidates to perform a certain task, further observing the easy-of-use of the prototype and the possible improvements that could be implemented for better understanding and/or pertaining to any hindrance the candidates may have.

Tasks for User and Participants:

  1. Login to the application
    • User and Participant logins
    • Identifying the separate administration login, that provides additional features/accessibility
  2. Match with a user on the home-screen of the application and start a conversation
    • Identifying the ease-of-use or difficulty in matching and starting a conversation
  3. Reporting a particular user/participant
    • Assessing the ease of performing this task (Is it easy to navigate and report?)
    • In situations, wherein the user/participant has been communicating inappropriately, and is to be reported.
  4. Editing/updating your profile
    • Pertaining to situations where the user/participant wants to upload a new photo or update their “About me” section
  5. Deleting the application
    • Assessing the easy of the deleting the application, if the user/participant doesn’t want to use the application any longer. (Is it easy to navigate this feature?)

Tasks for Administrator:

  1. Administration login
  2. Approving a user or a participant, and communicating with them
    • Assessing the ease of approving a user so they can schedule an initial meet-up appointment and start a new chat
  3. Getting notified of a reported user or participant
    • Processing a report of a user or participant, by deciding to “Forgive” or “Ban” them
  4. Only viewing participants or volunteers with a pending “Pending Approval” status
    • Easy to access only those with a “Pending Approval” status (Is it easy to navigate this feature for the admin?)

Based on the task description, the candidates would be asked the above questions consecutively in the very order. If the candidates face any difficulty to complete the asked task, the ineffectiveness would be noted and possible improvements could be discussed after the evaluation. Since for this project we created a high-fidelity prototype using Adobe XD and a low-fidelity prototype using Balsamiq Mockups, the evaluation tasks for the same volunteers will be done for both the prototypes, so that the prototypes can be accurately compared.

The evaluation would be concluded by thanking all the candidates for their time and suggestions regarding the application. This assessment promotes the approach of addressing the improvements that could be implemented to better the application to all its users.

Cognitive walkthrough

Cognitive Walkthrough for Volunteers/Participants:

  1. Login into the application by simulating typing. Click on the box that says “Email Address” and then tap the “Login” button.
  2. On the home screen, tap on the users profile. Read the profile by scrolling downwards. View all pictures of the user by tapping on the users picture. Scroll to the bottom of the users profile to “Like” or “Dislike”. To “Like” the user, tap the thumbs up button located at the bottom of the profile on the right side. To “Dislike” a user, tap the thumbs down button located at the bottom of the profile on the left side.
  3. To access the messages from the home screen, tap the messages button located at the top of the screen on the right side. To chat with Layne, tap on her picture or the text. To type a message, click on the “type a message” box. To unmatch Layne, click on her profile picture, located at the top of the chat in the middle of the screen. Scroll down to the bottom of her profile and click the “Unmatch” button located at the middle-bottom of her profile.
  4. To access messages from the home screen, tap the messages button located at the top of the screen on the right side. To chat with Alex, tap on his profile picture or the corresponding message from Alex. To report Alex, tap on the 3 dots located at the top of the chat on the right side. Select the option that best reflects the issue. Tap on the “Report User” button located at the bottom of the screen in the middle.
  5. From the home screen, click the “user” icon in the top left corner. Click the “Edit Profile” button from the list of buttons. To add a picture, click on the empty picture box, then tap the “upload picture” button or the “take a picture” button. To edit the “About me” section, tap on the text. To save changes, go back to the profile home page by clicking the back button located in the top left corner.
  6. To delete your account, from the home screen go to the profile page by clicking on the “user” icon located in the top left corner. Click on the red “Delete Account” button located at the bottom of the list. A “are you sure” notification will pop up. Click the “yes” button located on the right side of the notification.

Cognitive Walkthrough for Admins:

  1. At the bottom of the login screen, there is a “Admin Login”. To login as an Admin, tap on the text, then tap the “Login” button located just above the keyboard.
  2. From the home screen, to approve Makaila, tap on the “Manage” button. There is a blue “Approve User” button located in the middle of the screen. Tap on that button. To know if the user was approved, a small check mark will appear next to the “Approve User” text. To schedule an appointment with Makaila, there is a “Schedule new” appointment button located at the bottom of the list. To start a chat with Makaila, there is a “Message” button located in the middle of the list.
  3. From the home screen, tap on the “View Reports” button located under the “Volunteers/Participants/Appointments” tabs. To view the reported user and inquire about the issue, tap on the “Manage” button located to the right of the profile. To “Forgive” the user and allow them to continue using the applications, tap on the blue “Forgive” button located on the left side. To “Ban” a user from using the application, tap on the red “Ban” button located on the right side. The “Ban” button will bring up a notification box which asks the Admin if they are sure they want to ban the reported user. If the Admin is sure, tap on the “yes” button located on the right side. If the Admin does not want to ban the user, tap on the “no” button located on the left side.
  4. From the home screen, tap on the “Filter” button. The filter button will show a drop down menu of what you would like to filter. Tap on the “Pending” button to only show those users who are pending approval.

Quantitative results

We will collect quantitative data on the amount of time it takes to complete the task, number of times that they have to hit the back button, and a rating of which design they preferred, as well as qualitative data:

  • Which design implementation did you prefer? 1 or 2
  • On a scale of one to ten, how would you rate our aesthetic design?

Qualitative questions

We want to gather some outside suggestions and perspectives for our application. Since we are the designers, we have the opportunity improve the app and it’s interfaces, and as we have learned in class, designers are not great judges of an interface’s usability since we are not typical users. We will do a short survey after the participants complete the tasks that asks:

  • Do you have any suggestions to improve the interface?
  • Do you have any suggestions for additional features?

We will take their answers and preferences into consideration for future implementable features. Their feedback on the immediate existing interface will help us to polish the app, and the suggestions they have for future features will be put on our list of possibilities.

Summary of results

The user study was conducted similar to the evaluation script with both the low-prototypes and high-prototypes considered for the evaluation. The study was conducted in a controlled setting involving the users, in a laboratory environment, where the candidate’s process of completing the task was observed. Collection of data during the observation was done both quantitatively and qualitatively.

Quantitative Data
This data was collected by observing and measuring the user interaction using the Think Aloud technique, where we asked users to think aloud while performing the tasks, which helped during the evaluations as we observed any difficulties they may have had. We also measured the average time it took for each candidate to perform a task for both the high-fidelity prototype and the low-fidelity prototype. This aspect provided a contrast in terms of ease-of-use and accessibility, further giving an idea of which prototype was better implemented. The results obtained through this analysis gave the following:

Average time taken to complete a task

  • Low-Fidelity Prototype - Balsamiq Mockups:
    • Average time to report a participant or volunteer = 1:54 minutes
    • Average time to navigate the editing of profiles = 0:45 minutes
  • High-Fidelity Prototype - Adobe XD:
    • Average time to report a participant or volunteer = 1:10 minutes
    • Average time to navigate the editing of profiles = 0:20 minutes

Based on the data obtained from both the prototypes, it can be concluded that the high-fidelity prototype is clearly more efficient and thus lets the user complete the given task earlier.

Qualitative Data
The qualitative data we gathered through the evaluation was taking a screen recording of the user interaction with the application on both the prototypes, while they performed the tasks similar to the description in part 4. The purpose of collecting data in such a way was to gain insight of the effectiveness of the features and tools used for the application, for instance using the UVic colour pallet to indicate that the application was part of the University of Victoria program, navigating through the application symbols, finding the overall font and colour scheme easily readable and accessible.

Design recommendations

  • Using the variation prototype for your profile will reduce confusion as to editing profile, previewing profile, logging out and deleting account.
  • To reduce confusion about unmatching, instead of having the “...” button take you directly to reporting the user, there will be a screen that will have separate buttons. The buttons will say “Report User”, “View Users Profile”, and “Unmatch”.
  • The “Admin Login” needs to be a button instead of just text.
  • After clicking on a users profile that you would like to view, going back to the home screen was difficult for some users. Instead of having the back button in the middle of the screen, the back button will be placed in the top left corner to reduce confusion and be more consistent with the rest of the application.
  • An admin logout button should be added to the design.
  • Ability to swipe left or right on the potential matches profile instead of clicking the “Like” or “Dislike” buttons.
  • Clearer indication that the profiles on the home screen are people who are in your area, not people you have already “Liked” or “Disliked”.