UX Research Case Study: Usability Study of Akademi's.id App
A usability study is one of the research methods that massively used by UX designers and UX researchers. Designers and researchers do usability tests mainly because they want to uncover problems in the design, Discover opportunities to improve the design and learn about the users (both behaviour and preferences).
Back then, In March 2021, my team and I was doing the first usability studies on the Akademi's app as we found some research findings of technical difficulties in the app from the user, and we would like to validate the issues and uncover the design problems while also observing the user behaviour of using the app.
My Role
In this project, I act both as a researcher and designer. As a UX Designer instead of a consumer researcher, the stakeholder asks me to finish the project until the high fidelity and Prototype. I also become try to lead this project with the other 2 team members who do this project. And it turned out really great for them because they also lead other usability studies after finishing this project.
So after you know my role in this project, let’s dive into the project's background!
Why would we do the usability studies on the Akademis?App?
We found some feedback from the users during the previous research that they faced some technical difficulties during their journey on the tryout. At least from around 50 respondents from the survey (previous research survey), 10,2 % said they faced technical difficulties. So to validate the findings, we conduct the usability testing on one of the platforms that are Mobile App (Android)
Then What are we trying to learn from the Usability Testing?
From this usability testing, we would like to learn how easy or difficult for users to use Akademis.id App especially for searching new services that Akademis.id try to offers. To elaborate the goals, I tried to break down into 5 research question :
Research Question :
- How easy for the user to find the latest tryout available in the App?
- How fast participant finish the task that has been given?
- Is the content available on the app is already working as the user needs/want?
- How users interact with Akademis.id App?
- How is the user experience of using the Akademis.id app?
Remote Moderated Usability Testing is the method of usability that we used.
Usability testing is used because we want to see the behavioural result of the interaction between users and our products. We used success rate metrics from the task and time as the metrics of the app’s usability. From this testing, we found several problems that users address. We conduct the usability remotely because Akademis.id is implementing a fully remote working and my team was not in a place with me. It is more convenient for us because of our users in every place in Indonesia.
Why moderated? Because we would like to obtain what they do and we also need the qualitative data to make sure and discover more reason for what our participants have been done during the usability testing. My team and I consecutively become the facilitator, note-taker, and timekeeper for this usability testing in this project.
In this test, we decided to go with 5 participants. My team agrees that conducting a small test with different participants with maximal usability testing will be 5. As Nielsen Norman said, fewer participants are better to distribute the budget to test a lot of design than focusing all of the budgets on one study.
For the participants themselves, here are the participant demographic that we used for the usability testing :
- All of our users have ever user competitors app (Mostly Pahamify and Zenius)
- But only two users that recently used competitor apps, the others already like three months more not using the competitor apps.
- All of our participants have never used the Akademis.id app, so it is perfect for us to learn the usability of a new user on the app.
- And the last thing was all of our users is often to use a smartphone rather than Laptop.
Research Result: Usability Test Result
After we did to five participants, which I have already mentioned above, the demographic of the participants So far, there are 5 tasks that we gave to the participants :
Task 1 :User ingin mendaftar akun Akademis agar bisa mengetahui layanan yang dimiliki oleh Akademis melalui Aplikasi (Use Akademis and try to get the Akademis Account to browse the services)
Task 2 : User ingin mengetahui sebuah layanan yang dapat membantunya untuk mengetahui kemampuannya dalam menghadapi UTBK dengan soal yang terbaru(Use Akademis app to find the latest services that can measure your ability for UTBK)
Task 3 : User ingin mencari sebuah layanan yang dapat membantunya untuk mengetahui kemampuannya dalam menghadapi UTBK dengan soal yang terbaru secara cuma cuma (Use Akademis app to register the latest services that can your ability for UTBK for free)
Task 4 : User mencoba mengetahui berapa lama waktu yang dibutuhkan untuk mengerjakan tryout (Use Akademis app to find how long the services would be for the TKA Session)
Task 5 : User mengganti jurusan dan kampus yang ingin dituju di dalam aplikasi Akademis.id (Use Akademis app to change your dream universities and major in the app)
here is the result :
- The success rate metrics from all of the tasks were only 56% out of 100%. I calculated the success rate metrics based on the Nielsen Norman Article. It means that the results are not too good or too bad because it is above 50% (An bad website/app will have a success rate metrics below 50%), but still, it is far from great.
- From 5 task that we gave to the participants, mostly Task 2 and Task 4 is difficult for the participants. 3 out of 5 participants fail to complete tasks 2 and task 4. The task effectiveness from Task 2 and Task 4 was the lowest and that Is only 40%.
- We also measured the usability using the System usability test, and the result was C (68), and it’s not great but not bad either. But still, it is not great for us to deliver the best services to the students.
But What Happened with the Task 2 and Task 4?
Here are the explanation of the why Task 2 and Task 4 becomes the most difficult Task for the users :
- 3/5 Participants said it is too difficult to find the latest tryout that they could find in the mobile apps because the arrangement of the services is confusing. (Task 2)
During the Task 2 we notice that several things regarding the issues in the design process. Most of our users is trying to look at the banner first after they start the task, but unfortunately the banner is not updated yet with the latest services available so participants could not find the tryout overthere, Then what make it’s difficult to find is because the layout or the arrangement of the latest services was in the down not in the up. So they said that it’s not convienece to search the latest services if we have to scroll to down to find it.
“Mencari tryout terbaru susah karena yang baru di paling bawah harus scroll” (It’s difficult to find a new tryout because the newest is in the down side of the page and you have to scroll it if you want to find it)
“tata letak menu tryout membingungkan”
(The layout of the tryout menu is confusing)
“Tryoutnya kalau terkini itu dibawah karena agak pusing kalau mencari”
(The latest tryout is in the downside of the menu and it is so confusing to look for the newest one)
2. 3/5 Participants could not find the time for completion for every subject in the test in the app because the button is look like not “clickable”
this is happens because the design is look like not “clickable”. When they start to do the task, most of them actually still confused about the places to find the time for completion because it is not appear in the tryout page unless you click the blue bar called in the konten section which 3/5 participants think it is not click able.
“Button “TPA atau TPS” di dalam page Tryout tidak terlihat seperti button”
(The TPA or TPS button in the tryout page is not like a button)
“informasi waktu pengerjaan tryout tersembunyi”
(Information about the time for completing the tryout is hidden)
“Tulisan “TPS” di konten tryout tidak terlihat seperti button, membinungkan”
(a graphic with a “TPS” word on it is not look like a button, it’s confusing)
We also found others finding from this research that actually quite important for the design process especially for creating a user — friendly app and here is the finding :
- Registration flow in this app is complicated
- Microopy “Tryout tidak pakai ribet” is too vague
- Error sign during the registration is not visible
Defining the Problem Statement using “How Might We”
After discovering the problem on the Akademis.app by utilising the usability testing, I tried to create the problem statement before creating the recommendation for this research using a how might we concept and here is the problem statement
Problem : participants said it is too difficult to find the latest tryout in the Mobile App.
Problem Statement : How Might we make the flow to find the latest tryout easier?
Ideation :
- Update the banner regularly everytime there is a new tryout available
- Change the layout from the Tryout page in the App
Problem : participants Could not find the time for completion for every subject in the app
Problem Statement : How might we make the time for completion noticeable?
Ideation :
- Revamps the user interface in the selection tryout page by putting the time for completion outside the button.
Then because the stakeholder wants me to create the mockup of the design and accepted that the apps need to be revamps especially for the how to find the tryout and how to see the time for completion easier. So the first comes to mind was the user flow!
User Flow
This user flow is made because, we saw that the problem was happens because of the flow and process for the user before moving to the wireframes
So in this new user flow, I tried to divided into two process, the first one is the flow of looking for a tryout through the Home Page there is a section called “Tryout” then the second one is utilizing the search bar so they can search whatever services they want including the new one. Besides this one I also made several user flow for others journey as well that you could see down below :
Wireframe
These wireframes was createde after the lo — fi one and there will be a separated wireframe for each journey in the user flow. From the wireframes you could see that I tried to adjust so users could see the latest tryout from the beginning in the top.
Final Touch : Mock Up/ High Fidelity Wireframes
Before that I would like to show the previous design of the app :
Then this is the newer design fro the App :
By looking the design you could notice the difference between the original and the newer design where the newer design offers new layout and arrangement such as “Tryout Terdekat” which means is the latest services that Akademis could offers then in the Detail Tryout menu I tried to make a timeline so they users could see the timeline for the tryout and see the time for completion is needed to take the test.
you could also see others mock up others than this on my behance!
Conclusion
This research project actually was only discovering the design problems of the Akademis android app. But when I discussed with the stakeholders, they also requested the wireframe and also the design since in Akademis we don’t have any product design or any UI/UX Designer. But it is a fun project because I could involve in a different types of design thinking process , starts from emphatise, define, ideation then prototype.
One thing that could be done sooner was try to test the ideation to the users as soon as possible because when I had to take the project my stakeholder ask me to do another research and they say that the validation of the design will be done by others team. But overall it was a great project to learn more about the design thinking process implementation in the product design.
