Usability Testing (https://pbi.belajarbekerja.com/)

As a UI/UX Designer at Luarsekolah, I recently completed Usability Testing for the websites belajar.bekerja.com and pbi.belajarbekerja.com. The goal of this testing was to assess the usability and efficiency of both platforms, as well as to measure their SUS (System Usability Scale) scores.

For those planning to conduct Usability Testing, keep in mind that the process can be time-consuming. My suggestion is to create a dedicated timeline to help manage the testing process more efficiently. Here is an example of a Usability Testing timeline

Note: To learn more about Usability Testing, please read here: Usability Testing.

The method I used was the SUS Score Questionnaire, followed by in-depth user interviews. Here are the steps I took:

1. First, I defined the testing focus for both websites. The main focus areas were the payment flow for programs and the usability of the dashboard (such as ease of accessing internship tasks and learning materials).

2. After gathering the data to be tested, I prepared a list of questions. Below are some examples:

Positive Questions:

1. The process of purchasing an internship program was easy and smooth.

2. I could easily access the tasks I wanted to work on.

3. I was able to join internship meetings and evaluations easily and without issues.

4. I clearly knew where to find the history of uploaded tasks.

5. The platform design is comfortable and easy to use, providing a good experience.

Negative Questions:

1. I felt that the process of redeeming LS coins or using the referral program was unclear and difficult to understand.

2. I found it difficult to upload tasks.

3. I experienced trouble when trying to edit submitted tasks.

4. I had difficulty accessing the learning materials.

5. I struggled with the process of redeeming a code.

In this questionnaire, I combined both negative and positive statements. Why include both types? This combination helps reduce bias and improves the accuracy of usability testing results. Here are the detailed reasons:

• Avoiding Response Bias
  If all questions are phrased positively or negatively, users tend to give consistent responses without carefully considering each question. By mixing the formats, users are encouraged to think more critically before answering.

• Measuring Respondent Consistency
  By including both positive and negative questions, researchers can assess whether respondents are answering consistently. If there are conflicting answers (e.g., high scores for a positive question but low scores for a similar negative one), it may indicate inattention or bias.

• Reducing Skewed Responses
  Without variation, users may default to neutral or extreme responses (e.g., always agreeing or disagreeing). The combination of question types helps gather more balanced and valid data that reflects real user experiences.

• Ensuring Better Comprehension
  Users must read and understand each question more carefully. This ensures they are truly evaluating their experience with the product rather than answering automatically.

"The combination of positive and negative questions in SUS helps produce more accurate usability testing results that can be reliably used for design decisions."

After formulating the questions to be tested, I presented them to relevant stakeholders to ensure they understood the goals and content to be tested with respondents.

Once all the questions were approved by the relevant stakeholders, the next step was to input them into the questionnaire. I separated the negative and positive questions into different pages to reduce bias, improve user focus, and produce more accurate usability testing results.

To boost user participation in completing the questionnaire, I implemented a reward system. For those working in companies, rewards can be proposed to your division leader by explaining the purpose, so you don't need to spend personal funds to conduct the usability testing.

Here are the questionnaire links you can use as a reference:

1. Project-Based Internship Questionnaire

2. Job Connector Questionnaire

After distributing the questionnaire based on the previously planned timeline, here's how to process the collected data:

1. Gather all the scores provided by respondents.

2. Adjust each respondent's original score by subtracting 1 to scale the range from 0 to 4. This adjustment follows the standard System Usability Score (SUS) calculation method to ensure accuracy and consistency.

3. To get the individual SUS score, multiply each respondent’s total score by 2.5 to convert it to a 0–100 scale.

4. Finally, calculate the average of all respondents’ scores to get the final SUS Score.

We can also calculate the score for each question to identify specific problem areas experienced by users on the website platform. Here’s how to do it:

1. Add up all the scores obtained for each question.

2. Determine the multiplier to convert the results into a percentage (1–100%).
   Formula: (100 / Maximum total score per question)

3. The score for each question can then be calculated.

Here is the usability testing result for the website pbi.belajarbekerja.com:
PBI Calculation Results

Based on the analysis results, we were able to identify areas that require improvement on the pbi.belajarbekerja.com website. Key findings came from the questionnaire, particularly questions 6 and 10, which received the lowest scores. These two questions are related to the referral program redemption process and the class code redemption feature.

This indicates that users are experiencing difficulties or confusion when using these features, which means improvements are needed to enhance the overall user experience.

By identifying the existing issues, we can begin formulating the right questions for the in-depth interview sessions with users. The main focus should be on the sections with the lowest scores based on the questionnaire results, in order to explore the root causes more deeply. This will allow the improvement process to be more targeted and aligned with user needs.

In-Depth Interview Process:

The purpose of this process is to uncover the core pain points in the "referral program redemption and class code redemption" features, which received the lowest SUS scores.

Before conducting the interviews, we first need to create a timeline. Here is the timeline I used: In-depth interview timeline (see the "PBI interview" sheet). Once the timeline is set, the next step is to prepare an interview protocol to ensure a structured session. This protocol should focus on the pain points that have already been identified. Here is the Interview question protocol.

I also tested areas that received higher scores to ensure there are no hidden usability issues. It’s important to remember that the protocol serves only as a guide to keep the discussion on track. During the interview, aim for a relaxed, conversational tone—like chatting with a friend. This approach helps users feel more comfortable, allowing them to share their feedback or complaints more openly, which results in more accurate data.

After drafting the protocol, I created a list of respondents who gave low scores in the survey, then reached out to them to schedule interview sessions. To encourage participation, I offered a reward of IDR 50,000 to each respondent. I selected a total of 10 respondents—5 for exploring the pain points and 5 for A/B testing. The A/B testing aimed to compare the website before and after the improvements, in order to evaluate the effectiveness of the implemented solutions.

Once all appointments with the participants were confirmed, the interviews were ready to begin.

Problem / Pain Points

After conducting interviews with 5 respondents, we identified pain points across 9 key aspects:

Disclaimer: (2 Respondents) means that 2 respondents provided similar or identical feedback.

1. Aspect: Redeeming Regular Class Codes (Priority)

Pain Points:

• The redeem code menu on the landing page is not clear enough. Users expect the redeem feature to be more prominent and easier to access. (2 Respondents)

• The “Dashboard” button is hidden, making it difficult for users to find when attempting to redeem a code. (2 Respondents)

2. Aspect: Referral Program & LS Coin Redemption (Priority)

Pain Points:

• Users struggle to understand the referral mechanism and scheme, requiring them to reread the information multiple times. (4 Respondents)

• Confusion arises due to the presence of two types of LS Coins, separated between the PBI website and Luarsekolah. (1 Respondent)

3. Aspect: Redeeming Online Class Codes

Pain Points:

• Users face difficulties when redeeming codes on the Luarsekolah website, particularly in distinguishing between Prakerja and General classes. (5 Respondents)

• Users expect additional explanations for the class categories on the page displaying code copy cards. (1 Respondent)

• The "Get Code" button is unclear and confusing. (1 Respondent)

4. Aspect: Payment Flow

Pain Points:

• Users feel the button design is unclear, leading them to believe that only 4 projects are available. (1 Respondent)

5. Aspect: Meeting Schedule

Pain Points:

• Users are confused about the "Meeting" agenda, which is accessed through the "Webinar" menu. (1 Respondent)

• Users find it difficult to locate the "Webinar" menu. (1 Respondent)

6. Aspect: Handbook

Pain Points:

• Users assumed the handbook was just an image and didn’t realize they could scroll down. (1 Respondent)

• Users were confused about how to download the handbook file. (1 Respondent)

7. Aspect: Task Access

Pain Points:

• When a question includes a link, users have to copy and paste it manually, which creates additional effort. (1 Respondent)

8. Aspect: Task History Access

Pain Points:

• Users were confused because the “Task History” section is placed at the bottom of the page and requires scrolling. (3 Respondents)

9. Additional Insights

Pain Points:

• Users experienced issues accessing classes on the actual day. (1 Respondent)

• Users were confused by the multiple terms such as “Promo Code, Referral Code, etc.” and how to use them. (2 Respondents)

Ideate / Solution

After identifying the pain points in each aspect, we proceeded to the “Ideate” phase to find solutions through brainstorming and benchmarking sessions with other stakeholders. The following solutions were developed:

1. Aspect: Redeeming Regular Class Codes (Priority)

Pain Points:

• The redeem code menu on the landing page is not clear enough; users expect the menu to be more prominent for easier access. (2 Respondents)

• The “Dashboard” button is hidden, which hinders users from redeeming codes. (2 Respondents)

Solutions:

• Code redemption will be done only through the dashboard to avoid confusion, with added information after payment indicating that redemption should be done via the dashboard.

• Move the redeem code menu to the sidebar in the dashboard for easier access.

• Add a CTA button to access the dashboard on the landing page after users log in, making it easier to reach their dashboard.

2. Aspect: Referral Program & LS Coin Redemption (Priority)

Pain Points:

• Users find it difficult to understand the referral mechanism and scheme, requiring multiple readings. (4 Respondents)

• Users are confused by the presence of two separate LS Coins between the PBI website and Luarsekolah. (1 Respondent)

Solutions:

• Switch the layout between the referral mechanism section and the referral scheme.

• Break down the mechanism into bullet points to improve clarity.

• Add a sample case to explain how the referral scheme works.

3. Aspect: Redeeming Online Class Codes

Pain Points:

• Difficulty redeeming codes on the Luarsekolah website, especially in distinguishing between Prakerja and General class codes. (5 Respondents)

• Users expect additional explanation related to class categories on the code copy cards. (1 Respondent)

• The “Get Code” button is unclear and confusing. (1 Respondent)

Solutions:

• Add clear instructions on the online class page about the steps for redeeming codes.

• Label each class card with “Prakerja” or “General”.

• Redesign the “Get Code” button to make it more visible and intuitive.

4. Aspect: Payment Flow

Pain Point:

• Users feel that the button is unclear and thought only 4 projects were available. (1 Respondent)

Solutions:

• Make the slider button more prominent.

• Display favorite projects on the initial page.

5. Aspect: Meeting Schedule

Pain Points:

• Users are confused by the “Meeting” agenda, which is accessed through the “Webinar” menu. (1 Respondent)

• Users find it difficult to locate the “Webinar” menu. (1 Respondent)

Solutions:

• Rename the “Webinar” menu to “Pematerian” (Materials) so it aligns better with user expectations and content type.

• Merge the “Pematerian” and “Evaluation” menus into the sidebar for consistency and easier navigation.

6. Aspect: Handbook

Pain Points:

• Users thought the handbook was just an image and didn’t realize they could scroll down. (1 Respondent)

• Users were confused about how to download the handbook file. (1 Respondent)

Solutions:

• Add a visible scrollbar next to the handbook visual.

• Add a “Download” button on each handbook card.

7. Aspect: Task Access

Pain Point:

• If a question contains a link, users have to manually copy and paste it, creating extra steps. (1 Respondent)

Solution:

• Add a clickable hyperlink (href) directly within the task section, so users can access the link with one click.

8. Aspect: Task History Access

Pain Point:

• Users are confused because the “Task History” section is placed at the bottom and requires scrolling. (3 Respondents)

Solutions:

• Add a “Task History” section to each weekly task card.

• Create a new menu tab for “Task History” for easier navigation.

9. Additional Insights

Pain Points:

• Users encountered issues accessing a class on the scheduled day. (1 Respondent)

• Users were confused by the many terms like “Promo Code,” “Referral Code,” etc., and how to use them. (2 Respondents)

Solutions:

• Ensure the admin team always checks and verifies access links so that classes can be accessed as scheduled.

• Provide clearer explanations of the different code types and their respective uses.

A/B Testing Results

After identifying the solutions, the next step is to determine whether our solutions work for users by conducting A/B testing. This testing compares the old and new designs. A total of 4 respondents participated, as 1 respondent did not reply to follow-up confirmation. Below are the results of the A/B testing:

1. Aspect: Redeeming Regular Class Codes (Priority)

Pain Points:

• The redeem code menu on the landing page is not clearly visible; users expect the redeem feature to be more prominent and easier to access (2 respondents).

• The “Dashboard” button is hidden, making it difficult for users to redeem the code (2 respondents).

Solutions:

• Code redemption is moved to the dashboard to avoid user confusion. Clear instructions are added after the payment section to direct users to redeem only via the dashboard.

• The redeem code menu on the dashboard is moved to the sidebar for easier access.

• A CTA button to access the dashboard is added to the landing page after users log in.

A/B Testing Feedback:

• Users preferred the new design (4 respondents), although 1 respondent still requested the redeem menu on the landing page.

• Placing the redeem code menu in the sidebar made it easier for users to find and access it (3 respondents).

• The addition of a dashboard CTA button helped users directly access their dashboard (4 respondents).

2. Aspect: Referral Program & LS Coin Redemption (Priority)

Pain Points:

• Difficulty understanding the referral mechanism and scheme, requiring users to reread the information (4 respondents).

• Confusion due to the existence of two different types of LS Coins between the PBI and Luarsekolah websites (1 respondent).

Solutions:

• The section layout is rearranged to display the referral scheme before the mechanism.

• The referral mechanism is broken down into clear bullet points.

• A case example is added to illustrate how the referral system works.

A/B Testing Feedback:

• Users found the new design easier and more comfortable to understand, thanks to the bullet-point format and inclusion of case examples (4 respondents).

3. Aspect: Redeeming Online Class Codes

Pain Points:

• Difficulty redeeming codes on the Luarsekolah website, especially distinguishing between Prakerja and general classes (5 respondents).

• Users expected additional explanations on the card page regarding class categories (1 respondent).

• The "Get Code" button was unclear and confusing (1 respondent).

Solutions:

• Clearer instructions were added to the online class page outlining the code redemption steps.

• Cards are labeled as “Prakerja” or “General” to avoid confusion.

• The “Get Code” button is made more visually prominent.

A/B Testing Feedback:

• Users felt the latest design was more informative and user-friendly due to the added step-by-step instructions (4 respondents).

• Labeling the cards helped users identify class types more easily (4 respondents).

• The new prominent "Get Code" button (primary button) improved accessibility (4 respondents).

4. Aspect: Payment Flow

Pain Point:

• Users assumed only four projects were available due to an unclear button (1 respondent).

Solutions:

• The slider button was redesigned to be more prominent.

• Favorite projects are now displayed on the homepage.

A/B Testing Feedback:

• After relocating and improving the button’s visibility, users (2 respondents) found it easier to access and understood the section could be scrolled using the slider.

5. Aspect: Meetings

Pain Points:

• Users were confused by the “Meeting” agenda being accessed through the “Webinar” menu (1 respondent).

• Users found it hard to locate the “Webinar” menu (1 respondent).

Solutions:

• The “Webinar” menu was renamed to “Pematerian” (Lesson Material) to align with the terminology users were familiar with.

• Both “Pematerian” and “Evaluation” were moved to the sidebar for consistency and easier access.

A/B Testing Feedback:

• Renaming the menu helped users locate and access the meeting materials more easily (4 respondents).

• Grouping both features in the sidebar made it more intuitive and accessible (4 respondents).

6. Aspect: Handbook

Pain Points:

• Users thought the handbook was only an image and didn’t realize it could be scrolled (1 respondent).

• Users had trouble downloading the handbook (1 respondent).

Solutions:

• A visible scroll bar was added to the handbook display.

• A “Download” option was added to the handbook card.

A/B Testing Feedback:

• Making the scroll bar more visible helped users realize the handbook was scrollable (2 respondents).

• The new “Download” menu improved accessibility and made it easier to save the file (4 respondents).

7. Aspect: Task Access

Pain Point:

• When a task included a link, users had to copy and paste it manually, causing extra work (1 respondent).

Solution:

• Hyperlinked URLs were added to the task section so users could access them with a single click.

A/B Testing Feedback:

• All users benefited from clickable links in tasks, making it easier to access them directly.

8. Aspect: Task History Access

Pain Point:

• Users struggled to find the “Task History” because it was located at the bottom of the page and required scrolling (3 respondents).

Solution:

• “Task History” was added to each weekly task card and included in a new top-level menu bar.

A/B Testing Feedback:

• Users found it easier to locate their submitted task history when it was included in both the cards and a dedicated menu (4 respondents).

9. Additional Insights

Pain Points:

• Users couldn’t access their classes on the scheduled day (1 respondent).

• Users were confused by the many different terms such as “Promo Code,” “Referral Code,” etc., and didn’t understand how to use them (2 respondents).

Solutions:

• The admin team must ensure links are active and accessible according to the schedule.

• Provide detailed explanations for each type of code and how to use them.

A/B Testing Feedback:

• N/A

Conclusion:
Based on the results, all users preferred the new design. Most issues in each aspect were successfully addressed. A minor improvement suggested is to bring back the “Redeem Code” option on the landing page, giving users the choice to redeem either from the dashboard or directly from the landing page. That’s all from me. If you have any feedback or suggestions, feel free to reach out. Thank you!