Are Observation Checklists Failing Kenyan TVET Trainees? Here’s the Truth

Photo Credit: TVET-CDACC

Observation checklist in Kenyan TVETs is a structured, competency-based assessment tool for assessors. Assessors use it to evaluate the performance of trainees in real-time. It focuses on the specific units of competency, assessing compliance with safety protocols, effectiveness of procedures, and quality of the final outcomes to determine technical proficiency.

It consists of administrative details, unit of competency, tasks or steps, assessment criteria, marks assigned to each task, assessor’s and candidate’s feedback, and documentation requirements.

Get Customized PCs and Assessment Tools

An observation checklist in TVETS is a good concept in Competency-based Education when it is conducted in the right way. It provides the real competencies of students in specific units of competency. However, the use of observation checklists in the current implementation of CBET in Kenyan TVETs faces a myriad of challenges and does not reveal the true picture of the competencies of trainees. Some of these challenges encompass:

Subjectivity and Bias

The subjectivity and assessor bias are still the main issues of the application of observation checklists to determine competition in the TVET institutions. As much as these tools are meant to offer a systematic framework for appraising the performance of the trainees, they are very much dependent on the judgment of the assessor. This reliance leads to uneven grading, whereby different assessors can give varying grades to the same performance of competency. Moreover, personal prejudices (conscious or subconscious) affect the ways in which trainees are rated, which has an impact on the lack of fairness and objectivity. For this reason, competency-based evaluations may reveal observable differences in their results across institutions, leading to a diminishing of the reliability and standardization of such evaluations.

Photo Credit: TVET-CDACC

Get Customized PCs and Assessment Tools

Time management

Observation checklists in TVETs are structured and systematic assessments. For these reasons, they are very time-consuming to the assessors, particularly when they are dealing with a practical session and a large number of trainees. It is anticipated that every trainee is monitored individually relative to various performance parameters.  This takes a lot of time and attention, which cannot be provided by limited training plans. Consequently, the assessors might be in a hurry to complete the assessment exercise to meet all the trainees within the stipulated time. This may result in shallow evaluation where major features of competency, like accuracy, skills, and the ability to solve problems, are not well assessed. In other instances, trainers can tend to check the checklist mark without keenly monitoring the performance of the trainee, thus the validity and reliability of the assessment are compromised. Moreover, the need to fulfil the assessment schedule within a narrow time frame may diminish the chances of meaningful feedback and interaction between trainers and trainees. Assessors can be more oriented towards filling out the checklist instead of conducting some reflective dialogue with the learners or advising them in a positive way. This is against the very essence of Competency-Based Education and Training (CBET), that focusing on learning the skills.

Tedious or tiresome

The observation checklist consists of a lot of performance criteria to be followed and noted on each trainee during the practical sessions. When evaluations are carried out over a long duration of time or for a big group, it is possible to get tired of reviewing long lists of questions, which concern different trainees. Such a tedious and routine procedure decreases the concentration and attention to detail of the assessor. When fatigue crops in, it is more likely that one is going to miss critical criteria and make hasty judgments. Therefore, the reliability of the results can be undermined because of the quality and accuracy of the assessment. Moreover, it negatively affects the motivation and interest of the assessor due to the monotony of the observation checklists. As opposed to an important assessment of the trainee's competence, the process turns out to be a routine one where the documentation is to be completed. This makes the assessment less effective and can be a weakness for the assessor in terms of offering meaningful feedback to the trainees. Furthermore, the cost in the form of processing voluminous paperwork that comes with the checklists in observation can also add to exhaustion, especially in institutions where digital tools are not commonly applied. The whole process is further complicated by the fact that managing, storing, and reviewing a number of checklists regarding different trainees contributes to the workload.

Learn How to Develop Performance Criteria Weighting Tools

Omission of Critical Aspect of Competency

Although observation checklists intend to streamline assessment by subdividing tasks into easily observable forms, sometimes they involve measuring easy-to-measure activities, which leave out more complex aspects of competence. This leads to a neglect of critical thinking, the ability to solve problems, creativity, flexibility, and decision-making, which are very important attributes. A CBET framework competency is more than being able to do things right. It deals with the knowledge of the fundamental principles, reaction to the unforeseen circumstances, and the implementation of skills in the various real-life scenarios. Nevertheless, observation checklists are more likely to focus on procedure compliance, such as a trainee did certain things, instead of on the quality of performance or the rationale of actions. This bias results in insufficient evaluation that does not depict the actual ability of a trainee in a comprehensive manner. Some competencies are determined by the situation and might not be clearly observed at one time. For instance, the skill of a trainee to solve errors or to adjust to dynamic conditions might solely be found in particular situations, which might not be reflected in a standard checklist. Assessors can therefore fail to notice these main signs of competence when they do their evaluation using only checklists. The lack of these vital considerations compromises the soundness of the appraisal process and could lead to trainees being considered competent even in the absence of the vital skills at the workplace.

Lack of Depth in Skill Assessment

The development of observation checklists is generally designed to reflect the performance of certain steps or tasks of a trainee, usually in binary terms such as responses with Yes or No. Although this is a simpler way of assessing, it restricts the capability of the assessors to measure the quality, understanding, and mastery of the skill being portrayed. Competency-Based Education and Training (CBET) does not only focus on the performance of tasks, but also on the demonstration of proficiency, accuracy, and the ability to apply knowledge in various contexts. Observation checklists are inclined to address the superficial performance without addressing the most important aspects, like technique, efficiency, consistency, and logic in actions. An example is when a trainee does a job well but in an inefficient way, or he or she does not get the principle, which the checklist would not capture effectively. Moreover, complex skills are associated with various and connected elements that cannot be efficiently measured by items separated in checklists. Troubleshooting, innovation, and decision-making are the types of skills that cannot be examined and evaluated on the basis of ticking off actions that can be observed. Consequently, through this competency may end up being evaluated as competent using incomplete or shallow evidence of their competencies. Such triviality may also affect the quality and reliability of the assessment process in general because it fails to present a complete demonstration of the capacity of a trainee.

Photo Credit: TVET-CDACC

Get Free Performance Criteria Weighting Tools

Limited Feedback for Trainees

Even though these observation checklists are efficient in capturing certain tasks, they usually lack detailed information about the quality of work and aspects that need to be enhanced. In most instances, it takes assessors only to tick on the box or check off items as done without any explanatory remarks, and the trainees do not have much knowledge of their strengths and weaknesses. Feedback is also an important part of a Competency-Based Education and Training (CBET) framework, which is focused on skills development and ongoing enhancement. Nevertheless, the strict and time-based character of observation checklists restrains the possibility of substantial communication between evaluators and the trainees. This leads to the learners not getting positive advice on how to perfect their techniques, rectify their mistakes, or improve their overall performance. Moreover, a lack of discussion of feedback may prevent self-reflection and personal learning. It is possible that the trainees do not understand properly why they were rated as competent or not yet competent, and this makes them repeat their mistakes, and they develop skills at a slow pace. This confusion compromises the assessment process as a means of learning, and it becomes a formality instead of an act of development.

Buy One PC Weighting Or Candidate's Assessment Tool and Get One Free

Do not Mirror Well Real-world Performance  

The observation checklists evaluate certain tasks in a controlled training setting. However, in most cases, they do not reflect the dynamic and unpredictable aspects of real-life scenarios at work. In practice, multiple skills must be combined at the same time; employees have to react to unforeseen issues and make independent choices that are not well reflected in standardized checklist testing. Furthermore, observation checklists usually divide tasks into a fixed and sequential progression, which may not reflect the actual practice of work. With this systematic methodology, it is possible to develop a simulated environment where the trainees are concerned with filling in the checklist items instead of building up holistic competence. Consequently, a trainee might seem good in assessment and fail to work effectively under job conditions where he or she has to be flexible, adaptable, and solve problems. Additionally, real-life performance is usually accompanied by teamwork, communication, time pressure, and resource limits. These factors cannot be easily simulated or measured through the traditional observation checklists, which means that there is a discrepancy between the competency rated and the job preparedness. As a result, employers might discover that graduates are not practical and self-assured in the workplace, even though they have been certified to be competent. Such a misfit compromises the validity and usefulness of competency-based assessment in Kenyan TVETs.

Conclusion

Observation checklists in TVETs are effective in simple monitoring of trainees’ competencies in specific units. Nonetheless, they do not assess the comprehensive competency of trainees in Kenyan TVETs. For these reasons, there is a need of more integrative, multi-method approach that ensures that trainees learn and demonstrate real, job-ready skills.

Get Customized Free PC Weighting Tools