An evaluation of inter-rater reliability of the technicians in a manufacturing environment

Abstract

In the era of digital economy, Industry Revolution 4.0 has become the aim for manufacturing organisations in order to transform into a smart factory. With the advancement of technology, company engages in continuous improvement projects to ensure high quality products being manufactured. Assessing the strength of agreement between technicians’ ratings of quality problem identification results is of primary interest because an effective diagnostic procedure is dependent upon high levels of consistency between technicians. However, in practice, discrepancies are often observed between technicians’ ratings and it is considered as a major quality issue in monitoring the troubleshooting and repairs of the equipment. This has motivated us to evaluate the accuracy and agreement between technicians’ ratings. The primary objective of this study is to evaluate the interrater reliability of the technicians on the continuous improvement projects before actual implementation. A case study is conducted in one of the smart manufacturing companies in the Penang Free Trade Zone. This study utilised Fleiss’s Kappa analysis because it is suitable in situations where there are more than two raters, i.e. six technicians who are responsible to identify six problems simulated for a continuous improvement project. The findings of the study show good to excellent agreement and high accuracy in problem identification. Overall, the technicians are capable in understanding the newly developed troubleshooting and repairs database and able to carry out the continuous improvement project effectively. This outcome provides top management an insight for evidence-based decision making to thoroughly execute the newly developed digital database in smart manufacturing.