Rancang Bangun Sistem Otomasi Data Survei Berbasis Aturan untuk Struktur Data Hierarkis Root-Roster Modul Transformasi Data

Oleh :
Auliya' Nur 'afifah - V3423026 - Sekolah Vokasi

Statistics Indonesia (BPS) of Surakarta City faces challenges in the identification and validation process of survey data anomalies, which has traditionally been done manually using spreadsheets. The process of matching data against validation rules one by one, along with manually tracing irregularities back to the raw data, results in low time efficiency and a high risk of errors. As a solution, this research focuses on developing a Data Transformation Module within the GridRoot system to automate the data preparation and cleaning stages. Developed using the Scrum methodology, this module is implemented utilizing Python FastAPI as the primary data engine, supported by Laravel as the backend, Vue.js as the user interface, and MySQL as the database. The core performance of the Data Transformation Module lies in the execution of three automated ETL (Extract, Transform, Load) stages: rule extraction to convert validation syntax into Python scripts, variable mapping using a fuzzy matching algorithm, and data transformation that prepares integrated data and sets specific reference (index) columns to precisely track anomaly positions. All module functionalities have been evaluated using Black-box Testing, API Testing, and Usability Testing, and were declared successful. Through the implementation of this Data Transformation Module, the lengthy data cleaning preparation stages can be executed massively and automatically, making anomaly tracking back to the raw data much faster, more accurate, and more structured compared to the previous manual process.