Rethinking Data: API Scripting, Workflow Revisions, and Flood Risk Modeling
Welcome back to another weekly update from my internship at the KMUTT Geospatial Engineering and Innovation Center (KGEO). This week (October 12th to 16th) was a short but incredibly intense one, punctuated by local Thai holidays. It was a week entirely dedicated to taking a step back, identifying dataset flaws, and building a much more robust data pipeline for our flood risk prediction project.
A Reality Check and Restructuring the Workflow
Monday started with a reality check. After sitting down for a discussion with my internship mentor and lecturer, we identified a fundamental problem with the initial dataset I had built for the Nakhon Pathom and Sukhothai flood risk project.
Instead of pushing forward with flawed data, I made the decision to completely restructure the research methodology. Finding reliable, independent ground truth data for Sukhothai became the new absolute priority. I spent the entire day redesigning the workflow from scratch, creating a new master plan that would guide the rest of my week.
The GISTDA Data Heist and Pipeline Engineering
After a refreshing national holiday on Tuesday, I hit the ground running on Wednesday. My hunt for accurate ground truth flood data led me to GISTDA (Geo-Informatics and Space Technology Development Agency).
To my pleasant surprise, I discovered that fetching data from their endpoint did not require complex authentication or logins. Seizing the opportunity, I wrote a custom script to automatically call their API and download the necessary flood data. Once the raw data was secured, I went through a heavy data engineering process (cleaning, merging, and ultimately consolidating all the ground truth records into a single, structured CSV file). This step is a game-changer, as it will make feature engineering and model training significantly smoother moving forward.
Testing the Waters: Preliminary Model Execution
By Thursday, it was time to execute the new master plan. With the freshly structured CSV dataset ready, I initiated a preliminary training phase. I ran the gathered data through several traditional machine learning algorithms, as well as some heavier, complex learning models.
The goal here wasn't to find the perfect final model right away. Instead, this was a critical "sanity check" to validate whether the newly scraped dataset was viable, healthy, and ready for deeper, more advanced research actions in the coming weeks. Fortunately, the data responded well!
Wrapping Up a Short Week
Friday brought another local holiday to Bangkok, providing a much-needed long weekend to recharge. This week taught me a crucial lesson in research: sometimes, the best thing you can do is scrap your initial dataset, rethink your methodology, and build a better automated pipeline from the ground up. I am heading into next week feeling highly confident about our new data structure!

Comments
Post a Comment