Keeping track caring about web performance is hard with constantly changing standards, improving browsers, frameworks and devices. It gets even harder when you develop a tool meeting these changing requirements. Eight years ago, as an IT service provider, we were faced with the task of permanently monitoring the performance of one of the largest e-commerce platforms. After the initial use of WebPagetest, we quickly needed to develop our own features. What started as minor extensions became a separate project over time. In this talk, we would like to take you on the journey we have taken developing OpenSpeedMonitor. You will hear about some unexpected challenges, what we learned the hard way and why we would have failed years ago, if we didn't decide to develop FLOSS.