Updating a new version of a third-party library is traditionally not a trivial task. Github's Dependabot, Renovate, and similar services automatically create a new branch with the latest version of a library dependency and then execute project tests to detect any breaking changes. While such services are gaining a lot of traction, no study looks into whether test suites of average GitHub Projects have sufficient coverage and are adequate to detect incompatible library changes.

To better understand the state of test coverage and effectiveness of project test suites for detecting incompatible library changes in open-source projects, I will, in this talk, present a study comprising 262 Java projects on GitHub. By artificially injecting faulty changes in library dependencies, we identify that test suites, on average, have coverage of 58% of their direct and 20% of their transitive dependencies. The average test suite effectively detects 47% of faulty updates in direct dependencies and 35% in transitive dependencies. Based on our findings, I will explain recommendations for developers and toolmakers that could improve the reliability and expectations of automated dependency updating.