33
How much flakiness do you tolerate in end to end tests?
(programming.dev)
submitted
1 year ago* (last edited 1 year ago)
by
kersplort@programming.dev
to
c/experienced_devs@programming.dev
End to end and smoke tests give a really valuable angle on what the app is doing and can warn you about failures before they happen. However, because they're working with a live app and a live database over a live network, they can introduce a lot of flakiness. Beyond just changes to the app, different data in the environment or other issues can cause a smoke test failure.
How do you handle the inherent flakiness of testing against a live app?
When do you run smokes? On every phoenix branch? Pre-prod? Prod only?
Who fixes the issues that the smokes find?
If you want to level up your game, find a new job, or grow into a new role, by all means take a course or training on your own time. All of the concerns that you listed are probably worth spending dedicated time to upskill on.
If you stay in this field for much longer, you're going to run into a lot of cases where the thing you've been doing is replaced with the New Thing. The New Thing will have a couple new ideas, but will also fundamentally handle the same concerns as your Old Thing, often in similar ways. Don't spend your free time chasing the New Thing without getting something out of it - getting paid, making a project that you wanted to make anyways, contributing to New Thing open source projects.
If you sink work into the New Thing without anyone willing to pay for it, that's fine, but it means that you might never get someone to pay for it. Most companies are more than willing to hire experienced Old Thing devs on New Thing jobs, and will give you some time to skill up.