My major issue with this area is that almost every "speed test" is not measuring speed (I'm looking at you, Google).
They measure all sorts of other optimization things. We did an exercise to get a site to 100%, just to see what it would take.
Ends up we couldn't get to 100% because some of the scripts we load from Google don't have caching enabled, so they warned us about their own scripts
The two tools that I love are:
https://tools.pingdom.com - this is a raw speed test that shows you how fast or slow you are loading and shows the reason why, based on actual loading times of every image and script. Very useful for finding easily overlooked issues.
https://www.dareboost.com/en/compare - when the client says "yeah but Google page speed says blah blah" I use this tool to show that they're still loading faster than their competitors, despite what Google is recommending they do. Does brilliant visual side-by-side speed comparisons with a visual timeline to show first pixel, etc.
As for the main point: I live on my cell phone and I never blame slowness on the website, I am always blaming my WiFi or reception - I can't think of a single time where a website itself was slow enough that I recognized it being an issue. I know consumers are impatient, but I don't think 1 second difference on
first page load matters. The trick is to make sure caching ensures that every subsequent action is much faster.