….from:
Report finds almost all Android OEMs, not just Samsung, cheat on benchmarkshttp://9to5mac.com/2013/10/02/report-finds-almost-all-android-oems-not-just-samsung-cheat-on-benchmarks/
When Apple SVP Phil Schiller pointed us to a story earlier this week that Samsung was artificially inflatinghttp://9to5mac.com/2013/10/01/phil-schiller-calls-out-samsungs-note-3-benchmarking-shenanigans/ benchmark scores for its new Galaxy Note 3, many were quick to point out it wasn’t the first time Samsung had been caught engaged in such a practice. The same issue was discovered by AnandTech for the Galaxy S4 back in July, and today the site has an extensive report showinghttp://www.anandtech.com/show/7384/state-of-cheating-in-android-benchmarks that almost every Android smartphone manufacturer is shipping devices that do the same.
As pictured in the chart above, that includes the HTC One, HTC One mini, LG G2, Galaxy Tab 10.1, and many others. In fact, the only companies that appear to not be using the method is Apple and Motorola, as well as Google with its Nexus 4 and Nexus 7 devices:
We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop. With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung… None of the Nexus do, which is understandable since the optimization isn’t a part of AOSP. This also helps explain why the Nexus 4 performed so slowly when we reviewed it – this mess was going on back then and Google didn’t partake.
As noted in the report, the gains that OEMs are experiencing from the inflated scores are probably not worth the press they’ve been receiving. AnandTech points out that most of the inflated scores provide under a 10% increase in GPU and CPU performance benchmarks:
The hilarious part of all of this is we’re still talking about small gains in performance. The impact on our CPU tests is 0 – 5%, and somewhere south of 10% on our GPU benchmarks as far as we can tell. I can’t stress enough that it would be far less painful for the OEMs to just stop this nonsense and instead demand better performance/power efficiency from their silicon vendors.
You can check out the full report herehttp://www.anandtech.com/show/7384/state-of-cheating-in-android-benchmarks, which offers in-depth analysis on the optimizations it found for several devices across various benchmark tests.
[android-benchmarks-cheaters.png]
