General Gaming Article

Why Minimum FPS Can Be Misleading
Newegg Daily Deals: Corsair CX600 600W PSU, Samsung 31.5-Inch Curved Monitor, and More!
Check Out Gigabyte's Overclocking-Friendly Z170X SOC Force Motherboard
Mouse and Keyboard Support Headed to Xbox One, Will PC Streaming Come Next?
Microsoft Loses Big Overall Last Quarter But Sees Huge Gain in Surface Sales

Posted: 22 Jul 2015 02:50 PM PDT

What the numbers really mean

There are a few things in life that are constant: death, taxes, and hardware upgrades. Every time we experience a major shift in graphics fidelity, there's a requisite purchase of new hardware to keep up. Having now finished off the recent launches of Nvidia's GTX 980 Ti and AMD's Fury X, there's something we want to discuss a bit more that will affect future GPU reviews-or at least, it will affect our presentation of data. That topic is minimum frame rates.

Unlike average frame rates, usually expressed as FPS (frames per second), minimum frame rates are prone to some wild fluctuations between benchmark runs. The problem is that depending on built-in benchmarks and their reported frame rates isn't always reliable. Some games appear to sweep a few low results under the table, others report the absolute minimum frame rate, and still others use a sort of "average minimum" value. All of these can be useful to varying degrees, but the inconsistency between benchmarks is a real concern. To understand why, we need to talk about why minimum FPS matters and then look at a few benchmarks as examples.

The short summary is that minimum frame rates matter because they can cause a game to stutter. Imagine, as a worst-case scenario, a game where one frame renders at 20fps and then the next three frames render at 180fps. While the average frame rate would be 60fps, on a typical 60Hz display with VSYNC disabled you would see the first frame for three screen updates followed by one update showing parts of the three fast frames. Or let's take an even more extreme case: imagine a game that renders at 60fps for 19 frames and then 10fps for a single frame. The game would feel smooth for those 19 frames and then there would be a big stutter on the last frame. The average frame rate is still a respectable 48fps, but the minimum frame rate indicates there's a serious problem somewhere.

The difficulty is that minimum frame rates often don't come at regular intervals. The 19-to-1 ratio of the second example would be horrible if it happened, but it typically doesn't occur in normal gameplay. What's more likely is that you'll have games that run for hundreds or even thousands of frames at higher FPS values, but at times there's a scene transition or the GPU runs out of VRAM and you'll have some stuttering. Ideally, that's what we want to capture, but many games abstract the benchmark results into just minimum and average FPS. So let's look at a few examples.

First, Tomb Raider (2013) has a decent built-in benchmark. The test shows Lara Croft overlooking a scene of crashed boats, airplanes, etc., as the camera orbits around her. As the entire sequence consists of a single scene in the game, the loading of assets is done ahead of the benchmark run and the results are very consistent. If you run the test ten times, you might see a fluctuation of a few percent at most on both the average and minimum frame rates. Maximum frame rates may show greater variability, but few people are worried about the maximum FPS, so that's not a problem. The Tomb Raider benchmark at least is an example of a trustworthy minimum FPS result.

At the other end of the spectrum, the Unigine Heaven 4.0 benchmark has extremely unreliable minimum frame rates. The test consists of 26 scenes, but frame rates are captured during scene transitions-so for example, the first few scenes may have all their assets loaded into memory, but at some point there's a scene that has to load some assets. When this happens, there might be a single instance where the frame rate drops to 30fps. If the 30fps result happened consistently, it might be meaningful, but if it only occurs during a camera/scene change and it only happens for one or two frames out of thousands, it has little bearing on normal game play.

In between these extremes, there are other games where the built-in benchmark may have erratic minimum FPS results during the first few seconds of a benchmark. Shadow of Mordor is like this, as the level assets are still loading for the first few seconds. If you were actually playing Shadow of Mordor rather than just benchmarking it, you might have stuttering frame rates right as a saved game loads, but then for minutes or even hours afterward the frame rates would be higher and generally consistent. Run the built-in benchmark once and the minimum frame rate might show as 25fps. Run it three times in a row and you'll typically find that the second and third runs show significantly higher "minimums." But even running the test multiple times doesn't fully account for variations between runs.

Enter the 97 Percentile

The good news is that the problem with looking at pure minimums is a well-known fact, so clever statisticians have already had a solution for decades (or more): percentiles. The concept is easy enough: given a large enough set of numbers, sort them and the 97 percentile would be the number that is larger than 97 percent of the results. For minimum frame rates, we're going the other way and looking at the number that's smaller than 97 percent of results. Of course, you can make arguments for a different percentile-99 and 95 percentile are commonly used-but that's more debating semantics.

Some games even report percentiles already, e.g., GTAV's built-in benchmark reports 90-99 percentiles for the five test scenes, along with 50, 75, 80, and 85 percentiles for good measure. GTAV also goes one step further and reports the number of frames under 60fps and 30fps for each test sequence. Let's quickly look at what this means for a specific GPU tested in GTAV, the GTX 980 Ti running at 1080p.

GTAV GTX 980 Ti 1080p Results - Pass 4

Min	Avg	% > 60fps	% > 30fps	97%
8.6	83.1	93.9	99.8	58.8

If we only look at the minimum frame rate, it might appear that GTAV stutters a lot for this specific test case-the average FPS is a rather high 83, but minimum FPS is only 8.6! But looking closer, that minimum frame rate is quite rare, likely occurring only when a bunch of data has to be loaded into memory. Looking at the frame rates above 30fps, only 0.2 percent of frames were below that threshold, while 6.1 percent were below 60fps. And finally, the 97 percentile tells us that 97 percent of frames rendered at 58.8fps or faster.

GTAV GTX980Ti Graph

Of course, we could just present you with a complete graph of frame rates for the test, as shown above. The problem with this approach is that it makes comparing products difficult, especially for people who don't eat, breathe, and sleep statistics. We would need one chart per GPU per game, or perhaps we could do a few GPUs in each chart, but either way it quickly results in information overload. It also requires a lot more time to create all the charts, time which could be better spent in other endeavors. Using a 97 percentile result allows us to quickly get to the heart of the matter and provide a meaningful "typical minimum FPS" value.

Of course, totally ignoring all frame rates below the 97 percentile doesn't necessarily make sense either. Those slow frames are still present, and if there's enough of them-and if they're slow enough-it can dramatically impact the overall experience. Our solution is simple. Instead of looking at just the 97 percentile frame rate, we can find the average FPS for all frames that are slower than the 97 percentile. That way we don't miss out on the effect of a few very slow frames.

Check Out These Frame Rates

Here's where things get interesting. If we take the built-in benchmarks we're running for our GPU tests and log frame rates with FRAPS, this allows us to calculate the average FPS as well as the average FPS of the slowest three percent of frames-what we're calling the "average 97 percentile." We can also look at the true instantaneous minimum FPS according to FRAPS. You might assume all of the games report the absolute minimum FPS, but it turns out they all vary in how they're calculating minimums.

Some of the tests appear to "miss" counting certain frames while others seem to have some sort of percentile calculation in effect. The short summary is that most of the games show unwanted variations in this metric. But let's not jump ahead. In the charts below, we'll look at the reported "minimums" compared to our calculated "average 97 percentile" to show what's going on. We'll also provide separate commentary on each chart to discuss what we've noticed during testing.

Batman Origins Minimum Vs 97percentile

Starting with Batman: Arkham Origins, we immediately see that the "minimum" FPS reported by the built-in benchmark already appears to be doing some form of percentile calculation. Our "average 97 percentile" in all cases is lower than the game's reported minimum, and it would appear Arkham Origins is using a 90 percentile or similar; we've also noticed a tendency for the built-in benchmark to report a lower than expected "minimum" that only occurs between scenes. Looking at AMD and Nvidia GPUs, we have two different results. On the Nvidia side, our 97 percentile and the game's reported minimums are relatively close at higher resolutions, while on AMD hardware we see a relatively large discrepancy. This is one of several titles where the built-in benchmark provides somewhat "misleading" (or at least, not entirely correct) minimum frame rate reporting.

Hitman Absolution Minimum Vs 97percentile

Hitman: Absolution shows some of the same issues as Arkham Origins. The supposed minimum frame rates the built-in benchmark reports are either skipping some frames or using a percentile calculation. Interestingly, this appears to benefit AMD GPUs in particular at higher resolutions, where 4K on the 390X has a reported minimum that's 44 percent higher than our average 97 percentile calculation.

Shadow Mordor Minimum Vs 97percentile

Shadow of Mordor reverses the trend and we see universally higher frame rates compared to the reported minimum FPS. This time, however, AMD cards were being hurt by lower reported minimums. The reason is actually pretty straightforward: For the first few seconds after the benchmark begins, AMD GPUs in particular have a frame every half a second or so that takes longer to render. Once all the assets are loaded, things smooth out, but those early dips occur more frequently on Radeons. Also note that we were running this particular benchmark three times in succession at each resolution to try to stabilize the reported minimum FPS-if we only ran the test once, the reported minimums would be substantially lower on all GPUs.

Tomb Raider Minimum Vs 97percentile

Finally, we have a game where the reported minimum FPS closely matches our average 97 percentile calculations. It could be that Tomb Raider was already doing a percentile calculation, or more likely it's due to the fact that the benchmark scene isn't very dynamic, so the typical minimum FPS occurs for a longer period of time. The game also appears to pre-cache all assets, as much as possible, so there aren't any unusual spikes in FPS. The net result is that at most resolutions, we only see a 1-2fps difference, so the built-in gives meaningful and consistent results-at least for this particular game.

Unigine Heaven Minimum Vs 97percentile

Of the five games/engines we're testing for this article, Unigine Heaven has the least reliable/useful minimum FPS reporting. At lower resolutions on Nvidia hardware, a few dips in frame rate during scene transitions skew the numbers, so our 97 percentile results are substantially higher. Where things get interesting is on AMD GPUs, where the engine actually appears to miss some of the low frame rates. This might be something a driver update could address, but a detailed graph comparing the 980 Ti and Fury X will help illustrate the current problem better:

980ti vs FuryX Heaven

980ti vs FuryX Heaven 20s

We used two graphs because Heaven is a longer benchmark, and looking at the full 260-second chart doesn't clearly show what's happening. Zooming in on the first 20 seconds gives the proper view. Basically, while there are minor fluctuations in FPS on the GTX 980 Ti, on the Fury X (and other AMD GPUs show the same issue), there's a pattern that repeats every seven frames: one slow-to-render frame, one fast-to-render frame (most likely a runt frame), and then five frames at a mostly consistent rate. Right now, this is occurring throughout the 260-second test sequence. The result is a stuttering frame rate that's noticeable to the user, though thankfully Unigine is more of a tech demo/benchmark as opposed to an actual game.

Putting It All Together

"Hey, you in the back... WAKE UP!" Okay, sorry for the boring math diatribe, but sometimes it's important to understand what's going on and what it really means. This isn't intended as any form of manifesto on frame rates, and in fact this is a topic that has come up before. Nvidia even helped create some hardware and software to better report on what is happening on the end-user screen, called FCAT (Frame Capture Analysis Tool), but frankly it can be a pain to use. Our reason for talking about this is merely to shed some light, once more, on the importance of consistent frame rates.

We've been collecting data for a little while now, and since we were already running FRAPS for certain tests it makes sense to look at other places where it makes sense. We're planning to start using our 97 percentile "average minimum FPS" results for future GPU reviews, as it will help to eliminate some discrepancies in the reported frame rates from certain games (see above). It shouldn't radically alter our conclusions, but if there are driver issues (e.g., AMD clearly has something going wrong with Unigine Heaven 4.0 right now), looking at the reported minimum FPS along with the 97 percentile will raise a red flag.

If you read one of our GPU reviews in the near future and wonder why some of the minimum FPS results changed, this is why. It will also explain why some of our results won't fully line up with other results you might see reported-we're not willing to trust the reported minimums.

And if you really want to know how to calculate a similar number... fire up Excel, open your FRAPS frametimes CSV, and calculate the individual frame times in column C (e.g., C3 = B3-B2). Copy that formula down column C until the end of the collected data. Then the "average 97 percentile" is as follows, where "[C Data]" is the range of cells in column C containing individual frame times (e.g., "C3:C9289"):
=COUNTIF([C Data], ">"& PERCENTILE.INC([C Data],0.97))/SUMIF([C Data], ">"&PERCENTILE.INC([C Data],0.97))*1000

Newegg Daily Deals: Corsair CX600 600W PSU, Samsung 31.5-Inch Curved Monitor, and More!

Posted: 22 Jul 2015 01:08 PM PDT

Corsair CX600

Top Deal:

Whoa there playa, take that sucker out of your cart. Yes, we're talking about that 1500W power supply. Why? Well, are you planning to power your neighborhood? Oh, you're just preparing for Skylake -- we can dig it, but we can also see that single graphics card in your cart, and unless you're hiding a bazillion drives somewhere, you can save a lot by going with a more appropriate PSU selection. For example, check out today's top deal for a Corsair CX600 600W Power Supply for $57 with free shipping (normally $65 - use coupon code: [EMCAVNK32]; additional $20 mail-in-rebate). This is an 80 Plus Bronze certified model with 46A available on the +12V rail -- plenty for a modest configuration.

Other Deals:

Samsung S32E590C 31.5-inch Curved 4ms Widescreen LCD Monitor w/ Built-in Speakers for $400 with free shipping (normally $450 - use coupon code: [EMCAVNK25])

BenQ XR3501 Black 35-inch 144Hz 4ms 21:9 Ultra Wide Curve 2560X1080 Gaming Professional Monitor for $1000 with free shipping (normally $1060; USD $50 promotional gift card w/ purchase, limited offer)

WD Purple 4TB SATA 3.5-inch Internal Surveillance Hard Drive Bare Drive for $155 with free shipping (normally $165 - use coupon code: [ESCAVNK39])

Corsair RM750 750W ATX12V 80 Plus Gold Certified Full Modular Active PFC Power Supply for $110 with free shipping (normally $120 - use coupon code: [EMCAVNK45]; additional $30 Mail-in rebate)

Check Out Gigabyte's Overclocking-Friendly Z170X SOC Force Motherboard

Posted: 22 Jul 2015 12:35 PM PDT

Another board for Skylake-S

Gigabyte Z170X SOC Force

As we've already seen, Gigabyte is prepared for Intel's Skylake launch (presumably in just a couple of weeks) with a handful of motherboards already. In addition to the ones already shown, you can add the Z170X SOC Force to the growing list.

WCCFTech discovered a bunch of pictures of the board over at Thai-language website itfree4U. High end features abound in what appears to be a well thought out design on a board that's clearly aimed at overclockers, though you'll need a big case to run this bad boy—it's an Extended ATX (E-ATX) form factor motherboard.

As with all Z170X chipset boards for Skylake-S processors, Gigabyte's newest entry sports a new LGA 1151 socket designed for the next generation CPU family. Skylake will support both DDR3 and DDR4 memory, though enthusiast boards like the Z170X SOC Force will run with the latter—four DIMM slots in this instance, all located on one side of the CPU socket.

Looking around the socket reveals a 20 digital phase power design (16+4 CPU/PLL/memory). There are high end components throughout, including a Turbo BCLK chip for a bigger range of BCLK frequency turning (90-200MHz).

Gigabyte OC Touch

You can also see a familiar touch panel that's a holdover from previous SOC motherboards, along with a diagnostic LED, voltage check points, four PCI Express x16 slots, three PCi Express 3.0 x1 slots, eight SATA 6Gbps ports, three SATA Express ports, two USB 3.0 headers, two USB 2.0 headers, two USB 2.0 internal ports, and an I/O panel with four USB 3.0 ports, two USB 2.0 ports, OC button, PS/2 port, DVI/DisplayPort, HDMI, USB 3.0 Type-C, eSATA, GbE LAN, and audio ports.

All the overclocking goodies in the world won't do a lick of good if the architecture isn't flexible, though some early results are encouraging. Just a few days ago, an overclocker posted a CPU-Z image showing a Core i7-6700K running at nearly 5.2GHz at 1.35V using air cooling.

If final Skylake silicon overclocks anywhere near as well, it will be a lot of fun tinkering on motherboards like Gigabyte's Z170X SOC Force.

Follow Paul on Google+, Twitter, and Facebook

Mouse and Keyboard Support Headed to Xbox One, Will PC Streaming Come Next?

Posted: 22 Jul 2015 11:54 AM PDT

Putting the foundation in place

Xbox One Elite Controller

Exciting things are happening as we get closer to the launch of Windows 10. Just last week, Microsoft announced that all Xbox One owners could stream games from their Xbox One console to their Windows 10 PC, a feature that was first made available to Xbox One Preview participants a month ago. Now there's talk of adding keyboard and mouse support to the Xbox One, which could be in preparation of streaming games in the other direction.

News of the impending keyboard and mouse support came from Xbox boss Phil Spencer who was addressing a question posed on Twitter. A user asked if the new streaming functionality meant that mirroring PCs to an Xbox One with Windows 10 is a feature that's in the pipeline, and what Spencer had to say was encouraging.

Here's how the conversation went down on Twitter.

Tarkan Huseyin: Will we be able to mirror our PC to Xbox One with Windows 10?

Phil Spencer: Still finishing Xbox to Windows 10 streaming right now. I like the idea of Windows 10 > Xbox One streaming but don't have a plan yet.

Leandro G. Barbosa: For this [to work] Xbox [One] needs mouse support.

Phil Spencer: Yep, keyboard and mouse support for Xbox [One] would need to be there for this to work, those aren't far away.

This is encouraging because (A) Spencer seems to like the idea, (B) he used the world "yet," and (C) adding keyboard and mouse support gives the Xbox One the foundation it needs for streaming PC games on Windows 10 to the console.

Should this be put on the roadmap, it would be interesting to see what kind of feedback Microsoft would receive from system builders, as it could be a blow to the Steam Machine movement that's finally getting underway.

Of course, Microsoft might not care, as adding Windows 10 > Xbox One streaming could help boost console sales and perhaps take the top spot from Sony and its PlayStation 4 console (talking about sales, not which console is better).

Anyone have an thoughts on this?

Follow Paul on Google+, Twitter, and Facebook

Microsoft Loses Big Overall Last Quarter But Sees Huge Gain in Surface Sales

Posted: 22 Jul 2015 09:39 AM PDT

Surface rises up

Microsoft Sign

Microsoft posted a record $3.2 billion loss on revenues of $22.2 billion for the quarter ended June 30, 2015, though there's more to the story than that. Typically a company would be in big trouble if it was losing billions of dollars per quarter, but in this case, the results are indicative of a recent $7.5 billion write down Microsoft took related to its Nokia handset business.

Despite the massive loss, there are reasons for Microsoft to be optimistic. One of them is the continued rise in Surface sales. Surface revenue grew a whopping 117 percent to $888 million, which is thanks in large part to the Surface Pro 3 and Surface 3 launch. That's quite the turnaround for Microsoft's tablet efforts when you consider the $900 million charge it took on unsold Surface RT tablets just two short years ago.

Surface wasn't the only area posting gains. Microsoft's Xbox division saw revenue go up 27 percent on strong growth in consoles, Xbox Live transactions, and first party games, while search advertising revenue grew 21 percent with Bing grabbing a 20.3 percent share of the search market in the U.S.

The cloud was also kind to Microsoft. Nearly 3 million new Office 365 Consumer subscribers joined the fray, bringing the total number to 15.2 million, while commercial cloud revenue rose 88 percent.

"Our approach to investing in areas where we have differentiation and opportunity is paying off with Surface, Xbox, Bing, Office 365, Azure and Dynamics CRM Online all growing by at least double-digits," said Satya Nadella, chief executive officer at Microsoft. "And the upcoming release of Windows 10 will create new opportunities for Microsoft and our ecosystem."

These figures are the result of Nadella putting his stamp on Microsoft after inheriting a business with a shaky future from Steve Ballmer, who now owns the Los Angeles Clippers. In a recent letter to employees, Nadella warned that he would be making some "tough choices" to right the ship, which he followed through on by virtually disbanding the company's Nokia acquisition, resulting in 7,800 layoffs.

Follow Paul on Google+, Twitter, and Facebook

MMORPG Blog: Online Games MMO