Intel has assumed a lot if it comes to XeonPhi’s. One was that you will use it on dual-Xeon servers or workstations and that you already have a professional supplier of motherboards and other computer-parts. We can only guess why they’re not supporting non-professional enthusiasts who got the cheap XeonPhi.

After browsing half the internet to find an overview of motherboards, I eventually emailed Gigabyte, Asus and ASrock for more information for a desktop-motherboard that supports the blue thing. With the information I got, I could populate the below list. Like usual we share our findings with you.

Quote that applies here: “The main reason business grade computer supplies can be sold at a higher price is that the customers don’t know what they’re buying“. When I heard, I did not know why the customer is not well-informed – now I do.

Some remarks

Costs

The below motherboards cost a lot more than the gamer-motherboards. It starts around €300 and goes up to €800 and even over. Most cost in the price range of €400 and €600. For every buck above, you get really special features and more quality (like controllers not depending on the CPU to do the real work).

Yes, we also got a cheap XeonPhi for testing and were surprised by the overhead-costs being much larger than the actual try-out device itself. Especially because the Knights landing is simply not a really fast processor and not as easy to port software to as Intel promised (recently they say making it run is easy and after that you just need to optimize for performance), with a few exceptions.

Above 4GB decoding

This is the feature that’s needed and where you pay the €250 extra for. It means that 64 bits are used for direct addressing of the accelerator’s memory. Yes, that means extra overhead.

LGA 2011 vs LGA 2011-3

The “-3” means support for DDR4 from both the motherboard and the processor. The sockets sound as they’re almost the same, but they’re not.

If you want to quickly upgrade your computer to support, go for a LGA 2011 motherboard. Do check the number of lanes your CPU supports (see below).

PCIe lanes

All is mentioned at PCIe3 speeds, which is double the speed of PCIe2. With PCIe3 x16 you get theoretically 31 GB/s max due to overhead. For most GPUs and accelerators PCIe3 x8 (= PCIe2 x16) is enough as the internals of the device cannot handle more.

You can have 7x PCIe with a monstrous x16x16x16x16x16x16x16 support, but still the actual active lanes depends on the processor. Most high-end CPUs have 40 lanes, so you can have 2 accelerators at x16, plus an SSD at x4. All is mentioned at PCIe3 speeds, which is double the speed of PCIe2

Want more lanes? There are (more expensive) motherboards that have a special PCIe controller to supports even more lanes, which acts as a PCIe-switch. Most vendors are are pretty clear if a motherboards can support 3 or 4 GPUs at x16 – if it’s not mentioned (or a vague “3 way SLI/Crossfire support”, which can be 3×8), then simply assume it hasn’t got such controller. You can also get a dual-CPU, to double the number of PCIe-lanes. How and where the bottle-necks are in each of these cases, depends on the software/algorithm (type of communication with the CPU), how much lanes the controller has with the CPU.

PCIe

One thing they never really tell you in the specs, but only by image is that under the last PCIe slot there are many connectors which effectively makes use of the last one useless for double thick devices like the XeonPhi. Some motherboards have so many cables or two x16 slots next together, which is simply bad design when you know almost all high-end GPUs need two slots. So carefully check the images, before you buy.

If you want to learn more about PCIe, including the 64 bit addressing and lanes, then read Understanding Performance of PCI Express Systems by Jason Lawley from Xilinx.

Gigabyte

Below are three motherboards Gigabyte suggested.

• MW50:  1x LGA 2011-3,  4x PCIe (x16x16x8x4)
• MW70:  2x LGA 2011-3,  5x PCIe (x16x16x16x8x4), 3 way x16
• MU70:  1x LGA 2011-3,  4x PCIe (x16x16x8x2)

Only have one CPU to spare and chose a dual CPU board? Then “some PCIe, memory, network or I/O functions might be unavailable” – probably because there are less PCIe lanes. At least it works, when you want to do a relaxed upgrade.

Big plus for Gigabyte is the space they have for a double GPU for the last slot.

ASRock

In short: all “WS” motherboards. You will find on forums that the cheaper “X99 WS” won’t work and needs a special bios, but I asked and it seems to work out-of-the-box (untested). The WS-motherboards are on the whole price-range between €300 and €800, having all kinds of different options.

If you want 88 lanes with a single CPU, try the X99 WS-E10G with a 44 lane PCIe-switch. If you want a cheaper option with extra lanes, keep searching for other motherboards having the “PLX PEX 8747” controller or alike.

ASRock themselves suggest you use the X99 WS as a started motherboard, and then see what is needed for when you upgrade.

ASUS

ASUS seems to have the most info, with special “AVL lists” for each motherboard. So using Google, I could create the below list. As ASUS has very complete information, it’s easy to double check before you buy.

Motherboards with LGA 2011-3:

• Z10PA-D8: 2 x LGA 2011-3,  7x PCIe (x16x16)
• Z10PE-D16 WS:  2x LGA 2011-3,  x PCIe (x16x16), 4 way x16
• Z10PE-D8 WS:  x LGA 2011-3,  x PCIe (x16x16), 4 way x16

Motherboards with LGA 2011:

• Z9PE-D8 WS:  2x LGA 2011-3,  7x PCIe (x16x8x16x8x16x8x16)
• Z9PE-D16-10G:  2x LGA 2011-3,  5x PCIe (x2x16x8x16x16)
• P9X79 WS:  x LGA 2011-3, 7x PCIe (x16x8x16x8x16x8x16)
• P9X79-E WS:  x LGA 2011-3, 7x PCIe (x16x8x16x8x16x8x16), 4 way x16

Motherboards including case and power supply:

• RS92x-E7, 4x LGA 2011, rest unknown.
• ESC1000 G2, 1x LGA 2011, 6x PCIe (x16x16x16x16x16), 1350W
• ESC2000 G2, 2x LGA 2011, 7x PCIe (x16x16x16x16x16x8), 1350W
• ESC4000 G2, 2x LGA 2011, 8x PCIe (x16x8x16x8x16x8x16x8?), 4 way x16, 1620W, server model

More choice?

It seems you can hack your bios to show the option, as the support is sometimes already there. Hacking is always fun on motherboards that have support for a backup-bios and for people with time on their hands.

Also there are several more brands with good offerings. From the above three vendors I just got their business cards at ISC’15 and thus contact was easy. I’m happy to include others – just contact us or write in the comments. Also if you found more supported motherboards of the above three brands, the contact-page and the comments-section are for you.