I am attempting to run 3x Nvidia Grid K2 GPUs on a Supermicro C9X299-PGF motherboard. We are attempting to run these GPUs in pass-thru mode on ESXi 6.5 or 6.7 – I already have it running with one GPU. I am aware that the C9X299-PGF is not certified to run ESXi and this GPU configuration but I currently have one K2 GPU operational on ESXi (loading but haven’t verified fully functional)
The problem I have is sometimes when booting, the computer will POST up to DXE—BIOS PCI Bus Enumeration (94) then it will cycle (reboot). With one GPU, this may happen up to 3 times when it will finally boot completely. The POST cycling is random with sometimes booting immediately, sometimes 1 reboot, and up to 3 reboots. With 2 GPUs installed, the problem gets worse and with 3 GPUs, the system will never boot thru the POST, while cycling continuously. I have included a video showing the boot-up and the screen going black at which time it reboots. Be advised that I slowed down the video to see the POST messages.
The project mainly consists of you helping to adjust the BIOS configuration to make this work. You MUST HAVE KNOWLEGE of BIOS details and this hardware to take on this project and have completed something similar as I cannot take changes to brick the motherboard.
After much research, I found this to be a known issues and the following settings have been attempted:
Above 4G Decoding
MMIO High Granularity Size
BUT after a few attempts previously, the motherboard froze at Error 94 as stated above and would not reboot. I had to send the motherboard to you for repair so I am weary about messing too much with the BIOS as I don’t want to brick the motherboard again.
Our current configuration is as follows:
Motherboard: C9X299-PGF – updated to latest BIOS 2.0
CPU: Intel i9-9900X
Memory: Corsair Vengeance CMK64GX4m4D3000C16 – currently 64 Gig, eventually 128 Gig
GPU: Nvidia Grid K2 (attempts made with 1, 2 and 3 card configuration. A 3 card solution is required) GPUs have latest firmware and all rails powered (6 pin and 8 pin) are utilized.
Hard drive 1 (ESXi OS): Supermicro SATADOM 64 GB Internal Solid State Drive SSD-DM064-PHI
Hard drive 2 (Datashare): Samsung 970 EVO NVMe M.2 M2VLB1T0HALR (currently 1 in AHCI, eventually 2 in RAID)
NIC: Mellanox Cx354A ConnectX-3 InfiniBand
Please note that full payment will be made when the system completely boots allowing ESXi can pass-through the GPU to two VMs.
6 freelancers are bidding on average $166 for this job
Hi! You're never told about used power supply. Looks like you need at least 850W power supply for entire system. Let's discussin chat some technical details of your system. Thanks in advance Igor