Server Burn-In and Acceptance Testing Before Production

Why test a new server at all

Hardware vendors test what they ship. That does not tell you what happened in transit, in the rack, or with the exact mix of memory, drives, cards and firmware that landed in your order. A loose DIMM, a marginal power supply, a drive with a bad region or a GPU that errors under sustained load can all look healthy at the login prompt. The cheapest moment to find them is before production data lives on the machine, while you can still swap a part, return a unit or move a delivery date.

This is the testing step of a larger build. Our article on building a colocation footprint from an empty cage covers power, cooling and cabling around it. Here we stay inside the server: what to run, what each tool is good at, and what to write down.

Burn-in and acceptance: two different jobs

Burn-in is the stress. You run the machine hard, for a planned period, so that weak parts fail while it is still in the staging area. Acceptance is the decision. Someone with authority compares the results to criteria that were written before the first test started, and signs that the machine passed or did not.

Keeping them separate prevents a common failure. A run that produces no crash can still hide corrected memory errors, a drive that logged retries, or a GPU that reported errors the operating system shrugged off. Acceptance means you looked at those logs and compared them to a line you drew in advance.

Step 1: set the baseline before you stress anything

Before any test, record what the machine is. Serial numbers, part numbers, memory population, drive models, network cards, GPU models, and the versions of the system firmware, the management controller, the RAID or HBA firmware and each add-in card. Put the machine on the firmware baseline you intend to run in production, so that you test the configuration you will ship and not one you will change next week.

The DMTF publishes Redfish, a management standard for servers and other hardware. Its site lists the Redfish Specification (DSP0266, version 1.25.0 on the page we read), the Redfish Data Model Specification and a Redfish Schema Index. In practice, a Redfish-capable management controller lets a script pull inventory over the network in a structured form, the same way on hardware from different makers. We use that to capture a before-and-after inventory and attach it to the build record. Exact resource names and what each vendor implements vary, so check the schema index and your vendor's Redfish guide before you write the script.

Also read the management controller's own event log before you start and clear nothing. A part that was already throwing errors on delivery is evidence, and you want it in the record.

Step 2: memory with MemTest86

Memory faults are the classic cause of strange, unrepeatable crashes, so test memory first and test it outside the operating system. MemTest86 is a memory-testing program that boots from a USB flash drive. Its product page lists DDR2 through DDR5 and CAMM2 memory and ECC among the supported types, describes 14 test algorithms including SIMD, row hammer and DMA tests, and says it can decode a failed DIMM or chip from the error address. HTML reports are available, and customizable reports are a Pro edition feature.

Two cautions. First, the page does not describe ECC-specific behavior or error injection, so do not assume either; read the current MemTest86 documentation for your edition and confirm how it reports corrected and uncorrected errors on your platform. Second, memory that passes a standalone test can still show corrected errors later in the platform's own error log. Check the management controller's event log after the test, not only the MemTest86 screen.

Decide the pass rule up front. A common and defensible rule is that any uncorrectable error fails the DIMM, and that any corrected-error pattern the vendor's guidance calls out as a replacement trigger also fails it. Take the threshold from your server vendor and record it in the plan.

Step 3: CPU and system stress with stress-ng

Once memory is clean, boot the real operating system and load it. stress-ng is an open-source tool designed to exercise the physical subsystems of a computer as well as operating-system kernel interfaces. Its README describes more than 390 stress tests, including CPU, virtual memory, file system and cache tests, and a thermal-load capability.

The options we use most are documented in that README. The --cpu option starts CPU stressors, --vm starts virtual-memory stressors, and --vm-bytes sets how much memory they use. The --verify option turns on checking of results, so a stressor that computes a wrong answer is reported and not ignored. The --timeout option sets the run length. The --metrics option prints per-stressor results, and --tz reports thermal-zone temperatures.

The README is direct about two limits. stress-ng can report throughput, but it is not meant to be a precise benchmark. And some tests can make poorly designed hardware run hot and cause heavy system thrashing. Run it where cooling is already in service, with someone able to stop the test, and do not read its numbers as a performance guarantee.

Step 4: storage with fio

Drives are where early failures cost data, so test them before data arrives. fio, the Flexible I/O tester, simulates a configurable I/O workload described in a job file, using threads or processes to issue the I/O you specify. Its documentation covers the options you need for a burn-in profile: rw sets the pattern such as sequential or random reads and writes, bs sets the block size, direct requests non-buffered I/O, iodepth sets the queue depth for asynchronous engines, and numjobs runs identical clones in parallel. The runtime and time_based options keep a job looping for a set period, and ramp_time delays logging until the workload has settled.

For acceptance records, fio can write results in several formats, including JSON, through its output-format option, so each drive's result can be filed with the build. Run a short sequential and random profile on every drive, then a longer mixed profile on the full set at once, because a shared backplane or controller can fail only when all drives are busy.

fio can write to raw devices and destroy what is on them. Its documentation describes safeguards, such as a readonly mode and the allow_mounted_write setting, which defaults to false so that jobs writing to a mounted device or partition abort. Do not rely on them. Run destructive profiles only on drives you have confirmed are empty and unassigned, name the devices explicitly in the job file, and have a second person check the device list before the run.

After the storage run, read each drive's own health data and the controller log. Reallocated sectors, media errors or repeated command retries during the test are failures even when fio finished without complaint.

Step 5: GPUs with NVIDIA DCGM Diagnostics

GPU servers need a GPU-specific test. NVIDIA's Data Center GPU Manager (DCGM) includes diagnostics, run with the dcgmi diag command. The documentation we read says the NVIDIA Validation Suite is now called DCGM Diagnostics. It defines four run levels, set with -r: level 1 is a quick system validation, level 2 an extended validation, level 3 a longer hardware diagnostic and level 4 a longer-running extended diagnostic. The page gives expected durations for 4‑GPU and 8‑GPU systems, and says higher levels include the ones beneath them.

The diagnostics are made of plugins. The documentation describes a software deployment check, a diagnostic plugin that runs large matrix multiplications and checks memory read-back, a PCIe test for bandwidth, latency and replays, a GPU memory test that writes and reads patterns, targeted power and targeted stress plugins that push the GPUs toward a power or throughput target, a memory bandwidth test, and at level 4 a memory test and a pulse test that creates current spikes to check whether the power supply copes.

On errors, the same page is useful. It says the diagnostic plugin watches for XIDs, temperature violations and uncorrectable memory errors, and several other plugins fail on XIDs as well. An XID is an error report from the NVIDIA driver; NVIDIA keeps a separate XID Errors document that, per its landing page, explains what the messages are and how to analyze them with a catalog. The DCGM page has no dedicated ECC plugin. It says the GPU memory plugin detects memory mismatches and ECC errors, and it lets you set limits on single-bit error counts in the targeted stress and memory bandwidth tests. Error records carry a severity and category, and the page says the ISOLATE and RESET severities should be handled immediately.

Results come out as human-readable text, or as JSON with the -j option, with per-test pass or fail and information lines. A failed run that completes returns exit code 226. Capture the JSON, the exit code and the DCGM log file path with your build record. Then check the kernel log for XID messages, because a clean diagnostic summary with XIDs in the log is not a clean machine.

Defaults and durations differ by version and by GPU model. Read the DCGM release you actually install, and take the run level and test duration for your acceptance plan from NVIDIA's documentation for that release and from your server vendor.

Step 6: run it all together, then watch it

After subsystem tests pass, run a combined soak: CPU and memory stress, storage load and, where fitted, GPU load, at the same time, for a planned period. This is the test that shows power and cooling problems, because that is when the machine draws the most. Watch the power draw at the rack, the inlet and component temperatures, the fan speeds and the management controller event log through the whole run. Then reboot the machine several times and confirm it comes back with the same inventory, the same firmware versions and no new errors. Power-cycle behavior is part of acceptance, not an afterthought.

Step 7: write the acceptance criteria and keep the record

Acceptance criteria are short and specific. They say which tests ran, for how long, on which firmware baseline, and what counts as a pass. They name the failure conditions in plain terms: any uncorrectable memory error, any storage media error, any GPU XID, any thermal throttle during soak, any change in inventory after reboot, any unexplained controller event. They name who signs.

• The serial numbers and the inventory captured before and after testing.
• The firmware baseline and any version that was changed during the build.
• The tool versions and the exact command lines or job files used.
• Raw outputs: MemTest86 report, stress-ng metrics, fio JSON, DCGM JSON and exit code, and the controller event log from before and after.
• Environment readings during the soak, such as temperatures and power draw.
• Any part replaced, with the failed part's serial number and the vendor ticket.
• A pass or fail decision, the date, and the name of the person who made it.

Keep the record with the asset, not in someone's inbox. When a machine fails in year two, the first question is whether it was already marginal when it was built, and the record answers it.

Where teams go wrong

• Testing a different configuration from the one they ship, then upgrading firmware afterward.
• Reading only the tool's summary and not the controller log or kernel log.
• Running destructive disk tests on a machine that already holds data.
• Letting a pass count stand in for acceptance, with no written criteria.

How Clayton Cybernetics helps

Our datacenter builds team stages, burns in and documents servers before they enter a rack, and writes the acceptance plan with you so the criteria are agreed first. We tailor the test set to your hardware vendor's guidance and your workload, keep the raw results with the asset record, and hand over a written pass or fail. If you are planning a build or a refresh, ask us to review your current acceptance process.

Sources

• NVIDIA DCGM documentation: DCGM Diagnostics (dcgmi diag)
• NVIDIA: XID Errors documentation
• MemTest86 product page (PassMark)
• stress-ng README (Colin Ian King, GitHub)
• fio documentation (fio.readthedocs.io)
• DMTF Redfish standard

This article is general information, not vendor, warranty, electrical or safety advice. It describes tool documentation as read on November 8, 2026; options, defaults and supported hardware change by version. We give no pass thresholds, durations or temperature limits because they depend on your hardware vendor and workload. The DMTF Redfish documentation pages were partly unavailable to automated fetching and the NVIDIA XID page we reached was a landing page only; MemTest86's page did not describe ECC injection. Destructive storage tests erase data: run them only on empty devices you are authorized to wipe. Follow your hardware vendor's documentation. Last reviewed November 8, 2026.

Keep reading

Next · InfrastructureStructured Cabling That Survives the Next RefreshServers and switches are replaced every few years. Cabling stays in the walls, trays and racks much longer, and it is the part nobody wants to touch during an outage. Cabling that survives the next refresh is labeled so a stranger can read it, documented so the records match the racks, laid so fiber is not bent past its limit, and patched with rules that people actually follow. This article covers labeling and administration under TIA-606 at a high level, MPO trunks and the three polarity methods, bend radius and service loops, patching discipline, and the documentation that keeps all of it true. It is based on pages we read on November 8, 2026, and gives no distances because those depend on standards we could not read and on your cable's datasheet.9 min read

• SPF, DKIM and DMARC: Why Email Authentication Matters, and How a Resilient Exchange or Microsoft 365 Platform Is Run9 min
• Backups That Actually Restore: 3-2-1, Immutability, Restore Tests and Hurricane Season for Coastal Firms9 min