My Domain Controller Couldn't Find Itself for a Year

The short version

A routine pre-patch health check on my lab's only domain controller turned up something embarrassing: its AD-integrated DNS had never loaded, in the entire life of the domain.

The directory itself was healthy by every test I could run. The root cause was that the domain controller had lost its secure channel with itself. Its machine-account password no longer matched what Active Directory held for it, so the DNS Server service, running as SYSTEM, was being refused by its own directory, while every admin tool I ran worked fine.

The fix was fifteen minutes and one Microsoft KB (2751452): stop the Kerberos Key Distribution Center, reset the machine-account password with netdom resetpwd pointed at the DC's own IP address, and reboot. The zones loaded for the first time ever.

The environment

The lab is a 3-host vSAN ESA cluster running ESXi 9.1. Active Directory lives on a single domain controller, built early in the lab's life as a learning exercise. By design, AD is not the lab's DNS: two Pi-hole instances handle name resolution for the LAN, with hand-built static records for every lab host, plus conditional forwarding that sends queries for the AD domain to the domain controller itself.

That design is why the failure hid so well. At promotion time, the DC's DNS role was never fully finished; the thinking was "Pi-hole will be the DNS." Everything the lab uses resolves through the Pi-holes' static records, so day to day, nothing appeared wrong. Domain joins were flaky, once upon a time, and the joined machines were eventually unjoined because "it just caused issues."

Even the nightly backup of the DC (application-aware, using a domain credential) worked the whole time, because it connects by IP and static name and never needed AD's own DNS; nothing in the lab did.

How it surfaced

The incident started with the least dramatic sentence in systems administration: I ran a health check before patching.

dcdiag /q failed one test: Connectivity. The DC's own GUID record in _msdcs could not be resolved to an IP address. Resolving the domain name from the DC itself returned public internet addresses. Probing the DC's DNS service directly returned NXDOMAIN: the service was alive, answering, and empty. DNS Manager showed a red-badged server that "has not been configured," with zero zones loaded.

Everything that checked out

What followed was several hours of methodically proving that everything was fine, one subsystem at a time.

  • Services: DNS, NTDS, Netlogon, DFSR, ADWS. All running, all healthy.
  • The directory: LDAP binds worked locally and over the NIC. All naming contexts present. isSynchronized and isGlobalCatalogReady both true. A full dcdiag /v failed only Connectivity; Advertising, Netlogons, FSMO checks, all passed.
  • The zone data: The registry said four AD-integrated zones were configured. The zone objects existed in the directory, with two dozen child records in the main zone. Both application directory partitions existed and were hosted on the DC. Cross-reference records pointed at exactly the right places; replica-location attributes matched the DC's current NTDS Settings object letter for letter.
  • Permissions: The zone ACLs were pristine. My admin token carried Domain, Enterprise, and Schema Admins, elevated, and yet Get-DnsServerZone returned access denied, zone creation returned access denied, and dnscmd eventually stopped answering altogether.
  • Replacement: Restarting services didn't help. A reboot didn't help. Uninstalling and reinstalling the DNS role didn't help. Wiping the registry's zone table didn't help; the service re-derives its zones from AD, and still failed with zero local configuration. Deleting the zone objects themselves (after LDIFDE exports, because you export before you excise) didn't help.

Two real faults did turn up along the way, found by eye rather than by tool: the DC's subnet mask was wrong for the LAN's actual prefix length, and its network profile was set to Public. Both were worth fixing. Neither was the cause.

The error that finally said something

Event 4000 ("the DNS server was unable to open Active Directory") had been logging on a roughly ten-minute cycle for a year, carrying sub-error 9005. The headline said "can't open AD." The useful information was in Event 4007's XML payload: an invalid directory-partition registry state for the _msdcs zone's forest partition.

That's what led to KB 2751452, whose symptom list reads like a transcript of my afternoon: DNS zones don't load, events 4000 and 4007, the DNS console reports access denied, dnscmd is refused, and the service itself runs. Its stated cause #1: the DC/DNS server has lost its secure channel with itself. Cause #2, a lone DC pointing at itself for DNS, didn't fit: mine pointed at the Pi-holes.

The match was symptom-level. My 4000 sub-error wasn't a credential-class code, and the KB doesn't name one, so the secure channel stayed a prime suspect until the cure worked.

The secure-channel theory had come up hours earlier, and I'd set it aside as untestable. Test-ComputerSecureChannel finds its domain controller through the DC locator, which runs on DNS, the service that was down.

The fix

On the DC, elevated:

Stop-Service KDC -Force
netdom resetpwd /server:<DC's own IP> /userd:<domain>\Administrator /passwordd:*
Restart-Computer

Three details matter. First, the KDC has to be stopped, or the DC keeps trying to authenticate with the credential you're about to replace. Second, the server is specified by its own IP address, which is KB 2751452's instruction for a single-DC domain and also sidesteps the broken name resolution. Third, the password prompt rejected several attempts before accepting one, plausibly the broken channel refusing valid authentication until one attempt got through.

After the reboot, DNS Manager opened with no red badge and a complete console tree, for the first time all day. I recreated the zones as AD-integrated (they'd been deleted during the experiments), re-registered the DC's records, and restarted Netlogon so it could register itself.

Verification, in the present tense this time: dcdiag /q no longer fails Connectivity. gpupdate /force succeeds for both Computer and User policy. The original design also works end to end now: a LAN client's query for the domain travels through Pi-hole's conditional forwarding and lands on a zone that loads, and the LDAP SRV lookup for the domain answers for the first time in the domain's life.

The subplot: the public domain was mine

Early in the incident, the DC resolving its own domain to public IPs looked like a possible hijack or leak. The addresses belonged to my registrar. That was confusing for about a minute, until I remembered that I own the public version of this domain. I bought it during the lab build, through my registrar, never built anything on it, and it has worn the default "under construction" page ever since.

Owning the public name of your internal AD domain is what keeps that answer from belonging to a stranger; the textbook pattern goes one step further and puts AD on a subdomain of it. The "stranger" was me, and keeping that registration renewed is what keeps the joke a joke.

It has a job now, too: it hosts this site, a professional home for write-ups like this one that doesn't depend on any employer. Custom-domain email comes later, as a phase of a broader migration off rented platforms.

What I'm taking from it

  1. Check before you patch. A maintenance errand nobody needed to health-check first is the only reason a year-old silent failure was found on a quiet afternoon, before any crisis.
  2. A system nothing depends on can be broken forever without telling anyone. The lab routed around this failure so completely that a dead core service presented as "domain joins are flaky."
  3. When services fail and admins succeed, suspect the machine account. Everything I ran interactively worked; the DNS Server service, running as SYSTEM, was refused. That split is what a broken secure channel looks like.
  4. The test tool may need the broken thing. If your diagnostic locates its target through the failing service, an inconclusive result tells you about the tool's dependency and nothing about your theory.
  5. Read the event's data. The actionable detail was in an XML payload and a sub-error code, hours into an event that only said "unable to open AD."
  6. Dead ends are evidence. Every subsystem I cleared (partitions, cross-references, ACLs, the role reinstall) narrowed the suspect list until the only untested principal left was the machine credential itself.
  7. Own your namespace. See the subplot.

Sources

Updated Nov 17, 2026: narrowed several claims to what their sources support, added the Sources section, and edited the prose for clarity.


Daniel Barnes

Technical Support Engineer, VMware Cloud Foundation Division at Broadcom. Customer-facing work on VMware vSAN, ESXi, and VMware Cloud Foundation: storage, HCI, and virtualization troubleshooting. Views are my own and not necessarily those of Broadcom. I use AI tools for drafting and editing; the technical claims are mine, checked against public documentation and my own lab.

Started in 2018.

Homelab is a 3-host vSAN ESA cluster on ESXi 9.1 running self-hosted services (Nextcloud, Pi-hole DNS). That's where the write-ups come from.