Troubleshooting the DHCP Server#

SynergyCP deploys a PXE DHCP server in each IP Group. It only answers DHCP requests from servers that have an OS reload in progress, and it also acts as the DNS server for those servers during the install. This article covers how to check that it is running, inspect what it knows about, and watch it answer (or not answer) a booting server.

Please be advised that every statement of “DHCP server” refers to the DHCP server provided by SynergyCP for the purposes of automated OS reloads. It does not refer to any other DHCP server present on your network.

Where things live#

Everything below is run on the DHCP server itself over SSH (this is usually your master SynergyCP server, unless you installed it on a separate VM).

  • Configuration: /scp/pxe/dhcp/docker-compose.prod.yml
  • Docker Swarm service name: scp-bm-pxe-dhcp_api
  • HTTP API: port 3000 (the HTTP_PORT value in the compose file)
  • DNS: port 53, DHCP: ports 67/68

The API key is in the compose file. Grab it into a shell variable so you can paste the commands below as-is:

KEY=$(grep -E '^\s+API_KEY:' /scp/pxe/dhcp/docker-compose.prod.yml | awk '{print $2}')
IP=$(grep -E '^\s+IP_ADDR:' /scp/pxe/dhcp/docker-compose.prod.yml | awk '{print $2}')
echo "$IP $KEY"

The API key printed here must match the one saved under OS Reloads > Deploy DHCP Servers for this DHCP server in SynergyCP. If they differ, SynergyCP cannot push leases to this server and every OS reload in the IP Group will sit at 20% waiting for DHCP. Copy the key from the compose file into SynergyCP (not the other way around).

Is it running?#

Check that the Docker service has one running replica:

docker service ls | grep pxe-dhcp
docker service ps scp-bm-pxe-dhcp_api

If REPLICAS shows 0/1, look at the task error and the logs (see below). The most common causes are another process already bound to port 53 or 67 on the host (for example systemd-resolved, dnsmasq, or another dhcpd), or a wrong INTERFACE in the compose file after a NIC rename.

Then ask the API for its health. The version field tells you which release is deployed:

curl -s "http://$IP:3000/health?key=$KEY"

{"code": 200, "data": {"status": "healthy", "version": "2.0.0"}, "error": false, "messages": []}

A response of {"code": 401, ... "Invalid API key"} means the KEY you used does not match the compose file. No response at all means the container is not running, or a firewall is blocking port 3000.

What leases does it have?#

The DHCP server only holds a lease for each server that currently has an OS reload in progress. SynergyCP creates the lease when the install starts and deletes it when the install finishes. Listing the leases tells you whether SynergyCP successfully told this DHCP server about the machine you are trying to install:

curl -s "http://$IP:3000/leases?key=$KEY"

Pipe through python3 -m json.tool (or jq) for readable output:

curl -s "http://$IP:3000/leases?key=$KEY" | python3 -m json.tool

{
    "code": 200,
    "data": {
        "ac-1f-6b-12-34-56": {
            "mac": "ac-1f-6b-12-34-56",
            "ip": "203.0.113.50",
            "gateway": "203.0.113.1",
            "subnet_mask": "255.255.255.0",
            "file_server": "203.0.113.10",
            "filename": "ipxe.efi",
            "update_url": "https://api.example.com/api/server/123/install/456?key=...",
            "notified": false,
            "answer_token": "..."
        }
    },
    "error": false,
    "messages": []
}

What to check in the output:

  • The MAC is present. MACs are lowercase with dashes (ac-1f-6b-12-34-56). If the server you are installing is missing, SynergyCP never reached this DHCP server. Confirm the IP Group’s DHCP server points at this machine, the API key matches, and that port 3000 is reachable from the SynergyCP application server. Check System > Logs in SynergyCP for the failed request.
  • The MAC matches the NIC that is actually PXE booting. Watch the DHCP request in the IPMI console and compare. A common mistake is that SynergyCP has the MAC of the wrong port on a multi-port card.
  • ip, gateway and subnet_mask match the IP Entity assigned to the server.
  • file_server is the file server’s IP that the booting server can reach. This is what is handed out as the TFTP next-server. If it is a private IP the server cannot route to, set NEXT_SERVER in the compose file (see below).
  • notified flips to true after the server’s first successful DHCP ACK, at which point SynergyCP is told the server has booted and the install moves past 20%. If it stays false, the server never completed a DHCP exchange with this DHCP server (see “Watching DHCP traffic”).
  • An empty data: {} with an install stuck at 20% almost always means the API key mismatch described above.

Leases are held in memory only and are not written to disk. Restarting the container (including upgrading with the install script) empties the lease list and forgets the API hostname used for api.synergy. Any OS reload that was in progress at that moment will sit at 20% until it is restarted in SynergyCP, which recreates the lease. This is expected and is not a fault. Seeing an empty list right after a restart or upgrade does not indicate a problem.

Other useful read-only endpoints:

# IPs that are allowed to use this server for DNS (one per active lease)
curl -s "http://$IP:3000/dns/acl?key=$KEY"

Reading the logs#

The DHCP server logs to both the container and the host syslog, so either of these work:

docker service logs scp-bm-pxe-dhcp_api -f --tail 200
journalctl -t pxe-server -f

A healthy boot looks like this in the log. The giaddr is the relay agent’s IP when the server is on a different subnet, or 0.0.0.0 when it is on the same L2 segment as the DHCP server:

Received DHCP packet from ac:1f:6b:12:34:56, giaddr=203.0.113.1
DHCPDISCOVER from ac:1f:6b:12:34:56
Client architecture: UEFI (type 7), bootfile: ipxe.efi
Building OFFER for ac:1f:6b:12:34:56: IP=203.0.113.50, TFTP=203.0.113.10, file=ipxe.efi, giaddr=203.0.113.1, class=PXEClient
Sent DHCP OFFER to ac:1f:6b:12:34:56 for IP 203.0.113.50 (via relay 203.0.113.1)
DHCPREQUEST from ac:1f:6b:12:34:56
Sent DHCP ACK to ac:1f:6b:12:34:56 for IP 203.0.113.50 (via relay 203.0.113.1)
First ACK for ac:1f:6b:12:34:56, notifying update URL
Successfully notified update URL: https://api.example.com/...

After iPXE loads it does a second DHCP round. This time you should see iPXE client detected, serving URL: http://203.0.113.10:4002/pxelinux.cfg/01-ac-1f-6b-12-34-56.

Common log lines and what they mean:

  • Nothing at all when the server boots. The DHCP request never arrived. See “Watching DHCP traffic”.
  • Received DHCP packet from ... but no OFFER (with LOG_LEVEL: DEBUG you will also see No lease found for MAC). The DHCP server saw the request but has no lease for that MAC. Compare the MAC in the log with the leases list.
  • NACK - Invalid request: MAC=..., IP=... The server requested an IP that does not match its lease, usually a stale IP left over from a previous install or a different DHCP server on the network. It normally recovers on its own by re-DISCOVERing.
  • Failed to notify update URL ... The DHCP exchange worked but the callback to the SynergyCP API failed. The DHCP server needs outbound HTTPS to your API domain. If Cloudflare or a WAF sits in front of the API, it must allow this request.
  • DNS query from non-allowed IP A machine that does not currently have a lease asked this server for DNS. Harmless unless it is the server you are installing, in which case it was given a different IP than its lease (another DHCP server on the network).
  • Failed to resolve <hostname> for CNAME flattening The DHCP server cannot resolve your SynergyCP API domain using its upstream DNS servers (UPSTREAM_DNS). Installs will fail with Could not resolve host: api.synergy.

To get more detail, set LOG_LEVEL: DEBUG in /scp/pxe/dhcp/docker-compose.prod.yml and redeploy (see “Changing configuration”). Remember to set it back to INFO afterwards.

Watching DHCP traffic#

If the log shows nothing when the server boots, confirm whether DHCP packets are reaching the host at all:

apt-get install -y dhcpdump
dhcpdump -i eth0        # use the INTERFACE from the compose file

Or with tcpdump:

tcpdump -i eth0 -n -vvv port 67 or port 68

Boot the server from its IPMI console and watch for the BOOTPREQUEST:

  • No packets arrive. The network is not delivering the server’s DHCP broadcast to this host. Check the DHCP relay (ip helper-address or equivalent) on the server’s VLAN gateway points at this DHCP server’s IP, and that nothing between them filters UDP 67. Check the MTU on this host’s interface is 1500: ip addr | grep mtu | grep -v veth | grep -v docker.
  • Packets arrive but GIADDR is 0.0.0.0 on a relayed request. The relay is not populating giaddr, so the DHCP server cannot send the reply back through it. Fix the relay configuration.
  • CHADDR (client MAC) is not the MAC in SynergyCP. Fix the MAC on the server in SynergyCP or PXE boot from the right NIC.
  • You see a BOOTPREPLY from a different server IP. Another DHCP server on the network is answering. It must be removed from that VLAN or configured to ignore these servers, otherwise the booting machine may take its offer instead.
  • The server’s request is HTTPClient rather than PXEClient in the vendor class. The machine is in UEFI HTTP Boot mode. This DHCP server deliberately does not answer HTTP Boot with an HTTP Boot offer, so the firmware will time out and fall back to PXE on its own. If it never falls back, disable HTTP Boot / enable PXE in the BIOS boot options.

The DHCP server sends replies to relays on port 67, so if the request is seen but the reply never reaches the server, capture on the relay side too.

Is DNS working?#

During an install the server uses this DHCP server for DNS, and api.synergy must resolve to your SynergyCP API domain. Test from the DHCP server itself. Its own IP is not in the DNS ACL, so temporarily add it:

curl -s -X POST "http://$IP:3000/dns/acl?key=$KEY" -H 'Content-Type: application/json' -d "{\"ip\": \"$IP\"}"
dig @$IP api.synergy +short
dig @$IP google.com +short
curl -s -X DELETE "http://$IP:3000/dns/acl/$IP?key=$KEY"

api.synergy should return the IP of your API domain. If it returns nothing, the DHCP server does not know your API hostname yet. It learns it from the first lease created, or from the settings sync. Trigger a sync by editing the DHCP server under OS Reloads > Deploy DHCP Servers, changing the Hostname, saving, changing it back and saving again. If google.com also returns nothing, the DHCP server cannot reach its upstream DNS servers. See Could not resolve host: api.synergy for the client-side symptoms.

Changing configuration#

Configuration is entirely in /scp/pxe/dhcp/docker-compose.prod.yml. Edit the value under environment: and redeploy the stack for it to take effect:

cd /scp/pxe/dhcp
docker stack deploy --compose-file=docker-compose.prod.yml scp-bm-pxe-dhcp

Settings you are most likely to need:

  • INTERFACE and IP_ADDR if the host’s NIC name or IP changed. Symptom: the service restarts in a loop, or DHCP requests are seen by tcpdump but never logged.
  • NEXT_SERVER when the file server IP that SynergyCP hands out is not routable from the booting servers (for example the DHCP server is on a private subnet behind NAT). This overrides the TFTP address in every lease. Symptom: TFTP open timeout on the server’s console.
  • UPSTREAM_DNS if the defaults (8.8.8.8,8.8.4.4) are blocked from this host.
  • LOG_LEVEL: DEBUG while troubleshooting.

Re-running the install script also preserves these values and upgrades the image in place:

curl -sL https://install.synergycp.com/bm/pxe-dhcp-server.sh | bash -s latest

Still stuck?#

Collect the following and send it to SynergyCP support:

  • Output of /health and /leases (you can blank out update_url and answer_token)
  • docker service logs scp-bm-pxe-dhcp_api --tail 500 from around the time of the failed boot
  • A dhcpdump or tcpdump capture of the failed boot
  • The MAC address and IP Entity of the server in SynergyCP, and a screenshot of where the IPMI console stops
Backward Troubleshooting Port Forwarding Gateways Containerized Consoles Forward