I had an issue with one of my servers the other day: its power supply died unexpectedly during a scheduled restart. The poor thing never cam back up again.
Lucky for me, the data centre could simply swap out my hard disks and put them into another server. Although my data was save, the server wouldn’t connect to the network anymore – because it had a new MAC address. CentOS stores this value in two of its files, and when it changes (which is hardly ever the case), those files need to be updated.
Here’s how I fixed the problem. I did this on a CentOS 6 server, but it looks like the procedure is the same for CentOS 7.
Checking Interfaces and Connections
Before we begin, let’s see what the system looks like in its current (mis-)configured state. The following command will give us an overview:
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 8c:89:a5:1e:38:3d brd ff:ff:ff:ff:ff:ff
inet6 fe80::8e89:a5ff:fe1e:383d/64 scope link
valid_lft forever preferred_lft forever
3: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 8c:89:a5:1e:38:3e brd ff:ff:ff:ff:ff:ff
4: vlan3893@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
link/ether 8c:89:a5:1e:38:3d brd ff:ff:ff:ff:ff:ff
inet 10.31.53.8/24 brd 10.31.53.255 scope global vlan3893
inet6 fe80::8e89:a5ff:fe1e:383d/64 scope link
valid_lft forever preferred_lft forever
This tells us that eth0 appears to be unavailable, probably because it could not be brought up. We know why: because our MAC address has changed, and CentOS is trying to talk to hardware that doesn’t respond. Let’s fix that.
Detecting the current MAC address
There’s a system folder called /sys/class/net. In it we’ll find sub-folders corresponding to our connections (such as eth0, lo, wlan, etc). Each of these contains a file called address in which we can see our current MAC address. Those are updated on the fly and should always be up to date with whatever hardware we’re using.
The thing is though that a folder like eth0 will only exist if eth0 is indeed a valid connection. If eth0 is down, there’s no folder called eth0.
Hence, to have a look at all our current MAC addresses, we can read all address files like so:
cat /sys/class/net/*/address 8c:89:a5:1e:38:3d 8c:89:a5:1e:38:3e 00:00:00:00:00:00
The first one is the MAC address for our wired LAN connection, the second one is for the WLAN address and the third one for the local loopback.
Equipped with this knowledge, let’s tweak the system file that contains a record to the previous MAC address.
Updating CentOS with the current MAC address
The folks at the data centre were nice enough to tell me that all I had to do to connect my server to the network again was to delete a single file called
- /etc/udev/rules.d/XX-persistent-net.rules
This file is automatically rebuilt by CentOS and automatically updated with the current MAC address. XX stands for a priority number (70 in my case). In our case, the file looks like this:
# This file was automatically generated by the /lib/udev/write_net_rules
# program, run by the persistent-net-generator.rules rules file.
#
# You can modify it, as long as you keep each rule on a single
# line, and change only the value of the NAME= key.
# PCI device 0x8086:0x109a (e1000e)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:89:a5:1e:38:3d", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
# PCI device 0x8086:0x109a (e1000e)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="8c:89:a5:1e:38:3e", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
Notice that although the MAC addresses correspond to what we’ve discovered in the earlier step, all I seem to have here are eth1 and eth2 – no mention of eth0, because it’s currently down.
Deleting this file and restarting the server should have fixed my problem. But it didn’t. I also had to tweak my network scripts file.
Updating Network Scripts
There’s yet another configuration file in /etc/sysconfig/network-scripts. We’ll find several files here beginning with ifcfg-. Those are stored during the initial network setup and as it seems are not updated on the fly when a change in hardware occurs. Programme like nmtui update the settings in these files.
The one we’re interested in is ifcfg-eth0. Among many other things (depending on our hardware and setup), mine contained something like this:
IPV6INIT="yes" IPV6_AUTOCONF="no" NM_CONTROLLED="yes" ONBOOT="yes" HWADDR=8C:89:A5:1D:FA:DC
Note the last line beginning with HWADDR. This is the MAC address from the previous server. Let’s update this to match the current one. Now a server restart should do the trick of connecting us to the network again.
Summary
Here are the steps that brought my server back online:
- detect current MAC address with cat /sys/class/net/*/address
- remove /etc/udev/rules.d/70-persistent-net.rules
- update /etc/sysconfig/network-scripts/ifcfg-eth0 with the current MAC address
- restart the system
Further Reading
The following articles helped put me on the right track: