Part 2: Demystifying and Troubleshooting Name Resolution in Rocky Linux (TL'DR - It’s always DNS)

DNS Client and Resolvers

Alright, let's dive into the heart of DNS—clients and resolvers. Think of a DNS client as that friend who always knows the best places to eat. When you want to visit a website, your web browser asks a DNS client where to find it. The DNS client then queries a DNS resolver, which acts like a super smart assistant, knowing exactly whom to ask in the vast sea of DNS servers available on the internet. Together, they make sure that when you type ciq.com, google.com, or rockylinux.org, you’re seamlessly directed to the correct IP address.

Name Resolution

Now that we have a broad overview of what DNS clients and resolvers do, let's explore the different mechanisms for configuring them on our reference Linux distribution—Rocky Linux from CIQ (RLC). Specifically, we'll cover the name resolution configuration files and the userland DNS tools in the following sections.

Name Resolution configuration files

We'll start off with some key system-level configuration files that affect the behaviour of DNS clients on RLC. We refer to these configuration files as system-level because they are independent of the higher-level/userland DNS client tools.

The /etc/resolv.conf file

The /etc/resolv.conf file is like the little black book of your system—it tells your computer which DNS servers to use when it needs to resolve a domain name. Editing this file is a straightforward way to set your preferred DNS servers. Here's an excerpt:

nameserver 8.8.8.8
nameserver 1.1.1.1
search ciq.com

In the /etc/resolv.conf file, each nameserver entry defines a DNS server that the system should use for resolving domain names. The IP addresses 8.8.8.8 and 1.1.1.1 are the addresses of Google and Cloudflare's public DNS servers, respectively. Multiple nameservers can be listed for redundancy, and the system will query them in the order listed until it receives a valid response.

The final `search ciq.com` entry in the `/etc/resolv.conf` file allows you to specify a domain that will be appended to any hostname you try to resolve. For example, if you type `server1`, the system will automatically append `.ciq.com` to form `server1.ciq.com` before attempting to resolve it. This helps to save you the hassle of typing out lengthy domain names for resources that you need to access frequently.
So instead of typing and memorizing :

https://ghdghkjagggd4543222ggsjsjjjgdtgggahjjjjajjjkllshgdgdgdggd.ciq.com

you only need to type and memorize:

https://ghdghkjagggd4543222ggsjsjjjgdtgggahjjjjajjjkllshgdgdgdggd

Wow.... Thanks for nothing DNS!

In most Linux distributions like RLC, you can manually add your preferred DNS servers to this file.

CAUTION: Beware—the /etc/resolv.conf file can sometimes be overwritten by other network management applications - such as NetworkManager. If you want to make persistent changes, you'll need to be mindful and deliberate in your network configuration approach.

The /etc/nsswitch.conf file

The /etc/nsswitch.conf file controls the order in which name resolution methods are used. Think of it as a set of instructions that tells your system where to look first—should it check /etc/hosts before querying DNS servers?

Here’s a typical line from this file:

hosts: files dns

This line means that the system will first check the /etc/hosts file (files), and if it doesn’t find what it’s looking for, it will then query DNS servers (dns). Depending on your environment, tweaking this order can help optimize or troubleshoot your system’s name resolution functions.

The /etc/hosts File

The /etc/hosts file is your system’s personal IP address book. It allows you to manually map domain names to IP addresses, which can be especially useful for testing or when DNS isn’t available.

Here's a sample excerpt from an /etc/hosts file:

127.0.0.1   localhost

192.168.1.10   rlc-dev-server.local

75.2.60.5   ciq.com

192.168.1.100  google.com

In the /etc/hosts file above:

• 127.0.0.1 is the loopback address, commonly referred to as localhost. It allows the system to refer to itself.
• 192.168.1.10 is an internal IP address mapped to rlc-dev-server.local
• 75.2.60.5 is the IP address associated with ciq.com, allowing the system to resolve this domain without relying on an external DNS server.
• 192.168.1.100 is [WRONGLY] mapped to google.com;

This file is processed before any DNS query is sent out, making it a quick way to resolve local or custom domain names.

NOTE: Unless you have infinite resources and tools to replace the massive google.com behemoth, there probably aren't many good reasons to have the last entry shown in our /etc/hosts excerpt. The presence of that dummy entry in a system's /etc/hosts file guarantees that the real google.com services will never be reachable on the system!

You can use the getent hosts command to query the /etc/hosts file and other configured name services. For example:

getent hosts localhost

This command will return the IP address associated with localhost if it is defined in /etc/hosts or any other available name serviceName Resolution configuration tools

Name Resolution (DNS configuration tools)

This section goes over some user-land tools that can be used for configuring DNS related settings at the Network level.

NetworkManager

Network management on Linux distributions has historically been done via a hodgepodge of low-level subsystems, tools/applications and configuration files. Over time, the tools and specific network related configuration files have varied and many have come and gone, while others have stuck around and gone on to become the standard on popular distribution families. Sample popular network management tools include - Wicd, NetworkManager, Connman, Netplan, systemd-networkd. NetworkManager is the default network management tool on Rocky Linux, RLC, RHEL, CentOS stream systems.

NetworkManager (NM) is a system network service used for managing network devices and connections on modern Linux distributions. Amongst other things, NM can be used for ensuring that DNS and other network configurations are applied correctly and persistently. On Rocky Linux, you can use the nmcli utility to manage DNS settings.

For example, assuming you have an existing NetworkManager connection defined and named eth0, you can specify the DNS servers (8.8.8.8 and 1.1.1.1) to be associated with the particular connection by running the following nmcli commands:

nmcli con mod 'eth0' ipv4.dns "8.8.8.8 1.1.1.1"
nmcli con up 'eth0'

Using NetworkManager to handle DNS ensures that changes are persistent across reboots.

TIP: ip Vs nmcli

ip and nmcli are popular command-line tools used for configuring and managing network settings on Linux distributions. When it comes to DNS, nmcli (and NetworkManager) takes the spotlight, as ip can't handle DNS server configurations. The ip command is a lower-level networking tool, focusing on advanced network functions like route manipulation, address assignment and device management.

systemd-resolved

systemd-resolved is a modern systemd-based service that simplifies client-side DNS management and provides resolver services to local applications on Linux systems like RLC.

Instead of manually editing configuration files, systemd-resolved helps streamline DNS resolution, making it easy to integrate different network interfaces and manage caching.

TIP: systemd-resolved is not installed by default on RLC. You can quickly install it using the DNF package manager by running dnf -y install systemd-resolved.

Configuring systemd-resolved on RLC

The primary configuration file for systemd-resolved is /etc/systemd/resolved.conf.

To enable and configure systemd-resolved, use the following commands:

sudo systemctl enable systemd-resolved
sudo systemctl start systemd-resolved

Once enabled, you can link /etc/resolv.conf to use systemd-resolved:

sudo ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf

This setup allows systemd-resolved to manage your DNS settings seamlessly.

Viewing DNS Settings

We've explored the system level configuration files that influence the name resolution behavior on Linux systems and we also briefly touched on some popular network management tools used for specifying the DNS server(s) used by a typical RLC system.

With these important configurations being set in various files and by various tools, it is often desirable to be able to see the net result. Besides standard command line utilities (such as grep, cat, awk, sed, less, more and so on) that you readily have available for manually parsing configuration files, you also have purpose built utilities tools like resolvectl at your disposal. resolvectlis your go-to command for this sort of holistic introspection!

So, want to check which DNS server your system is currently using? Type:

resolvectl status

This will display detailed information about the DNS servers in use, making troubleshooting a breeze.

TIP: The resolvectl command is provided by the systemd-resolved package. If you don't have systemd-resolved installed, the resolvectl utility will be missing from your system!

Testing DNS Configurations

To make sure everything is set up correctly, you’ll need some trusty tools to test your DNS configuration:

• nslookup: A simple legacy command to query DNS servers.

  nslookup ciq.com
  

• dig: A powerful tool that provides more detailed output and supports many rich options.

  dig google.com
  

• host: Another handy command to resolve domain names.

  host rockylinux.org
  

• resolvectl: Provided you have systemd-resolved running, resolvectl can used for performing quick DNS lookups.

  resolvectl query cnn.com
  

These tools can help you verify that your DNS settings are working as expected and troubleshoot any issues.

Troubleshooting common name resolution issues

DNS issues are universal and can be incredibly frustrating. This likely explains why numerous memes, haikus, and personal accounts of DNS problems causing medium to large-scale outages are so popular online. For example, here's a well-known DNS haiku widely attributed to Kyle Simpson:

It’s not DNS. There is no way it’s DNS. It was DNS.

Common problems include incorrect DNS server settings, connectivity issues, or outdated cache. Here are some quick troubleshooting steps you can try in a logical sequence:

• Check `/etc/resolv.conf` file to ensure the right DNS servers are being used.

  grep nameserver /etc/resolv.conf
  

• Check NetworkManager settings to ensure the right DNS servers have been configured at the network configuration level. For example if you have an active Networkmanager connection defined and named "eth0", you can query for the value of the ipv4.dns setting for the connection by running:

  nmcli --fields ipv4.dns  con  show "eth0"
  

• Use the ping utility to check reachability of your configured DNS servers. For example if your system to is configured to query a DNS server on your Local area network (LAN) with the IP address 192.168.1.1, you can try pinging it via:

  ping 192.168.1.1
  

• If your local DNS server is unreachable or unavailable, you can test DNS name resolution by temporarily specifying an external (and trusted) DNS server that is almost always guaranteed to be available.

  Use the popular dig command and specify 8.8.8.8 as the temporary DNS server by running:

  dig @8.8.8.8 rockylinux.org
  

• Flush the DNS cache if you suspect outdated information is causing issues.

  sudo systemd-resolve --flush-caches
  

• And if any of these simple tests fail - then you know "It's DNS Blame Time". And the next drastic solution is to setup and manage your very own DNS server!

Fun Fact:

Why is 8.8.8.8 so popular? Google’s public DNS server at 8.8.8.8 is widely used because it’s fast, reliable, and easy to remember. Plus, it’s Google, so you know they’ve got some serious infrastructure behind it.

What's Next? Setting Up Your Own DNS Server

Now that we’ve covered DNS clients/resolvers and how to configure them on RLC, it’s time to AMP things up a bit. In Part 3, we’ll explore setting up and managing our own DNS servers. Stay tuned.