Recently I’ve been exploring different flavors of linux, some exceptionally small, some that just run entirely in memory, some that come only with the core packages. By far my favorite is puppy linux, with tiny-core in second.
On that journey it prompted me to explore some of my favorite technical documentation about systems engineering and observability by Brendan Gregg. This article is about the tools I installed after revisiting BPF and Systems Performance books.
When your system is experiencing problems, the last thing you want to do is try to install tools on something that is already malfunctioning. Emergency changes can introduce unplanned risk to your system.
Large companies (FANG) have operating system teams and platform engineers, that ensure systems have packages, but for the everyone else this might not be the case.
For container environments it likely makes sense to create a privileged debugging container that has full access to the system and be deployed when needed.
It’s important to understand that there is a quadrant of tools and analysis features, it is also possible that tools can fit in more than one quadrant.
| Package | Provides | Notes |
|---|---|---|
| procps | ps(1), vmstat(8), uptime(1), top(1) | - Identify runaway processes consuming CPU or memory. - Check system load averages. - Monitor context switches and run queue pressure. - Observe swap activity and memory shortages. |
| util-linux | dmesg(1), lsblk(1), lscpu(1) | - Review kernel warnings and hardware failures. - Verify storage configuration. - Inspect CPU topology and capabilities. |
| iproute2 | ip(8), ss(8), nstat(8), tc(8) | - Investigate network connectivity. - Analyze socket usage. - Identify connection storms. - Examine packet drops and TCP statistics. |
| numactl | numastat(8) | NUMA (Non-Uniform Memory Architecture) scheduling or memory placement policy. In addition it can set persistent policy for shared memory segments or files. |
| tcpdump | tcpdump(8) | - Verify network traffic is reaching a service. - Capture protocol exchanges. - Investigate re-transmissions and connection failures. |
| linux-tools-common linux-tools-$(uname -r) |
perf(1), turbostat(8) | Hardware performance counters and low-level kernel profiling capabilities. |
| bpfcc-tools (bcc) | opensnoop(8), execsnoop(8), runqlat(8), softirqs(8), hardirqs(8), ext4slower(8), ext4dist(8), biotop(8), biosnoop(8), biolatency(8), tcptop(8), tcplife(8), trace(8), argdist(8), funccount(8), profile(8), etc. |
Visibility into kernel and application behavior without requiring custom kernel modules |
| bpftrace | bpftrace, basic versions of opensnoop(8), execsnoop(8), runqlat(8), biosnoop(8), etc. |
Create custom observability scripts |
| trace-cmd | trace-cmd(1) | kernel-level tracing |
| nicstat | nicstat(1) | Provides network-device-level statistics that are easier to interpret than raw interface counters. |
| ethtool | ethtool(8) | understanding physical NIC behavior. |
| tiptop | tiptop(1) | PMU/PMC top dynamic real-time view of the tasks running in the system |
| cpuid | cpuid(1) | - Which instruction sets are available? - What microarchitecture is this system using? |
| msr-tools | rdmsr(8), wrmsr(8) | CPU digging Low-level access to CPU Model-Specific Registers between the bios and the kernel. |
These tools also require specific kernel configuration.
Don’t make changes in your prod environment without consulting your ADR and org. 