In this article, we will take a look at how service management is used in Linux, first by defining it, understanding its behavior and configuration, and some useful commands.
Note: Im running a CentOS9 Stream in this article, It came out in December 2021, I will not be covering UNIX or Windows, only Linux
We will be focusing on the newer system systemd - we will not talk about its ancestor (sysV) or competitors (upstart..)
Definition
All processes running on the system are child processes of the systemd init process. It is an init system used to boostrap you from the kernel to user space.
systemd is rather new, as It was developped in 2010 and stable as of 2015. It is rapidly becoming the most used init process, but has some concerns on its inner security.
It will be considered as a system and service manager. For those that haven’t noticed, It is a play on words refering to its quick adapting ability, but for real, the d in systemd refers to daemon, and If you need to refer to it, call it System daemon, not System D.
It is hard to define systemd as a good or bad news for Linux. While It is praised by developers and users alike for its reliable parallelism during boot and centralized management of processes, daemons, services and mount points, It strayed away from the UNIX philosophy by its design (e.g. mission creep/bloat feeling).
We will focus on the service management part on systemd and the init process will be covered in another article.
What does systemd manage ?
First, It does NOT manage anything in /etc/init.d/. You should never add configuration there, as It will have no effect. There is backwards compatibility with SysV init scripts, but they will not be searched in this deprecated directory.
You can explore /etc/systemd/ instead, It gives an idea of the inner architecture
You can analyze each file full configuration with systemd-analyze cat-config [PATH_TO_FILE]
At its core, systemd manages units, and there is 11 available, highlighted in bold are the most important:
| Units | Description |
|---|---|
| Service | Start and control daemons and processes they consist of, most used type |
| Socket | Encapsulate local IPC or network sockets in the system, useful for socket-based activation |
| Target | Useful to group other units through a chain of dependencies |
| Device | Expose kernel devices and may be used to implement device-based activation |
| Mount | Control mount points in the file system |
| Automount | For on-demand mounting of file systems (hot-plug..) |
| Timer | Useful for triggering activation of other units based on timers |
| Swap | Similar to mount units, encapsulate memory swap components of the operating system |
| Path | Used to activate other services when file system objects change |
| Slice | Group and manage processes and resources (cgroups) |
| Scope | Similar to service units, but for foreign processes management (e.g. init) |
Units are named after their configuration files, which can include positive or negative requirement dependencies as well as ordering.
One example of such unit is the cron daemon, used for scheduling your batch.
To put it into perspective, my CentOS uses 370 unit files, with 141 loaded units
How are units made ?
Each units is configured through a plain-text files, with a .ini type syntax.
systemd stores them in three location, depending on their usage
| Path | Type of unit |
|---|---|
| /usr/lib/systemd/system | systemd default unit distributed by RPM packages |
| /run/systemd/system | Systemd unit files created at run time. This directory takes precedence over the default |
| /etc/systemd/system | Systemd unit files created by systemctl enable as well as unit files added for extending a service. This directory takes precedence over the run-time one |
An example of crond.service unit configuration, part of the default configuration:
You can notice its scheduling dependencies on After= and WantedBy ; It is also able to restart itself on a specific timer.
Note: It’s common case to have those service units configuration created in the three locations, and there is a link on higher priority paths towards the default one. Be careful when you change it, as anything written in the default unit path (in /usr/lib/) will be overwritten at each OS update.
Well known units
If you are interested in systemd service architecture, try out systemctl list-dependencies
Among all the running units (and not only services), there is a few interesting ones that you might already know:
| Unit | Type | Used for |
|---|---|---|
| proc-sys-fs-binfmt_misc | automount | all block/character devices |
| run-user-xxxx | mount | storing files used by running processes for that user |
| sys-kernel-debug | mount | Kernel debug file system |
| sys-kernel-tracing | mount | Kernel Trace file system |
| init | scope | System and service manager |
| auditd | service | Security auditing service |
| crond | service | Command scheduler |
| firewalld | service | Dynamic firewall daemon |
| NetworkManager | service | Handles network configuration |
| sshd | service | OpenSSH server daemon |
| systemd-journald | service | Journal service |
| systemd-logind | service | user login management |
| user@xxxx | service | User manager for UID xxxx |
| user | slice | User and session slice |
| dbus | socket | D-Bus System Message Bus Socket |
| network | target | Network component |
There is many more, and you can check a unit’s critical chain with systemd-analyze critical-chain [unit]
If you need to have better boot time, you can check what take the longest.
Possible state of units
Let’s preface it by saying systemd does not communicate with services that have not been started by it. All is managed through a process PID, and used to query and manage the service. If you did not start your daemon through systemd, It will be impossible to query its state through systemd commands.
There is two different types of state available on an unit:
- Loaded : If the unit file has been found, and It is enabled
- Active : If the unit is running or stopped
Since It is required to load an unit before running it, you need to consider both types to get a unit’s status.
Unit loading
A unit has to be loaded if you need to run it. This state depends entirely on the unit configuration, and It can be one of the following:
- loaded
- not-found (e.g. not found in the three possibles paths)
- bad-setting
- error (e.g. set a masked service as enabled for example)
- masked
An example of bad-setting, where we missed the initial / in the path
After correcting the issue:
A note on mask : It is a stronger version of disable ; It links the unit file to /dev/null, making it impossible to start them. It prohibits all kinds of activation, including enablement or manual activation. It can be used to prevent accidentally using a service that conflicts with a running one.
Once It is loaded, It will look in its configuration for its enablement state ([Install] section of the unit file). It hooks the unit into its various suggested places (e.g. the unit is automatically started on boot or when a particular kind of hardware is plugged in).
Unit activation
Your unit can be either started or stopped, but It can actually be more refined than that. A unit possesses two levels of activation state:
- High-level (often called active), and can take the following state:
- active
- inactive
- failed
- reloading
- activating
- deactivating
- Low-level called sub, whose values depend on unit-type specific detailed state of the unit You can check what substates is available per unit type with
systemctl --state=help
Create our own service unit
Let’s create our own service unit, which will simply repeat a ping/pong in a file. It will help us understand its state, and how to use a custom file as a service. We could imagine implementing a socket, device, or any other types of units.
Two things to note before we start:
- System services are unable to read from the standard input stream, and when started, It connects its standard input to /dev/null to prevent any interaction.
- System services do not inherit any context (e.g. environment variables like HOME or PATH) from the invoking user and their session. It runs in a clean execution context. You can check this out by typing
envfor your local environment variables andsystemctl show-environmentfor systemd environment variables in your shell.
Now, let’s create our service unit, in /etc/systemd/system
We will call it pingpong.service:
We write a simple script, which repeatedly write ping/pong into a named pipe
Let’s start it ! We check its status beforehand
We can see It created a pipe successfully
The script is not perfect, as It will be blocked if someone else access this pipe, as read is blocking. The script location is also not perfect, as It depends on a non-sudo user, and represents a security risk, It would be better to review its permission and put it in /usr/local/bin
Let’s stop it, and check It deleted our named pipe
All good ! It’s also possible to create a unit template, if you want to create a skeleton of your units.
Useful commands
To reload and apply unit changes:
1
2
systemctl daemon-reload
systemctl restart [UNIT_NAME]
To have an overview of overridden/modified unit files, use systemd-delta
To view groupings of processes system-wise, use systemd-cgls
Credits
https://en.wikipedia.org/wiki/Systemd
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/chap-managing_services_with_systemd
https://www.computernetworkingnotes.com/linux-tutorials/
https://www.freedesktop.org/software/systemd/man/systemctl.html