How to use systemd for system administration on linux

Basic Usage of systemctl

systemctl is the command that substitutes the old "/etc/init.d/foo start/stop", but also does a lot more, as you can learn from its man page.

Some basic use-cases are:

• systemctl - list all loaded units and their state (where unit is the term for a job/service)
• systemctl list-units - list all units
• systemctl start [NAME...] - start (activate) one or more units
• systemctl stop [NAME...] - stop (deactivate) one or more units
• systemctl disable [NAME...] - disable one or more unit files
• systemctl list-unit-files - show all installed unit files and their state
• systemctl --failed - show which units failed during boot
• systemctl --type=mount - filter for types; types could be: service, mount, device, socket, target
• systemctl enable debug-shell.service - start a root shell on TTY 9 for debugging

For more convinience in handling units, there is the package systemd-ui, which is started as user with the commandsystemadm.

Switching runlevels, reboot and shutdown are also handled by systemctl:

• systemctl isolate graphical.target - take you to what you know as init 5, where your X-server runs
• systemctl isolate multi-user.target - take you to what you know as init 3, TTY, no X
• systemctl reboot - shut down and reboot the system
• systemctl poweroff - shut down the system

All these commands, other than the ones for switching runlevels, can be executed as normal user.

Basic Usage of journalctl

systemd does not only boot machines faster than the old init system, it also starts logging much earlier, including messages from the kernel initialization phase, the initial RAM disk, the early boot logic, and the main system runtime. So the days where you needed to use a camera to provide the output of a kernel panic or otherwise stalled system for debugging are mostly over.

With systemd, logs are aggregated in the journal which resides in /var/log/. To be able to make full use of the journal, we first need to set it up, as Debian does not do that for you yet:

# addgroup --system systemd-journal
# mkdir -p /var/log/journal
# chown root:systemd-journal /var/log/journal
# gpasswd -a $user systemd-journal

That will set up the journal in a way where you can query it as normal user. Querying the journal with journalctl offers some advantages over the way syslog works:

• journalctl --all - show the full journal of the system and all its users
• journalctl -f - show a live view of the journal (equivalent to "tail -f /var/log/messages")
• journalctl -b - show the log since the last boot
• journalctl -k -b -1 - show all kernel logs from the boot before last (-b -1)
• journalctl -b -p err - shows the log of the last boot, limited to the priority "ERROR"
• journalctl --since=yesterday - since Linux people normally do not often reboot, this limits the size more than -b would
• journalctl -u cron.service --since='2014-07-06 07:00' --until='2014-07-06 08:23' - show the log for cron for a defined timeframe
• journalctl -p 2 --since=today - show the log for priority 2, which covers emerg, alert and crit; resembles syslogpriorities emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7)
• journalctl > yourlog.log - copy the binary journal as text into your current directory

Journal and syslog can work side-by-side. On the other hand, you can remove any syslog packages like rsyslog orsyslog-ng once you are satisfied with the way the journal works.

For very detailed output, append "systemd.log_level=debug" to the kernel boot-time parameter list, and then run:

# journalctl -alb

Log levels can also be edited in /etc/systemd/system.conf.

Analyzing the Boot Process with systemd

systemd allows you to effectively analyze and optimize your boot process:

• systemd-analyze - show how long the last boot took for kernel and userspace
• systemd-analyze blame - show details of how long each service took to start
• systemd-analyze critical-chain - print a tree of the time-critical chain of units
• systemd-analyze dot | dot -Tsvg > systemd.svg - put a vector graphic of your boot process (requires graphvizpackage)
• systemd-analyze plot > bootplot.svg - generate a graphical timechart of the boot process

systemd has pretty good documentation for such a young project under heavy developement. First of all, there is the0pointer series by Lennart Poettering. The series is highly technical and quite verbose, and holds a wealth of information. Another good source is the distro agnostic Freedesktop info page with the largest collection of links to systemd resources, distro specific pages, bugtrackers and documentation. A quick glance at:

# man systemd.index

will give you an overview of all systemd man pages. The command structure for systemd for various distributions is pretty much the same, differences are found mainly in the packaging.

kSar sar grapher – A Graphical interface for sysstat sar data

sysstat sar provides command line based monitoring data. Those who are new or migrating from Windows or MAC and used to the graphical output, it might get confusing and boring. Hence the development of kSar sar grapher. kSar sar grapher is a graphing tool that can graph for Linux, MAC and Solaris sar outputs. Using KSar you can output graphs to a pdf file. kSar sar grapher is developed byAlexandre Cherif and uses a BSD license for distribution.

kSar sar grapher is a java based application that graphs your sar data.

You can load data from three method :

1. local file
2. local command execution and
3. remote command execution via SSH

kSar sar grapher supports the sar output of the following OS:

1. Solaris 8, 9 and 10
2. Mac OS/X 10.4+
3. Linux (Systat Version >= 5.0.5)
4. AIX (4.3 & 5.3)
5. HPUX 11.00+

So let’s get onto business and install kSar sar grapher.

Step 1: How to download ksar using wget?

kSar sar grapheris not available in Linux repositories. You have the go to the Sourceforge project page and download it manually.

Link: http://sourceforge.net/projects/ksar/

Run the following command to download it to any folder. I’ve downloaded it to my /root directory. At the time of writing this guide, kSar v 5.0.6 was available.

root@kali:~# wget http://downloads.sourceforge.net/project/ksar/ksar/5.0.6/kSar-5.0.6.zip

Step 2: How to unpack kSar zip file?

There’s no installation for kSar sar grapher. You just unpack it and put in any directory. I prefer /opt folder (opt = optional packages).

Run the following command to unzip kSar-5.0.6.zip and move the extracted directory to /opt folder.

root@kali:~# unzip kSar-5.0.6.zip
root@kali:~#
root@kali:~# mv kSar-5.0.6 /opt/
root@kali:~#
root@kali:~# cd /opt/kSar-5.0.6/

Step 2.1: kSar dependencies and requirements

kSar sar grapher got only one dependency as far I can see. You just to install Java JDK on your machine. There are lot’s of guides on how to do that. Those who are Kali Linux used like me should use this guide to install Java JDK.

Minimum Requirements:

1. sysstat or SAR in the system
2. JRE Version 1.5 or +

Step 3: How to run kSar?

By default the run.sh file is not executable. You need to chmod +x it.

root@kali:/opt/kSar-5.0.6/# ls
root@kali:/opt/kSar-5.0.6/#
root@kali:/opt/kSar-5.0.6/# chmod +x run.sh
root@kali:/opt/kSar-5.0.6/#
root@kali:/opt/kSar-5.0.6/# ./run.sh

Once kSar is running, leave the terminal open, if you close the terminal kSar will close too.

Step 4: How to generate graphs using kSar?

As I’ve mentioned before, you can generate sar graphs using kSar from three different inputs:

1. Local text file
2. Remote command execution via SSH and
3. Local command execution.

Let’s go through them all.

Step 4.1 How to generate graphs using kSar from local text file?

Run the following command to generate a sar file that will be used in kSar sar grapher.

First, you need to grab sar command statistics. Type the following command to get stats, enter (type it on your server):

root@kali:~# LC_ALL=C sar -A > /root/sar.data.txt

I’ve saved this file in my /root directory.

In kSar, click on Data > Load from text file option.

Browse to /root folder and open sar.data.txt file.

Now double click on kSar icon (it changes when the file is loaded) and select Processes. Click on each options (memory, swap, page etc.). You should be able to drag your mouse to zoom in or out.

Step 4.2 How to generate graphs using kSar via Launch SSH command?

What SSH command does is it allows you to connect to any machine that is running a SSH server and view and generate SAR graphs remotely or locally is realtime. There’s two step to this option:

1. You connect to a machine via SSH
2. You run a SAR command which then collects data.

In kSar, click on Data > Launch SSH command.

On the next popup window, it should come up a SSH connection popup.

In my case I am just going to login to localhost (which is own machine). If you know IP address for a remote host (a remote host can be anything from a Linux machine, Webserver, Solaris machine) running SAR. In my case I just press  YES .

Next window asks me if I am to trust the authenticity key for the host. Because it’s my OWN machines, I am going to press  YES .

If you are not able to connect, via SSH command, that just means following following 3 things:

1. The remote server does not have SSH server enabled
2. You are not allowed
3. A Firewall is blocking connection

In case it’s your local machine, you MUST install and enable openssh-server on your local machine.

Next popup should ask you for your password. Type in the correct password and press  OK .

On the next popup window, it should come up with  SAR -A command by default. Press OK  now.

Give it a few minutes and you should see the graphs are loaded.

Step 4.3 How to generate graphs using kSar using local SSH command?

For Step 4.2 I’ve used root@localhost which logs on to my own machine. If you are really going to view your own machines graphs only, then Running Local command is the better and faster way. It allows you to view your own workstations SAR graphs in a nice and easy to understand graphical interface.

In kSar, click on Data > Run Local command.

On the next popup window, it should come up with  SAR -A command by default. Press OK  now.

Give  it a few minutes and you should see the graphs are loaded.

How to zoom in and out on kSar graphs?

If you drag your mouse on the graphs, you can zoom on that part of it. To unzoom, do the drag your mouse to the opposite direction. Another way is to right click and select zoom.

Command specific to kSar:

Running kSar GUI:

You can run the GUI with 4 ways:

1. GUI only : java -jar kSar-x.x.x.jar
2. GUI default data grabbing : java -jar kSar-x.x.x.jar -startup
3. GUI with the “one login for all” function : java -jar kSar-x.x.x.jar -wizard
4. GUI and collect : java -jar kSar-x.x.x.jar -input 'ssh:// or file:// or cmd://'
   • eg. : java -jar kSar-x.x.x.jar -input 'cmd:///usr/bin/sar -f /tmp/mysar.file'

Running kSar from command line

To run kSar sar grapher on the command line, you MUST specify input and output argument:

example: java -jar kSar-x.x.x.jar -input 'file:///var/log/sa/sarXX' -outputPDF today.pdf

kSar help

To show kSar sar grapher help menu/instructions

java -jar kSar-x.x.x.jar  -help

Using shortcut XML

You can make template of data collection via XML (see contrib/shortcut.xml). This shortcut can be open with (-startup or -wizard cf. upper)

Replace current shortcut

java -jar kSar-x.x.x.jar -replaceShortcut newxml_shortcut_file.xml

Add some shortcut

java -jar kSar-x.x.x.jar -addShortcut newxml_shortcut_file.xml

ADMIN commands:

Clear all the preferences on kSar sar grapher

java -jar kSar-x.x.x.jar -ClearAllPrefs

That’s it. Thanks for reading. Please share.