System-call budgets for Linux jobs

Stops one busy job from slowing every other job on a Linux machine: each job gets a budget for expensive system calls. Linux 5.5 and up, no kernel module.

Three uses for SyscallGuard

Some system calls are expensive: flushing files to disk, creating files, starting processes. A job that makes thousands of them slows down everything else on the same Linux machine. SyscallGuard gives each job a budget for those calls, so the busy job slows down instead.

Keeps a build fast next to a noisy logger

At priority 2 the logger gets the smaller share, so its costly calls wait while the build at priority 8 keeps going.

Holds a batch job to a fixed rate

A batch job gets a fixed 500 cost units a second instead of a share that changes with the other jobs.

Contains a script you do not trust

Over budget, its attempts to start new processes fail at once. sgctl kill stops its whole process group.

How the sgd supervisor works

Start sgd once, then start each job with sg-launch. Cheap calls run as usual; each expensive one is charged to its job, and sgd lets it run, makes it wait, or refuses it.

See what sgd decides

What sgd does with a costly call. It charges the job, then decides.

sgd allows, holds or rejects calls

Within budget the call runs. Short of it, the call waits up to 2 s, then fails with EAGAIN.

sgd lends budget between jobs

An idle job lends to a short one, keeping 30% in reserve. Half of each later refill repays the loan.

SyscallGuard policies are sgd options

Borrowing, paced rejections and synchronous wake-up are on by default; one flag on sgd turns each off.

sgd defaults
3 of 5 on
  • Borrowingon; -B turns it off
  • Paced rejectionsheld 10 ms; -R 0 replies at once
  • Synchronous wake-uprequested on Linux 6.6+; -W turns it off
  • Prometheus metricsoff; -m port serves 127.0.0.1
  • Event logoff; -l file writes JSON lines

sgctl kill stops a runaway job

sgctl kill signals the job's whole process group, so a fork storm ends in one command.

  • sgctl kill 2SIGKILL to the job’s whole process group
  • sgctl kill 2 15Or a chosen signal (15 = SIGTERM)
  • sg-launch -f rejectFork fails with EAGAIN at once when over budget

Syscall costs measured on your kernel

The seed costs only rank the call classes. sg_calibrate.py times every costly call with bpftrace and writes the median, in microseconds, to a file sgd loads.

sg_calibrate.py measures costs with eBPF

Run your own workload under bpftrace. The median latency of each costly call becomes its cost, 1 unit to the microsecond.

On WSL2, kernel 6.6, fsync came out at 1,504 units against a seed of 20. Two runs agreed within 3%.

Needs root, and the file belongs to the machine it was measured on.

Read sg_calibrate.py
confwsl2-6.6.confpysg_calibrate.pyWSL2
# name             cost  median_ns   samples  seed# name          cost  seed
read                  1        533     17121     1read             1     1
write                 3       2621     15154     1write            3     1
openat                4       3725     31832     3openat           4     3
mkdir                71      70762     15071     3mkdir           71     3
fsync              1504    1504367       954    20fsync         1504    20
fdatasync           167     167185       941    20fdatasync      167    20
vfork               181     180972       470    30vfork          181    30
clone               167     167184       470    30clone          167    30
execve              112     111550       470    40execve         112    40
clone_thread         45      45426       472    10clone_thread    45    10
fsync: 1,504 units, seed 20.

Questions

The limits are in the README too. It builds with gcc and make; the tools need Python 3.

LinuxPythonPrometheusGitHub
Read the design notes

Which kernels does it run on?

Linux 5.5 and up, 5.19 recommended, 6.6 for synchronous wake-up. x86-64 and aarch64.

Do I need root?

Not to enforce budgets. Only calibration needs root, for bpftrace.

Why does io_uring_setup fail?

It returns EPERM, because one io_uring_enter could submit many operations past per-call accounting.

What happens if sgd exits?

Costly calls of still-running jobs fail with ENOSYS. A filter cannot be removed from a running process.

Where should I try fork storms?

In a VM or a spare machine. Budgets slow a storm; sgctl kill ends it.