1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
|
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Preventing out-of-memory (OOM) errors on Linux</title>
<style>
@font-face {
font-family: 'Quattrocento';
font-weight: 400;
src: url(/fonts/quattrocento-latin.woff2) format('woff2');
font-display: swap;
}
@font-face {
font-family: 'Open Sans';
font-weight: 300;
src: url(/fonts/open-sans-latin.woff2) format('woff2');
font-display: swap;
}
body {
font-family: 'Quattrocento', serif;
font-size: 20px;
line-height: 1.5em;
margin: 0 auto;
max-width: 60ch;
padding: 1em 0.5em;
background: #fff;
color: #111;
}
@media (min-width: 40em) { body { padding: 1em 2em; } }
@media (prefers-color-scheme: dark) {
body { background: #1a1a1a; color: #ddd; }
a { color: #adf; }
a:visited { color: #c9a0ff; }
hr { border-top-color: #444; }
}
h1, h2 { font-family: 'Open Sans', sans-serif; }
h1 { font-weight: 300; }
h2 { font-weight: 400; margin-top: 2em; }
.meta { font-size: 0.85em; color: #666; margin-top: -0.5em; }
@media (prefers-color-scheme: dark) { .meta { color: #999; } }
hr { border: none; border-top: 1px solid #ccc; margin: 2em 0; }
pre {
background: #f5f5f5;
border: 1px solid #ddd;
border-radius: 3px;
padding: 0.75em 1em;
overflow-x: auto;
font-size: 0.8em;
line-height: 1.4em;
}
code { font-size: 0.85em; }
pre code { font-size: 1em; }
@media (prefers-color-scheme: dark) {
pre { background: #222; border-color: #444; }
}
</style>
</head>
<body>
<h1>Preventing out-of-memory (OOM) errors on Linux</h1>
<p class="meta">2020-01-25 · tags: linux</p>
<hr>
<p>I've been running out of memory more and more often lately. I don't use any swap
space because I am of the opinion that 16GB of memory should be sufficient for most
daily and professional tasks. Which is generally true, however sometimes I have a
runaway process filling my memory. Emacs is very good at doing this for example,
prone to filling your RAM when you open JSON files with very long lines.</p>
<p>In theory, the kernel OOM killer should come in and save the day, but the Linux OOM
killer is notorious for being extremely … conservative. It will try to free every
internal structure it can before even thinking about touching any userspace processes.
At that point, the desktop usually stopped responding minutes ago.</p>
<p>Luckily the kernel provides memory statistics for the whole system, as well as single
process, and the <a href="https://github.com/rfjakob/earlyoom"><code>earlyoom</code></a> tool uses those to
keep memory usage under a certain limit. It will start killing processes, "heaviest"
first, until the given upper memory limit is satisfied again.</p>
<p>On NixOS, I set:</p>
<pre><code>{
services.earlyoom = {
enable = true;
freeMemThreshold = 5; # <%5 free
};
}</code></pre>
<p>and after activation, this simple test shows whether the daemon is working:</p>
<pre><code>$ tail /dev/zero
fish: "tail /dev/zero" terminated by signal SIGTERM (Polite quit request)</code></pre>
<p><code>tail /dev/zero</code> searches for the last line of the file <code>/dev/zero</code>, and since it
cannot know that there is no next line and no end to the stream of <code>\0</code> this file
produces, it will fill the RAM as quickly as physically possible. Before it can fill
it completely, <code>earlyoom</code> recognizes that the limit was breached, singles out the
<code>tail</code> command as the process using the most amount of memory, and sends it a
<code>SIGTERM</code>.</p>
</body>
</html>
|