Adaptive Connection Pool with Dual EMA Signals
A common pattern in networked applications is maintaining a pool of reusable connections — database handles, HTTP clients, IMAP sessions. The challenge: how many connections should the pool hold? Too few and callers wait. Too many and you waste server resources and connection limits.
This article describes an adaptive pool implementation that automatically sizes itself between a configured minimum and maximum, driven by two independent signals measured passively during normal operation.
The Problem with Fixed-Size Pools
A fixed pool of N connections is always wrong for at least part of the time. During a traffic burst you need more; during quiet periods you're holding open connections that could be closed. Dynamic sizing is the right answer, but most approaches get the signal wrong.
What not to measure
Queue length (how many callers are waiting) is a lagging indicator. You only know there's a problem after callers are already blocked. And it only captures contention at the semaphore — if you have 10 idle connections and 10 simultaneous requests, all 10 get served instantly with zero queue length, yet the pool might still be the right size.
Utilisation ratio (outstanding / size) has the opposite problem: it fires on every burst even when nobody is waiting, causing grow/shrink oscillation. A burst that uses all 10 connections but serves them all in 1ms looks the same as one where callers wait 500ms.
Two Independent Signals
The solution is to separate the two concerns:
- Should the pool grow? — measure contention: how long did callers wait?
- Should the pool shrink? — measure over-provisioning: how idle is the pool?
These are independent questions with independent answers.
The pressure signal (grow)
pressure = EMA of min(log(1 + waitMs) / log(1 + targetMs), 1.0)
waitTime is the actual wall-clock time each caller spent blocked waiting for
a slot. targetWait is your acceptable latency budget per connection acquisition
(e.g. 20ms if your page load target is 100ms).
The signal uses a logarithmic scale rather than a linear one. Connection wait times span many orders of magnitude: nanoseconds when a slot is immediately available, milliseconds under moderate load, seconds under heavy load. On a linear scale, the difference between 0ms and 1ms is lost in the noise when the target is 20ms (sample = 0.05). On a log scale that same difference registers clearly.
With targetWait = 20ms (log1p(20) ≈ 3.045):
| Wait time | linear sample | log sample |
|---|---|---|
| 0ns (instant) | 0.000 | 0.000 |
| 1ms | 0.050 | 0.228 |
| 5ms | 0.250 | 0.589 |
| 10ms | 0.500 | 0.787 |
| 20ms (target) | 1.000 | 1.000 |
| 50ms+ | → 1.0 (capped) | → 1.0 (capped) |
A 1ms wait produces a log sample of 0.228 vs linear's 0.050 — four times more sensitive to short waits that would otherwise be invisible.
Grow by 1 when pressure > 0.3 — callers are consistently experiencing
meaningful wait times relative to the target latency.
The key insight: this is sampled after unblocking from the semaphore, so it directly measures the thing you care about. No proxies.
The idleness signal (shrink)
idleness = EMA of (idleConnections / size)
Sampled in release, after the connection is returned to the pool. This
measures what fraction of the pool is currently sitting idle.
- All connections in use → sample = 0 → idleness decays
- Half the pool idle → sample = 0.5 → idleness rises
- All connections idle → sample = 1.0 → idleness rises fast
Shrink by 1 when idleness > 0.5 — more than half the pool is
consistently idle after releases.
Why this combination avoids oscillation
Consider a burst of 10 simultaneous requests against a pool of 10 connections:
-
All 10 requests arrive, all 10 connections are claimed instantly.
pressure ≈ 0(no waiting),idleness ≈ 0(all in use). No grow, no shrink. -
Requests complete, connections are released one by one. After each release,
len(conns)grows: 1/10, 2/10, 3/10... Idleness EMA starts rising. -
After the burst, all 10 connections are idle.
idleness → 1.0. Shrink fires — pool reduces toward the appropriate steady-state size. -
Next burst arrives. If waiting occurs, pressure rises → grow fires.
The pool tracks actual usage patterns rather than instantaneous snapshots.
Implementation
Data structure
type pool struct {
min, max int
mu sync.Mutex
size int // current target: min ≤ size ≤ max
outstanding int // connections held by callers
draining int // slots scheduled for permanent removal
pressure float64 // EMA grow signal
idleness float64 // EMA shrink signal
// Semaphore: RECEIVE to claim a slot, SEND to return one.
// Buffered at max so size can grow up to max without reallocation.
sem chan struct{}
conns chan *Connection
}
The semaphore channel uses an important convention: receive = claim a slot,
send = return a slot. This is the opposite of the more common "send to
acquire" pattern. The reason: we want <-sem to block when no slots are
available (pool is full), which means the channel must be empty when full.
Pre-filling with initSize items makes initSize slots available immediately.
Initialisation
func newPool(min, max int) *pool {
initSize := min(min*3, max) // start warmed up but not at max
p := &pool{
min: min,
max: max,
size: initSize,
sem: make(chan struct{}, max), // buffered at max, never reallocated
conns: make(chan *Connection, max),
}
for i := 0; i < initSize; i++ {
p.sem <- struct{}{} // pre-fill initSize slots
}
return p
}
Starting at min * 3 (capped at max) gives the pool a warm start — enough
capacity for a typical burst without opening connections upfront.
acquire
func (p *pool) acquire() (*Connection, error) {
start := time.Now()
<-p.sem // block until a slot is available
waited := time.Since(start)
p.mu.Lock()
p.outstanding++
// Update pressure EMA using log-scaled wait time.
// log1p(x) = log(1+x), avoids log(0) and spreads out short waits.
waitMs := waited.Seconds() * 1000
targetMs := targetWait.Seconds() * 1000
sample := min(math.Log1p(waitMs)/math.Log1p(targetMs), 1.0)
p.pressure = alpha*sample + (1-alpha)*p.pressure
// Grow if callers are waiting too long.
if p.pressure > growThreshold && p.size < p.max {
p.size++
p.sem <- struct{}{} // mint a new slot
}
p.mu.Unlock()
// Reuse idle connection or open new one.
select {
case c := <-p.conns:
return c, nil
default:
return openNewConnection()
}
}
The pressure sample is taken after unblocking. If we waited, the sample is high. If we got a slot instantly, it's near zero. This correctly distinguishes "pool is big enough" from "pool is too small".
Growing is done by sending one extra item to sem — this is safe because sem
is buffered at max, and the mutex ensures size ≤ max is maintained.
release
func (p *pool) release(c *Connection) {
go func() { // fire-and-forget
// Health check before touching pool state.
if err := c.Ping(); err != nil {
c.Close()
// Return the slot — pool size is unchanged, a fresh connection
// will be opened on next acquire.
p.mu.Lock(); p.outstanding--; p.mu.Unlock()
p.sem <- struct{}{}
return
}
p.mu.Lock()
p.outstanding--
shouldDrain := p.draining > 0
if shouldDrain { p.draining-- }
p.mu.Unlock()
if shouldDrain {
c.Close()
return // slot NOT returned — pool permanently shrinks by 1
}
// Return connection.
p.conns <- c
// Sample idleness and check for shrink.
p.mu.Lock()
idleSample := float64(len(p.conns)) / float64(max(p.size, 1))
p.idleness = alpha*idleSample + (1-alpha)*p.idleness
if p.idleness > shrinkThreshold && p.size > p.min && p.draining == 0 {
p.size--
p.draining++
}
p.mu.Unlock()
p.sem <- struct{}{} // return the slot
}()
}
release is fire-and-forget — the caller returns immediately. A goroutine
performs the health check and idleness sampling in the background.
The shrink mechanism: instead of immediately removing a slot, we increment
draining. The next release goroutine that finds draining > 0 discards its
connection and does not return the slot to sem. This permanently reduces the
pool's capacity by 1. The draining counter ensures we only shrink by 1 per
decision, even if multiple releases happen before the drain completes.
The health check: sending a NOOP (or Ping) verifies the connection is
still alive. If it fails, the connection is discarded but the slot is returned
to sem — the pool size is unchanged, and the next acquire will open a fresh
connection. This is important for resilience: if the laptop sleeps overnight and
all connections go stale, they are discarded one by one on use, each returning
their slot, so the pool remains fully functional.
Observed behaviour
Here is a real log trace from the implementation serving an email web client, showing two load bursts separated by a quiet period:
19:23:19 grow size=11 pressure=0.32 waited=94ms
19:23:19 grow size=12 pressure=0.45 waited=93ms ← cascade: each waiter
19:23:19 grow size=13 pressure=0.36 waited=1µs ← unblocks and samples
19:23:19 acquire new (size=13 outstanding=13 waited=752ns)
← pressure now < 0.3, stops
19:23:27 shrink size=12 idleness=0.54
19:23:29 shrink size=11 idleness=0.61 ← quiet period: slow drain
19:24:22 shrink size=10 idleness=0.60
... (continues shrinking every ~30-60s) ...
19:25:07 shrink size=5 idleness=0.53
19:25:08 grow size=6 pressure=0.36 waited=343ms ← new burst hits small pool
19:25:08 grow size=7 pressure=0.49 waited=299ms
19:25:16 grow size=8 pressure=0.39 waited=932ns ← further grow from next request
19:25:16 grow size=9 pressure=0.31 waited=1µs
19:25:23 shrink size=8 idleness=0.51 ← settles, starts shrinking again
The cascade grow
When multiple requests are waiting simultaneously, each one that unblocks contributes an EMA sample:
- First waiter waited 94ms → high pressure sample → grow (size 10→11)
- Second waiter waited 93ms (slightly less, new slot was available) → still high → grow (10→11→12)
- Third waiter now gets a slot instantly (1µs) → low sample → pressure drops below threshold → grow stops
This is the self-limiting property: each grow immediately unblocks one waiter, whose lower wait time brings the pressure back down. The pool grows by exactly as many steps as needed to drain the queue, then stops.
The slow shrink
After the burst, the 8 remaining idle connections produce idleness ≈ 0.6
on each release. The EMA smoothing means it takes ~5-10 releases before the
threshold is reliably exceeded, so shrinks fire roughly every 30-60 seconds
during quiet periods — gradual, not immediate. This is intentional: a pool
that shrinks aggressively after every quiet moment would oscillate.
Grow/shrink symmetry
The two signals operate on different timescales by design. Pressure spikes fast (a single 94ms wait pushes the EMA well above threshold) so growth is quick. Idleness accumulates slowly (each release only nudges the EMA by alpha=0.2) so shrinkage is gradual. The asymmetry means the pool is always ready to serve a burst but doesn't hold excess connections forever.
Tuning
| Parameter | Description | Typical value |
|---|---|---|
alpha |
EMA smoothing factor | 0.2 (≈ last 5 events) |
targetWait |
Acceptable wait per acquire | 20ms |
growThreshold |
Pressure level that triggers grow | 0.3 |
shrinkThreshold |
Idleness level that triggers shrink | 0.5 |
min |
Minimum pool size | 2–3 |
max |
Maximum pool size | 10–25 |
initSize |
Starting size | min(min*3, max) |
The EMA alpha of 0.2 means recent events have more influence but a single anomalous event doesn't cause an immediate resize. Lower alpha = more conservative (slower to react), higher = more aggressive (faster but noisier).
Properties
No oscillation: grow and shrink are driven by different signals measured
at different times. A burst that causes all connections to be claimed but
serves them quickly will see pressure ≈ 0 (no waiting) and idleness → 0
(all in use) — neither threshold fires during the burst.
Lazy: no connections are opened until needed. The pool starts with initSize
slots (semaphore capacity) but opens actual connections on demand.
Self-healing after sleep/disconnect: dead connections are discarded and their slots returned, keeping the pool at full capacity. The pool seamlessly replenishes as connections are reused.
Bounded: the pool never exceeds max connections. sem is buffered at
max and the mutex ensures size ≤ max before any grow operation.
Non-blocking release: callers are never delayed by the health check. The background goroutine handles the round-trip independently.