A Node.js API can return 30-second responses while its CPU chart sits below 20%, and it can also peg a core at 100% while users see no unusual delay. The difference is often visible within one minute if you measure event-loop delay and event-loop utilization beside request timings instead of treating CPU as the verdict.
This matters because users do not experience “CPU utilization.” They experience a checkout request that stops responding, a WebSocket message that arrives late, or a health check that times out after a deploy. In a Node.js process, a short stretch of synchronous JavaScript can delay every request sharing that event loop, while a slow database or third-party API can make one request painfully slow without preventing the process from accepting and progressing other work.
Start with the user-visible failure, not the host graph
Suppose an endpoint normally has a p95 latency of 80 ms. At 14:10, its p95 rises to 8 seconds. A CPU dashboard shows 18% usage for the container, so the first instinct is often to rule out application saturation and investigate the database. That conclusion is premature.
A single Node.js process normally executes JavaScript callbacks on one main event-loop thread. If one callback performs 400 ms of synchronous parsing, serialization, compression, cryptography, or an accidental large loop, the process may use only a fraction of the CPU allocated to a multi-core container. Yet requests that need that event loop during those 400 ms must wait.
Conversely, consider an endpoint that calls a payment provider with a 10-second timeout. The process can spend most of those 10 seconds waiting on network I/O. Users still see 10-second latency, but event-loop delay can remain low and the event loop can spend little time active.
| User symptom | Likely metric pattern | First investigation target |
|---|---|---|
| Many unrelated routes become slow together | Elevated event-loop delay; event-loop utilization often elevated | Synchronous JavaScript, excessive callback work, CPU contention |
| One dependency-backed route becomes slow | Low event-loop delay; low or moderate utilization; long dependency span | Database, DNS, remote API, connection pool, network path |
| Process CPU is high but latency is stable | Event-loop delay remains near baseline | Background work, another process thread, capacity trend rather than immediate incident |
Measure two event-loop signals because they answer different questions
Event-loop delay asks: “How late was the event loop in getting a chance to run?” Node’s monitorEventLoopDelay() records a histogram of this delay. A rising p95 or p99 is strong evidence that timers and queued callbacks are not getting scheduled promptly.
Event-loop utilization (ELU) asks: “During this interval, how much time did the event loop spend active rather than idle?” Node exposes it through performance.eventLoopUtilization(). A high ELU means the loop was busy; it does not by itself prove that users were slow. A scheduled job can make ELU high without affecting a low-volume API, while a short but severe synchronous pause can create a bad tail-latency spike that an average smooths away.
CPU measures something else: CPU time consumed by the process or host. It cannot tell you whether the main JavaScript thread was unavailable when an incoming request needed it. Container CPU percentages are especially easy to misread when limits allow multiple cores or when the reported value is averaged over a long interval.
Use all three measurements, but assign each a job:
- Request latency: what users observed, split by route and status code.
- Event-loop delay: whether the JavaScript scheduler was delayed.
- ELU: whether the loop was active versus idle over the interval.
- CPU: whether the process or machine was consuming compute capacity.
Add a small event-loop probe before changing production code
The following module uses Node’s built-in node:perf_hooks APIs. It emits a snapshot every 10 seconds. The delay histogram reports nanoseconds, so divide by 1e6 to export milliseconds. Keep the logging or metrics exporter outside request handlers; calculating and emitting a metric once per request creates precisely the extra work you are trying to inspect.
import { monitorEventLoopDelay, performance } from 'node:perf_hooks';
const loopDelay = monitorEventLoopDelay({ resolution: 20 });
loopDelay.enable();
let previousElu = performance.eventLoopUtilization();
setInterval(() => {
const currentElu = performance.eventLoopUtilization();
const elu = performance.eventLoopUtilization(currentElu, previousElu);
previousElu = currentElu;
const metrics = {
event_loop_delay_p50_ms: loopDelay.percentile(50) / 1e6,
event_loop_delay_p95_ms: loopDelay.percentile(95) / 1e6,
event_loop_delay_p99_ms: loopDelay.percentile(99) / 1e6,
event_loop_delay_max_ms: loopDelay.max / 1e6,
event_loop_utilization: elu.utilization
};
console.log(JSON.stringify(metrics));
loopDelay.reset();
}, 10_000).unref();A 20 ms resolution is a practical starting point for detecting delays that are meaningful to an HTTP service. Do not interpret a single p99 value without comparing it with your normal baseline and the same 10-second window’s request volume. A quiet service can show awkward-looking percentiles from a small sample of scheduling observations; a sustained rise correlated with route latency is the useful signal.
Scenario one: a blocked loop makes unrelated requests slow
Imagine an API with GET /catalog and POST /reports/export. The catalog route reads from a cache and usually returns in 40 ms. The export route receives a large JSON payload and synchronously transforms it before returning a CSV file.
function buildExport(payload) {
// Illustrative anti-pattern: synchronous work on the request path.
return JSON.stringify(payload)
.replaceAll('"', '')
.toUpperCase();
}The exact operation is not the point. Real incidents often involve JSON.parse() or JSON.stringify() on unexpectedly large bodies, synchronous filesystem APIs, expensive regular expressions, image processing, compression, or a loop over a very large in-memory collection. While that JavaScript runs, Node cannot execute the callback that completes an otherwise fast /catalog request.
During a burst of exports, you would expect a recognizable shape: export latency rises first, catalog latency rises too, event-loop delay percentiles climb, and ELU rises toward 1 for the affected process. CPU may rise, but it may not look dramatic at the container level. If the container can use four CPUs and one event-loop thread is fully occupied, a dashboard may display roughly one core’s worth of work rather than an alarming “100%.”
The key decision is not “add more CPU.” First find the synchronous region. A CPU profile captured during the incident is more valuable than a generic profile collected at idle, because it identifies the functions consuming the blocked interval.
Scenario two: I/O-heavy latency leaves the event loop healthy
Now use the same service, but make POST /checkout call a remote fraud-scoring API. At 14:10, the provider’s response time increases from 120 ms to 7 seconds. Checkout p95 climbs sharply, support tickets arrive, and the application’s HTTP request count may remain normal.
In this case, the Node process normally starts an outbound request and yields control while it waits for the network response. Other callbacks can run. Event-loop delay remains close to its baseline, ELU may remain low or moderate, and unrelated routes such as /catalog can continue to meet their latency target.
The correct question changes from “what blocked JavaScript?” to “where did this request spend its 7 seconds?” Instrument the dependency boundary. Record the remote service name, operation, timeout outcome, retry count, and duration. If you use distributed tracing, inspect the span for the outbound HTTP call rather than only the top-level route span.
Do not make the common overcorrection of treating low ELU as proof of health. Low ELU plus slow checkout means the Node runtime likely had capacity, but users still had a production incident. The remediation is usually a dependency timeout, a circuit breaker, a queueing decision, a fallback, or a provider investigation—not worker threads for the application.
Use a diagnosis matrix during the incident
The fastest debugging path is to compare measurements from the same time window. Use 10-second or 30-second windows for event-loop metrics and align them with route p95/p99 latency, error rate, and dependency duration. Do not compare a one-minute CPU average to a five-minute latency chart and infer causation.
| Event-loop delay | ELU | Route pattern | Most likely next step |
|---|---|---|---|
| High | High | Many routes degrade | Capture CPU profile; find synchronous or CPU-heavy application work |
| High | Not consistently high | Short tail-latency spikes across routes | Look for periodic synchronous tasks, garbage-collection pressure, or host scheduling contention |
| Low | Low | One route is slow | Inspect database, remote HTTP, DNS, queues, and connection acquisition time |
| Low | High | Latency stable or only a CPU-heavy route is affected | Profile workload; assess capacity and isolate noncritical background work |
| High | High | Only one route appears slow | Check whether that route triggers shared synchronous work or causes response serialization pressure |
This is a decision rule, not a substitute for traces. High delay tells you that the runtime was unable to schedule promptly; it does not name the offending function. Low delay tells you to stop blaming the event loop first, not to stop investigating latency.
Correlate metrics with request and dependency timing
Event-loop data becomes actionable when it answers a request-level question. Add a request ID to application logs, measure total route duration, and measure major dependency calls separately. For an Express-style handler, the shape is simple: record a timestamp at entry, record timestamps before and after the database or HTTP call, then log the route duration when the response finishes.
app.get('/checkout/:id', async (req, res, next) => {
const started = performance.now();
try {
const fraudStarted = performance.now();
const fraudResult = await scoreFraud(req.params.id);
const fraudMs = performance.now() - fraudStarted;
res.json({ fraudResult });
console.log({
route: '/checkout/:id',
total_ms: performance.now() - started,
fraud_ms: fraudMs
});
} catch (error) {
next(error);
}
});If total duration is 7,050 ms and fraud_ms is 6,980 ms while event-loop delay is normal, the case is straightforward. If dependency time is 80 ms but total time is 3,000 ms and event-loop delay spikes during the same interval, inspect work before sending the response: transformations, template rendering, serialization, logging, and response compression are frequent places to look.
This correlation also prevents a costly team failure mode: the application team blames the database because latency is high, while the database team sees normal query duration. Event-loop delay provides evidence about whether the application itself was able to process the query result promptly.
Fix the blocked-loop case without merely moving the bottleneck
Once a profile identifies synchronous work, choose a fix based on whether the result must be returned in the current request. For an export that takes seconds, making the HTTP request wait is usually the wrong product boundary even if you move the computation elsewhere.
- Reduce the work: paginate large responses, limit request-body size, avoid serializing fields the client does not need, and replace pathological algorithms.
- Chunk the work: process a bounded batch, yield back to the event loop, then continue. This improves fairness but can increase total completion time.
- Use worker threads for CPU-bound JavaScript: keep expensive computation off the main event loop, but limit concurrency so workers do not exhaust CPU.
- Queue long jobs: return a job ID, process exports asynchronously, and let the client poll or receive a completion notification.
- Replace synchronous APIs: remove synchronous filesystem and cryptographic operations from request paths where asynchronous alternatives fit.
The tradeoff people skip is operational ownership. Worker threads introduce message-passing and memory-transfer concerns. Queues introduce retry, idempotency, retention, and monitoring requirements. Still, these costs are usually easier to control than allowing one customer’s large export to delay every customer’s API call.
Fix the I/O-heavy case with deadlines and isolation
For slow dependency calls, adding worker threads does not make the remote service respond sooner. Instead, place an explicit deadline around the outbound operation and decide what the user should receive when the deadline expires. A timeout without a product decision simply converts “slow” into “failed” at a later point.
Start by separating timings that are often lumped into “HTTP latency”: connection acquisition, DNS lookup, TCP/TLS connection, time to first response byte, full response body, and retry delay. Your HTTP client or tracing system may expose some of these directly; at minimum, log dependency name, duration, status, and timeout outcome.
Then protect the healthy parts of the service. Bound concurrent calls to a degraded provider, avoid unbounded retry loops, and consider a fallback response when the business operation permits it. For checkout fraud scoring, “manual review required” may be safer than a blanket allow or deny decision. For product recommendations, a cached or empty recommendation set may be acceptable.
Event-loop metrics still matter here because they stop you from applying the wrong repair. If delay is normal and only the fraud-dependent route is slow, optimize the provider contract and failure policy first. Scaling Node replicas can increase the number of simultaneous slow calls and make the provider’s overload worse.
Build the dashboard and alert you will use this week
Create one dashboard row per service instance or pod and one aggregated service view. Keep route latency, event-loop delay, ELU, CPU, error rate, and dependency duration on aligned time axes. Per-instance visibility is important: one unhealthy process can produce tail latency while an average across many replicas hides it.
- Deploy the
monitorEventLoopDelay()and ELU probe to one noncritical service or a small canary set. - Record p50, p95, p99, and maximum event-loop delay in milliseconds every 10 seconds.
- Add route p95/p99 latency split by route, plus duration metrics for the service’s two most important dependencies.
- Run a controlled test: send concurrent requests while triggering known CPU-heavy work, then repeat with an intentionally delayed dependency in a test environment.
- Write a runbook rule: elevated delay across unrelated routes means profile application work; normal delay with a slow dependency span means investigate the dependency path.
- Alert on sustained deviation from your own baseline, not an arbitrary universal event-loop-delay number.
The useful outcome is not a prettier Node.js dashboard. It is a faster first decision during an incident: isolate CPU-bound work from the event loop when the scheduler is delayed, or stop tuning JavaScript when the runtime is healthy and time is disappearing in I/O. That distinction turns “CPU looks fine” from a dead end into evidence.