Inline Caching: How JITs Make Dynamic Calls Fast

The Problem: Dynamic Dispatch Has No Fixed Address

In a statically typed language, point.x compiles to a single load: the compiler knows point is a Point, that Point lays out x at byte offset 0, and emits mov rax, [rdi]. Method calls resolve through a vtable — one indirect load and one indirect jump — because a C++ or Java object's class is fixed for its lifetime.

Dynamic languages destroy that certainty. In JavaScript, obj.x could hit an own property, a property inherited from anywhere along a mutable prototype chain, a getter, or nothing at all — and two objects reaching the same obj.x site may have completely different layouts. The straightforward implementation stores each object as a hash map from string names to values, so every access becomes: hash the name, probe the object's dictionary, and if it misses, walk the prototype chain repeating the probe at each link. That is dozens of cycles and several dependent memory loads for what static code did in one instruction.

The insight behind inline caching is that this dynamism is theoretical. At a given source location, the same handful of object shapes show up over and over: a loop over an array of records touches objects that are all built the same way. Measurements going back to Deutsch and Schiffman found that the receiver's class at a message-send site is unchanged from the previous execution roughly 95% of the time. Inline caching is the machinery for cashing in on that stability.

Hidden Classes: 'Has Property X' as Pointer Identity

An inline cache is only cheap if the check that "this object matches the cached case" is cheap. Comparing property dictionaries would defeat the purpose. The enabling data structure — introduced as maps in the SELF language (Chambers, Ungar & Lee, OOPSLA 1989) and now called hidden classes in V8, Shapes in SpiderMonkey, and Structures in JavaScriptCore — factors an object's structure out of the object itself.

Every object carries a single hidden pointer to its shape. The shape is an immutable descriptor that says: these property names exist, in this order, at these offsets, with these attributes. Objects that were built the same way share the exact same shape object, so the question "does this object have property x, and where?" collapses to a single pointer comparison: if two objects' shape pointers are identical, their layouts are identical.

• Transition trees. Adding a property does not mutate the shape — it transitions to a new shape via a labeled edge. Starting from the empty shape and adding x then y always lands on the same {x,y} shape. This is why constructors that initialize the same fields in the same order produce monomorphic code, and why adding fields in different orders, or conditionally, quietly splits your objects into several shapes.
• Fast vs. dictionary mode. Properties live inline in the object's slots (a fixed offset) as long as it stays in the shape system. Objects used as growable maps — many properties, deletions, string keys computed at runtime — fall back to dictionary mode, a real hash table, and lose the O(1) shape check.
• Prototype validity. A property found on a prototype is safe to cache only while the prototype chain is unchanged. V8 attaches a prototype validity cell so the cache verifies one cell instead of re-walking the chain; mutating any prototype invalidates every cache that depended on it.

The Inline Cache: Writing the Answer at the Call Site

The original mechanism, from L. Peter Deutsch and Allan Schiffman's Efficient Implementation of the Smalltalk-80 System (POPL 1984), is beautifully direct. Each message-send site in the compiled code starts as a call to the general lookup routine. On the first execution it performs the full method search, then backpatches the call instruction in place to (a) remember the receiver's class and (b) jump straight to the resolved method. The cache is literally inline in the instruction stream — hence the name.

On subsequent calls the patched site runs a tiny prologue: compare the receiver's class to the cached class; if they match, the method runs immediately; if not, fall back to lookup and re-patch. For property access (a load or store IC), the modern shape of the fast path is:

; obj in rdi, cache holds cachedShape and cachedOffset
cmp   [rdi + kShapeOffset], cachedShape   ; is obj the shape we saw?
jne   miss                               ; no -> slow path, update IC
mov   rax, [rdi + cachedOffset]          ; yes -> load the field directly

That is the whole win: a compare, a predictable branch, and a load — two or three cycles — replacing a hash lookup and a chain walk. The cost is a small amount of self-modifying code (or, in modern engines, a per-site feedback slot and a dispatchable stub), plus the discipline of invalidating caches when shapes or prototypes change. Because the check is a value comparison, a mispredicted or changed shape is safe — it just falls through to the miss path — which is what lets the whole scheme be an optimization rather than a source of bugs.

Monomorphic, Polymorphic, Megamorphic

A site's behavior is classified by how many shapes it has seen. This taxonomy — and the crucial polymorphic case — comes from Urs Hölzle, Craig Chambers and David Ungar's Optimizing Dynamically-Typed Object-Oriented Languages With Polymorphic Inline Caches (ECOOP 1991).

• Monomorphic (one shape). The common, ideal case: one cached shape, one compare, one load. Most access sites in real programs are monomorphic.
• Polymorphic (a few shapes). When a handful of shapes recur — say a site iterating over both Circle and Square objects — the cache grows into a polymorphic inline cache (PIC): a small table of (shape → handler) entries checked in turn. Cost is a short linear scan, still far cheaper than a dictionary lookup, and it preserves the type information the optimizer wants.
• Megamorphic (too many shapes). Past a small threshold — V8 caps polymorphism at roughly 4 before declaring the site megamorphic — keeping per-site entries stops paying off. The site switches to a generic stub that consults a process-wide megamorphic cache: a fixed-size hash table keyed by (shape, property name). Lookups are still O(1) expected, but the constant is larger, the result is not inlined, and the optimizer can no longer speculate on the type.

The states are a one-way ratchet in most engines: a site that has gone megamorphic does not usually snap back, because the shapes it saw are genuinely diverse. Recognizing megamorphic sites is a standard performance-debugging step — a single hot function that dispatches over dozens of shapes (a generic serializer, an ORM, a framework's render loop) can dominate a profile purely because its ICs collapsed.

Feeding the JIT: Type Feedback and Speculation

Inline caches do double duty. Beyond speeding up individual sites, they are the primary type-feedback source for the optimizing compiler. Each site's IC state — which shapes appeared, how many, and what they resolved to — is recorded in a per-function feedback vector (V8) or CacheIR stub chain (SpiderMonkey). When a function gets hot, the optimizing tier reads that history and speculates.

If a load site was monomorphic on shape S, the optimizer emits the field access as a direct load guarded by a cheap shape check, and installs a deoptimization point: should an object of an unexpected shape ever arrive, execution bails out of the optimized code back to the interpreter or baseline tier, which handles the general case and re-profiles. This is exactly the speculation that lets V8's TurboFan or SpiderMonkey's Warp inline a method call and then inline through it, treating a dynamic dispatch as if it were a static one — with a guard as the safety net.

The engineering has converged on layered tiers, each leaning on the same feedback. In V8: Ignition (bytecode interpreter) collects IC feedback, Sparkplug and Maglev are fast/mid-tier JITs, and TurboFan is the top optimizer. SpiderMonkey's modern design expresses every inline cache as a sequence of CacheIR operations, so the same cache description drives both the baseline stubs and the Warp optimizer — a notable unification of what used to be duplicated logic. HotSpot's JVM applies the same principle to invokevirtual/invokeinterface, using per-site type profiles to inline monomorphic and bimorphic calls with a guard.

Real Systems, Costs, and Failure Modes

Inline caching is now table stakes for any fast dynamic-language runtime. V8 (Chrome, Node.js, Edge), SpiderMonkey (Firefox), and JavaScriptCore (Safari) all combine hidden classes with monomorphic/polymorphic/megamorphic ICs. PyPy uses maps for Python instance dictionaries and attribute caches; Ruby's YJIT (shipped in Ruby 3.1+) and TruffleRuby rely on inline caches for method dispatch and instance-variable access; the JVM has used inline caches for virtual dispatch for decades.

The failure modes are the mirror image of the wins, and they are the practical reason "write your objects consistently" is folk performance advice:

• Shape churn. Initializing fields in different orders, adding properties after construction, using delete, or attaching properties conditionally splits one logical type into many shapes, pushing sites polymorphic or megamorphic. Using an object as a hash map (dynamic string keys) forces dictionary mode and abandons inline caching entirely — use a real Map instead.
• Megamorphic hot paths. Highly generic code — a logging framework, a deep-equality routine, a template renderer walking arbitrary user objects — sees so many shapes that its ICs give up. The fix is often to specialize or monomorphize the hot loop.
• Deoptimization loops. If speculation keeps being violated (an object occasionally arrives with an off-shape), the engine can thrash between optimizing and deoptimizing. Engines add deopt counters and eventually stop re-optimizing a site.
• Invalidation storms. Mutating a widely used prototype (monkey-patching Array.prototype) invalidates every dependent cache and validity cell across the program, forcing mass re-lookup.

None of these are correctness bugs — the shape guard makes every miss safe — but they explain the large, sometimes 10× performance gaps between code that keeps its sites monomorphic and code that does not. That an idea this small underpins so much of the modern web is the quiet triumph of the technique.

The three states an inline cache moves through as it sees more object shapes

State	Shapes cached	Lookup cost	What the JIT does with it
Uninitialized (cold)	0	Full lookup: hash the name, walk the prototype chain	Records the first shape and patches the site
Monomorphic	1	1 shape compare + 1 load at a constant offset (~2–3 cycles)	Speculates hard: inlines the load, guards on the shape
Polymorphic	2 – ~4	Linear scan of cached shapes, then load	Inlines a short dispatch, or a few guarded cases
Megamorphic	too many (> ~4)	Generic stub → global hash cache keyed by (shape, name)	Gives up on speculation; emits a generic call