The LuaJIT NYI That Silently Poisoned an Unrelated Hot Loop

Posted by Sander Bos on with 0 Comment

If you haven’t used Lua before, it’s the go-to embedded scripting language for games like Factorio and World of Warcraft, and for applications like Neovim and OpenResty. LuaJIT, its just-in-time compiler, is widely praised for its speed, so it’s easy to assume your code is already running as fast as possible. In reality, you can unknowingly cripple performance by hitting one of LuaJIT’s NYIs.

NYI stands for “Not Yet Implemented,” meaning operations that LuaJIT cannot translate into optimized machine code. What happens next depends on the specific NYI, and untangling those differences is the subject of this post.

I ran into this while benchmarking grug-for-lua, my Lua implementation of the grug modding language. The same benchmark, with identical code and inputs, sometimes reported 6 billion iterations and other times only 300 million, a 20× difference. The culprit turned out to be a seemingly harmless operation in one part of the code that silently caused LuaJIT to blacklist a function that an unrelated hot loop later depended on.

This post follows that investigation. It aims to be approachable even if you’ve never touched LuaJIT or compilers before, so it links to background reading wherever something isn’t explained in full. We’ll look at two NYI-related pitfalls, and how to let your CI guard against them.

The suspect code

All game functions are called through Entity:_run_game_fn, which looked roughly like this:

function Entity:_run_game_fn(name, ...)
    local game_fn = self.file.game_fns[name]

    local ok, result = pcall(game_fn, self.state, ...)
end

It was called like so:

self:_run_game_fn(call_expr.fn_name, unpack(args))

At first glance, nothing looks unusual. pcall stands for “protected call” and works like a try/catch: it calls game_fn and catches any error instead of letting it propagate. Forwarding arguments with unpack and receiving them with ... is a common Lua idiom, but inside a LuaJIT hot loop this perfectly valid code triggers one of the compiler’s least obvious optimization pitfalls.

Trace stitching

LuaJIT is a tracing JIT compiler: as your program runs, it records the hottest paths of execution (traces) and compiles them into optimized machine code. A list of features it cannot record into these traces is documented on the LuaJIT Not Yet Implemented wiki page, and unpack is one of them, marked 2.1 stitch. Instead of failing immediately, LuaJIT performs a stitch, which lets it resume trace recording after the NYI instruction executes. For implementation details, see the NYI and Trace Stitching section of LuaJIT Internals (Pt. 2/3): Fighting the JIT Compiler.

My initial assumption was that the stitch itself caused the slowdown by temporarily dropping back to the interpreter. But when I profiled the benchmark with LuaJIT 2.1.1774896198 using -jv, unpack never appeared in the trace log at all.

Reading the trace log

The trace log told a different story. A trace failed immediately after returning from the callee, followed by repeated “NYI: return to lower frame” messages, and eventually a notice that the function had been blacklisted from compilation.

To isolate it, I wrote an MRE (minimal reproducible example):

-- empty_fn takes no args here because many game functions don't
-- take args either; this MRE is meant to mirror that.
-- Note: replacing () with (...) makes the benchmark report fast times,
-- since it avoids the NYI shown below.
local function empty_fn() end

local function run_unpack()
    pcall(empty_fn, unpack({}))
end

-- On my laptop (AMD Ryzen AI 9 HX 370): below ~70k iterations it's
-- reliably fast, above ~90k it's reliably slow, and 80k lands in
-- the nondeterministic zone in between.
for _ = 1, 80000 do
    run_unpack()
end

local start = os.clock()

-- Because of the previous loop, this unrelated hot loop is a coin toss
-- between being JIT-compiled (fast) or stuck in the interpreter (slow).
for _ = 1, 100000000 do
    empty_fn()
end

print(os.clock() - start)
Running luajit -jv pcall_mre.lua produces two very different outcomes depending on chance. Here’s a fast run:
[TRACE   1 pcall_mre.lua:12 return]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
...
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE   2 pcall_mre.lua:4 return]
[TRACE   3 pcall_mre.lua:22 loop]
0.050325

The key observation is that unpack never appears in the trace log. Instead, the recorder “forgets” that execution is inside a pcall, resuming recording as if that pcall had never happened. The failure doesn’t happen at unpack itself; it happens later, when execution returns through that pcall.

LuaJIT gives pcall its own dedicated frame type, FRAME_PCALL, and when the trace recorder later returns through that frame, it expects to find the recorded caller context needed to unwind correctly. Because trace stitching resumes recording as a fresh root trace, the new trace has no record of entering the FRAME_PCALL, so recording aborts with exactly the error shown above: “NYI: return to lower frame”.

A slow run diverges right after that same [TRACE 2 pcall_mre.lua:4 return] line:

[TRACE   1 pcall_mre.lua:12 return]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
...
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE   2 pcall_mre.lua:4 return]
[TRACE --- pcall_mre.lua:2 -- NYI: return to lower frame]
[TRACE --- pcall_mre.lua:22 -- blacklisted at pcall_mre.lua:2]
[TRACE --- pcall_mre.lua:22 -- blacklisted at pcall_mre.lua:2]
...
[TRACE --- pcall_mre.lua:22 -- blacklisted at pcall_mre.lua:2]
[TRACE --- pcall_mre.lua:22 -- blacklisted at pcall_mre.lua:2]
0.725689

Whether empty_fn ends up blacklisted depends on how many stitch attempts through the pcall complete before the benchmark loop begins, an outcome governed by the JIT’s internal heuristics rather than anything in the code. After enough failed attempts, LuaJIT blacklists empty_fn‘s bytecode, so the hot loop calling it can no longer be JIT-compiled and falls back to the interpreter, even though the loop itself never touches pcall or unpack. That’s why it suffers the 14× slowdown. As a sanity check, calling an identical empty_fn2() inside the hot loop stays fast instead, since it was never blacklisted.

The fix

The fix in db94c5a removes the final unpack() call and replaces it with generated wrapper functions:

local loader = loadstring or load

local pcall_wrappers = {}

local function get_pcall_wrapper(arg_count)
    if pcall_wrappers[arg_count] then
        return pcall_wrappers[arg_count]
    end

    local arg_list = {}
    for i = 1, arg_count do
        arg_list[i] = "args[" .. i .. "]"
    end

    -- Generate a specialized wrapper to avoid `unpack` (which triggers a LuaJIT NYI).
    -- Example (arg_count=2): return function(fn, args) return pcall(fn, args[1], args[2]) end
    local args_str = #arg_list > 0 and (", " .. table.concat(arg_list, ", ")) or ""
    local code = string.format("return function(fn, args) return pcall(fn%s) end", args_str)

    local wrapper = loader(code)()
    pcall_wrappers[arg_count] = wrapper
    return wrapper
end

local function run_unpack()
    local args = {}
    local wrapper = get_pcall_wrapper(#args)
    wrapper(empty_fn, args)
end

Instead of forwarding arguments through unpack, each arity gets a specialized wrapper that indexes the args table directly. The wrappers are cached by argument count, so the code generation cost is paid once while execution remains fully traceable by LuaJIT. pcall was never the problem here; it’s fully traceable by LuaJIT. Only unpack triggers the stitch, which is why each generated wrapper still calls pcall to preserve the original error handling.

You can compare db94c5a against its parent 4523ea9 to see the difference directly. In benchmarks/minimal, runs of 4523ea9 fluctuate wildly between fast and slow depending on whether the earlier blacklist is triggered. With db94c5a, the performance is stable, and the 20× variance disappears. However, preventing that silent trace blacklisting was only the first step in fully optimizing the hot loop.

The second NYI: closures

While investigating the blacklist issue, I started checking every hot path in grug-for-lua for other NYIs. That uncovered a second one, triggered by closures (nested functions), which carries an even steeper penalty. Its failure mode is different: rather than poisoning an unrelated loop, it slows down the very loop it appears in. At first glance, it looks like code a compiler should optimize away, similar to what C++’s as-if rule permits for code with no observable side effects.

It’s worth its own minimal reproducible example, closure_mre.lua:

local start = os.clock()

for _ = 1, 100000000 do
    local function nested() end
    nested()
end

print(os.clock() - start)

With nested() defined inside the loop, luajit -jv closure_mre.lua produces this trace log:

[TRACE --- closure_mre.lua:3 -- NYI: bytecode FNEW   at closure_mre.lua:4]
[TRACE --- closure_mre.lua:3 -- NYI: bytecode FNEW   at closure_mre.lua:4]
[TRACE   1 closure_mre.lua:4 return]
[TRACE --- closure_mre.lua:3 -- NYI: bytecode FNEW   at closure_mre.lua:4]
[TRACE --- closure_mre.lua:3 -- NYI: bytecode FNEW   at closure_mre.lua:4]
...
[TRACE --- closure_mre.lua:3 -- NYI: bytecode FNEW   at closure_mre.lua:4]
[TRACE --- closure_mre.lua:3 -- NYI: bytecode FNEW   at closure_mre.lua:4]
1.781446

This is the slow run: repeatedly allocating the closure prevents proper JIT compilation, resulting in a 60× slowdown compared to the optimized version.

Move nested() outside the loop, and the trace output changes completely:

[TRACE   1 closure_mre.lua:4 loop]
0.030344

nested takes no arguments, reads no upvalues, and does nothing in its body, yet LuaJIT still preserves Lua 5.1 semantics. As the Lua 5.1 Reference Manual states, every new function object is distinct from any previously existing one:

Two objects are considered equal only if they are the same object. Every time you create a new object (a table, userdata, thread, or function), this new object is different from any previously existing object.

In theory, LuaJIT could optimize the allocation away, much as it already does for tables using its Allocation Sinking optimization.

Unlike unpack, FNEW (function new) is a hard NYI rather than a stitched one. As soon as the recorder reaches it, recording aborts outright. The NYI wiki correspondingly lists its Compiled? status as no. There’s no race to win or lose here; the loop is simply stuck in the interpreter every time. The only change that matters is moving nested outside the loop, so the loop body becomes a plain function call with no per-iteration allocation, making it 60× faster.

Catching NYIs early

I configure my CI’s build.yml to run benchmarks under luajit -jv. It lets the build fail if any NYI other than the known-sporadic return to lower frame case shows up in the trace log, or if any blacklisted line shows up. That way, any NYI or blacklisting introduced into a hot path gets caught automatically.

# Remove `NYI: return to lower frame` lines from the log.
CLEANED_LOG=$(sed '/NYI: return to lower frame/d' luajit.log)

# Check if any other NYI lines remain.
if echo "$CLEANED_LOG" | grep -q 'NYI'; then
    echo "ERROR: Unexpected NYI detected in LuaJIT trace output" >&2
    echo "$CLEANED_LOG" | grep 'NYI' >&2
    exit 1
fi

# Check if any blacklisted lines remain.
if echo "$CLEANED_LOG" | grep -q 'blacklisted'; then
    echo "ERROR: Blacklisted function detected in LuaJIT trace output" >&2
    echo "$CLEANED_LOG" | grep 'blacklisted' >&2
    exit 1
fi

Getting unpack() out of the NYI list

As an homage to Cloudflare’s LuaJIT Hacking: Getting next() out of the NYI list, I named my pull request perf: get unpack() out of the NYI list. It targets luajit2, OpenResty’s fork of LuaJIT, rather than upstream LuaJIT, since upstream only allows collaborators to open pull requests there. I plan to send the patch to LuaJIT’s creator, Mike Pall, over LuaJIT’s mailing list in the future. In the meantime, luajit2 is a reasonable place to land the fix first: per its own repository description, it states: “It is not to be considered a fork, since we still regularly synchronize changes from the upstream LuaJIT project.”

Here’s the new recff_unpack function the PR adds, which teaches the trace recorder to compile unpack directly instead of falling back to a stitch:

/* unpack(t, [i, [j]]) */
static void LJ_FASTCALL recff_unpack(jit_State *J, RecordFFData *rd)
{
  TRef trtab = J->base[0];
  TRef tri = J->base[1];
  TRef trj = J->base[2];
  RecordIndex ix;
  GCtab *t;
  int32_t i, e, k;
  if (!tref_istab(trtab)) return;  /* Interpreter will throw. */
  t = tabV(&rd->argv[0]);
  if (tref_isnil(tri)) i = 1;
  else {
    i = argv2int(J, &rd->argv[1]);
    if (tref_isk(tri))
      emitir(IRTGI(IR_EQ), tri, lj_ir_kint(J, i));
  }
  if (!tref_isnil(trj)) {  /* trj set guarantees tri was too. */
    e = argv2int(J, &rd->argv[2]);
    if (!tref_isk(trj))
      emitir(IRTGI(IR_EQ), trj, lj_ir_kint(J, e));
  } else {  /* Guard the length, since it wasn't given as a constant. */
    TRef trlen = emitir(IRTI(IR_ALEN), trtab, TREF_NIL);
    e = (int32_t)lj_tab_len(t);
    emitir(IRTGI(IR_EQ), trlen, lj_ir_kint(J, e));
  }
  if (i > e) { rd->nres = 0; return; }
  int32_t maxn = LJ_MAX_JSLOTS - (int32_t)J->baseslot;
  uint32_t span = (uint32_t)e - (uint32_t)i;  /* n - 1, exact & signed overflow-free. */
  if (maxn <= 0 || span >= (uint32_t)maxn)
    lj_trace_err_info(J, LJ_TRERR_STACKOV);
  int32_t n = (int32_t)span + 1;  /* safe: span < maxn <= LJ_MAX_JSLOTS here. */
  ix.tab = trtab; ix.idxchain = 0; ix.val = 0;
  settabV(J->L, &ix.tabv, t);
  rd->nres = n;
  for (k = 0; k < n; k++) {
    ix.key = lj_ir_kint(J, i + k);
    setintV(&ix.keyv, i + k);
    J->base[k] = lj_record_idx(J, &ix);
  }
}

Passing the existing test suite wasn’t enough to convince me recff_unpack was correct for every edge case, so I wrote unimut (universal mutator, designed to work with any programming language) to mutation test it. unimut systematically mutates the function’s logic and checks whether the test suite still catches each change, surfacing gaps that plain line and branch coverage can’t:

The surviving mutants are expected. They involve checks against internal LuaJIT implementation details that Lua-level tests can’t, or shouldn’t, cover.

Even once this is merged, the get_pcall_wrapper workaround from earlier in this post isn’t going anywhere. Almost nobody uses OpenResty’s luajit2 fork specifically, and plenty of programs that do embed some LuaJIT never update the version they ship. This fix lets a small slice of users get unpack compiled for free, while the wrapper workaround stays the practical fix for everyone else for the foreseeable future.

Conclusion

The broader danger of tracing JITs is that performance bugs don’t always show up where you’d expect. With unpack, an innocent call silently blacklisted a completely unrelated hot loop, so benchmarks and profilers pointed at the wrong place entirely. With closures, the cost landed exactly where you’d expect, yet still went unnoticed because nothing in the code looked wrong.

Manual spot-checks cannot catch either failure mode. Instead, use luajit -jv to treat compiler trace output as a testable artifact in CI, so any regressions caused by NYIs get caught automatically instead of slipping back in silently.

If you want to dig deeper into NYIs, check out Cloudflare’s LuaJIT Hacking: Getting next() out of the NYI list and api7.ai’s The JIT Compiler’s Drawback: Why Avoid NYI?.

unpack is now off the list, with tests and reproduction steps in the PR if you want to see it in action. Plenty of NYIs are still sitting on the LuaJIT Not Yet Implemented wiki page, closures included. If you’re looking for an excuse to get your hands dirty in lj_record.c, that page is a good place to start.

 

Written by Sander Bos

Sander Bos is a software engineer at Stream HPC, where he’s worked on GPU benchmarking across HIP, CUDA, Vulkan, OpenCL, and SYCL. He wrote primbench, an open-source GPU benchmarking library, for AMD. Outside of work, he’s spent the last two years building grug, an open-source modding language designed so that a mod written today still runs unchanged in 100 years.