ZJIT: Add diff tool for running benchmarks between git refs (ruby#16160)

For Shopify#822

Adds a small script for automatically running benchmarks between two git refs.

Example output (doubled an increment to `inline_cfunc_optimized_send_count` to illustrate stat diffs):
```
$ ./zjit_diff --before master
I, [2026-03-03T10:24:52.691660 #48554]  INFO -- : Existing worktree found at /var/folders/l_/p20zlp5j3wx76vr9ggzgzcmr0000gn/T/ruby-zjit-before
HEAD is now at 7a3940e Unify rb_node_list_new and rb_node_list_new2
I, [2026-03-03T10:24:52.729815 #48554]  INFO -- : Found existing build for master, skipping build
I, [2026-03-03T10:24:52.729881 #48554]  INFO -- : Existing worktree found at /var/folders/l_/p20zlp5j3wx76vr9ggzgzcmr0000gn/T/ruby-zjit-after
HEAD is now at 3859f12966 dup
I, [2026-03-03T10:24:52.801287 #48554]  INFO -- : Found existing build for HEAD, skipping build
I, [2026-03-03T10:24:52.801354 #48554]  INFO -- : Running benchmarks
I, [2026-03-03T10:24:52.801404 #48554]  INFO -- : ruby-bench already cloned, pulling from upstream

<... benchmark run logs ...>

before: ruby 4.1.0dev (2026-03-03T14:05:00Z master 7a3940e) +ZJIT dev +PRISM [arm64-darwin25]
last_commit=Unify rb_node_list_new and rb_node_list_new2
after: ruby 4.1.0dev (2026-03-03T15:17:20Z :detached: 3859f12966) +ZJIT dev +PRISM [arm64-darwin25]

----------  -------------  -------------  -------------  ------------
bench         before (ms)     after (ms)  after 1st itr  before/after
lobsters     747.9 ± 2.9%   757.7 ± 2.4%          0.983   0.987
railsbench  1183.4 ± 1.9%  1214.5 ± 2.4%          1.001   0.974 (*)
----------  -------------  -------------  -------------  ------------

Legend:
- after 1st itr: ratio of before/after time for the first benchmarking iteration.
- before/after: ratio of before/after time. Higher is better for after. Above 1 represents a speedup.
- ***: p < 0.001, **: p < 0.01, *: p < 0.05 (Welch's t-test)

Output:
data/zjit_diff.json
================================================================================
ZJIT Stats Comparison
================================================================================

  before (baseline):
    ruby 4.1.0dev (2026-03-03T14:05:00Z master 7a3940e) +ZJIT dev +PRISM [arm64-darwin25]
last_commit=Unify rb_node_list_new and rb_node_list_new2
  after:
    ruby 4.1.0dev (2026-03-03T15:17:20Z :detached: 3859f12966) +ZJIT dev +PRISM [arm64-darwin25]

BENCHMARK TIMINGS (lower is better)
--------------------------------------------------------------------------------
  lobsters:
    before               avg:   0.748s  min:   0.706s  ★ (baseline)
    after                avg:   0.758s  min:   0.735s       +1.3% (slower)
  railsbench:
    before               avg:   1.183s  min:   1.158s  ★ (baseline)
    after                avg:   1.214s  min:   1.182s       +2.6% (slower)

MEMORY USAGE
--------------------------------------------------------------------------------
  lobsters:
    before               maxrss:    548.3MB  zjit_mem:     68.6MB
    after                maxrss:    520.8MB  zjit_mem:     68.3MB
  railsbench:
    before               maxrss:    208.3MB  zjit_mem:     29.3MB
    after                maxrss:    205.7MB  zjit_mem:     29.3MB

SEND COUNTERS (showing differences > 5.0%)
--------------------------------------------------------------------------------
  lobsters:
    inline_cfunc_optimized_send_count      64,066,387 ( 37.3%) →    105,498,619 ( 61.4%) ▲  +64.7%
    optimized_send_count                  136,557,743 ( 79.5%) →    177,989,922 (103.6%) ▲  +30.3%
    send_count                            171,791,578 (100.0%) →    213,223,697 (124.1%) ▲  +24.1%
  railsbench:
    inline_cfunc_optimized_send_count     130,638,873 ( 36.3%) →    225,643,322 ( 62.8%) ▲  +72.7%
    optimized_send_count                  306,690,230 ( 85.3%) →    401,694,689 (111.8%) ▲  +31.0%
    send_count                            359,393,477 (100.0%) →    454,397,936 (126.4%) ▲  +26.4%

SUMMARY COUNTERS (showing differences > 5.0%)
--------------------------------------------------------------------------------
  railsbench:
    invalidation_time                             7ms →            7ms ▲   +5.3%

COMPILE HIR PASS TIMINGS (showing differences > 5.0%)
--------------------------------------------------------------------------------
  lobsters:
    eliminate_dead_code_time                    245ms →          232ms ▼   -5.3%
```

Author: 39bytes

tool/zjit_diff.rb

@@ -0,0 +1,272 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'fileutils'
+require 'optparse'
+require 'tmpdir'
+require 'logger'
+require 'digest'
+require 'shellwords'
+
+GitRef = Struct.new(:ref, :commit_hash)
+
+RUBIES_DIR = File.join(Dir.home, '.zjit-diff')
+BEFORE_NAME = 'ruby-zjit-before'
+AFTER_NAME = 'ruby-zjit-after'
+
+LOG = Logger.new($stderr)
+
+def macos?
+  Gem::Platform.local == 'darwin'
+end
+
+class CommandRunner
+  def initialize(quiet: false)
+    @quiet = quiet
+  end
+
+  def cmd(*args, **options)
+    options[:out] ||= @quiet ? File::NULL : $stderr
+    options = options.merge(exception: true)
+    system(*args, **options)
+  end
+end
+
+class ZJITDiff
+  DATA_FILENAME = File.join('data', 'zjit_diff')
+  RUBY_BENCH_REPO_URL = 'https://github.com/ruby/ruby-bench.git'
+
+  def initialize(before_hash:, after_hash:, runner:, options:)
+    @before_hash = before_hash
+    @after_hash = after_hash
+    @runner = runner
+    @options = options
+  end
+
+  def bench!
+    LOG.info('Running benchmarks')
+    ruby_bench_path = @options[:bench_path] || setup_ruby_bench
+    run_benchmarks(ruby_bench_path)
+  end
+
+  private
+
+  def run_benchmarks(ruby_bench_path)
+    Dir.chdir(ruby_bench_path) do
+      @runner.cmd({ 'RUBIES_DIR' => RUBIES_DIR },
+                  './run_benchmarks.rb',
+                  '--chruby',
+                  "before::#{@before_hash} --zjit-stats;after::#{@after_hash} --zjit-stats",
+                  '--out-name',
+                  DATA_FILENAME,
+                  *@options[:bench_args],
+                  *@options[:name_filters])
+
+      @runner.cmd('./misc/zjit_diff.rb', "#{DATA_FILENAME}.json", out: $stdout)
+    end
+  end
+
+  def setup_ruby_bench
+    path = File.join(Dir.tmpdir, 'ruby-bench')
+    if Dir.exist?(path)
+      LOG.info('ruby-bench already cloned, pulling from upstream')
+      Dir.chdir(path) do
+        @runner.cmd('git', 'pull')
+      end
+    else
+      LOG.info("ruby-bench not cloned yet, cloning repository to #{path}")
+      @runner.cmd('git', 'clone', RUBY_BENCH_REPO_URL, path)
+    end
+    path
+  end
+end
+
+class RubyWorktree
+  attr_reader :hash
+
+  BREW_REQUIRED_PACKAGES = %w[openssl readline libyaml].freeze
+
+  def initialize(name:, ref:, runner:, force_rebuild: false)
+    @path = File.join(Dir.tmpdir, name)
+    @ref = ref
+    @force_rebuild = force_rebuild
+    @runner = runner
+    @hash = nil
+
+    setup_worktree
+  end
+
+  def build!
+    Dir.chdir(@path) do
+      configure_cmd_args = ['--enable-zjit=dev', '--disable-install-doc']
+      if macos?
+        brew_prefixes = BREW_REQUIRED_PACKAGES.map do |pkg|
+          `brew --prefix #{pkg}`.strip
+        end
+        configure_cmd_args << "--with-opt-dir=#{brew_prefixes.join(':')}"
+      end
+      configure_cmd_hash = Digest::MD5.hexdigest(configure_cmd_args.join(''))
+
+      build_cmd_args = ['-j', 'miniruby']
+      build_cmd_hash = Digest::MD5.hexdigest(build_cmd_args.join(''))
+
+      @hash = "#{configure_cmd_hash}-#{build_cmd_hash}-#{@ref.commit_hash}"
+      prefix = File.join(RUBIES_DIR, @hash)
+
+      if Dir.exist?(prefix) && !@force_rebuild
+        LOG.info("Found existing build for #{@ref.ref}, skipping build")
+        return
+      end
+
+      @runner.cmd('./autogen.sh')
+
+      cmd = [
+        './configure',
+        *configure_cmd_args,
+        "--prefix=#{prefix}"
+      ]
+
+      @runner.cmd(*cmd)
+      @runner.cmd('make', *build_cmd_args)
+      @runner.cmd('make', 'install')
+    end
+  end
+
+  private
+
+  def setup_worktree
+    if Dir.exist?(@path)
+      LOG.info("Existing worktree found at #{@path}")
+      Dir.chdir(@path) do
+        @runner.cmd('git', 'checkout', @ref.commit_hash)
+      end
+    else
+      LOG.info("Creating worktree for ref '#{@ref.ref}' at #{@path}")
+      @runner.cmd('git', 'worktree', 'add', '--detach', @path, @ref.commit_hash)
+    end
+  end
+end
+
+def clean!
+  [BEFORE_NAME, AFTER_NAME].each do |name|
+    path = File.join(Dir.tmpdir, name)
+    if Dir.exist?(path)
+      LOG.info("Removing worktree at #{path}")
+      system('git', 'worktree', 'remove', '--force', path)
+    end
+  end
+
+  if Dir.exist?(RUBIES_DIR)
+    LOG.info("Removing ruby installations from #{RUBIES_DIR}")
+    FileUtils.rm_rf(RUBIES_DIR)
+  end
+
+  bench_path = File.join(Dir.tmpdir, 'ruby-bench')
+  return unless Dir.exist?(bench_path)
+
+  LOG.info("Removing ruby-bench clone at #{bench_path}")
+  FileUtils.rm_rf(bench_path)
+end
+
+def parse_ref(ref)
+  out = `git rev-parse --verify #{ref}`
+  return nil unless $?.success?
+
+  GitRef.new(ref: ref, commit_hash: out.strip)
+end
+
+DEFAULT_BENCHMARKS = %w[lobsters railsbench].freeze
+
+options = {}
+
+subtext = <<~HELP
+  Subcommands:
+     bench :  Run benchmarks
+     clean :  Clean temporary files created by benchmarks
+  See '#{$PROGRAM_NAME} COMMAND --help' for more information on a specific command.
+HELP
+
+top_level = OptionParser.new do |opts|
+  opts.banner = "Usage: #{$PROGRAM_NAME} [options]"
+  opts.separator('')
+  opts.separator(subtext)
+end
+
+subcommands = {
+  'bench' => OptionParser.new do |opts|
+    opts.banner = "Usage: #{$PROGRAM_NAME} [options] <benchmarks to run>"
+
+    opts.on('--before REF', 'Git ref for ruby (before)') do |ref|
+      git_ref = parse_ref ref
+      if git_ref.nil?
+        warn "Error: '#{ref}' is not a valid git ref"
+        exit 1
+      end
+
+      options[:before] = git_ref
+    end
+
+    opts.on('--after REF', 'Git ref for ruby (after)') do |ref|
+      git_ref = parse_ref ref
+      if git_ref.nil?
+        warn "Error: '#{ref}' is not a valid git ref"
+        exit 1
+      end
+
+      options[:after] = git_ref
+    end
+
+    opts.on('--bench-path PATH',
+            'Path to an existing ruby-bench repository clone ' \
+            '(if not specified, ruby-bench will be cloned automatically to a temporary directory)') do |path|
+      options[:bench_path] = path
+    end
+
+    opts.on('--bench-args ARGS', 'Args to pass to ruby-bench') do |bench_args|
+      options[:bench_args] = bench_args.shellsplit
+    end
+
+    opts.on('--force-rebuild',
+            'Force building ruby again instead of using even if existing builds exist in the cache at ~/.diffs') do
+      options[:force_rebuild] = true
+    end
+
+    opts.on('--quiet', 'Silence output of commands except for benchmark result') do
+      options[:quiet] = true
+    end
+
+    opts.separator('')
+    opts.separator('If no benchmarks are specified, the benchmarks that will be run are:')
+    opts.separator(DEFAULT_BENCHMARKS.join(', '))
+  end,
+  'clean' => OptionParser.new do |opts|
+  end
+}
+
+top_level.order!
+command = ARGV.shift
+subcommands[command].order!
+
+case command
+when 'bench'
+  options[:name_filters] = ARGV.empty? ? DEFAULT_BENCHMARKS : ARGV
+  options[:after] ||= parse_ref('HEAD')
+
+  runner = CommandRunner.new(quiet: options[:quiet])
+
+  before = RubyWorktree.new(name: BEFORE_NAME,
+                            ref: options[:before],
+                            runner: runner,
+                            force_rebuild: options[:force_rebuild])
+  before.build!
+  after = RubyWorktree.new(name: AFTER_NAME,
+                           ref: options[:after],
+                           runner: runner,
+                           force_rebuild: options[:force_rebuild])
+  after.build!
+
+  zjit_diff = ZJITDiff.new(runner: runner, before_hash: before.hash, after_hash: after.hash, options: options)
+  zjit_diff.bench!
+when 'clean'
+  clean!
+end