AssertionError: Cannot find component 'SSL' for 'OpenSSL.SSL.Context depdendt on SCCs ordering in incremental checks

[cburroughs]

Crash Report

I'm running mypy through the Pants build system on an internal Python monorepo. You can think of Pants as roughly in the vein of Bazel, or Buck2 if you are more familiar with those systems. Notably Pants runs typechecking (along with most other operations) in an ephemeral sandbox with the minimum dependency set needed. So if I have src/{foo,bar,bax} and I check src/foo only the transitive dependencies of src/foo are included (and not bar or bax). While it is common to typecheck the whole monorepo, it is also common to just check the little library in the corner you are working on, or dynamically check what has changed since trunk. This means that the packages that mypy can see in the environment can vary significantly on each run. This is historically a source of anecdotal reports of "spooky" issues with the mypy cache pantsbuild/pants#18519 -- some of wich have been fixed; thank you!

(An alternative design would be to use a separate cache per target combination, but that would produce poor hit rates and an explosion of storage).

This issue attempts to describe one such issue that leads to an AssertionError when type checking code that uses pyopenssl. I think this is likely the same issue as #19477. I've attempted fixes that suggest overlap with #16214, but I think that is related as opposed to identical.

Traceback

Traceback (most recent call last):
  File "/workspace/build/CI/MY_REPO/pants.d.cache/pex_root/venvs/3/00d861817b8c511dac47c6904fb3ad1319d7fd53/a94ba71a28dab5f2849156666f783a089f13d400/pex", line 380, in <module>
    boot(
  File "/workspace/build/CI/MY_REPO/pants.d.cache/pex_root/venvs/3/00d861817b8c511dac47c6904fb3ad1319d7fd53/a94ba71a28dab5f2849156666f783a089f13d400/pex", line 368, in boot
    sys.exit(func())
             ^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/main.py", line 135, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/main.py", line 219, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 351, in build
    result = build_inner(
             ^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 448, in build_inner
    graph = dispatch(sources, manager, stdout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 3214, in dispatch
    process_graph(graph, manager)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 3625, in process_graph
    done, still_working, results = manager.wait_for_done(graph)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 1101, in wait_for_done
    process_stale_scc(graph, next_scc, self)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 3755, in process_stale_scc
    process_fresh_modules(graph, sorted(prev_scc.mod_ids), manager)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 3703, in process_fresh_modules
    graph[id].fix_cross_refs()
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/build.py", line 2364, in fix_cross_refs
    fixup_module(self.tree, self.manager.modules, self.options.use_fine_grained_cache)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 53, in fixup_module
    node_fixer.visit_symbol_table(tree.names, tree.fullname)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 160, in visit_symbol_table
    self.visit_type_info(value.node)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 73, in visit_type_info
    self.visit_symbol_table(info.names, info.fullname)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 162, in visit_symbol_table
    value.node.accept(self)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/nodes.py", line 1164, in accept
    return visitor.visit_decorator(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 192, in visit_decorator
    d.func.accept(self)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/nodes.py", line 1004, in accept
    return visitor.visit_func_def(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 170, in visit_func_def
    func.type.accept(self.type_fixer)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/types.py", line 2337, in accept
    return visitor.visit_callable_type(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 281, in visit_callable_type
    argt.accept(self)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/types.py", line 3394, in accept
    return visitor.visit_union_type(self)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 374, in visit_union_type
    it.accept(self)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/types.py", line 1686, in accept
    return visitor.visit_instance(self)
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 245, in visit_instance
    inst.type = lookup_fully_qualified_typeinfo(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/fixup.py", line 383, in lookup_fully_qualified_typeinfo
    stnode = lookup_fully_qualified(name, modules, raise_on_missing=not allow_missing)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/build/CI/MY_REPO/.cache/pants/named_caches/pex_root/venvs/3/s/77d10232/venv/lib/python3.12/site-packages/mypy/lookup.py", line 53, in lookup_fully_qualified
    assert key in names, f"Cannot find component {key!r} for {name!r}"
           ^^^^^^^^^^^^
AssertionError: Cannot find component 'SSL' for 'OpenSSL.SSL.Context'

To Reproduce

Alas! I have been unable so far to create a deterministic self contained reproduction. I'm going to instead give a qualitative explanation that I hope explains what is going on from post-hoc cache inspection, trace logs, and smashing LLMs against them. In my monorepo I've rigged a setup to generate each combination of available targets (so src/*, src/foo, src/bar, src/foo & src/bar, etc in my example above). I then run this in N shards and see what pops up. This deterministically produces the error (shard 107 is always one of the ones that hits it), but not in a way I've been able to isolate and extract stand alone. I've then downloaded the trace logs and caches for local analysis.

The trace that doesn't hit this error looks something like:

Line 14669: TRACE: Found fresh SCC (OpenSSL)
Line 14711: TRACE: Found fresh SCC (OpenSSL.crypto)
Line 14751: TRACE: Found fresh SCC (OpenSSL.SSL)
...
Line 18530: LOG:  Processing 131 fresh SCCs
Line 18563: TRACE: Load tree OpenSSL
Line 18682: TRACE: Load tree OpenSSL.crypto
Line 18685: TRACE: Load tree OpenSSL.SSL      ← All three loaded in same batch

While an error trace is along the lines of:

Line 14305: TRACE: Found fresh SCC (OpenSSL)
Line 15297: TRACE: Found fresh SCC (OpenSSL.crypto)
Line 15340: TRACE: Found fresh SCC (OpenSSL.SSL)
...
Line 16649: LOG:  Processing 63 fresh SCCs
Line 16668: TRACE: Load tree OpenSSL          ← Only parent loaded
Line 16764: TRACE: Load tree redis.client     ← Has OpenSSL.SSL.Context type ref
Line 16870: AssertionError                    ← fix_cross_refs fails

LLM "Why OpenSSL.SSL Wasn't Loaded" ascii art

The dependency chain is:

my_src.mcp_server.prompts (NEW, no cache)
  └── imports → my_src.mcp_server.config (interface changed)
        └── imports → my_src.mcp_server.metrics (stale)
              └── imports → ... → redis.client (fresh, from cache)
                                    └── imports → OpenSSL (parent package)
                                    └── type_ref → "OpenSSL.SSL.Context" (NOT an import!)

Since `redis.client` only **imports** `OpenSSL` (the parent), not `OpenSSL.SSL`, the SCC dependency graph doesn't include `OpenSSL.SSL`. The type reference `"OpenSSL.SSL.Context"` is stored as a string in the cache and isn't tracked as a dependency.

redis/client.py for reference.

LLM "When the Bug Triggers" Summary

The bug triggers when:
1. **New source files** are added that have no cache
2. Those files transitively depend on `redis.client` (through some import chain)
3. `redis.client` is in a fresh SCC that gets loaded
4. `OpenSSL.SSL` is in a different fresh SCC that is NOT in the current batch
5. During `fix_cross_refs` for `redis.client`, the lookup for `OpenSSL.SSL.Context` fails

I'm not an expert at the mypy codebase and I've tried to be clear where I'm leaning on LLMs. I'm happy to run any PRs through my internal setup. For transparency -- and not because any human is obliged to read it -- here is a consolidated LLM generated summary: https://gist.github.com/cburroughs/a255831b2780aafe128c500ea43bb4e8

Your Environment

• Mypy version used: 1.19.1; master at and branched from 67b9c81dd0fa7f3193d6d3448704a01808620158
• Mypy command-line flags: --explicit-package-bases -vv --sqlite-cache
• Mypy configuration options from mypy.ini (and other config files): Edited sample: https://gist.github.com/cburroughs/6435cc46d75f0fff7ec151a5f580ddc6
• Python version used: 3.12.12
• Operating system and version: Linux 6.12.58

[ilevkivskyi]

Thanks for nice repro! For this kind of issues having a clear repro is half of the fix. I am quite sure this is the same issue as #19477, so will close this as a duplicate.

[ilevkivskyi]

@cburroughs I finally started looking into this. First, unfortunately, your PR is not really a fix. Second, something goes very wrong even on "successful" runs. Namely I see in your logs:

Line 18563: TRACE: Load tree OpenSSL
Line 18682: TRACE: Load tree OpenSSL.crypto
Line 18685: TRACE: Load tree OpenSSL.SSL

but these modules should not be loaded in this order, because OpenSSL depends on OpenSSL.SSL (it imports it, see https://github.com/pyca/pyopenssl/blob/main/src/OpenSSL/__init__.py#L8). Mypy should normally detect this kind of imports, otherwise this crash would be waaaay more common. So this must be something specific to pyopenssl. I looked at what do I actually have in mypy site-packages, and I was really surprised to see both:

~/venvs/mypy_dev_3_12/lib/python3.12/site-packages/OpenSSL

and

~/venvs/mypy_dev_3_12/lib/python3.12/site-packages/OpenSSL-stubs

the latter looks incomplete, for example OpenSSL-stubs/__init__.pyi is empty. I have no idea why do I have the -stubs. My guess is it is coming from an older version of pyopenssl, and this somehow confuses mypy (although I still can't reproduce the crash locally).

@cburroughs Could you please check what is your situation with pyopenssl? Namely where mypy finds the sources for pyopenssl?

[cburroughs]

Thanks for taking a look!

First, unfortunately, your PR is not really a fix.

I take it you mean #20658? I linked to this ticket to provide further context on my test setup, but that PR is intended to fix #16214

I have no idea why do I have the -stubs

So, now I have the same question. I have a lockfile, so to be pedantically specific I have pyopenssl-25.3.0-py3-none-any.whl and types_pyOpenSSL-24.1.0.20240722-py3-none-any.whl. But, when I go to https://pypi.org/project/types-pyOpenSSL/ today I see "Note: The pyOpenSSL package includes type annotations or type stubs since version 24.2.1. Please uninstall the types-pyOpenSSL package if you use this or a newer version." If I remove types_pyOpenSSL-24.1.0.20240722-py3-none-any.whl from my lockfile I have so far been unable to reproduce the issue. I

For the banal question of "Why do I have types-pyOpenSSL in my lockfile?" the answer is "I put it there" But, I take it this is not the expected way mypy is supposed to work in this case,

If I crack open the last release of the stubd I see an empty __init__.pyi file, but I'm unsure how to interpret that.

$ ls -lh types-pyOpenSSL-24.1.0.20240722/OpenSSL-stubs/
total 103K
-rw-r----- 1 ecsb ecsb 7.8K Jul 21  2024 crypto.pyi
-rw-r----- 1 ecsb ecsb    0 Jul 21  2024 __init__.pyi
-rw-r----- 1 ecsb ecsb  302 Jul 21  2024 METADATA.toml
-rw-r----- 1 ecsb ecsb    8 Jul 21  2024 py.typed
-rw-r----- 1 ecsb ecsb  113 Jul 21  2024 rand.pyi
-rw-r----- 1 ecsb ecsb 7.2K Jul 21  2024 SSL.pyi
-rw-r----- 1 ecsb ecsb  130 Jul 21  2024 version.pyi

[ilevkivskyi]

I linked to this ticket to provide further context on my test setup, but that PR is intended to fix #16214

Yeah, I understand. I wanted to say that PR is not really a valid fix for either of the issues (and I still think the two may actually have the same root cause).

If I crack open the last release of the stubd I see an empty __init__.pyi file, but I'm unsure how to interpret that.

You should interpret that as a bug in types-PyOpenSSL. That said mypy shouldn't crash on it. I think I may have an idea about what exactly happens and how to fix it.

[ilevkivskyi]

Well, I found some self-contained test case that crashes with this traceback, but I am still not sure this is the same one

[case testIndirectDependencyDeletedModule]
import a
[file a.py]
import b
[file a.py.2]
import OpenSSL.SSL
import b
[file b.py]
import foo
import OpenSSL
y: OpenSSL.SSL.Context
[file OpenSSL/__init__.pyi]
[file OpenSSL/SSL.pyi]
import c
class Context: ...
[file c.py]
import d
[file c.py.2]
import d  # touch
[file d.py]
import e
[file d.py.2]
import e  # touch
[file e.py]
[file foo.py]
import baz
[file baz.py]
import OpenSSL.SSL
[file baz.py.2]
[out]
[out2]

I will try to write a fix for this one, and then we will see if it will fix this and/or google crash.

[ilevkivskyi]

@cburroughs Could you please check if #20735 has any effect on your repro(s)?

[ilevkivskyi]

Hm, I just checked, it looks like my PR doesn't fix the google-related crash (but at least I can repro it locally now). Could you please still check your repro with types-PyOpenSSL (the one with empty __init__.pyi)?

[cburroughs]

I tried with #20735 + types-PyOpenSSL back in. I didn't see AssertionError: Cannot find component 'SSL' for 'OpenSSL.SSL.Context' but I did see several instances of:

Traceback (most recent call last):
  File "/repo/pants.d.cache/pex_root/venvs/3/f4e6bb5593a1365e7e4fa7634d9beab4f6c30a33/a94ba71a28dab5f2849156666f783a089f13d400/pex", line 380, in <module>
    boot(
  File "/repo/pants.d.cache/pex_root/venvs/3/f4e6bb5593a1365e7e4fa7634d9beab4f6c30a33/a94ba71a28dab5f2849156666f783a089f13d400/pex", line 368, in boot
    sys.exit(func())
             ^^^^^^
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/__main__.py", line 15, in console_entry
    main()
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/main.py", line 135, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/main.py", line 219, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/build.py", line 371, in build
    result = build_inner(
             ^^^^^^^^^^^^
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/build.py", line 468, in build_inner
    graph = dispatch(sources, manager, stdout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/build.py", line 3473, in dispatch
    process_graph(graph, manager)
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/build.py", line 3904, in process_graph
    stale, fresh = find_stale_sccs(ready, graph, manager)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/repo/.cache/pants/named_caches/pex_root/venvs/3/s/4e82d9cf/venv/lib/python3.12/site-packages/mypy/build.py", line 3803, in find_stale_sccs
    if graph[id].priorities[dep] == PRI_INDIRECT:
       ~~~~~~~~~~~~~~~~~~~~^^^^^
KeyError: 'builtins'

[ilevkivskyi]

@cburroughs Thank you! I have just updated the PR, could you please try again now?

[ilevkivskyi]

This is the last commit 52622da

[cburroughs]

At 52622da + types-PyOpenSSL I see zeroOpenSSL.SSL.Context or KeyError: 'builtins' crashes. (And as you noted no change on AssertionError: Cannot find module for google) 👍