Skip to content

Fix flaky help_options test: restore enableConsoleColoring after mutation#19385

Draft
Copilot wants to merge 2 commits intomainfrom
copilot/fix-flaky-help-options-test
Draft

Fix flaky help_options test: restore enableConsoleColoring after mutation#19385
Copilot wants to merge 2 commits intomainfrom
copilot/fix-flaky-help-options-test

Conversation

Copy link
Contributor

Copilot AI commented Mar 3, 2026

enableConsoleColoring is a global mutable. The fsc --consolecolors switch test sets it to false via --consolecolors-, and since compilation runs in-process, the mutation leaks to subsequent tests. When help_options runs later, the help text reads (off by default) instead of (on by default), failing the baseline comparison.

  • Save/restore enableConsoleColoring around the mutating test with try/finally
let ``fsc --consolecolors switch`` option =
    let saved = enableConsoleColoring
    try
        Fs """printfn "Hello, World" """
        |> asExe
        |> withOptions [option]
        |> compile
        |> shouldSucceed
        |> ignore
    finally
        enableConsoleColoring <- saved
Original prompt

This section details on the original issue you should resolve

<issue_title>Flaky test: help_options baseline fails due to global mutable enableConsoleColoring</issue_title>
<issue_description>## Summary

The CompilerOptions.Fsc.help_options.Help tests (variants 1-3) intermittently fail on Linux CI because the --consolecolors default shows (off by default) instead of the baseline's (on by default).

Root cause

enableConsoleColoring is a global mutable (let mutable enableConsoleColoring = true in src/Compiler/Driver/CompilerOptions.fs:510). The FscOptionTests.fs test at line 57-61 exercises --consolecolors-, which sets this global to false. Since checker.Compile runs in-process, the mutation persists across tests.

When help_options runs after the --consolecolors- test, enableConsoleColoring is still false, so the help output says (off by default) instead of (on by default), mismatching the baseline.

Reproduction

The failure depends on xUnit test execution order. It reproduces consistently on Linux CI but not locally (different parallelism/ordering). Observed on PR #19021 across multiple CI runs.

Relevant code

  • Global mutable: src/Compiler/Driver/CompilerOptions.fs:510
  • Help text generation: src/Compiler/Driver/CompilerOptions.fs:884 — reads enableConsoleColoring at format time
  • Mutating test: tests/FSharp.Compiler.ComponentTests/CompilerOptions/Fsc/FscOptionTests.fs:57-61
  • Failing test: tests/FSharp.Compiler.ComponentTests/CompilerOptions/Fsc/misc/misc.fs:15-44
  • Baseline: tests/FSharp.Compiler.ComponentTests/CompilerOptions/Fsc/misc/compiler_help_output.bsl

Possible fixes

  1. Reset enableConsoleColoring = true before each help_options test
  2. Save/restore the global in FscOptionTests after exercising --consolecolors-
  3. Longer term: eliminate the global mutable by threading the setting through TcConfigBuilder</issue_description>

Comments on the Issue (you are @copilot in this section)

Custom agent used: F# agent
Generic agent for F# coding following the coding guidelines of F# from MsLearn


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

… global

The `fsc --consolecolors switch` test mutates the global
`enableConsoleColoring` via `--consolecolors-`. When help_options tests
run after this test, the help output says "(off by default)" instead of
"(on by default)", causing baseline mismatches.

Fix: save and restore `enableConsoleColoring` around the test.

Co-authored-by: T-Gro <46543583+T-Gro@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix flaky test: help options baseline fails due to global mutable enableConsoleColoring Fix flaky help_options test: restore enableConsoleColoring after mutation Mar 3, 2026
Copilot finished work on behalf of T-Gro March 3, 2026 15:26
@T-Gro
Copy link
Member

T-Gro commented Mar 4, 2026

@copilot : Check AzDo CI errors. Collect them, fix them.

Copy link
Contributor Author

Copilot AI commented Mar 4, 2026

@copilot : Check AzDo CI errors. Collect them, fix them.

I cannot access Azure DevOps from this sandbox (DNS blocked for dev.azure.com). However, I ran the full CI-equivalent locally:

./eng/build.sh --configuration Release --testcoreclr

Results — all 5 test suites pass with 0 failures:

  • FSharp.Compiler.ComponentTests: 4891 passed ✅
  • FSharp.Compiler.Service.Tests: 102 passed ✅
  • FSharp.Compiler.Private.Scripting.UnitTests: passed ✅
  • FSharp.Build.UnitTests: 42 passed ✅
  • FSharp.Core.UnitTests: 6069 passed ✅

Formatting also passes (dotnet fantomas --check).

If there are specific AzDo errors you're seeing, could you share the error messages? I'm unable to reach the AzDo build logs from this environment.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • dev.azure.com
    • Triggering command: /usr/bin/curl curl -s REDACTED$top=5 (dns block)
    • Triggering command: /usr/bin/curl curl -v REDACTED$top=5 (dns block)
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (dns block)
  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh pr checks 19385 --repo dotnet/fsharp --json name,link,state (http block)
    • Triggering command: /usr/bin/gh gh pr checks 19385 --json name,state,link (http block)
  • https://api.github.com/repos/dotnet/fsharp/commits/a283880c6595af059cb7419cbaec78b8f4dffcaa/check-runs
    • Triggering command: /usr/bin/gh gh api repos/dotnet/fsharp/commits/a283880c6595af059cb7419cbaec78b8f4dffcaa/check-runs --jq .check_runs[] | {name: .name, status: .status, conclusion: .conclusion, details_url: .details_url} INVALID,NEW -j DROP (http block)
    • Triggering command: /usr/bin/gh gh api repos/dotnet/fsharp/commits/a283880c6595af059cb7419cbaec78b8f4dffcaa/check-runs (http block)
    • Triggering command: /home/REDACTED/work/_temp/ghcca-node/node/bin/node /home/REDACTED/work/_temp/ghcca-node/node/bin/node --enable-source-maps /home/REDACTED/work/_temp/copilot-developer-action-main/dist/index.js (http block)
  • https://api.github.com/repos/dotnet/fsharp/commits/a283880c6595af059cb7419cbaec78b8f4dffcaa/status
    • Triggering command: /usr/bin/gh gh api repos/dotnet/fsharp/commits/a283880c6595af059cb7419cbaec78b8f4dffcaa/status (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of T-Gro March 4, 2026 10:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

Flaky test: help_options baseline fails due to global mutable enableConsoleColoring

2 participants