feat: add vllm-metal

[doringeman]

You can build and install it manually using

make vllm-metal-build && make vllm-metal-install && MODEL_RUNNER_PORT=8080 make run

or, let DMR pull vllm-metal from Hub (https://hub.docker.com/layers/docker/model-runner/vllm-metal-v0.1.0-20260126-121650/images/sha256-9f4eeb45c168889baa14e935f5a9200bf1dcc6ab9d5be392fb1092a414fcc8cf from https://github.com/vllm-project/vllm-metal/releases/tag/v0.1.0-20260126-121650)

rm -rf ~/.docker/model-runner/vllm-metal && MODEL_RUNNER_PORT=8080 make run

or, install vllm-metal form local source (for vllm-metal development)

make vllm-metal-dev VLLM_METAL_PATH=../vllm-metal && MODEL_RUNNER_PORT=8080 make run

---

For running vllm-metal Python 3.12 is required, the same used for compiling the wheel.

brew install python@3.12

The vllm-metal files are downloaded and installed at ~/.docker/model-runner/vllm-metal.

---

Then, pull and use an MLX model.

MODEL_RUNNER_HOST=http://localhost:8080 docker model pull hf.co/mlx-community/llama-3.2-1b-instruct-4bit

curl -X POST http://localhost:8080/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
      "model": "hf.co/mlx-community/llama-3.2-1b-instruct-4bit",
      "messages": [{"role": "user", "content": "Hello, who are you?"}],
      "max_tokens": 512
    }'

At the moment vllm-metal doesn't seem to support streaming, so the CLI needs a tweak.

make -C cmd/cli install

MODEL_RUNNER_HOST=http://localhost:8080 docker model run hf.co/mlx-community/llama-3.2-1b-instruct-4bit "hi, who are you?"

[gemini-code-assist]

Summary of Changes

Hello @doringeman, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the model-runner's capabilities by integrating vllm-metal as a new inference backend, specifically targeting macOS ARM64 for optimized performance on Apple Silicon. It includes comprehensive tooling for building and installing this backend via new Makefile targets. Furthermore, the changes improve the robustness of the API client by enabling it to process both streaming and non-streaming responses, and enhance the model scheduling system to intelligently select the most appropriate backend based on platform and model format.

Highlights

• vllm-metal Backend Integration: Introduces a new backend for vllm-metal, specifically optimized for macOS ARM64 (Apple Silicon) to leverage Metal performance for accelerated inference.
• Makefile Enhancements: Adds new make targets (vllm-metal-build, vllm-metal-install, vllm-metal-clean) for streamlined management of the vllm-metal installation lifecycle.
• Flexible API Response Handling: Updates the API client to gracefully handle both streaming (Server-Sent Events) and non-streaming JSON responses from OpenAI-compatible LLM servers, improving compatibility.
• Intelligent Backend Prioritization: Modifies the model scheduling logic to prioritize vllm-metal for safetensors models on macOS, followed by MLX, and then vLLM or SGLang on Linux, ensuring optimal backend selection.
• Sandbox Configuration Flexibility: Adjusts the macOS sandbox creation to allow processes to run without sandboxing when no specific configuration is provided, accommodating the vllm-metal Python virtual environment.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces support for the vllm-metal backend, which is a great addition for Apple Silicon users. The changes are comprehensive, including Makefile targets for building and installing, the backend implementation itself, and necessary adjustments in the CLI and scheduler to integrate it. The approach of distributing the Python dependencies via a Docker image is clever. My review includes a few points for improvement: a bug fix for handling streaming responses, a suggestion to improve Makefile maintainability, and minor enhancements to error handling and security in the new backend code.

[sourcery-ai]

New security issues found

[doringeman]

None of these flagged security issues doesn't take user input. 🤔

[sourcery-ai]

Hey - I've found 4 security issues, and left some high level feedback:

Security issues:

• Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code. (link)
• Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code. (link)
• Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code. (link)
• Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code. (link)

General comments:

• The new non-streaming handling in ChatWithMessagesContext relies on the first line not starting with data: , which can be brittle for chunked or pretty-printed JSON responses; consider detecting streaming via headers (e.g., Content-Type or Transfer-Encoding) or attempting JSON decode first before falling back to SSE parsing.
• Python 3.12 detection and venv setup logic is now duplicated across the Makefile targets, vllmmetal backend (findSystemPython / downloadAndExtract), and build-vllm-metal-tarball.sh; consolidating this into a single reusable helper or clearly aligning the flows would reduce the risk of version skew or behavior drift.
• In sandbox_darwin.Create, when configuration is empty the process is no longer sandboxed but error messages still refer to a "sandboxed process"; updating the error text or differentiating the code path would make behavior clearer and easier to debug.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The new non-streaming handling in `ChatWithMessagesContext` relies on the first line not starting with `data: `, which can be brittle for chunked or pretty-printed JSON responses; consider detecting streaming via headers (e.g., `Content-Type` or `Transfer-Encoding`) or attempting JSON decode first before falling back to SSE parsing.
- Python 3.12 detection and venv setup logic is now duplicated across the Makefile targets, `vllmmetal` backend (`findSystemPython` / `downloadAndExtract`), and `build-vllm-metal-tarball.sh`; consolidating this into a single reusable helper or clearly aligning the flows would reduce the risk of version skew or behavior drift.
- In `sandbox_darwin.Create`, when `configuration` is empty the process is no longer sandboxed but error messages still refer to a "sandboxed process"; updating the error text or differentiating the code path would make behavior clearer and easier to debug.

## Individual Comments

### Comment 1
<location> `pkg/inference/backends/vllmmetal/vllmmetal.go:141` </location>
<code_context>
		out, err := exec.CommandContext(ctx, pythonPath, "--version").Output()
</code_context>

<issue_to_address>
**security (go.lang.security.audit.dangerous-exec-command):** Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code.

*Source: opengrep*
</issue_to_address>

### Comment 2
<location> `pkg/inference/backends/vllmmetal/vllmmetal.go:209` </location>
<code_context>
	venvCmd := exec.CommandContext(ctx, pythonPath, "-m", "venv", v.installDir)
</code_context>

<issue_to_address>
**security (go.lang.security.audit.dangerous-exec-command):** Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code.

*Source: opengrep*
</issue_to_address>

### Comment 3
<location> `pkg/inference/backends/vllmmetal/vllmmetal.go:274` </location>
<code_context>
	cmd := exec.CommandContext(ctx, v.pythonPath, "-c", "import vllm_metal")
</code_context>

<issue_to_address>
**security (go.lang.security.audit.dangerous-exec-command):** Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code.

*Source: opengrep*
</issue_to_address>

### Comment 4
<location> `pkg/sandbox/sandbox_darwin.go:127` </location>
<code_context>
		command = exec.CommandContext(ctx, name, arg...)
</code_context>

<issue_to_address>
**security (go.lang.security.audit.dangerous-exec-command):** Detected non-static command inside Command. Audit the input to 'exec.Command'. If unverified user data can reach this call site, this is a code injection vulnerability. A malicious actor can inject a malicious script to execute arbitrary code.

*Source: opengrep*
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[doringeman]

@sourcery-ai resolve

[doringeman]

@sourcery-ai dismiss

[ilopezluna]

I'm super excited with this one 👏

[ericcurtin]

This LGTM, want to answer @ilopezluna 's queries @doringeman ?

[doringeman]

@sourcery-ai dismiss

[ilopezluna]

this is super nice 🔥