Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
4afeb9b
Initial plan
Copilot Jan 23, 2026
c979d8c
Implement core Electron architecture with overlay and chat windows
Copilot Jan 23, 2026
1c0d40e
Add comprehensive documentation and fine grid support
Copilot Jan 23, 2026
6445155
Address code review feedback: remove unused deps, improve error handling
Copilot Jan 23, 2026
402a0ca
Fix grid labeling logic and improve code organization
Copilot Jan 23, 2026
cc985ed
Add comprehensive implementation summary
Copilot Jan 23, 2026
0affc49
Add quick start guide and project status documentation
Copilot Jan 23, 2026
6345882
Add final visual summary
Copilot Jan 23, 2026
148bc7b
Merge pull request #1 from TayDa64/copilot/implement-headless-overlay…
TayDa64 Jan 24, 2026
2b56488
Add transparent overlay proof screenshot
TayDa64 Jan 24, 2026
d6ac0ba
Initial commit for Liku AI System monorepo
TayDa64 Jan 24, 2026
5a32167
feat: Fix overlay fine dots, improve key bindings, fix chat window
TayDa64 Jan 25, 2026
a4c33ab
feat: Fix overlay keyboard shortcuts using globalShortcut API
TayDa64 Jan 25, 2026
8ca66b6
feat: Add GitHub Copilot OAuth integration
TayDa64 Jan 25, 2026
aa3648f
fix: Change single-key shortcuts to Ctrl+Alt modifiers
TayDa64 Jan 26, 2026
05ce620
Fix overlay initialization crash and coordinate drift, add system aut…
TayDa64 Jan 27, 2026
4c853f4
Add shared grid math and local fine grid; expand PR template
TayDa64 Jan 27, 2026
630d93d
Address review comments and unused code
TayDa64 Jan 28, 2026
6780b49
Initial plan
Copilot Jan 28, 2026
937de87
Add inspect overlay layer with region boxes, tooltip, and AI integration
Copilot Jan 28, 2026
e17acaf
Fix code review issues: bounds checks, deprecations, error handling, …
Copilot Jan 28, 2026
1e58c7d
Merge pull request #3 from TayDa64/copilot/add-inspect-overlay-feature
TayDa64 Jan 28, 2026
b64d116
refactor(ui-automation): modularize into 21 files for extensibility
TayDa64 Jan 28, 2026
d89840d
Add CLI commands, UI automation enhancements, and baseline app docs
TayDa64 Jan 29, 2026
1e49da2
Add multi-agent system with Supervisor-Builder-Verifier architecture
TayDa64 Jan 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# Precision Grounding + Inspect Overlay (Opus Execution Plan)

## Summary
- Align grid math across overlay, main, and AI prompts using shared constants.
- Add local fine grid around the cursor for precise targeting without full-grid noise.
- Introduce devtools-style inspect overlays (actionable element boxes + metadata).
- Ensure AI uses the same visual grounding as the user.

## Goals / Non-Goals
**Goals**
- User and AI see the same targeting primitives (grid + inspect metadata).
- Fine precision selection without needing full fine-grid visibility.
- Deterministic coordinate mapping across renderer/main/AI prompt.

**Non-Goals**
- Full external app DOM access (we rely on OCR + visual detection).
- Replacing the grid system entirely.

## Problem
- Fine dots do not appear around the cursor, preventing high-precision selection.
- AI coordinate grounding drifts due to mismatched math across modules.
- AI lacks the same visualization/inspection context the user sees.

## Approach
1) Shared grid math module used by renderer, main, and AI prompt.
2) Local fine-grid rendering around cursor in selection mode.
3) Inspect layer backed by visual-awareness to surface actionable regions.
4) AI prompt + action executor aligned to overlay math and inspect metadata.

## Key Changes (Planned)
- `src/shared/grid-math.js`: canonical grid constants + label → pixel conversion.
- `src/renderer/overlay/overlay.js`: local fine-grid render + shared math usage.
- `src/renderer/overlay/preload.js`: expose grid math to renderer safely.
- `src/main/system-automation.js`: unify coordinate mapping.
- `src/main/ai-service.js`: ground prompts + fine label support.
- `src/main/index.js`: optional inspect toggle + overlay commands.
- `src/main/visual-awareness.js`: actionable element detection + metadata surface.

## Implementation Plan
**Phase 1: Grounding & Precision**
- [x] Shared grid math module and renderer/main integration.
- [x] Local fine-grid around cursor with snap highlight.
- [ ] Add label→pixel IPC from main to overlay to guarantee exact mapping.
- [ ] Add fine label rendering on hover (C3.12) in local grid.

**Phase 2: Inspect Overlay (Devtools‑Style)**
- [ ] Add inspect toggle command and UI indicator.
- [ ] Visual-awareness pass: actionable region detection (buttons, inputs, links).
- [ ] Overlay layer draws bounding boxes + tooltip with text/role/confidence.
- [ ] Selection handoff: click through to element center.

**Phase 3: AI Grounding + Action Execution**
- [ ] Include inspect metadata + screen size in AI context.
- [ ] Prefer inspect targets; fallback to grid only if needed.
- [ ] Add “precision click” action with safety confirmation.

## UX Notes
- Inspect mode should be visually distinct (e.g., cyan boxes, tooltip anchored).
- Local fine grid should fade in/out smoothly and never block click-through.
- Keep overlays under 16ms frame budget; throttle redraw to pointer move.

## Testing
**Unit**
- Grid label conversions (coarse + fine).
- Shared constants remain consistent across renderer/main/AI.

**Manual**
- Cursor-local fine dots appear in selection mode and track cursor.
- Background click-through still works in both modes.
- Inspect overlay alignment with visible UI elements.

**Regression**
- Coarse grid rendering.
- Pulse effect visibility.
- Safety confirmation flow intact.

## Risks / Mitigations
- DPI scaling drift → use Electron `screen.getPrimaryDisplay().scaleFactor`.
- Performance → local fine grid only; throttled draw.
- Overlay click-through → hide overlay only at click execution.

## Observability / Debugging
- Add a debug overlay toggle for grid math readouts.
- Log label→pixel conversions when in inspect mode.
- Capture last 10 action targets in memory for post-mortem.

## Opus Notes (Websearch Required)
- Verify Electron overlay best practices (`setIgnoreMouseEvents` behavior).
- Validate DPI/scaling guidance for Windows and macOS.
- Check common patterns for devtools-like overlays.

## Checklist
- [ ] Shared grid math used everywhere (renderer, main, AI prompt).
- [ ] Local fine grid visible and performant.
- [ ] Inspect overlay works and aligns with AI context.
- [ ] AI actions target inspect regions with correct coordinates.
- [ ] Tests updated/added and passing.
Loading