bases/fjord
Watch
0
0
Fork
You've already forked fjord
 0
Code Issues 3 Pull requests Projects Releases Packages Wiki Activity

Window overview: a drawer page listing what's open on the desktop #3

New issue
Open
opened 2026-10-10 21:37:06 +02:00 by arne · 1 comment
arne commented 2026-10-10 21:37:06 +02:00
Owner
Copy link

This was generated by AI during triage.

Triage: enhancement, needs-triage (the repo has no labels yet). Not ready to start: the scope and the privacy trade-offs need deciding first.

Want

An overview of what's open on the desktop: every niri window, grouped by workspace in strip order. As far as possible it should also show what's in each window: every browser tab, and a synopsis of what a terminal is showing.

What we know (verified on the running system, niri unstable 2026-08-02)

niri gives us, per window: id, title, app_id, pid, workspace_id, is_focused, is_floating, is_urgent, focus_timestamp, and layout (pos_in_scrolling_layout = [column, row], tile and window sizes). The data comes from niri msg --json windows or the event stream Niri.qml already reads.

  • WindowsChanged arrives on connect. Updates should come from WindowOpenedOrChanged, WindowClosed and WindowLayoutsChanged, but none of those were observed yet. Check them on the wire before relying on them, given the WindowFocusChanged history in NOTES.md.
  • pid is shared: every Firefox window has the same pid, and so does every Ghostty window. /proc data is per app, not per window, for both.
  • niri can't tell us anything about what's inside a window.

Browser tabs:

  • Firefox writes every window and tab to ~/.mozilla/firefox/default/sessionstore-backups/recovery.jsonlz4 (mozlz4: mozLz40\0 + 4-byte size + an LZ4 block). It's refreshed about every 15 seconds. Decoding needs an LZ4 decoder, which isn't on the wrapper's PATH today.
  • Proposed way to tie it to niri: a niri Firefox window's title is <selected tab title> — Mozilla Firefox. Not yet confirmed.
  • Chromium's Sessions/Session_* files are an undocumented binary format.
  • A WebExtension with native messaging would give live tabs for both browsers. Firefox release requires signing it (unlisted AMO), and Chromium needs some way to load an unpacked extension.
  • Ruled out: a remote debugging port. Any local process could then drive the logged-in browser.

Terminal contents:

  • Ghostty has no API on Linux for reading what a window shows.
  • herdr does: herdr pane read <id> --source visible, and herdr agent list gives agent state. So realistically, only terminals running inside herdr can get a synopsis.
  • Not yet known: how a herdr pane maps to a specific Ghostty window. Check herdr api snapshot.
  • Thumbnails through niri msg action screenshot-window would also write every capture to the clipboard, and from there into clipboard history.

Open questions (why this needs more thought)

  1. Scope. Is this a plain window list (cheap, everything above the line is known), or the full "what's inside" version? Could the list ship first?
  2. Where it lives. A drawer page next to home? Its own key? Should it act on windows (focus, close) or only show them?
  3. Synopsis: LLM or not. herdr agent state plus last command and exit status (from a richer fish title) may be enough. An LLM summary (claude -p --model haiku, or the API) costs money and time.
  4. Privacy. A synopsis sends terminal screens to an API, and those can hold tokens, env output and private data. Should it be opt-in per pane? Redacted? A local model? Tab URLs and titles are sensitive too, even if they never leave the machine.
  5. Browser tab source. Firefox's session file (no install, ~15 s lag, Firefox only) or an extension (live, both browsers, signing and packaging work)?
  6. Architecture. A Python helper (fjordxy-windows, like fjordxy-settings.py) that joins niri, tabs, herdr and cached summaries into one JSON document, drawn by a thin QML page? Or a Niri.qml window mirror plus per-app providers in QML?

Spikes before deciding (all read-only)

  • Decode recovery.jsonlz4 and confirm the selected-tab title ↔ niri title match.
  • Run herdr api snapshot and see whether its panes can be tied to Ghostty windows.
  • Capture the event stream while opening and closing windows, changing a title and moving a column, and record the payloads in NOTES.md.
> *This was generated by AI during triage.* **Triage:** enhancement, needs-triage (the repo has no labels yet). Not ready to start: the scope and the privacy trade-offs need deciding first. ## Want An overview of what's open on the desktop: every niri window, grouped by workspace in strip order. As far as possible it should also show what's *in* each window: every browser tab, and a synopsis of what a terminal is showing. ## What we know (verified on the running system, niri unstable 2026-08-02) **niri gives us, per window:** `id`, `title`, `app_id`, `pid`, `workspace_id`, `is_focused`, `is_floating`, `is_urgent`, `focus_timestamp`, and `layout` (`pos_in_scrolling_layout` = [column, row], tile and window sizes). The data comes from `niri msg --json windows` or the event stream `Niri.qml` already reads. - `WindowsChanged` arrives on connect. Updates should come from `WindowOpenedOrChanged`, `WindowClosed` and `WindowLayoutsChanged`, but **none of those were observed yet**. Check them on the wire before relying on them, given the `WindowFocusChanged` history in NOTES.md. - `pid` is shared: every Firefox window has the same pid, and so does every Ghostty window. `/proc` data is per app, not per window, for both. - niri can't tell us anything about what's inside a window. **Browser tabs:** - Firefox writes every window and tab to `~/.mozilla/firefox/default/sessionstore-backups/recovery.jsonlz4` (mozlz4: `mozLz40\0` + 4-byte size + an LZ4 block). It's refreshed about every 15 seconds. Decoding needs an LZ4 decoder, which isn't on the wrapper's PATH today. - Proposed way to tie it to niri: a niri Firefox window's title is `<selected tab title> — Mozilla Firefox`. *Not yet confirmed.* - Chromium's `Sessions/Session_*` files are an undocumented binary format. - A WebExtension with native messaging would give live tabs for both browsers. Firefox release requires signing it (unlisted AMO), and Chromium needs some way to load an unpacked extension. - Ruled out: a remote debugging port. Any local process could then drive the logged-in browser. **Terminal contents:** - Ghostty has no API on Linux for reading what a window shows. - herdr does: `herdr pane read <id> --source visible`, and `herdr agent list` gives agent state. So realistically, only terminals running inside herdr can get a synopsis. - *Not yet known:* how a herdr pane maps to a specific Ghostty window. Check `herdr api snapshot`. - Thumbnails through `niri msg action screenshot-window` would also write every capture to the clipboard, and from there into clipboard history. ## Open questions (why this needs more thought) 1. **Scope.** Is this a plain window list (cheap, everything above the line is known), or the full "what's inside" version? Could the list ship first? 2. **Where it lives.** A drawer page next to `home`? Its own key? Should it act on windows (focus, close) or only show them? 3. **Synopsis: LLM or not.** herdr agent state plus last command and exit status (from a richer fish title) may be enough. An LLM summary (`claude -p --model haiku`, or the API) costs money and time. 4. **Privacy.** A synopsis sends terminal screens to an API, and those can hold tokens, `env` output and private data. Should it be opt-in per pane? Redacted? A local model? Tab URLs and titles are sensitive too, even if they never leave the machine. 5. **Browser tab source.** Firefox's session file (no install, ~15 s lag, Firefox only) or an extension (live, both browsers, signing and packaging work)? 6. **Architecture.** A Python helper (`fjordxy-windows`, like `fjordxy-settings.py`) that joins niri, tabs, herdr and cached summaries into one JSON document, drawn by a thin QML page? Or a `Niri.qml` window mirror plus per-app providers in QML? ## Spikes before deciding (all read-only) - Decode `recovery.jsonlz4` and confirm the selected-tab title ↔ niri title match. - Run `herdr api snapshot` and see whether its panes can be tied to Ghostty windows. - Capture the event stream while opening and closing windows, changing a title and moving a column, and record the payloads in NOTES.md.
arne commented 2026-10-10 23:53:36 +02:00
Author
Owner
Copy link

This was generated by AI during triage.

Direction change: terminal content comes from Rex

Superlogical's Rex will replace herdr (see #4), so the terminal half of this issue is designed against Rex, not herdr. That changes several things above:

  • Terminal status comes mainly from programs reporting it. Rex implements the Program Status Protocol (OSC 7501): a program reports idle / working / done / blocked (with a kind: permission, question or auth) / error, plus a one-line msg and optional progress. Rex delivers these as program_status_changed events. This mostly settles open questions 3 and 4: the program chooses what to disclose, and nothing leaves the machine. An LLM synopsis would only fill in for programs that don't report status. The herdr pane read route is dropped.
  • The shared-pid problem goes away for terminals. Rex reports each block's foreground process (name, cwd, pid) through terminal.process_changed, and directory changes through pwd_changed.
  • Live data: rex events --all --json prints one JSON object per line, like niri's stream.
  • OSC 7501's states are worth using for every row, not only Rex ones. In particular, done stays until the user has seen it, which fits how Notify.qml marks messages read.
  • New open question, the biggest one: how a niri window maps to a Rex session, window or block. It depends on what the Linux client turns out to be. Tracked in #4.
  • Proposed terminology (#4): a desktop is a session, and everything on it, terminal or GUI app, is a block. If that holds, this overview becomes a list of sessions and their blocks.

Rendering risk (applies whatever the source)

Window titles, tab titles and any msg are untrusted text. QML's Text defaults to AutoText, which renders anything that looks like HTML as rich text, so these fields need textFormat: Text.PlainText. OSC 7501 also asks that displayed status name the terminal it came from, so one program can't pose as another.

Spike list, updated

  • Dropped: herdr api snapshot.
  • Unchanged: decode Firefox's session file and confirm the title match, and capture niri window events on the wire.
  • New, once Rex is out for Linux: map niri windows to Rex blocks, and check which programs actually emit OSC 7501.
> *This was generated by AI during triage.* ## Direction change: terminal content comes from Rex Superlogical's Rex will replace herdr (see #4), so the terminal half of this issue is designed against Rex, not herdr. That changes several things above: - **Terminal status comes mainly from programs reporting it.** Rex implements the [Program Status Protocol (OSC 7501)](https://www.superlogical.com/rex/docs/build/program-status): a program reports `idle` / `working` / `done` / `blocked` (with a `kind`: `permission`, `question` or `auth`) / `error`, plus a one-line `msg` and optional `progress`. Rex delivers these as `program_status_changed` events. This mostly settles open questions 3 and 4: the program chooses what to disclose, and nothing leaves the machine. An LLM synopsis would only fill in for programs that don't report status. The `herdr pane read` route is dropped. - **The shared-pid problem goes away for terminals.** Rex reports each block's foreground process (`name`, `cwd`, `pid`) through `terminal.process_changed`, and directory changes through `pwd_changed`. - **Live data:** `rex events --all --json` prints one JSON object per line, like niri's stream. - **OSC 7501's states are worth using for every row,** not only Rex ones. In particular, `done` stays until the user has seen it, which fits how `Notify.qml` marks messages read. - **New open question, the biggest one: how a niri window maps to a Rex session, window or block.** It depends on what the Linux client turns out to be. Tracked in #4. - **Proposed terminology (#4):** a desktop is a session, and everything on it, terminal or GUI app, is a block. If that holds, this overview becomes a list of sessions and their blocks. ## Rendering risk (applies whatever the source) Window titles, tab titles and any `msg` are untrusted text. QML's `Text` defaults to `AutoText`, which renders anything that looks like HTML as rich text, so these fields need `textFormat: Text.PlainText`. OSC 7501 also asks that displayed status name the terminal it came from, so one program can't pose as another. ## Spike list, updated - Dropped: `herdr api snapshot`. - Unchanged: decode Firefox's session file and confirm the title match, and capture niri window events on the wire. - New, once Rex is out for Linux: map niri windows to Rex blocks, and check which programs actually emit OSC 7501.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
screen-recording
niri-wallpaper-in-overview
niri-kdl-escape-hatch
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
bases/fjord#3
No description provided.