TVMux Semantic Annotations via Custom Escape Sequences

This document outlines the design for a semantic annotation system using custom DCS (Device Control String) escape sequences to inject contextual metadata into terminal recordings.

Overview

The goal is to embed structured, machine-readable annotations into terminal recordings that provide rich semantic context for downstream analysis tools like bittty. These annotations are invisible to standard terminal emulators but preserved in asciinema cast files.

Technical Foundation

DCS (Device Control String) Format

We use DCS sequences for better terminal compatibility and structured parameter support:

ESC P tvmux://event-type?params ESC \

Why DCS over APC:

URL-Style Routing

The URL format provides:

Event Categories

Phase 1: Foundation Events (High Priority)

Basic Context

tvmux://timestamp?unix=1642123456&iso=2024-01-13T15:30:56Z
tvmux://pane?session=main&window=1&pane=2&active=true
tvmux://process?pid=12345&cmd=vim&args=file.txt&ppid=12300

Manual Annotations

tvmux://note?text=debugging%20auth%20issue&category=work&priority=high
tvmux://bell?from=main:1:3&reason=build_complete&urgent=false

System Events

tvmux://clipboard?action=copy&length=42&hash=sha256:abc123
tvmux://notification?app=slack&title=Message%20from%20Alice&urgent=false

Phase 2: Shell Integration (Medium Priority)

Command Lifecycle

tvmux://command/start?cmd=grep&args=-r%20pattern%20.&pid=12345&cwd=/home/user/project
tvmux://command/end?cmd=grep&exit_code=0&duration=1.2s&output_lines=15
tvmux://search/history?query=git%20log&matches=15&selected=git%20log%20--oneline

Context Changes

tvmux://cwd?from=/home/user&to=/home/user/project&method=cd
tvmux://git?branch=feature-xyz&repo=myproject&dirty=true&ahead=2&behind=0
tvmux://env?var=NODE_ENV&value=development&previous=production

Phase 3: Advanced Integration (Lower Priority)

Content & References

tvmux://link?url=https://example.com&context=opened_from_terminal&title=Example%20Site
tvmux://file?path=/etc/passwd&hash=sha256:abc123&semantic_id=config_file_123&type=system_config
tvmux://manpage?page=grep&section=1&context=referenced&query=regular%20expressions

Voice & Deferred Notes

tvmux://voice/start?note_id=uuid123&duration_estimate=30s&trigger=hotkey
tvmux://voice/end?note_id=uuid123&transcription_pending=true&duration=28.5s
tvmux://voice/transcribed?note_id=uuid123&text=remembered%20to%20fix%20auth%20bug&language=en

Advanced Context

tvmux://search/web?query=python%20async%20patterns&engine=google&results_count=10
tvmux://download?url=https://releases.ubuntu.com/file.iso&size=3.2GB&destination=/tmp
tvmux://package?action=install&name=nginx&manager=apt&version=1.18.0

Implementation Strategy

Phase 1: Core Infrastructure (Week 1)

  1. DCS Sequence Utilities 
    def encode_tvmux_annotation(event_type: str, **kwargs) -> str:
    """Encode semantic data as DCS sequence with URL format"""

def write_annotation_to_fifo(fifo_path: Path, event_type: str, **kwargs):
    """Write annotation to recording stream"""

def parse_tvmux_annotation(dcs_string: str) -> Dict[str, Any]:
    """Parse DCS back into structured data for analysis"""
  2. Configuration System 
    [annotations]
enabled = true
format = "url"  # Future: support json, keyvalue
timestamp_all_events = true

[annotations.events]
timestamp = true
pane = true
process = true
note = true
bell = true
clipboard = false      # Privacy-sensitive, off by default
notification = false   # Off by default

[annotations.clipboard]
max_length = 100
hash_content = true

[annotations.notification]
apps = ["slack", "discord"]  # Whitelist specific apps
urgent_only = true
  3. Integration Points
    • Pane switching in _dump_pane()
    • Tmux hook callbacks for system events
    • Recording start/stop events

Phase 2: Shell Integration (Weeks 2-3)

  1. PS1 Integration - Detect context changes via prompt hooks
  2. Command Wrappers - Override common commands with annotation functions: 
    # Shell function wrapper example
original_grep() { command grep "$@"; }
grep() {
    echo -e "\ePtvmux://command/start?cmd=grep&args=$(url_encode "$*")\e\\"
    local start_time=$(date +%s.%N)
    local exit_code
    original_grep "$@"
    exit_code=$?
    local duration=$(echo "$(date +%s.%N) - $start_time" | bc)
    echo -e "\ePtvmux://command/end?cmd=grep&exit_code=$exit_code&duration=${duration}s\e\\"
    return $exit_code
}
  3. Directory Change Detection - Hook cd, pushd, popd
  4. Git Branch Monitoring - PS1 integration + git command wrappers

Phase 3: Advanced Features (Week 4+)

  1. Desktop Integration
    • D-Bus monitoring for notifications (org.freedesktop.Notifications)
    • X11/Wayland clipboard event detection
    • XDG desktop file associations
  2. Voice Notes System
    • Hotkey integration (tmux key binding)
    • Audio recording to temporary files
    • Whisper transcription pipeline
    • Note ID linking and resolution
  3. TUI Application Hooks
    • LD_PRELOAD wrapper for ncurses functions
    • Python decorators for textual applications
    • Semantic markup injection at interaction points

Configuration Examples

Basic Setup (Privacy-Focused)

[annotations]
enabled = true

[annotations.events]
timestamp = true
pane = true
process = true
note = true
# All system integration disabled for privacy
clipboard = false
notification = false
command = false

Full Development Setup

[annotations]
enabled = true
timestamp_all_events = true

[annotations.events]
timestamp = true
pane = true
process = true
note = true
bell = true
command = true
cwd = true
git = true
clipboard = true
notification = true

[annotations.clipboard]
enabled = true
max_length = 200
hash_long_content = true

[annotations.notification]
enabled = true
apps = ["slack", "discord", "teams"]
urgent_only = false
include_body = false  # Privacy

Use Cases & Benefits

For bittty Terminal Player

For Analysis & Documentation Tools

For AI Training & Assistance

Technical Considerations

Performance

Privacy & Security

Compatibility

Future Extensions

Advanced Semantic Analysis

Integration Ecosystem

This design provides a foundation for incredibly rich, semantically-aware terminal recordings that could revolutionize how we analyze, replay, and learn from terminal sessions. The URL-based approach offers familiar semantics while maintaining extensibility and human readability.