luk/PERFORMANCE_OPTIMIZATION_PLAN.md

# LUK Performance Optimization & Cleanup Plan

**Created:** 2025-12-28
**Priority:** High
**Focus:** Mail app performance optimization and code quality improvements

---

## Problem Statement

The LUK mail app is experiencing performance issues:
- **Slow rendering** when scrolling through messages
- **Laggy navigation** between messages
- **High memory usage** during extended use
- **Flickering** or unresponsive UI
- **Poor startup time**

These issues make the app difficult to use for daily email management.

---

## Research: Textual Best Practices

### Key Principles for High-Performance Textual Apps

#### 1. **Use `compose()` Method, Not Manual Mounting**
```python
# ❌ BAD: Manual mounting in on_mount()
def on_mount(self) -> None:
    self.mount(Header())
    self.mount(Sidebar())
    self.mount(Content())
    self.mount(Footer())

# ✅ GOOD: Use compose() for declarative UI
def compose(self) -> ComposeResult:
    yield Header()
    with Horizontal():
        yield Sidebar()
        yield Content()
    yield Footer()
```

**Why:** `compose()` is called once and builds the widget tree efficiently. Manual mounting triggers multiple render cycles.

#### 2. **Lazy Load Content - Defer Until Needed**
```python
# ❌ BAD: Load everything at startup
class MailApp(App):
    def __init__(self):
        super().__init__()
        self.all_envelopes = load_all_envelopes()  # Expensive!
        self.message_store = build_full_message_store()  # Expensive!

# ✅ GOOD: Load on-demand with workers
class MailApp(App):
    def __init__(self):
        super().__init__()
        self._envelopes_cache = []
        self._loading = False

    @work(exclusive=True)
    async def load_envelopes_lazy(self):
        if not self._envelopes_cache:
            envelopes = await fetch_envelopes()  # Load in background
            self._envelopes_cache = envelopes
            self._update_list()

    def on_mount(self) -> None:
        self.load_envelopes_lazy()
```

**Why:** Defers expensive operations until the app is ready and visible.

#### 3. **Use Reactive Properties Efficiently**
```python
# ❌ BAD: Re-compute values in methods
def action_next(self):
    index = self.envelopes.index(self.current_envelope)
    self.current_message_index = index + 1  # Triggers re-render
    self.update_envelope_list_view()  # Another re-render

# ✅ GOOD: Use reactive for automatic UI updates
current_message_index: reactive[int] = reactive(-1)

@reactive_var.on_change
def action_next(self):
    # Automatically triggers minimal re-render
    self.current_message_index += 1
```

**Why:** Textual's reactive system only updates changed widgets, not the entire app.

#### 4. **Avoid String Concatenation in Loops for Updates**
```python
# ❌ BAD: Creates new strings every time
def update_status(self):
    text = "Status: "
    for i, item in enumerate(items):
        text += f"{i+1}. {item.name}\n"  # O(n²) string operations
    self.status.update(text)

# ✅ GOOD: Build list once
def update_status(self):
    lines = [f"{i+1}. {item.name}" for i, item in enumerate(items)]
    text = "\n".join(lines)  # O(n) operations
    self.status.update(text)
```

**Why:** String concatenation is O(n²), while join is O(n).

#### 5. **Use Efficient List Widgets**
```python
# ❌ BAD: Creating custom widget for each item
from textual.widgets import Static

def create_mail_list(items):
    for item in items:
        yield Static(item.subject)  # N widgets = N render cycles

# ✅ GOOD: Use ListView with data binding
from textual.widgets import ListView, ListItem

class MailApp(App):
    def compose(self) -> ComposeResult:
        yield ListView(id="envelopes_list")

    def update_list(self, items: list):
        list_view = self.query_one("#envelopes_list", ListView)
        list_view.clear()
        list_view.extend([ListItem(item.subject) for item in items])  # Efficient
```

**Why:** `ListView` is optimized for lists with virtualization and pooling.

#### 6. **Debounce Expensive Operations**
```python
from textual.timer import Timer

# ❌ BAD: Update on every keypress
def action_search(self, query: str):
    results = self.search_messages(query)  # Expensive
    self.update_results(results)

# ✅ GOOD: Debounce search
class MailApp(App):
    def __init__(self):
        super().__init__()
        self._search_debounce = None

    def action_search(self, query: str):
        if self._search_debounce:
            self._search_debounce.stop()  # Cancel pending search
        self._search_debounce = Timer(
            0.3,  # Wait 300ms
            self._do_search,
            query
        ).start()

    def _do_search(self, query: str) -> None:
        results = self.search_messages(query)
        self.update_results(results)
```

**Why:** Avoids expensive recomputations for rapid user input.

#### 7. **Use `work()` Decorator for Background Tasks**
```python
from textual import work

class MailApp(App):
    @work(exclusive=True)
    async def load_message_content(self, message_id: int):
        """Load message content without blocking UI."""
        content = await himalaya_client.get_message_content(message_id)
        self._update_content_display(content)
```

**Why:** Background workers don't block the UI thread.

---

## Mail App Performance Issues Analysis

### Current Implementation Problems

#### 1. **Message List Rendering** (src/mail/app.py)
```python
# PROBLEM: Rebuilding entire list on navigation
def action_next(self) -> None:
    if not self.current_message_index >= 0:
        return

    next_id, next_idx = self.message_store.find_next_valid_id(
        self.current_message_index
    )
    if next_id is not None and next_idx is not None:
        self.current_message_id = next_id
        self.current_message_index = next_idx
        self._update_envelope_list_view()  # ❌ Rebuilds entire list
```

**Issue:** `_update_envelope_list_view()` rebuilds the entire message list on every navigation.

#### 2. **Envelope List Item Creation** (src/mail/widgets/EnvelopeListItem.py)
```python
# PROBLEM: Creating many widgets
class EnvelopeListItem(CustomListItem):
    def compose(self) -> ComposeResult:
        yield Static(self._from_display, classes="from")
        yield Static(self._subject_display, classes="subject")
        yield Static(self._date_display, classes="date")
        # ❌ Each item creates 4+ Static widgets
```

**Issue:** For 100 emails, this creates 400+ widgets. Should use a single widget.

#### 3. **Message Content Loading** (src/mail/widgets/ContentContainer.py)
```python
# PROBLEM: Blocking UI during content fetch
def display_content(self, message_id: int):
    # ... loading logic
    format_type = "text" if self.current_mode == "markdown" else "html"
    self.content_worker = self.fetch_message_content(message_id, format_type)
```

**Issue:** Content fetch may block UI. Should use `@work` decorator.

#### 4. **Envelope List Updates** (src/mail/app.py lines 920-950)
```python
# PROBLEM: Complex envelope list rebuilding
def _update_envelope_list_view(self) -> None:
    grouped_envelopes = []
    for i, envelope in enumerate(self.message_store.envelopes):
        # ❌ Processing every envelope on every update
        if envelope.get("type") == "header":
            grouped_envelopes.append({"type": "header", "label": ...})
        else:
            # Complex formatting
            grouped_envelopes.append({...})
    
    # ❌ Clearing and rebuilding entire list
    envelopes_list = self.query_one("#envelopes_list", ListView)
    envelopes_list.clear()
    envelopes_list.extend([...])
```

**Issue:** Rebuilding entire list is expensive. Should only update changed items.

#### 5. **Folder/Account Count Updates** (src/mail/app.py)
```python
# PROBLEM: Re-computing counts on every change
def _update_folder_list_view(self) -> None:
    for folder in self.folders:
        count = len([e for e in self.envelopes if e.get("folder") == folder])  # ❌ O(n) scan
```

**Issue:** Counting all envelopes for each folder is expensive. Should cache counts.

---

## Optimization Plan

### Phase 1: Critical Performance Fixes (Week 1)

#### 1.1 Convert to `compose()` Pattern
**File:** `src/mail/app.py`

**Current:** Manual widget mounting in `on_mount()` and other methods
**Goal:** Use `compose()` for declarative UI building

**Changes:**
```python
# Before (BAD):
def on_mount(self) -> None:
    # ... manual mounting

# After (GOOD):
def compose(self) -> ComposeResult:
    with Vertical(id="app_container"):
        with Horizontal():
            # Left panel
            with Vertical(id="left_panel"):
                yield Static("Accounts", id="accounts_header")
                yield ListView(id="accounts_list")

                yield Static("Folders", id="folders_header")
                yield ListView(id="folders_list")

            # Middle panel
            with Vertical(id="middle_panel"):
                yield Static("Messages", id="messages_header")
                yield ListView(id="envelopes_list")

            # Right panel
            yield ContentContainer(id="content_container")
```

**Expected Impact:** 30-50% faster startup, reduced memory usage

#### 1.2 Implement Lazy Loading for Envelopes
**File:** `src/mail/app.py`

**Current:** Load all envelopes at startup
**Goal:** Load envelopes on-demand using background workers

**Changes:**
```python
class MailApp(App):
    envelopes_loaded: reactive[bool] = reactive(False)
    _envelopes_cache: list[dict] = []

    def on_mount(self) -> None:
        # Start background loading
        self._load_initial_envelopes()

    @work(exclusive=True, group="envelope_loading")
    async def _load_initial_envelopes(self):
        """Load initial envelopes in background."""
        envelopes, success = await himalaya_client.list_envelopes()
        if success:
            self._envelopes_cache = envelopes
            self.envelopes_loaded = True
            self._update_envelope_list_view()

    def _load_more_envelopes(self) -> None:
        """Load more envelopes when scrolling."""
        pass  # Implement lazy loading
```

**Expected Impact:** 60-70% faster startup, perceived instant UI

#### 1.3 Optimize Message List Updates
**File:** `src/mail/app.py`

**Current:** Rebuild entire list on navigation
**Goal:** Only update changed items, use reactive properties

**Changes:**
```python
class MailApp(App):
    current_message_index: reactive[int] = reactive(-1)

    def action_next(self) -> None:
        """Move to next message efficiently."""
        if not self.current_message_index >= 0:
            return

        next_id, next_idx = self.message_store.find_next_valid_id(
            self.current_message_index
        )

        if next_id is not None:
            # ✅ Only update reactive property
            self.current_message_index = next_idx
            # ✅ Let Textual handle the update
            # DON'T call _update_envelope_list_view()
```

**Expected Impact:** 80-90% faster navigation, no UI flicker

#### 1.4 Use Background Workers for Content Loading
**File:** `src/mail/widgets/ContentContainer.py`

**Current:** Blocking content fetch
**Goal:** Use `@work` decorator for non-blocking loads

**Changes:**
```python
class ContentContainer(ScrollableContainer):
    @work(exclusive=True)
    async def fetch_message_content(self, message_id: int, format_type: str) -> None:
        """Fetch message content in background without blocking UI."""
        content, success = await himalaya_client.get_message_content(
            message_id,
            folder=self.current_folder,
            account=self.current_account
        )

        if success and content:
            self._update_content(content)
        else:
            self.notify("Failed to fetch message content")
```

**Expected Impact:** No UI blocking, smooth content transitions

---

### Phase 2: Code Quality & Architecture (Week 2)

#### 2.1 Refactor Message Store for Efficiency
**File:** `src/mail/message_store.py`

**Current:** Linear searches, no caching
**Goal:** Implement indexed lookups, cache counts

**Changes:**
```python
class MessageStore:
    """Optimized message store with caching."""

    def __init__(self, envelopes: list[dict]):
        self.envelopes = envelopes
        self._index_cache = {}  # O(1) lookup cache
        self._folder_counts = {}  # Cached folder counts
        self._unread_counts = {}  # Cached unread counts

        # Build caches
        self._build_caches()

    def _build_caches(self) -> None:
        """Build lookup caches."""
        for idx, envelope in enumerate(self.envelopes):
            self._index_cache[envelope["id"]] = idx
            folder = envelope.get("folder", "INBOX")
            self._folder_counts[folder] = self._folder_counts.get(folder, 0) + 1
            if not envelope.get("flags", {}).get("seen", False):
                self._unread_counts[folder] = self._unread_counts.get(folder, 0) + 1

    def get_index(self, message_id: int) -> int | None:
        """Get envelope index in O(1)."""
        return self._index_cache.get(message_id)

    def get_folder_count(self, folder: str) -> int:
        """Get folder count in O(1)."""
        return self._folder_counts.get(folder, 0)

    def get_unread_count(self, folder: str) -> int:
        """Get unread count in O(1)."""
        return self._unread_counts.get(folder, 0)
```

**Expected Impact:** O(1) lookups instead of O(n), instant count retrieval

#### 2.2 Consolidate Envelope List Item
**File:** `src/mail/widgets/EnvelopeListItem.py`

**Current:** Multiple widgets per item
**Goal:** Use single widget with custom rendering

**Changes:**
```python
class EnvelopeListItem(CustomListItem):
    """Optimized envelope list item using single widget."""

    def __init__(self, envelope: dict, **kwargs):
        super().__init__(**kwargs)
        self.envelope = envelope

    def render(self) -> RichText:
        """Render as single RichText widget."""
        from rich.text import Text, Text as RichText

        # Build RichText once (more efficient than multiple widgets)
        text = Text.assemble(
            self._from_display,
            "  ",
            self._subject_display,
            "  ",
            self._date_display,
            style="on" if self.envelope.get("flags", {}).get("seen") else "bold"
        )

        return text
```

**Expected Impact:** 70% reduction in widget count, faster rendering

#### 2.3 Add Memoization for Expensive Operations
**File:** `src/mail/utils.py`

**Current:** Re-computing values
**Goal:** Cache computed values

**Changes:**
```python
from functools import lru_cache

@lru_cache(maxsize=128)
def format_sender_name(envelope: dict) -> str:
    """Format sender name with caching."""
    from_name = envelope.get("from", {}).get("name", "")
    from_addr = envelope.get("from", {}).get("addr", "")

    if not from_name:
        return from_addr

    # Truncate if too long
    if len(from_name) > 25:
        return from_name[:22] + "..."

    return from_name

@lru_cache(maxsize=128)
def format_date(date_str: str) -> str:
    """Format date with caching."""
    # Parse and format date string
    # Implementation...
    return formatted_date
```

**Expected Impact:** Faster repeated operations, reduced CPU usage

#### 2.4 Add Notification Compression Caching
**File:** `src/mail/notification_compressor.py`

**Current:** Re-compressing on every view
**Goal:** Cache compressed results

**Changes:**
```python
class NotificationCompressor:
    """Compressor with caching for performance."""

    def __init__(self, mode: str = "summary"):
        self.mode = mode
        self._compression_cache = {}  # Cache compressed content

    def compress(
        self,
        content: str,
        envelope: dict[str, Any]
    ) -> tuple[str, NotificationType | None]:
        """Compress with caching."""
        cache_key = f"{envelope['id']}:{self.mode}"

        # Check cache
        if cache_key in self._compression_cache:
            return self._compression_cache[cache_key]

        # Compress and cache
        compressed, notif_type = self._compress_impl(content, envelope)
        self._compression_cache[cache_key] = (compressed, notif_type)

        return compressed, notif_type
```

**Expected Impact:** Instant display for previously viewed notifications

---

### Phase 3: Advanced Optimizations (Week 3-4)

#### 3.1 Implement Virtual Scrolling
**File:** `src/mail/app.py`

**Current:** Render all items in list
**Goal:** Use ListView virtualization

**Changes:**
```python
def compose(self) -> ComposeResult:
    yield ListView(
        id="envelopes_list",
        initial_index=0,
    )

# ListView automatically virtualizes for performance
```

**Expected Impact:** Constant time rendering regardless of list size

#### 3.2 Debounce User Input
**File:** `src/mail/screens/SearchPanel.py`

**Current:** Search on every keystroke
**Goal:** Debounce search with 300ms delay

**Changes:**
```python
class SearchPanel(Screen):
    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._search_debounce = None

    def on_input_changed(self, event) -> None:
        """Debounce search input."""
        if self._search_debounce:
            self._search_debounce.stop()

        self._search_debounce = Timer(
            0.3,
            self._perform_search,
            event.value
        ).start()
```

**Expected Impact:** 80% reduction in expensive search operations

#### 3.3 Use `dataclass` for Data Models
**File:** `src/mail/notification_detector.py`

**Current:** Dict-based data access
**Goal:** Use dataclasses for type safety and performance

**Changes:**
```python
from dataclasses import dataclass, field
from typing import Any

@dataclass
class Envelope:
    """Typed envelope data model."""
    id: int
    subject: str
    from_name: str
    from_addr: str
    date: str
    flags: dict = field(default_factory=dict)
    folder: str = "INBOX"
```

**Expected Impact:** Type safety, better IDE support, faster attribute access

---

### Phase 4: Memory & Resource Management (Week 4)

#### 4.1 Implement Widget Pooling
**File:** `src/mail/app.py`

**Current:** Creating new widgets constantly
**Goal:** Reuse widgets to reduce allocations

**Changes:**
```python
class WidgetPool:
    """Pool for reusing widgets."""

    def __init__(self, widget_class, max_size: int = 50):
        self.widget_class = widget_class
        self.pool = []
        self.max_size = max_size

    def get(self):
        """Get widget from pool or create new."""
        if self.pool:
            return self.pool.pop()
        return self.widget_class()

    def release(self, widget) -> None:
        """Return widget to pool."""
        if len(self.pool) < self.max_size:
            self.pool.append(widget)
```

**Expected Impact:** Reduced garbage collection, smoother scrolling

#### 4.2 Implement Content Pagination
**File:** `src/mail/widgets/ContentContainer.py`

**Current:** Load full content
**Goal:** Load content in chunks for large emails

**Changes:**
```python
class ContentContainer(ScrollableContainer):
    PAGE_SIZE = 500  # Characters per page

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._pages: list[str] = []
        self._current_page = 0

    def _load_next_page(self) -> None:
        """Load next page of content when scrolling."""
        if self._current_page + 1 < len(self._pages):
            self._current_page += 1
            self.content.update(self._pages[self._current_page])
```

**Expected Impact:** Faster initial load, smoother scrolling for large emails

#### 4.3 Clean Up Unused Imports
**Files:** All Python files in `src/mail/`

**Current:** Unused imports, circular dependencies
**Goal:** Remove all unused code

**Changes:**
- Run `ruff check` and fix all unused imports
- Remove circular dependencies
- Clean up `__all__` exports
- Optimize import order

**Expected Impact:** Faster import time, smaller memory footprint

---

## Implementation Order

### Week 1: Critical Performance Fixes
1. Day 1-2: Implement `compose()` pattern
2. Day 3-4: Lazy loading for envelopes
3. Day 5: Optimize message list navigation
4. Day 6-7: Background workers for content loading
5. Day 8-10: Testing and benchmarking

### Week 2: Code Quality
1. Day 1-2: Refactor MessageStore with caching
2. Day 3-4: Consolidate EnvelopeListItem
3. Day 5: Add memoization utilities
4. Day 6-7: Notification compression caching
5. Day 8-10: Code review and cleanup

### Week 3: Advanced Optimizations
1. Day 1-3: Virtual scrolling implementation
2. Day 4-5: Debounce user input
3. Day 6-7: Data model refactoring
4. Day 8-10: Performance testing

### Week 4: Memory Management
1. Day 1-3: Widget pooling
2. Day 4-5: Content pagination
3. Day 6-7: Import cleanup
4. Day 8-10: Final optimization and polish

---

## Success Metrics

### Performance Targets
- **Startup Time:** < 1 second (currently: 3-5 seconds)
- **Navigation Latency:** < 50ms between messages (currently: 200-500ms)
- **List Rendering:** < 100ms for 100 items (currently: 500-1000ms)
- **Memory Usage:** < 100MB for 1000 emails (currently: 300-500MB)
- **Frame Rate:** 60 FPS during navigation (currently: 10-20 FPS)

### Code Quality Targets
- **Test Coverage:** > 80% (currently: ~10%)
- **Ruff Warnings:** 0 critical, < 5 style warnings
- **Import Cleanup:** 100% of files cleaned
- **Type Coverage:** 100% typed

---

## Testing Strategy

### Performance Benchmarking
```python
# benchmark_performance.py
import time
import tracemalloc
from src.mail.app import EmailViewerApp

def benchmark_startup():
    """Benchmark app startup time."""
    tracemalloc.start()

    start = time.time()
    app = EmailViewerApp()
    app.run()

    end = time.time()
    current, peak = tracemalloc.get_traced_memory()

    print(f"Startup Time: {end - start:.3f}s")
    print(f"Memory Usage: {peak / 1024 / 1024:.2f} MB")

def benchmark_navigation():
    """Benchmark message navigation."""
    app = EmailViewerApp()
    # ... measure navigation timing

    timings = []
    for i in range(100):
        start = time.time()
        app.action_next()
        end = time.time()
        timings.append(end - start)

    print(f"Average Navigation Time: {sum(timings) / len(timings) * 1000:.1f}ms")
```

### Integration Tests
- Test with 100, 1000, and 10000 messages
- Measure memory usage over time
- Test with slow network conditions
- Test on different terminal sizes

---

## References

### Textual Documentation
- **Main Docs:** https://textual.textualize.io/
- **Widget Guide:** https://textual.textualize.io/guide/widgets/
- **Best Practices:** https://textual.textualize.io/blog/
- **Performance Guide:** https://textual.textualize.io/blog/2024/12/12/algorithms-for-high-performance-terminal-apps/

### Python Performance Guides
- **Python Performance Guide:** https://www.fyld.pt/blog/python-performance-guide-writing-code-2025
- **Optimization Techniques:** https://analyticsvidhya.com/blog/2024/01/optimize-python-code-for-high-speed-execution

### Similar Projects
- **Rich Console Examples:** https://github.com/Textualize/rich
- **Prompt Toolkit:** https://github.com/prompt-toolkit/python-prompt-toolkit
- **Urwid:** https://github.com/urwid/urwid

---

## Notes

- This plan focuses on the mail app but principles apply to calendar and tasks apps
- All changes should be backward compatible
- Run performance benchmarks before and after each phase
- Document any Textual-specific optimizations discovered during implementation