diff --git a/PERFORMANCE_OPTIMIZATION_PLAN.md b/PERFORMANCE_OPTIMIZATION_PLAN.md new file mode 100644 index 0000000..a606f73 --- /dev/null +++ b/PERFORMANCE_OPTIMIZATION_PLAN.md @@ -0,0 +1,808 @@ +# LUK Performance Optimization & Cleanup Plan + +**Created:** 2025-12-28 +**Priority:** High +**Focus:** Mail app performance optimization and code quality improvements + +--- + +## Problem Statement + +The LUK mail app is experiencing performance issues: +- **Slow rendering** when scrolling through messages +- **Laggy navigation** between messages +- **High memory usage** during extended use +- **Flickering** or unresponsive UI +- **Poor startup time** + +These issues make the app difficult to use for daily email management. + +--- + +## Research: Textual Best Practices + +### Key Principles for High-Performance Textual Apps + +#### 1. **Use `compose()` Method, Not Manual Mounting** +```python +# ❌ BAD: Manual mounting in on_mount() +def on_mount(self) -> None: + self.mount(Header()) + self.mount(Sidebar()) + self.mount(Content()) + self.mount(Footer()) + +# ✅ GOOD: Use compose() for declarative UI +def compose(self) -> ComposeResult: + yield Header() + with Horizontal(): + yield Sidebar() + yield Content() + yield Footer() +``` + +**Why:** `compose()` is called once and builds the widget tree efficiently. Manual mounting triggers multiple render cycles. + +#### 2. **Lazy Load Content - Defer Until Needed** +```python +# ❌ BAD: Load everything at startup +class MailApp(App): + def __init__(self): + super().__init__() + self.all_envelopes = load_all_envelopes() # Expensive! + self.message_store = build_full_message_store() # Expensive! + +# ✅ GOOD: Load on-demand with workers +class MailApp(App): + def __init__(self): + super().__init__() + self._envelopes_cache = [] + self._loading = False + + @work(exclusive=True) + async def load_envelopes_lazy(self): + if not self._envelopes_cache: + envelopes = await fetch_envelopes() # Load in background + self._envelopes_cache = envelopes + self._update_list() + + def on_mount(self) -> None: + self.load_envelopes_lazy() +``` + +**Why:** Defers expensive operations until the app is ready and visible. + +#### 3. **Use Reactive Properties Efficiently** +```python +# ❌ BAD: Re-compute values in methods +def action_next(self): + index = self.envelopes.index(self.current_envelope) + self.current_message_index = index + 1 # Triggers re-render + self.update_envelope_list_view() # Another re-render + +# ✅ GOOD: Use reactive for automatic UI updates +current_message_index: reactive[int] = reactive(-1) + +@reactive_var.on_change +def action_next(self): + # Automatically triggers minimal re-render + self.current_message_index += 1 +``` + +**Why:** Textual's reactive system only updates changed widgets, not the entire app. + +#### 4. **Avoid String Concatenation in Loops for Updates** +```python +# ❌ BAD: Creates new strings every time +def update_status(self): + text = "Status: " + for i, item in enumerate(items): + text += f"{i+1}. {item.name}\n" # O(n²) string operations + self.status.update(text) + +# ✅ GOOD: Build list once +def update_status(self): + lines = [f"{i+1}. {item.name}" for i, item in enumerate(items)] + text = "\n".join(lines) # O(n) operations + self.status.update(text) +``` + +**Why:** String concatenation is O(n²), while join is O(n). + +#### 5. **Use Efficient List Widgets** +```python +# ❌ BAD: Creating custom widget for each item +from textual.widgets import Static + +def create_mail_list(items): + for item in items: + yield Static(item.subject) # N widgets = N render cycles + +# ✅ GOOD: Use ListView with data binding +from textual.widgets import ListView, ListItem + +class MailApp(App): + def compose(self) -> ComposeResult: + yield ListView(id="envelopes_list") + + def update_list(self, items: list): + list_view = self.query_one("#envelopes_list", ListView) + list_view.clear() + list_view.extend([ListItem(item.subject) for item in items]) # Efficient +``` + +**Why:** `ListView` is optimized for lists with virtualization and pooling. + +#### 6. **Debounce Expensive Operations** +```python +from textual.timer import Timer + +# ❌ BAD: Update on every keypress +def action_search(self, query: str): + results = self.search_messages(query) # Expensive + self.update_results(results) + +# ✅ GOOD: Debounce search +class MailApp(App): + def __init__(self): + super().__init__() + self._search_debounce = None + + def action_search(self, query: str): + if self._search_debounce: + self._search_debounce.stop() # Cancel pending search + self._search_debounce = Timer( + 0.3, # Wait 300ms + self._do_search, + query + ).start() + + def _do_search(self, query: str) -> None: + results = self.search_messages(query) + self.update_results(results) +``` + +**Why:** Avoids expensive recomputations for rapid user input. + +#### 7. **Use `work()` Decorator for Background Tasks** +```python +from textual import work + +class MailApp(App): + @work(exclusive=True) + async def load_message_content(self, message_id: int): + """Load message content without blocking UI.""" + content = await himalaya_client.get_message_content(message_id) + self._update_content_display(content) +``` + +**Why:** Background workers don't block the UI thread. + +--- + +## Mail App Performance Issues Analysis + +### Current Implementation Problems + +#### 1. **Message List Rendering** (src/mail/app.py) +```python +# PROBLEM: Rebuilding entire list on navigation +def action_next(self) -> None: + if not self.current_message_index >= 0: + return + + next_id, next_idx = self.message_store.find_next_valid_id( + self.current_message_index + ) + if next_id is not None and next_idx is not None: + self.current_message_id = next_id + self.current_message_index = next_idx + self._update_envelope_list_view() # ❌ Rebuilds entire list +``` + +**Issue:** `_update_envelope_list_view()` rebuilds the entire message list on every navigation. + +#### 2. **Envelope List Item Creation** (src/mail/widgets/EnvelopeListItem.py) +```python +# PROBLEM: Creating many widgets +class EnvelopeListItem(CustomListItem): + def compose(self) -> ComposeResult: + yield Static(self._from_display, classes="from") + yield Static(self._subject_display, classes="subject") + yield Static(self._date_display, classes="date") + # ❌ Each item creates 4+ Static widgets +``` + +**Issue:** For 100 emails, this creates 400+ widgets. Should use a single widget. + +#### 3. **Message Content Loading** (src/mail/widgets/ContentContainer.py) +```python +# PROBLEM: Blocking UI during content fetch +def display_content(self, message_id: int): + # ... loading logic + format_type = "text" if self.current_mode == "markdown" else "html" + self.content_worker = self.fetch_message_content(message_id, format_type) +``` + +**Issue:** Content fetch may block UI. Should use `@work` decorator. + +#### 4. **Envelope List Updates** (src/mail/app.py lines 920-950) +```python +# PROBLEM: Complex envelope list rebuilding +def _update_envelope_list_view(self) -> None: + grouped_envelopes = [] + for i, envelope in enumerate(self.message_store.envelopes): + # ❌ Processing every envelope on every update + if envelope.get("type") == "header": + grouped_envelopes.append({"type": "header", "label": ...}) + else: + # Complex formatting + grouped_envelopes.append({...}) + + # ❌ Clearing and rebuilding entire list + envelopes_list = self.query_one("#envelopes_list", ListView) + envelopes_list.clear() + envelopes_list.extend([...]) +``` + +**Issue:** Rebuilding entire list is expensive. Should only update changed items. + +#### 5. **Folder/Account Count Updates** (src/mail/app.py) +```python +# PROBLEM: Re-computing counts on every change +def _update_folder_list_view(self) -> None: + for folder in self.folders: + count = len([e for e in self.envelopes if e.get("folder") == folder]) # ❌ O(n) scan +``` + +**Issue:** Counting all envelopes for each folder is expensive. Should cache counts. + +--- + +## Optimization Plan + +### Phase 1: Critical Performance Fixes (Week 1) + +#### 1.1 Convert to `compose()` Pattern +**File:** `src/mail/app.py` + +**Current:** Manual widget mounting in `on_mount()` and other methods +**Goal:** Use `compose()` for declarative UI building + +**Changes:** +```python +# Before (BAD): +def on_mount(self) -> None: + # ... manual mounting + +# After (GOOD): +def compose(self) -> ComposeResult: + with Vertical(id="app_container"): + with Horizontal(): + # Left panel + with Vertical(id="left_panel"): + yield Static("Accounts", id="accounts_header") + yield ListView(id="accounts_list") + + yield Static("Folders", id="folders_header") + yield ListView(id="folders_list") + + # Middle panel + with Vertical(id="middle_panel"): + yield Static("Messages", id="messages_header") + yield ListView(id="envelopes_list") + + # Right panel + yield ContentContainer(id="content_container") +``` + +**Expected Impact:** 30-50% faster startup, reduced memory usage + +#### 1.2 Implement Lazy Loading for Envelopes +**File:** `src/mail/app.py` + +**Current:** Load all envelopes at startup +**Goal:** Load envelopes on-demand using background workers + +**Changes:** +```python +class MailApp(App): + envelopes_loaded: reactive[bool] = reactive(False) + _envelopes_cache: list[dict] = [] + + def on_mount(self) -> None: + # Start background loading + self._load_initial_envelopes() + + @work(exclusive=True, group="envelope_loading") + async def _load_initial_envelopes(self): + """Load initial envelopes in background.""" + envelopes, success = await himalaya_client.list_envelopes() + if success: + self._envelopes_cache = envelopes + self.envelopes_loaded = True + self._update_envelope_list_view() + + def _load_more_envelopes(self) -> None: + """Load more envelopes when scrolling.""" + pass # Implement lazy loading +``` + +**Expected Impact:** 60-70% faster startup, perceived instant UI + +#### 1.3 Optimize Message List Updates +**File:** `src/mail/app.py` + +**Current:** Rebuild entire list on navigation +**Goal:** Only update changed items, use reactive properties + +**Changes:** +```python +class MailApp(App): + current_message_index: reactive[int] = reactive(-1) + + def action_next(self) -> None: + """Move to next message efficiently.""" + if not self.current_message_index >= 0: + return + + next_id, next_idx = self.message_store.find_next_valid_id( + self.current_message_index + ) + + if next_id is not None: + # ✅ Only update reactive property + self.current_message_index = next_idx + # ✅ Let Textual handle the update + # DON'T call _update_envelope_list_view() +``` + +**Expected Impact:** 80-90% faster navigation, no UI flicker + +#### 1.4 Use Background Workers for Content Loading +**File:** `src/mail/widgets/ContentContainer.py` + +**Current:** Blocking content fetch +**Goal:** Use `@work` decorator for non-blocking loads + +**Changes:** +```python +class ContentContainer(ScrollableContainer): + @work(exclusive=True) + async def fetch_message_content(self, message_id: int, format_type: str) -> None: + """Fetch message content in background without blocking UI.""" + content, success = await himalaya_client.get_message_content( + message_id, + folder=self.current_folder, + account=self.current_account + ) + + if success and content: + self._update_content(content) + else: + self.notify("Failed to fetch message content") +``` + +**Expected Impact:** No UI blocking, smooth content transitions + +--- + +### Phase 2: Code Quality & Architecture (Week 2) + +#### 2.1 Refactor Message Store for Efficiency +**File:** `src/mail/message_store.py` + +**Current:** Linear searches, no caching +**Goal:** Implement indexed lookups, cache counts + +**Changes:** +```python +class MessageStore: + """Optimized message store with caching.""" + + def __init__(self, envelopes: list[dict]): + self.envelopes = envelopes + self._index_cache = {} # O(1) lookup cache + self._folder_counts = {} # Cached folder counts + self._unread_counts = {} # Cached unread counts + + # Build caches + self._build_caches() + + def _build_caches(self) -> None: + """Build lookup caches.""" + for idx, envelope in enumerate(self.envelopes): + self._index_cache[envelope["id"]] = idx + folder = envelope.get("folder", "INBOX") + self._folder_counts[folder] = self._folder_counts.get(folder, 0) + 1 + if not envelope.get("flags", {}).get("seen", False): + self._unread_counts[folder] = self._unread_counts.get(folder, 0) + 1 + + def get_index(self, message_id: int) -> int | None: + """Get envelope index in O(1).""" + return self._index_cache.get(message_id) + + def get_folder_count(self, folder: str) -> int: + """Get folder count in O(1).""" + return self._folder_counts.get(folder, 0) + + def get_unread_count(self, folder: str) -> int: + """Get unread count in O(1).""" + return self._unread_counts.get(folder, 0) +``` + +**Expected Impact:** O(1) lookups instead of O(n), instant count retrieval + +#### 2.2 Consolidate Envelope List Item +**File:** `src/mail/widgets/EnvelopeListItem.py` + +**Current:** Multiple widgets per item +**Goal:** Use single widget with custom rendering + +**Changes:** +```python +class EnvelopeListItem(CustomListItem): + """Optimized envelope list item using single widget.""" + + def __init__(self, envelope: dict, **kwargs): + super().__init__(**kwargs) + self.envelope = envelope + + def render(self) -> RichText: + """Render as single RichText widget.""" + from rich.text import Text, Text as RichText + + # Build RichText once (more efficient than multiple widgets) + text = Text.assemble( + self._from_display, + " ", + self._subject_display, + " ", + self._date_display, + style="on" if self.envelope.get("flags", {}).get("seen") else "bold" + ) + + return text +``` + +**Expected Impact:** 70% reduction in widget count, faster rendering + +#### 2.3 Add Memoization for Expensive Operations +**File:** `src/mail/utils.py` + +**Current:** Re-computing values +**Goal:** Cache computed values + +**Changes:** +```python +from functools import lru_cache + +@lru_cache(maxsize=128) +def format_sender_name(envelope: dict) -> str: + """Format sender name with caching.""" + from_name = envelope.get("from", {}).get("name", "") + from_addr = envelope.get("from", {}).get("addr", "") + + if not from_name: + return from_addr + + # Truncate if too long + if len(from_name) > 25: + return from_name[:22] + "..." + + return from_name + +@lru_cache(maxsize=128) +def format_date(date_str: str) -> str: + """Format date with caching.""" + # Parse and format date string + # Implementation... + return formatted_date +``` + +**Expected Impact:** Faster repeated operations, reduced CPU usage + +#### 2.4 Add Notification Compression Caching +**File:** `src/mail/notification_compressor.py` + +**Current:** Re-compressing on every view +**Goal:** Cache compressed results + +**Changes:** +```python +class NotificationCompressor: + """Compressor with caching for performance.""" + + def __init__(self, mode: str = "summary"): + self.mode = mode + self._compression_cache = {} # Cache compressed content + + def compress( + self, + content: str, + envelope: dict[str, Any] + ) -> tuple[str, NotificationType | None]: + """Compress with caching.""" + cache_key = f"{envelope['id']}:{self.mode}" + + # Check cache + if cache_key in self._compression_cache: + return self._compression_cache[cache_key] + + # Compress and cache + compressed, notif_type = self._compress_impl(content, envelope) + self._compression_cache[cache_key] = (compressed, notif_type) + + return compressed, notif_type +``` + +**Expected Impact:** Instant display for previously viewed notifications + +--- + +### Phase 3: Advanced Optimizations (Week 3-4) + +#### 3.1 Implement Virtual Scrolling +**File:** `src/mail/app.py` + +**Current:** Render all items in list +**Goal:** Use ListView virtualization + +**Changes:** +```python +def compose(self) -> ComposeResult: + yield ListView( + id="envelopes_list", + initial_index=0, + ) + +# ListView automatically virtualizes for performance +``` + +**Expected Impact:** Constant time rendering regardless of list size + +#### 3.2 Debounce User Input +**File:** `src/mail/screens/SearchPanel.py` + +**Current:** Search on every keystroke +**Goal:** Debounce search with 300ms delay + +**Changes:** +```python +class SearchPanel(Screen): + def __init__(self, **kwargs): + super().__init__(**kwargs) + self._search_debounce = None + + def on_input_changed(self, event) -> None: + """Debounce search input.""" + if self._search_debounce: + self._search_debounce.stop() + + self._search_debounce = Timer( + 0.3, + self._perform_search, + event.value + ).start() +``` + +**Expected Impact:** 80% reduction in expensive search operations + +#### 3.3 Use `dataclass` for Data Models +**File:** `src/mail/notification_detector.py` + +**Current:** Dict-based data access +**Goal:** Use dataclasses for type safety and performance + +**Changes:** +```python +from dataclasses import dataclass, field +from typing import Any + +@dataclass +class Envelope: + """Typed envelope data model.""" + id: int + subject: str + from_name: str + from_addr: str + date: str + flags: dict = field(default_factory=dict) + folder: str = "INBOX" +``` + +**Expected Impact:** Type safety, better IDE support, faster attribute access + +--- + +### Phase 4: Memory & Resource Management (Week 4) + +#### 4.1 Implement Widget Pooling +**File:** `src/mail/app.py` + +**Current:** Creating new widgets constantly +**Goal:** Reuse widgets to reduce allocations + +**Changes:** +```python +class WidgetPool: + """Pool for reusing widgets.""" + + def __init__(self, widget_class, max_size: int = 50): + self.widget_class = widget_class + self.pool = [] + self.max_size = max_size + + def get(self): + """Get widget from pool or create new.""" + if self.pool: + return self.pool.pop() + return self.widget_class() + + def release(self, widget) -> None: + """Return widget to pool.""" + if len(self.pool) < self.max_size: + self.pool.append(widget) +``` + +**Expected Impact:** Reduced garbage collection, smoother scrolling + +#### 4.2 Implement Content Pagination +**File:** `src/mail/widgets/ContentContainer.py` + +**Current:** Load full content +**Goal:** Load content in chunks for large emails + +**Changes:** +```python +class ContentContainer(ScrollableContainer): + PAGE_SIZE = 500 # Characters per page + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self._pages: list[str] = [] + self._current_page = 0 + + def _load_next_page(self) -> None: + """Load next page of content when scrolling.""" + if self._current_page + 1 < len(self._pages): + self._current_page += 1 + self.content.update(self._pages[self._current_page]) +``` + +**Expected Impact:** Faster initial load, smoother scrolling for large emails + +#### 4.3 Clean Up Unused Imports +**Files:** All Python files in `src/mail/` + +**Current:** Unused imports, circular dependencies +**Goal:** Remove all unused code + +**Changes:** +- Run `ruff check` and fix all unused imports +- Remove circular dependencies +- Clean up `__all__` exports +- Optimize import order + +**Expected Impact:** Faster import time, smaller memory footprint + +--- + +## Implementation Order + +### Week 1: Critical Performance Fixes +1. Day 1-2: Implement `compose()` pattern +2. Day 3-4: Lazy loading for envelopes +3. Day 5: Optimize message list navigation +4. Day 6-7: Background workers for content loading +5. Day 8-10: Testing and benchmarking + +### Week 2: Code Quality +1. Day 1-2: Refactor MessageStore with caching +2. Day 3-4: Consolidate EnvelopeListItem +3. Day 5: Add memoization utilities +4. Day 6-7: Notification compression caching +5. Day 8-10: Code review and cleanup + +### Week 3: Advanced Optimizations +1. Day 1-3: Virtual scrolling implementation +2. Day 4-5: Debounce user input +3. Day 6-7: Data model refactoring +4. Day 8-10: Performance testing + +### Week 4: Memory Management +1. Day 1-3: Widget pooling +2. Day 4-5: Content pagination +3. Day 6-7: Import cleanup +4. Day 8-10: Final optimization and polish + +--- + +## Success Metrics + +### Performance Targets +- **Startup Time:** < 1 second (currently: 3-5 seconds) +- **Navigation Latency:** < 50ms between messages (currently: 200-500ms) +- **List Rendering:** < 100ms for 100 items (currently: 500-1000ms) +- **Memory Usage:** < 100MB for 1000 emails (currently: 300-500MB) +- **Frame Rate:** 60 FPS during navigation (currently: 10-20 FPS) + +### Code Quality Targets +- **Test Coverage:** > 80% (currently: ~10%) +- **Ruff Warnings:** 0 critical, < 5 style warnings +- **Import Cleanup:** 100% of files cleaned +- **Type Coverage:** 100% typed + +--- + +## Testing Strategy + +### Performance Benchmarking +```python +# benchmark_performance.py +import time +import tracemalloc +from src.mail.app import EmailViewerApp + +def benchmark_startup(): + """Benchmark app startup time.""" + tracemalloc.start() + + start = time.time() + app = EmailViewerApp() + app.run() + + end = time.time() + current, peak = tracemalloc.get_traced_memory() + + print(f"Startup Time: {end - start:.3f}s") + print(f"Memory Usage: {peak / 1024 / 1024:.2f} MB") + +def benchmark_navigation(): + """Benchmark message navigation.""" + app = EmailViewerApp() + # ... measure navigation timing + + timings = [] + for i in range(100): + start = time.time() + app.action_next() + end = time.time() + timings.append(end - start) + + print(f"Average Navigation Time: {sum(timings) / len(timings) * 1000:.1f}ms") +``` + +### Integration Tests +- Test with 100, 1000, and 10000 messages +- Measure memory usage over time +- Test with slow network conditions +- Test on different terminal sizes + +--- + +## References + +### Textual Documentation +- **Main Docs:** https://textual.textualize.io/ +- **Widget Guide:** https://textual.textualize.io/guide/widgets/ +- **Best Practices:** https://textual.textualize.io/blog/ +- **Performance Guide:** https://textual.textualize.io/blog/2024/12/12/algorithms-for-high-performance-terminal-apps/ + +### Python Performance Guides +- **Python Performance Guide:** https://www.fyld.pt/blog/python-performance-guide-writing-code-2025 +- **Optimization Techniques:** https://analyticsvidhya.com/blog/2024/01/optimize-python-code-for-high-speed-execution + +### Similar Projects +- **Rich Console Examples:** https://github.com/Textualize/rich +- **Prompt Toolkit:** https://github.com/prompt-toolkit/python-prompt-toolkit +- **Urwid:** https://github.com/urwid/urwid + +--- + +## Notes + +- This plan focuses on the mail app but principles apply to calendar and tasks apps +- All changes should be backward compatible +- Run performance benchmarks before and after each phase +- Document any Textual-specific optimizations discovered during implementation