luk/PROJECT_PLAN.md

LUK Project Plan

This document outlines planned improvements across the LUK applications (Mail, Calendar, Sync, Tasks).

---

Bug Fixes

Tasks App - Table Not Displaying (FIXED)

Priority: Critical
File: src/tasks/app.py

The tasks app was not showing the task table due to a CSS grid layout issue. The grid layout with mixed dock and grid positioning caused the main content area to have 0 height.

Fix: Changed from grid layout to horizontal layout with docked header/footer.

---

Sync App

Performance Improvements

1. Parallelize Message Downloads

Priority: High
Files: src/services/microsoft_graph/mail.py, src/services/microsoft_graph/client.py

Currently, message downloads are sequential (for loop with await). This is a significant bottleneck.

Current code pattern:

for msg_id in message_ids:
    content = await client.get_message(msg_id)
    await save_to_maildir(content)

Proposed changes:

1. Increase semaphore limit in client.py from 2 to 5 concurrent HTTP requests
2. Batch parallel downloads using asyncio.gather() with batches of 5-10 messages
3. Batch sync state writes every N messages instead of after each message (currently an I/O bottleneck in archive sync)

Example implementation:

BATCH_SIZE = 5

async def fetch_batch(batch_ids):
    return await asyncio.gather(*[client.get_message(id) for id in batch_ids])

for i in range(0, len(message_ids), BATCH_SIZE):
    batch = message_ids[i:i + BATCH_SIZE]
    results = await fetch_batch(batch)
    for content in results:
        await save_to_maildir(content)
    # Write sync state every batch instead of every message
    save_sync_state()

2. Optimize Maildir Writes

Priority: Medium
File: src/utils/mail_utils/maildir.py

The save_mime_to_maildir_async function is a potential bottleneck. Consider:

• Batching file writes
• Using thread pool for I/O operations

---

CLI Improvements

3. Default to TUI Mode

Priority: Medium
File: src/cli/sync.py

Currently luk sync requires subcommands. Change to:

• luk sync → Opens TUI (interactive mode) by default
• luk sync --once / luk sync -1 → One-shot sync
• luk sync --daemon / luk sync -d → Daemon mode
• luk sync status → Show sync status

---

UI Consistency

4. Navigation and Styling

Priority: Low
File: src/cli/sync_dashboard.py

• Add j/k keybindings for list navigation (vim-style)
• Use border: round in TCSS for consistency with other apps
• Add .border_title styling for list containers

5. Notifications Toggle

Priority: Low
Files: src/cli/sync_dashboard.py, src/utils/notifications.py

Add a UI switch to enable/disable desktop notifications during sync.

---

Inter-Process Communication (IPC)

Overview

Priority: High
Goal: Enable real-time updates between apps when data changes:

• Mail app refreshes when sync downloads new messages
• Tasks app refreshes when syncing or when tasks are added from mail app
• Calendar app reloads when sync updates events

Platform Options

macOS Options

1. Distributed Notifications (NSDistributedNotificationCenter)

   • Native macOS IPC mechanism
   • Simple pub/sub model
   • Lightweight, no server needed
   • Python: Use pyobjc (Foundation.NSDistributedNotificationCenter)
   • Example: CFNotificationCenterPostNotification()

2. Unix Domain Sockets

   • Works on both macOS and Linux
   • Requires a listener process (daemon or embedded in sync)
   • More complex but more flexible
   • Can send structured data (JSON messages)

3. Named Pipes (FIFO)

   • Simple, works on both platforms
   • One-way communication
   • Less suitable for bidirectional messaging

4. File-based Signaling

   • Write a "dirty" file that apps watch via watchdog or fsevents
   • Simple but has latency
   • Works cross-platform

Linux Options

1. D-Bus

   • Standard Linux IPC
   • Python: Use dbus-python or pydbus
   • Powerful but more complex
   • Not available on macOS by default

2. Unix Domain Sockets

   • Same as macOS, fully compatible
   • Recommended for cross-platform compatibility

3. systemd Socket Activation

   • If running as a systemd service
   • Clean integration with Linux init

Recommended Approach

Use Unix Domain Sockets for cross-platform compatibility:

# Socket path
SOCKET_PATH = Path.home() / ".cache" / "luk" / "ipc.sock"

# Message types
class IPCMessage:
    MAIL_UPDATED = "mail.updated"
    TASKS_UPDATED = "tasks.updated"
    CALENDAR_UPDATED = "calendar.updated"

# Sync daemon sends notifications
async def notify_mail_updated(folder: str):
    await send_ipc_message({"type": IPCMessage.MAIL_UPDATED, "folder": folder})

# Mail app listens
async def listen_for_updates():
    async for message in ipc_listener():
        if message["type"] == IPCMessage.MAIL_UPDATED:
            self.load_messages()

Implementation files:

• src/utils/ipc.py - IPC client/server utilities
• src/cli/sync_daemon.py - Add notification sending
• src/mail/app.py - Add listener for mail updates
• src/tasks/app.py - Add listener for task updates
• src/calendar/app.py - Add listener for calendar updates

---

Calendar App

Visual Improvements

1. Current Time Hour Line Styling

Priority: High
File: src/calendar/widgets/WeekGrid.py

The current time indicator hour line should have a subtle contrasting background color to make it more visible.

2. Cursor Hour Header Highlighting

Priority: Medium
File: src/calendar/widgets/WeekGrid.py

The hour header at cursor position should have a brighter background, similar to how the day header is highlighted when selected.

---

Layout Improvements

3. Responsive Detail Panel

Priority: Medium
Files: src/calendar/app.py, src/calendar/widgets/

When the terminal is wider than X characters (e.g., 120), show a side-docked detail panel for the selected event instead of a modal/overlay.

4. Sidebar Mini-Calendar

Priority: Medium
Files: src/calendar/app.py, src/calendar/widgets/MonthCalendar.py

When the sidebar is toggled on, display a mini-calendar in the top-left corner showing:

• Current day highlighted
• The week(s) currently visible in the main WeekGrid pane
• Click/navigate to jump to a specific date

Implementation:

• Reuse existing MonthCalendar widget in compact mode
• Add reactive property to sync selected week with main pane
• Add to sidebar composition when toggled

---

Microsoft Graph Integration

5. Calendar Invites Sidebar

Priority: Medium
Files: src/calendar/app.py, src/services/microsoft_graph/calendar.py

Display a list of pending calendar invites from Microsoft Graph API in a sidebar panel:

• List pending invites (meeting requests)
• Show invite details (organizer, time, subject)
• Accept/Decline/Tentative actions
• No sync needed - fetch on-demand from API

API Endpoints:

• GET /me/calendar/events?$filter=responseStatus/response eq 'notResponded'
• POST /me/events/{id}/accept
• POST /me/events/{id}/decline
• POST /me/events/{id}/tentativelyAccept

UI:

• Toggle with keybinding (e.g., i for invites)
• Sidebar shows list of pending invites
• Detail view shows full invite info
• Action buttons/keys for response

---

Search Feature

6. Calendar Search

Priority: Medium
Files: src/calendar/app.py, src/services/khal/client.py (if khal supports search)

Add search functionality if the underlying backend (khal) supports it:

• / keybinding to open search input
• Search by event title, description, location
• Display search results in a modal or replace main view
• Navigate to selected event

Check: Does khal search command exist?

---

Help System

7. Help Toast (Keep Current Implementation)

Priority: Low
File: src/calendar/app.py

The ? key shows a help toast using self.notify(). This pattern should be implemented in other apps (Mail, Tasks, Sync) for consistency.

---

Mail App

Layout Fixes

1. Remove Envelope Icon/Checkbox Gap

Priority: High
File: src/mail/widgets/EnvelopeListItem.py

There's a 1-character space between the envelope icon and checkbox that should be removed for tighter layout.

---

Theme Improvements

2. Replace Hardcoded RGB Colors

Priority: High
File: src/mail/email_viewer.tcss

Multiple hardcoded RGB values should use Textual theme variables for better theming support:

Line	Current	Replacement
6	border: round rgb(117, 106, 129)	border: round $border
46	background: rgb(55, 53, 57)	background: $surface
52, 57	RGB label colors	Theme variables
69	background: rgb(64, 62, 65)	background: $panel
169, 225, 228, 272, 274	Various RGB colors	Theme variables

---

Keybindings

3. Add Refresh Keybinding

Priority: Medium
File: src/mail/app.py

Add r keybinding to refresh/reload the message list.

4. Add Mark Read/Unread Action

Priority: Medium
Files: src/mail/app.py, src/mail/actions/ (new file)

Add action to toggle read/unread status on selected message(s).

---

Search Feature

5. Mail Search

Priority: Medium
Files: src/mail/app.py, backend integration

Add search functionality if the underlying mail backend supports it:

• / keybinding to open search input
• Search by subject, sender, body
• Display results in message list
• Check: Does himalaya or configured backend support search?

---

UI Enhancements

6. Folder Message Counts

Priority: Medium
Files: src/mail/app.py, src/mail/widgets/

Display total message count next to each folder name (e.g., "Inbox (42)").

7. Sort Setting in Config/UI

Priority: Low
Files: src/mail/config.py, src/mail/app.py

Add configurable sort order (date, sender, subject) with UI toggle.

---

Message Display

8. URL Compression in Markdown View

Priority: Medium
Files: src/mail/widgets/ContentContainer.py, src/mail/screens/LinkPanel.py

Compress long URLs in the markdown view to ~50 characters with a nerdfont icon. The _shorten_url algorithm in LinkPanel.py can be reused.

Considerations:

• Cache processed markdown to avoid re-processing on scroll
• Store URL mapping for click handling

9. Remove Emoji from Border Title

Priority: Low
File: src/mail/widgets/ContentContainer.py or EnvelopeHeader.py

Remove the envelope emoji prefix before message ID in border titles.

10. Enhance Subject Styling

Priority: Medium
File: src/mail/widgets/EnvelopeHeader.py

• Move subject line to the top of the header
• Make it bolder/brighter for better visual hierarchy

---

Tasks App

Search Feature

1. Task Search

Priority: Medium
Files: src/tasks/app.py, src/services/dstask/client.py

Add search functionality:

• / keybinding to open search input
• Search by task summary, notes, project, tags
• Display matching tasks
• Check: dstask likely supports filtering which can be used for search

Implementation:

• Add search input widget (TextInput)
• Filter tasks locally or via dstask command
• Update table to show only matching tasks
• Clear search with Escape

---

Help System

2. Implement Help Toast

Priority: Low
File: src/tasks/app.py

Add ? keybinding to show help toast (matching Calendar app pattern). Note: This is already implemented in the current code.

---

Cross-App Improvements

1. Consistent Help System

Implement ? key help toast in all apps using self.notify():

• Mail: src/mail/app.py
• Tasks: src/tasks/app.py (already has it)
• Sync: src/cli/sync_dashboard.py

2. Consistent Navigation

Add vim-style j/k navigation to all list views across apps.

3. Consistent Border Styling

Use border: round and .border_title styling consistently in all TCSS files.

4. Consistent Search Interface

Implement / keybinding for search across all apps with similar UX:

• / opens search input
• Enter executes search
• Escape clears/closes search
• Results displayed in main view or filtered list

---

Implementation Priority

Phase 1: Critical/High Priority

1. Tasks App: Fix table display (DONE)
2. Sync: Parallelize message downloads (DONE - connection pooling + batch size increase)
3. Mail: Replace hardcoded RGB colors (DONE - already using theme variables)
4. Mail: Remove envelope icon/checkbox gap (DONE)
5. Calendar: Current time hour line styling (DONE - added surface background)
6. IPC: Implement cross-app refresh notifications (DONE)

Phase 2: Medium Priority

1. Sync: Default to TUI mode (DONE - already implemented)
2. Calendar: Cursor hour header highlighting (DONE)
3. Calendar: Responsive detail panel
4. Calendar: Sidebar mini-calendar
5. Calendar: Calendar invites sidebar
6. Mail: Add refresh keybinding (DONE - r key)
7. Mail: Add mark read/unread action (DONE - u key)
8. Mail: Folder message counts (DONE)
9. Mail: URL compression in markdown view (DONE)
10. Mail: Enhance subject styling (DONE)
11. Mail: Search feature
12. Tasks: Search feature (DONE - / key with live filtering)
13. Calendar: Search feature (DONE - / key using khal search)

Phase 3: Low Priority

1. Sync: UI consistency (j/k navigation, borders)
2. Sync: Notifications toggle
3. Calendar: Help toast (already implemented, replicate to other apps)
4. Mail: Remove emoji from border title
5. Mail: Sort setting in config/UI
6. Cross-app: Consistent help, navigation, and styling

---

Technical Notes

IPC Implementation Details

For macOS-first with Linux compatibility:

# src/utils/ipc.py
import asyncio
import json
from pathlib import Path

SOCKET_PATH = Path.home() / ".cache" / "luk" / "ipc.sock"

class IPCServer:
    """Server for sending notifications to listening apps."""
    
    async def broadcast(self, message: dict):
        """Send message to all connected clients."""
        pass

class IPCClient:
    """Client for receiving notifications in apps."""
    
    async def listen(self, callback):
        """Listen for messages and call callback."""
        pass

Backend Search Capabilities

Backend	Search Support
dstask	Filter by project/tag, summary search via shell
himalaya	Check himalaya search command
khal	Check khal search command
Microsoft Graph	Full text search via $search parameter

---

Notes

• All UI improvements should be tested with different terminal sizes
• Theme changes should be tested with multiple Textual themes
• Performance improvements should include before/after benchmarks
• New keybindings should be documented in each app's help toast
• IPC should gracefully handle missing socket (apps work standalone)
• Search should be responsive and not block UI

---

Mail Rendering Improvements

Smart Notification Email Compression (COMPLETED)

Priority: High Files: src/mail/notification_detector.py, src/mail/notification_compressor.py, src/mail/config.py

Problem: Transactional notification emails from tools like Renovate, Jira, Confluence, and Datadog are hard to parse and display poorly in plain text terminal environments.

Solution: Implemented intelligent notification detection and compression system:

1. Notification Type Detection

   • Automatically detects emails from:
     • GitLab (pipelines, MRs, mentions)
     • GitHub (PRs, issues, reviews)
     • Jira (issues, status changes)
     • Confluence (page updates, comments)
     • Datadog (alerts, incidents)
     • Renovate (dependency updates)
     • General notifications (digests, automated emails)
   • Uses sender domain and subject pattern matching
   • Distinguishes between similar services (e.g., Jira vs Confluence)

2. Structured Summary Extraction

   • Type-specific extractors for each platform
   • Extracts: IDs, titles, status changes, action items
   • Preserves important links for quick access

3. Terminal-Friendly Formatting

   • Two compression modes:
     • summary: Brief one-page view
     • detailed: Structured table format
   • Markdown rendering with icons and clear hierarchy
   • Shows notification type, key details, actions
   • Footer indicates compressed view (toggle to full with m)

4. Configuration Options

   [content_display]
   compress_notifications = true
   notification_compression_mode = "summary"  # "summary", "detailed", or "off"
   

5. Features

   • Zero-dependency (no LLM required, fast)
   • Rule-based for reliability
   • Extensible to add new notification types
   • Preserves original content for full view toggle
   • Type-specific icons using NerdFont glyphs

Implementation Details:

• NotificationType dataclass for type definitions
• is_notification_email() for detection
• classify_notification() for type classification
• extract_notification_summary() for structured data
• NotificationCompressor for formatting
• DetailedCompressor for extended summaries
• Integrated with ContentContainer widget
• 13 unit tests covering all notification types

Future Enhancements:

• Add LLM-based summarization option (opt-in)
• Learn notification patterns from user feedback
• Custom notification type definitions
• Action-based email filtering (e.g., "archive all Renovate emails")
• Bulk actions on notification emails (archive all, mark all read)
• Notification grouping in envelope list
• Statistics on notification emails (counts by type, frequency)

---

---

Note-taking integration and more cross-app integrations

I like the tdo (https://github.com/2KAbhishek/tdo) program for managing markdown notes with fzf and my terminal text editor. It makes it easy to have a "today" note and a list of todos. Perhaps we can gather those todos from the text files in the $NOTES_DIR and put them into the task list during regular sync - and when users mark a task complete the sync can find the text file it was in and mark it complete on that line of markdown text. We need a little ore features for the related annotations then, because when I press n in the notes app we would want to actually open the textfile that task came from, not just make another annotation. So we would need a special cross-linking format for knowing which tasks came from a $NOTES_DIR sync. And then we can work on the same IPC scenario for tasks that were created in the email app. Then those tasks should be connected so that when the user views the notes on those tasks they see the whole email. That would be simpe enough if we just copied the email text into an annotation. But maybe we need a way to actually change the selected message ID in the mail app if it's open. So if the user activates the "open" feature on a task the related email will be displayed in the other terminal window where the user has mail open.

---

Polish for release and Auth

• Authentication isn't working if the user is in a different directory than expected. Store the auth token in a ~/.local directory so it's consistently availabe in every dir.
• Setup scripts, look at popular modern python CLI or TUI projects and create an install or first run script that works for most scenarios, using uv or pip to install globally for users, or maybe get a homebrew installation to work? Get users to set up himalaya and khal and dstask but ask them for their choice once we have other backends available.
• Documentation files in the repo, and documentation site to publish.

---

Library Updates & Python Version Review

Priority: Medium (Scheduled Review)

Periodically review the latest releases of heavily-used libraries to identify:

• Bug fixes that address issues we've encountered
• New features that could improve the codebase
• Deprecation warnings that need to be addressed
• Security updates

Key Libraries to Review

Library	Current Use	Review Focus
Textual	All TUI apps	New widgets, performance improvements, theming changes, CSS features
aiohttp	Microsoft Graph API client	Async improvements, connection pooling
msal	Microsoft authentication	Token caching, auth flow improvements
rich	Console output (via Textual)	New formatting options
orjson	Fast JSON parsing	Performance improvements
pyobjc (macOS)	Notifications	API changes, compatibility

Textual Changelog Review Checklist

When reviewing Textual releases, check for:

1. New widgets - Could replace custom implementations
2. CSS features - New selectors, pseudo-classes, properties
3. Theming updates - New theme variables, design token changes
4. Performance - Rendering optimizations, memory improvements
5. Breaking changes - Deprecated APIs, signature changes
6. Worker improvements - Background task handling

Python Version Upgrade

Current Status

• Check .python-version and pyproject.toml for current Python version
• Evaluate upgrade to Python 3.13 or 3.14 when stable

Python 3.13 Features to Consider

• Improved error messages
• Type system enhancements (typing module improvements)
• Performance optimizations (PEP 709 - inline comprehensions)

Python 3.14 Considerations

• Status: Currently in alpha/beta (as of Dec 2024)
• Expected stable release: October 2025
• Recommendation: Wait for stable release before adopting
• Pre-release testing: Can test compatibility in CI/CD before adoption

Upgrade Checklist

1. Review Python release notes for breaking changes
2. Check library compatibility (especially pyobjc, textual, msal)
3. Update .python-version (mise/pyenv)
4. Update pyproject.toml requires-python field
5. Run full test suite
6. Test on both macOS and Linux (if applicable)
7. Update CI/CD Python version

Action Items

1. Quarterly Review - Schedule quarterly reviews of library changelogs
2. Dependabot/Renovate - Consider adding automated dependency update PRs
3. Changelog Reading - Before updating, read changelogs for breaking changes
4. Test Coverage - Ensure adequate test coverage before major updates