Fix N+1 query pattern in attachment fetching for activities

[Copilot]

The activities API was executing O(n) database queries to fetch attachments, calling get_attachments() once per comment/communication in a loop. For leads with many activities, this caused significant overhead.

Changes

New batch fetching function

• Added get_attachments_batch(doctype, names) that fetches attachments for multiple documents in a single query using SQL IN clause
• Returns dictionary mapping document names to their attachment lists

Updated get_opportunity_activities() and get_lead_activities()

• Batch fetch all Comment attachments (comments + info_logs) before processing loops
• Batch fetch all Communication attachments before processing loops
• Replace per-item get_attachments() calls with O(1) dictionary lookups

Before:

for comment in docinfo.comments:
    activity = {
        "attachments": get_attachments("Comment", comment.name),  # N+1 query
        ...
    }

After:

comment_names = [c.name for c in docinfo.comments]
comment_attachments = get_attachments_batch("Comment", comment_names)  # Single query

for comment in docinfo.comments:
    activity = {
        "attachments": comment_attachments.get(comment.name, []),  # O(1) lookup
        ...
    }

Impact

Reduces queries from O(n) to O(1) — exactly 2 queries per function regardless of activity count. Estimated 15-25% performance improvement for activity timeline loads.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>N+1 Query Pattern in get_attachments() Called Per Comment</issue_title>
<issue_description>

Metadata

• File(s): next_crm/api/activities.py:127, 139, 157, 311, 331
• Category: Database/API
• Severity: Medium
• Effort to Fix: Medium
• Estimated Performance Gain: 15-25%

Problem Description

The get_attachments() function is called inside loops for each comment, info_log, and communication. For a lead/opportunity with many activities, this creates numerous additional database queries.

Impact Analysis

• CPU Impact: Medium - Query execution overhead
• Memory Impact: Low - Small result sets per query
• Database Impact: Medium - O(n) queries where n = number of comments + communications
• User Experience Impact: Medium - Activity timeline loads slowly

Code Location

# next_crm/api/activities.py:120-130 - Called per comment
for comment in docinfo.comments:
    activity = {
        "name": comment.name,
        "activity_type": "comment",
        # ...
        "attachments": get_attachments("Comment", comment.name),  # N+1
    }

# next_crm/api/activities.py:132-142 - Called per info_log
for info_log in docinfo.info_logs:
    activity = {
        # ...
        "attachments": get_attachments("Comment", info_log.name),  # N+1
    }

# next_crm/api/activities.py:144-163 - Called per communication
for communication in docinfo.communications + docinfo.automated_messages:
    activity = {
        # ...
        "attachments": get_attachments("Communication", communication.name),  # N+1
    }

Root Cause

1. get_attachments() called inside multiple loops
2. No batch fetching of attachments for all comments/communications at once
3. Same pattern repeated in both get_opportunity_activities() and get_lead_activities()

Proposed Solution

Batch fetch attachments before the loops:

def get_attachments_batch(doctype, names):
    """Batch fetch attachments for multiple documents"""
    if not names:
        return {}
    
    attachments = frappe.db.get_all(
        "File",
        filters={
            "attached_to_doctype": doctype,
            "attached_to_name": ["in", names]
        },
        fields=[
            "name", "file_name", "file_type", "file_url",
            "file_size", "is_private", "creation", "owner",
            "attached_to_name"
        ],
    )
    
    # Group by attached_to_name
    result = {}
    for att in attachments:
        key = att["attached_to_name"]
        if key not in result:
            result[key] = []
        result[key].append(att)
    
    return result


def get_opportunity_activities(name):
    # ... existing setup code ...

    # Batch fetch attachments for all comments
    comment_names = [c.name for c in docinfo.comments] + [i.name for i in docinfo.info_logs]
    comment_attachments = get_attachments_batch("Comment", comment_names)
    
    # Batch fetch attachments for all communications
    comm_names = [c.name for c in docinfo.communications + docinfo.automated_messages]
    comm_attachments = get_attachments_batch("Communication", comm_names)

    # Now use the pre-fetched attachments
    for comment in docinfo.comments:
        activity = {
            "name": comment.name,
            "activity_type": "comment",
            "creation": comment.creation,
            "owner": comment.owner,
            "content": comment.content,
            "attachments": comment_attachments.get(comment.name, []),  # O(1) lookup
            "is_lead": False,
        }
        activities.append(activity)

    for info_log in docinfo.info_logs:
        activity = {
            "name": info_log.name,
            "activity_type": "comment",
            "creation": info_log.creation,
            "owner": info_log.owner,
            "content": info_log.content,
            "attachments": comment_attachments.get(info_log.name, []),  # O(1) lookup
            "is_lead": False,
        }
        activities.append(activity)

    for communication in docinfo.communications + docinfo.automated_messages:
        activity = {
            "activity_type": "communication",
            "communication_type": communication.communication_type,
            "creation": communication.creation,
            "data": {
                "subject": communication.subject,
                "content": communication.content,
                # ...other fields...
                "attachments": comm_attachments.get(communication.name, []),  # O(1) lookup
            },
            "is_lead": False,
        }
        activities.append(activity)

    # ... rest of function ...

Implementation Steps

1. Create get_attachments_batch() helper function
2. Collect all comment/info_log/communication names before loops
3. Batch fetch attachments grouped by doctype
4. Replace get_attachments() calls with dictionary lookups
5. Apply same pattern to `...

• Fixes N+1 Query Pattern in get_attachments() Called Per Comment #332

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.