Skip to content

Memory is critically overloaded WARNING #550

@stephenleo

Description

@stephenleo

I'm running the quickstart code in WSL2 on Windows 11. Crawlee is not using the full available memory in WSL2. I have 15Gi total memory in WSL2

$  free -mh
               total        used        free      shared  buff/cache   available
Mem:            15Gi       2.7Gi       8.9Gi       3.0Mi       3.9Gi        12Gi
Swap:          4.0Gi          0B       4.0Gi

Crawlee is only using 3.88GB for some reason.

[crawlee._autoscaling.snapshotter] WARN  Memory is critically overloaded. Using 5.24 GB of 3.88 GB (135%). Consider increasing available memory.

How can I make Crawlee use a larger percentage of available memory?

Code (same code as the quickstart):

import asyncio

from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext


async def main() -> None:
    # PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.
    crawler = PlaywrightCrawler()

    # Define a request handler to process each crawled page and attach it to the crawler using a decorator.
    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        # Extract relevant data from the page context.
        data = {
            'url': context.request.url,
            'title': await context.page.title(),
        }
        # Store the extracted data.
        await context.push_data(data)
        # Extract links from the current page and add them to the crawling queue.
        await context.enqueue_links()

    # Add first URL to the queue and start the crawl.
    await crawler.run(['https://crawlee.dev'])


if __name__ == '__main__':
    asyncio.run(main())

Metadata

Metadata

Assignees

No one assigned

    Labels

    t-toolingIssues with this label are in the ownership of the tooling team.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions