-
Notifications
You must be signed in to change notification settings - Fork 535
Closed
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.
Description
I'm running the quickstart code in WSL2 on Windows 11. Crawlee is not using the full available memory in WSL2. I have 15Gi total memory in WSL2
$ free -mh
total used free shared buff/cache available
Mem: 15Gi 2.7Gi 8.9Gi 3.0Mi 3.9Gi 12Gi
Swap: 4.0Gi 0B 4.0GiCrawlee is only using 3.88GB for some reason.
[crawlee._autoscaling.snapshotter] WARN Memory is critically overloaded. Using 5.24 GB of 3.88 GB (135%). Consider increasing available memory.How can I make Crawlee use a larger percentage of available memory?
Code (same code as the quickstart):
import asyncio
from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext
async def main() -> None:
# PlaywrightCrawler crawls the web using a headless browser controlled by the Playwright library.
crawler = PlaywrightCrawler()
# Define a request handler to process each crawled page and attach it to the crawler using a decorator.
@crawler.router.default_handler
async def request_handler(context: PlaywrightCrawlingContext) -> None:
# Extract relevant data from the page context.
data = {
'url': context.request.url,
'title': await context.page.title(),
}
# Store the extracted data.
await context.push_data(data)
# Extract links from the current page and add them to the crawling queue.
await context.enqueue_links()
# Add first URL to the queue and start the crawl.
await crawler.run(['https://crawlee.dev'])
if __name__ == '__main__':
asyncio.run(main())Metadata
Metadata
Assignees
Labels
t-toolingIssues with this label are in the ownership of the tooling team.Issues with this label are in the ownership of the tooling team.