A high-performance web crawler written in Rust, designed for efficient and reliable web scraping with support for Tor network integration.
- 🚀 Fast and efficient web crawling using Rust
- 🔍 HTML parsing and processing capabilities
- 🕸️ Tor network support for anonymous crawling
- 🐳 Docker containerization for easy deployment
- 🌸 Bloom filter implementation for efficient URL tracking
- ⚡ Asynchronous operation with Tokio runtime
- 🦀 Rust (latest stable version)
- 🐳 Docker and Docker Compose (for containerized deployment)
- 🔒 Tor (for anonymous crawling)
- Clone the repository:
git clone https://github.com/yourusername/DeepStalk.git
cd DeepStalk- Build the project:
cargo build --release- Build and run using Docker Compose:
docker-compose up --build./target/release/crawle-rsThe project includes a utility script run_with_input.sh that allows for automated control of the crawler:
./run_with_input.shThis script:
- 🎮 Runs the crawler in the background
- ⏱️ Automatically sends 'q' input every minute
- 🔄 Provides a way to gracefully control the crawler's operation
docker run -it deepstalk- 📂
src/- Source code directory - 📦
target/- Build output directory - 🐳
Dockerfile- Container configuration - 🐳
docker-compose.yml- Docker Compose configuration - 📄
Cargo.toml- Rust project configuration and dependencies
- 🌸 fastbloom - For efficient URL tracking
- 🔍 html5ever - HTML parsing
- 🧩 lol_html - HTML processing
- 🌐 reqwest - HTTP client
- ⚡ tokio - Asynchronous runtime
- 🔗 url - URL parsing and manipulation
This project is licensed under the terms of the license included in the repository.
Contributions are welcome! Please feel free to submit a Pull Request.
This project includes Tor network support for anonymous crawling. Please use responsibly and in accordance with applicable laws and regulations.