Scrapy

Scrapy is the high-level, open-source Python framework for fast, scalable web crawling and structured data extraction.

Scrapy is a powerful, asynchronous application framework for web scraping and general-purpose web crawling (using 'Spiders'). Built on Python and the Twisted networking engine, it handles concurrent requests efficiently: it doesn't wait for a request to finish before sending the next one. This architecture ensures high performance and scalability for large-scale data projects. Developers use CSS selectors or XPath expressions for precise data extraction, then utilize built-in components like Item Pipelines for post-processing and Feed Exports to output scraped data directly to formats like JSON, CSV, or XML.

https://scrapy.org