A Python-based web scraper that extracts product data and HTML content from the Shophive website for analysis and automation. It takes approximately 7-10 minutes for 100 pages with 6000+ products.
- Scrapes product titles, prices, and links
- Handles pagination across multiple pages
- Extracts raw HTML for deeper analysis
- Saves cleaned data into structured format (CSV)
- Basic error handling for failed requests
- Python
- requests
- BeautifulSoup
- pandas
Clone the repository:
git clone https://github.com/horridhaider/shophive-scraper.git
cd shophive-scraperInstall dependencies:
pip install -r requirements.txtEdit the config file for custom: -Base URL -Start Page -End Page
Run the scraper:
python shopping.pyThe scraper generates a CSV file containing:
-Product Name -Price -Product URL
Example:
| Product Name | Price | URL |
|---|---|---|
| iPhone 11 | PKR 150,000 | https://shophive.com/... |
