📰 Build an Automated Economic Times News Scraper in Python

If you have ever wanted to collect news data for analysis, machine learning, or just to build your own custom news feed, web scraping is the way to go. In this project, we are going to look at a beginner-friendly Python script that automates the extraction of news articles from the Economic Times.

---

🚀 What This Project Does

This script is designed to be a continuous, automated news collector. Here is exactly what it handles behind the scenes:

• Reads RSS Feeds: It taps into the official Economic Times RSS feeds across multiple categories (Markets, Tech, Startups, Politics, etc.).
• Scrapes Full Articles: It doesn't just stop at the summary. It visits the actual article link to extract the full body text.
• Extracts Metadata: It grabs the title, author, published time, and even the thumbnail image.
• Saves to JSON: It neatly organizes and saves the scraped data into specific category files (e.g., tech_industry.json).

---

⚙️ The Setup & Requirements

Before running the code, you will need to install two powerful Python libraries. Open your terminal and run:

pip install requests beautifulsoup4

💡 Beginner Friendly Settings:
The script is built with a dedicated "Beginner Settings" block at the very top. You can easily change how many articles it downloads (ARTICLES_PER_CATEGORY), where to save them (OUTPUT_DIR), and whether it should run forever or just once (RUN_CONTINUOUSLY).

---

💻 The Python Code

Below is the full source code for the scraper, pulled directly from my GitHub repository. Feel free to copy it or clone the repo!

---

📊 Expected Output

When you run the script, it will print a clean, readable log in your terminal showing exactly what it is doing. It skips duplicates automatically so you never save the same article twice!

============================================================ Cycle #1 | 2024-05-18 10:00:00 ============================================================ [1/33] Category: TOP_NEWS Fetching: https://economictimes.indiatimes.com/... Saved: Sensex crashes 800 points amid global sell-off... Fetching: https://economictimes.indiatimes.com/... Saved: RBI announces new guidelines for digital lending... Waiting 3.0s before the next category... ------------------------------------------------------------ Cycle #1 complete. DB snapshot: top_news -> 2 articles (NEWS_DATA_ET\top_news.json) ------------------------------------------------------------ Sleeping for 200s before the next cycle...

The JSON Data Structure

If you open one of the generated JSON files (like top_news.json), you will see beautifully structured data ready for your next big project:

[ { "title": "Sensex crashes 800 points amid global sell-off", "url": "https://economictimes.indiatimes.com/...", "category": "Markets", "author": "ET Markets Web Team", "published_time": "Sat, 18 May 2024 09:30:00 IST", "modified_time": "2024-05-18T09:35:00Z", "keywords": [ "Sensex", "Nifty", "Stock Market" ], "short_desc": "Indian equity indices plunged on Saturday...", "long_desc": "Indian equity indices plunged on Saturday following weak global cues...", "thumbnail": "https://img.etimg.com/thumb/msid-12345.jpg", "source": "Economic Times" } ]

---

🚀 Next Steps

This data is perfect for building a custom news dashboard, running Natural Language Processing (NLP) to perform sentiment analysis on the stock market, or feeding into an AI model. Happy scraping!