Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,36 +128,46 @@ The output file contains the following columns:
- `latest` is intended for latest-news monitoring and does not require keywords.
- Latest mode currently starts with a smaller subset of sources than the full search catalog.

## Supported Websites (48)
## Supported Websites (61)

[Antara News](https://antaranews.com),
[Bali Post](https://www.balipost.com),
[BBC News](https://bbc.com),
[Berita Jatim](https://beritajatim.com),
[BeritaSatu](https://www.beritasatu.com),
[Bisnis.com](https://bisnis.com),
[Bloomberg Technoz](https://bloombergtechnoz.com),
[CNA Indonesia](https://www.cna.id),
[CNBC Indonesia](https://cnbcindonesia.com),
[CNN Indonesia](https://cnnindonesia.com),
[DailySocial](https://news.dailysocial.id),
[Detik](https://detik.com),
[Fajar](https://fajar.co.id),
[Galamedia](https://galamedia.pikiran-rakyat.com),
[Gatra](https://www.gatra.net),
[Hukumonline](https://www.hukumonline.com),
[Grid](https://www.grid.id),
[Harian Jogja](https://www.harianjogja.com),
[Hipwee](https://www.hipwee.com),
[IDN Times](https://idntimes.com),
[iNews](https://inews.id),
[Investor Daily](https://investor.id),
[Jawapos](https://jawapos.com),
[Jakarta Globe](https://jakartaglobe.id),
[Jakarta Post](https://thejakartapost.com),
[Jakarta Selaras](https://jakarta.selaras.co.id),
[Jawapos](https://jawapos.com),
[JPNN](https://jpnn.com),
[Kaltim Post](https://kaltimkece.borneo24.com),
[Katadata](https://katadata.co.id),
[KBR](https://kbr.id),
[Kompas](https://kompas.com),
[Kontan](https://kontan.co.id),
[Kumparan](https://kumparan.com),
[Liputan6](https://liputan6.com),
[Media Indonesia](https://mediaindonesia.com),
[Merdeka](https://merdeka.com),
[Metro TV News](https://metrotvnews.com),
[Niaga.Asia](https://www.niaga.asia),
[Mojok](https://mojok.co),
[Mongabay Indonesia](https://mongabay.co.id),
[Okezone](https://okezone.com),
[Pantau.com](https://www.pantau.com),
Expand All @@ -167,10 +177,12 @@ The output file contains the following columns:
[Republika](https://republika.co.id),
[RM.ID](https://rm.id),
[RRI](https://rri.co.id),
[RMOL](https://rmol.id),
[SINDOnews](https://sindonews.com),
[Suara](https://suara.com),
[Suara Merdeka](https://suaramerdeka.com),
[Surabaya Pagi](https://surabayapagi.com),
[SWA](https://swa.co.id),
[Tempo](https://tempo.co),
[Tirto](https://tirto.id),
[Tribunnews](https://tribunnews.com),
Expand Down
22 changes: 22 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.8.6] - 2026-05-16

### Added
- Added Fajar (`fajar`) scraper with search and latest support
- Added Mojok (`mojok`) scraper with search and latest support
- Added Grid (`grid`) scraper with search and latest support
- Added Hipwee (`hipwee`) scraper with search and latest support
- Added Jakarta Globe (`jakartaglobe`) scraper with search and latest support
- Added RMOL (`rmol`) scraper with tag-based search and latest support
- Added CNA Indonesia (`cnaindonesia`) scraper (latest-only; search uses Algolia JS)
- Total stable scrapers: 60

### Notes
- Fajar: WordPress search on fajar.co.id; Sulawesi regional gap
- Mojok: WordPress search on mojok.co; satire/opinion/youth angle
- Grid: /search?q= endpoint; grid.id; lifestyle/pop culture gap
- Hipwee: WordPress search on hipwee.com; youth demographic angle
- Jakarta Globe: /search/{keyword} custom endpoint; English coverage alternative
- RMOL: /tag/{keyword} tag-based search; political commentary/analysis
- CNA Indonesia: Drupal + Algolia JS; latest-only via /terbaru; international SE Asia perspective
- Niaga.Asia: WordPress /?s= search; business/economy focus; Kalimantan coverage

## [0.8.5] - 2026-05-16

### Added
Expand Down
2 changes: 1 addition & 1 deletion docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Test that everything works:
# Check available scrapers
newswatch --list_scrapers

# Should show 42 stable scrapers
# Should show 60 stable scrapers
```

## Your First Scraping Session
Expand Down
109 changes: 64 additions & 45 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

news-watch scrapes structured news data from Indonesia's top news websites with keyword/date search and latest-news monitoring.

The current stable release supports 48 query-backed Indonesian news scrapers. No investigating or quarantined sources remain.
The current stable release supports 61 Indonesian news scrapers with search or latest mode.

## Installation

Expand Down Expand Up @@ -49,50 +49,69 @@ print(len(latest))

## Supported News Sources

### Stable (strict keyword search)

| Source | Domain |
|--------|--------|
| Antara News | antaranews.com |
| BBC News | bbc.com |
| BeritaJatim | beritajatim.com |
| Bisnis.com | bisnis.com |
| Bloomberg Technoz | bloombergtechnoz.com |
| CNBC Indonesia | cnbcindonesia.com |
| CNN Indonesia | cnnindonesia.com |
| Detik | detik.com |
| Galamedia | galamedia.pikiran-rakyat.com |
| IDN Times | idntimes.com |
| iNews | inews.id |
| Investor Daily | investor.id |
| Jawapos | jawapos.com |
| Jakarta Post | thejakartapost.com |
| JPNN | jpnn.com |
| Katadata | katadata.co.id |
| Kompas | kompas.com |
| Kontan | kontan.co.id |
| Kumparan | kumparan.com |
| Liputan6 | liputan6.com |
| Media Indonesia | mediaindonesia.com |
| Merdeka | merdeka.com |
| Metro TV News | metrotvnews.com |
| Mongabay Indonesia | mongabay.co.id |
| Okezone | okezone.com |
| Pikiran Rakyat | pikiran-rakyat.com |
| Poskota | poskota.co.id |
| Republika | republika.co.id |
| RM.ID | rm.id |
| RRI | rri.co.id |
| SINDOnews | sindonews.com |
| Suara | suara.com |
| Suara Merdeka | suaramerdeka.com |
| Surabaya Pagi | surabayapagi.com |
| Tempo | tempo.co |
| Tirto | tirto.id |
| Tribunnews | tribunnews.com |
| TVOne | tvonenews.com |
| TVRI News | tvrinews.id |
| Viva | viva.co.id |
| Source | Slug | Search | Latest | Notes |
|--------|------|--------|--------|-------|
| Antara News | `antaranews` | Yes | Yes | Strict search |
| BBC News | `bbc` | Yes | Yes | Strict search |
| Bali Post | `balipost` | No | Yes | Latest-only |
| Berita Jatim | `beritajatim` | Yes | Yes | Strict search |
| BeritaSatu | `beritasatu` | Yes | Yes | Strict search |
| Bisnis.com | `bisnis` | Yes | Yes | Browser required; Strict search |
| Bloomberg Technoz | `bloombergtechnoz` | Yes | Yes | Strict search |
| CNA Indonesia | `cnaindonesia` | No | Yes | Latest-only |
| CNBC Indonesia | `cnbcindonesia` | Yes | Yes | Strict search |
| CNN Indonesia | `cnnindonesia` | Yes | Yes | Strict search |
| DailySocial | `dailysocial` | Yes | Yes | Strict search |
| Detik | `detik` | Yes | Yes | Strict search |
| Fajar | `fajar` | Yes | Yes | Strict search |
| Galamedia | `galamedia` | Yes | Yes | Strict search |
| Gatra | `gatra` | Yes | Yes | Strict search |
| Grid | `grid` | Yes | Yes | Strict search |
| Harian Jogja | `harianjogja` | Yes | Yes | Strict search |
| Hipwee | `hipwee` | Yes | Yes | Strict search |
| IDN Times | `idntimes` | Yes | Yes | Browser required; Strict search |
| Investor.id | `investor` | Yes | Yes | Strict search |
| JPNN (Jawa Pos News Network) | `jpnn` | Yes | Yes | Strict search |
| Jakarta Globe | `jakartaglobe` | Yes | Yes | Strict search |
| Jakarta Selaras | `jakartaselarascoid` | Yes | Yes | Tag/sitemap filtered |
| Jawa Pos | `jawapos` | Yes | Yes | Strict search |
| KBR | `kbr` | Yes | Yes | Strict search |
| Kaltim Post (Borneo24) | `kaltimpost` | Yes | Yes | Strict search |
| Katadata | `katadata` | Yes | Yes | Strict search |
| Kompas | `kompas` | Yes | Yes | Strict search |
| Kontan | `kontan` | Yes | Yes | Strict search |
| Kumparan | `kumparan` | Yes | Yes | Strict search |
| Liputan6 | `liputan6` | Yes | Yes | Browser required; Strict search |
| Media Indonesia | `mediaindonesia` | Yes | Yes | Strict search |
| Merdeka | `merdeka` | Yes | Yes | Strict search |
| MetroTV News | `metrotvnews` | Yes | Yes | Strict search |
| Mojok | `mojok` | Yes | Yes | Strict search |
| Mongabay Indonesia | `mongabay` | Yes | Yes | Strict search |
| Niaga.Asia | `niagaasia` | Yes | Yes | Strict search |
| Okezone | `okezone` | Yes | Yes | Strict search |
| Pantau.com | `pantau` | Yes | Yes | Strict search |
| Pikiran Rakyat | `pikiranrakyat` | Yes | Yes | Browser required; Strict search |
| Poskota | `poskota` | Yes | Yes | Strict search |
| Project Multatuli | `projectmultatuli` | Yes | Yes | Strict search |
| RM.ID (Rakyat Merdeka) | `rmid` | Yes | Yes | Strict search |
| RMOL | `rmol` | Yes | Yes | Tag/sitemap filtered |
| RRI (RRI.co.id) | `rri` | Yes | Yes | Strict search |
| Republika | `republika` | Yes | Yes | Browser required; Strict search |
| SINDOnews | `sindonews` | Yes | Yes | Strict search |
| SWA | `swa` | Yes | Yes | Strict search |
| Suara | `suara` | Yes | Yes | Browser required; Strict search |
| Suara Merdeka | `suaramerdeka` | Yes | Yes | Strict search |
| Surabaya Pagi | `surabayapagi` | Yes | Yes | Strict search |
| TVOne | `tvone` | Yes | Yes | Strict search |
| TVRI News | `tvrinews` | Yes | Yes | Strict search |
| Tempo | `tempo` | Yes | Yes | Strict search |
| The Jakarta Post | `jakartapost` | Yes | Yes | Browser required; Strict search |
| Tirto | `tirto` | Yes | Yes | Browser required; Strict search |
| Tribunnews | `tribunnews` | Yes | Yes | Strict search |
| VOA Indonesia | `voaindonesia` | Yes | Yes | Strict search |
| VOI.id | `voi` | Yes | Yes | Strict search |
| Viva | `viva` | Yes | Yes | Strict search |
| iNews | `inews` | Yes | Yes | Strict search |

## Important Considerations

Expand Down
2 changes: 1 addition & 1 deletion docs/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ Some sources (e.g., Tirto) may require browser automation. Ensure Playwright is
playwright install chromium
```

All 42 scrapers are stable and included in the runtime. No investigating or quarantined sources remain.
All 60 scrapers are stable and included in the runtime. No investigating or quarantined sources remain.

Try:
```bash
Expand Down
100 changes: 100 additions & 0 deletions src/newswatch/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,17 @@ class ScraperEntry:
note="promoted to stable; rebuilt with sitemap keyword filtering; 2026-04-18",
supports_latest=True,
),
ScraperEntry(
"fajar",
"Fajar",
"fajar", "FajarScraper",
status="stable",
strict_search=True,
concurrency=5,
smoke_keyword="ekonomi",
supports_latest=True,
note="new; WordPress /?s= search; Sulawesi regional gap; 2026-05-16",
),
ScraperEntry(
"galamedia",
"Galamedia",
Expand All @@ -121,6 +132,28 @@ class ScraperEntry:
note="promoted to stable; /search?q= endpoint + div.latest__item + keyword-in-title filtering; 2026-04-24",
supports_latest=True,
),
ScraperEntry(
"grid",
"Grid",
"grid", "GridScraper",
status="stable",
strict_search=True,
concurrency=5,
smoke_keyword="ekonomi",
supports_latest=True,
note="new; /search?q= endpoint; /read/{id}/{slug} article pattern; lifestyle/pop culture gap; 2026-05-16",
),
ScraperEntry(
"hipwee",
"Hipwee",
"hipwee", "HipweeScraper",
status="stable",
strict_search=True,
concurrency=5,
smoke_keyword="politik",
supports_latest=True,
note="new; WordPress /?s= search; youth/satire/opinion angle; 2026-05-16",
),
ScraperEntry(
"idntimes",
"IDN Times",
Expand All @@ -143,6 +176,17 @@ class ScraperEntry:
note="promoted to stable; no-result gate added 2026-04-18",
supports_latest=True,
),
ScraperEntry(
"jakartaglobe",
"Jakarta Globe",
"jakartaglobe", "JakartaGlobeScraper",
status="stable",
strict_search=True,
concurrency=5,
smoke_keyword="indonesia",
supports_latest=True,
note="new; /search/{keyword} custom endpoint; English coverage; 2026-05-16",
),
ScraperEntry(
"jakartapost",
"The Jakarta Post",
Expand Down Expand Up @@ -233,6 +277,39 @@ class ScraperEntry:
note="flaky for finance terms",
supports_latest=True,
),
ScraperEntry(
"mojok",
"Mojok",
"mojok", "MojokScraper",
status="stable",
strict_search=True,
concurrency=5,
smoke_keyword="politik",
supports_latest=True,
note="new; WordPress /?s= search; satire/opinion/youth angle; 2026-05-16",
),
ScraperEntry(
"niagaasia",
"Niaga.Asia",
"niagaasia", "NiagaAsiaScraper",
status="stable",
strict_search=True,
concurrency=5,
smoke_keyword="ekonomi",
supports_latest=True,
note="new; WordPress /?s= search; business/economy; Kalimantan coverage; 2026-05-17",
),
ScraperEntry(
"jakartaselarascoid",
"Jakarta Selaras",
"jakartaselarascoid", "JakartaSelarasScraper",
status="stable",
strict_search=False,
concurrency=5,
smoke_keyword="ekonomi",
supports_latest=True,
note="new; custom CMS; RSS/sitemap keyword filtering; /detail/{id}/{slug} articles; 2026-05-17",
),
ScraperEntry(
"mongabay",
"Mongabay Indonesia",
Expand Down Expand Up @@ -274,6 +351,18 @@ class ScraperEntry:
note="promoted to stable; Playwright CSE bypass CF 1015; 2026-04-23",
supports_latest=True,
),
ScraperEntry(
"cnaindonesia",
"CNA Indonesia",
"cnaindonesia", "CNAIndonesiaScraper",
status="stable",
strict_search=False,
supports_search=False,
concurrency=5,
smoke_keyword="politik",
supports_latest=True,
note="new; Drupal + Algolia JS search; latest-only via /terbaru; 2026-05-16",
),
ScraperEntry(
"poskota",
"Poskota",
Expand All @@ -295,6 +384,17 @@ class ScraperEntry:
note="promoted to stable; rebuilt with Playwright tag page + keyword URL filter; 2026-04-18",
supports_latest=True,
),
ScraperEntry(
"rmol",
"RMOL",
"rmol", "RMOLScraper",
status="stable",
strict_search=False,
concurrency=5,
smoke_keyword="politik",
supports_latest=True,
note="new; /tag/{keyword} tag-based search; political commentary; 2026-05-16",
),
ScraperEntry(
"rri",
"RRI (RRI.co.id)",
Expand Down
Loading
Loading