edward/newegg-hdd
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Switch to Playwright to bypass Newegg bot detection, closes #2

Browse source 
Newegg now blocks requests-based scraping; replace with Playwright
using headless Chromium with mouse simulation to pass bot detection.
Also fix hardcoded build output path, use os.makedirs for nested dirs,
update category labels (HDD/SATA SSD/NVMe SSD), drop near-empty 2.5"
internal and laptop HDD categories, and fix invalid HTML in index
template (h2 inside table cells).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Edward Betts 2026-04-03 15:06:49 +01:00
parent 55bb3697b6
commit 2dc799ecaa
3 changed files with 62 additions and 42 deletions
Download patch file Download diff file Expand all files Collapse all files

2
requirements.txt
View file

@ -1,4 +1,4 @@
requests
playwright
lxml
jinja2
daiquiri