edward/newegg-hdd
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Fix Playwright timeout on slow servers

Browse source 
wait_until="networkidle" requires zero network activity for 500ms,
which times out on ad-heavy pages when running on a server. Switch to
domcontentloaded and wait explicitly for div.item-container instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Edward Betts 2026-04-03 15:32:38 +01:00
parent d161e82865
commit d4308685f7
1 changed files with 2 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

3
crawl.py
View file

@ -74,7 +74,8 @@ def get_product_list(n: str, page: Optional[int] = None) -> str:
"Object.defineProperty(navigator, 'webdriver', {get: () => undefined})" "Object.defineProperty(navigator, 'webdriver', {get: () => undefined})"
) )
pg = context.new_page() pg = context.new_page()
pg.goto(url, wait_until="networkidle", timeout=60000) pg.goto(url, wait_until="domcontentloaded", timeout=60000)
pg.wait_for_selector("div.item-container", timeout=60000)
if "areyouahuman" in pg.url: if "areyouahuman" in pg.url:
logger.info("bot detection triggered, simulating mouse movement...") logger.info("bot detection triggered, simulating mouse movement...")