Scripts

Google Scholar citations

Refresh _data/citations.json (profile stats, per-paper counts, citing works, leaderboard):

python3 scripts/update_scholar_citations.py
python3 scripts/update_scholar_citations.py --delay 2.5

The fetcher is ordinary HTTP (stdlib only). Headless Chromium gets a Scholar CAPTCHA from the same address that still serves the pages to a normal request.

Options:

• --delay 2.5 — pause between requests
• --no-merge — do not union with existing citing-paper lists
• --dry-run — print JSON to stdout

Scheduled refresh (polsci server)

The weekly refresh runs on the polsci server, not on GitHub Actions. GitHub runners are often rate-limited; the university server is not, as long as the request is plain HTTP rather than headless Chrome.

Setup on polsci (~/scholar-updater/):

• venv/ — Python venv with Playwright + Chromium headless shell
• site/ — clone of this repo, pushed via a repo-scoped deploy key (~/.ssh/id_ed25519_ghdeploy, write access to this repository only)
• update-citations.sh — fetch/reset to origin/master, scrape, commit + push only if _data/citations.json changed
• cron: Mondays 06:15 (server time), logs to ~/scholar-updater/cron.log

The GitHub Action (.github/workflows/update-scholar-citations.yml) remains as a manual fallback (workflow_dispatch only) and will usually fail with a CAPTCHA. Merge-on-truncate keeps previous citing-paper lists if a scrape is partial.