--- title: "Poor man's selenium" summary: "When chrome-driver just decides to not work on your development workstation today" date: "2025-11-09T00:00:00-08:00" tags: ["why, just why", "webdev"] categories: ["Web development"] ShowToc: false --- Sometimes if you just need some light web scraping, nothing too complicated. Other times, chrome webdriver will just decide to not work, but that's pretty rare. Or you really need to extract some useful bits of information, but you don't have selenium already installed handy. Maybe you're on a borrowed laptop that resets on every checkout. In these cases, the devtools console will do. No need to take out selenium. The essence here is you can interact with the page in a few basic, but often good enough ways: - `document.querySelector`, obviously - `outerHTML` on HTMLElement to stringify a DOM node. Do some string manipulation to clean it up. - optionally do some string or DOM processing (clone the node first) - when done, copy the array to clipboard. Now you have scraped output! - simulate user clicking with `click()` on HTMLElement - This might not always work, so your milage may vary. - study their code, and emulate some behaviors yourself to "interact" with the page - For example, if button fetches HTML and replaces a subtree, you can just do the fetch yourself. - Only really possible if it's using jQuery or simple vanilla JS. - for navigation... - Replace `window.location.href` to navigate, but **be careful**: this wipes out your devtools console. If you really need to do this, spending some time to craft a full script that you can just paste into console will pay off. - Modify `` with `target="_blank"` to batch open links in new tabs. - call `setTimeout` recursively - This isn't super easy to code. Consider quickly cobble together an async/await helper like https://stackoverflow.com/a/33292942 ... ... ... Ok I'll confess, the real reason is chrome webdriver broke on me. As they say: I did this not because it was easy, but because I thought it would be easy. Easier than selenium, which honestly I'm not sure if that's true. I guess it was kind of fun?