Remote Browser Control with WebKit Debug and JavaScript
In this post I’ll show you how to remotely control your Google Chrome browser using JavaScript and scrape some data, even if it’s on an AJAX powered website or behind HTTPS authentication. Nice?
I LOVE WebKit and now it got even sweeter. Very recently there was a small buzz about a new feature: WebKit Remote Debugging that allows to use Element Inspector remotely! (since it’s essentially just a web page and some javascript + websockets). What’s the real icing on the cake for me, is the ability to plug-in a different interface. I wrote a simple interface that can execute several pieces of JavaScript and return the values back to you. Here’s how to set it up.
Open Terminal and find the Google Chrome executable. To use remote debugging, run it with a special parameter:
$ ./Google Chrome –remote-debugging-port=9222
If you get:
Try symlinking Versions:
The browser should start normally. Now go to a different browser, e.g. Safari and check out:
Select a page and you should see the Inspector.
That’s all nice and neat. But let’s see my remote script: http://gist.github.com/972742
- To connect to the debugger, we’re using WebSockets. Change the page number based on the link from “http://localhost:9222″. Every “Tab” has a different “Page” number.
// Set page number! var host = "ws://localhost:9222/devtools/page/5"; socket = new WebSocket(host);
- To execute JavaScript I wrapped a JSON-RPC-like command into a method with callback. More protocol schema here.
function remoteEval(scriptString,callback) {
seqCallback[seqNo] = callback;
socket.send('{"seq":'+seqNo+',"domain":"Runtime",'+
'"command":"evaluate","arguments":{"expression":"'+
scriptString.replace(/["]/g,'\"')
+'","objectGroup":"console","includeCommandLineAPI":false}}');
seqNo++;
}
- And this is how a sample script works:
remoteOnLoad = function(result) {
// We might've ended up on the login page, so let's log in!
if (remoteURL.indexOf("ServiceLogin") > 0 )
remoteEval( " document.getElementById('Email').value = 'username'; "+
" document.getElementById('Passwd').value = 'password'; "+
" document.getElementsByTagName('form')[0].submit(); "
, function(result) { alert(result); } );
// We're home!
if (remoteURL.indexOf("mail.google.com/mail/") > 0 )
remoteEval( " try { document.getElementById('canvas_frame')"+
".contentWindow.document.getElementsByClassName( 'md' )[0].innerText } catch(e) { -1 }"
, function(result) { // Waiting for AJAX. Try again in 2 sec.
if (result == -1) { setTimeout(remoteOnLoad,2000); }
else { alert(result); } } );
// This should return scraped information
// about your data usage on gmail!
// E.g. You're currently using 150MB out of 7000MB
} ;
// This happenes first:
remoteEval(" location.href = 'http://gmail.com' ");
//Let's go to Gmail!
Don’t know how about you, but I just wanna wrap it in Node.js, run it somewhere on Linux with a dummy X11 server for Chrome and write lot’s of crazy tasks, so it does it all for me! Imagine e.g. that instead of that alert(result) you make it a WebHook or a service?
. Or maybe you have to deploy your application to even more than two machines? Well, here’s a small cheat sheet.
)
