2013-09-29:
Setting request headers on Selenium Webdriver with Ruby

Abstract | History/Motivation | The Proxy Solution | Other Solutions

Abstract

Selenium Webdriver (2.0) is the industry standard for web testing and it supports many browsers (Chrome, Firefox, IE, Safari, etc..).

Unfortunately Webdriver does not support setting/modifying the request headers or getting the response codes or response headers, and in fact the Selenium team has made it very clear that this will not be supported in a heavily contentious debate with the users.

History/Motivation

I have a project where I need to use a proxy with Selenium Webdriver that uses authorization and it needs to be fully automated. Normally I would add an HTTP header such as: 
Proxy-Authorization: Basic ZG8geW91IHRoaW5rIEkgYW0gc3R1cGlkPw==
Unfortunately, Webdriver does not allow the user to change the HTTP headers, and it never will. (and you can't get the response headers or status either)

You can pass the user@passwd in the URL on some browsers, but that doesn't work on all browsers, and I think that's only for normal authentication and not proxy authentication. Regardless, it didn't work for me.

My next thought was that since I could alter the proxy source, I could send the authorization (I couldn't remove authorization since I needed to distinguish between users) using a cookie, but that would require setting a cookie before each web page, and Webdriver does not allow that, and it never will.

To further complicate the matters, I also need to set the referer for the pages in my testing since the referer field is used by many websites and I need the testing to be like real usage.

The Proxy Solution

So now what? I tried to find an alternative to Selenium, but I needed cross-browser support, and there didn't seem to be any reasonable options. I looked into a number of Selenium solutions, but most of them were Firefox only. Finally I settled on using a second, local proxy, and after trying to deal with other people's code, I realized that I could simplify my ruby proxy code (which is threaded and supports the vast majority of sites) to make a local proxy which could rewrite headers, do authorization, add cookies, set the referer, and so on.

And best of all I realized that since I could just run the proxy in a thread inside my test/Selenium code, then I could control the headers directly, though I also wrote code that would let me change the headers by calling a special proxy URL as well, in case I wanted to run it as a separate script.

First the script is available as a standalone simple proxy, you can set proxy or headers in the script or else by calling the proxy URL, as an example, setting the proxy and an authorization header: 

http://localproxy/proxy=getdave.com:8080
http://localproxy/addHdr=Proxy-Authorization:Basic%20ZG8geW91IHRoaW5rIEkgYW0gc3R1cGlkPw==
Here's the script - incidentally this also works as just a simple proxy and there aren't many working, simple ruby proxy examples available (using just TCPSocket): And here's a full Selenium Webdriver example that starts up the proxy in a thread, sets up the proxy settings (to use another proxy with authorization) and then uses chrome to fetch that page (first through the local proxy and then through the external proxy). It also sets the referer, so you could easily remove the setting of the external proxy if you just want to use the local proxy to rewrite headers for a regular Selenium test.

Other Solutions
URL user/password
Example: 'http://user@password:somedomain.com/'
https://github.com/webmetrics/browsermob-proxy
Ruby: http://elementalselenium.com/tips/17-retrieve-http-status-codes
ProxyLight
http://www.supermind.org/blog/968/howto-collect-webdriver-http-request-and-response-headers
Javascript/Firefox
http://stackoverflow.com/questions/6478672/how-to-send-an-http-requestheader-using-selenium-2
Selenium RC
http://blog.mogotest.com/2010/06/23/how-to-perform-basic-auth-in-selenium/

Back to Solutions.

DaveSource.com - Dave's geek site GetDave.com - all the current Dave Pointers. MarginalHacks - I have an elegant script for that, but it's too small to fit in the margin.