I have been hard at work the last two months realizing many improvements to the underlying CGI Proxy code that I have always wanted to do but never had time for.

As a result we now have:

  • Rudimentary Hotmail support
  • A cleaner looking interface
  • Improved RC4 url handling
  • Proper proxying of meta refresh tags
  • Various techniques to reduce abuse and thereby improve performance for everyone
  • Improved Youtube support
  • HTML obfuscation
  • Better support for SSL
  • Several CGI Proxy Bug Fixes
  • Backend caching improvements
  • Various changes that make general maintenance much easier
  • URL Expiration

All of the above things represent a major breakthrough that will allow us to provide much better performance to end users while maintaining the profitability of providing our free service. It also means that one major part of developing free proxies is done. With the backend having most of the features and bug fixes that I’ve wanted for a long time, I can now focus my efforts on building the front end to this system, which will allow me to give free proxy hosting to anyone who wants it. Our anticipated release date is early September.

One change in particular that I’m proud of is the HTML Obfuscation routine. This encodes all the html on every page you view via proxy into hex code. In this way, content based filtering solutions will be utterly frustrated when they try to categorize the content on any page you view via proxy. Your browser can decode all this content on the fly by using a simple javascript routine, however, web based filtering solutions will be unable to do this, as they are not sophisticated enough to decode javascript. This technology is optionally available on the home pages and ancillary pages on the proxies I run, both for myself and for others. I have activated obfuscation support on all the proxies owned by myself. This means that web based filters will have to manually categorize any of my proxies, keeping them from being blocked for longer periods of time and at a larger number of institutions and networks. As a noticed side benefit, obfuscating all html code improved the performance of all proxies by eliminating abusive bot access to the proxies I run. I suppose the vast majority of bots cannot parse links on pages that were dynamically inserted with javascript. So in addition to protecting user’s privacy and making my site harder to block, I scored a big win on performance by eliminating a huge source of automated and abusive access.
Nothing comes for free however, and this particular technique has a couple of drawbacks.

  • First of all, because all page content is wrapped in a layer of javascript, it makes it nearly impossible to support proxying of javascript code via my service. Javascript does not interpret page elements that have been dynamically written to the page in the same way that it can reference elements that were hard coded into the page. As a result, most websites out there would have to have massive changes to their codebase made in order to be compatible with the html obfuscation system. Because I am no javascript expert, and because javascript behaves inconsistantly in general and especially accross platforms, support for javascript proxying would almost certainly come at a cost of disabling obfuscation.
  • Secondly, wrapping all html in a hex encoding greatly increases bandwidth usage, since the majority of characters that used to take up just one byte now are represented by three characters apiece. To deal with this problem, I had to enable Gzip compression on my proxy, which necessarily increases cpu usage, but dealt with the huge bandwidth usage increase from this change.
  • Third, by obfuscating the home pages of the proxies, the content of my sites becomes invisible to search engines and all bots. This means that the benefit of having my sites stay unblocked longer is offset by the corresponding reduction in traffic that search engines can send me, because there are virtually no keywords whatsoever that spiders can use to categorize my site. Users who use contextual advertising on their homepage (such as adsense) need to be careful, because enabling obfuscation on the entirety of those pages will make it impossible for the contextual ad network to properly target your ads.

I believe that the massive improvements in user privacy, unblock-ability, and reduction in abuse more than make up for the above tradeoffs, but I felt I should explain them since they are significant. When I launch, users will be able to use tags on their home pages to decide what parts (if any) of their home page should be obfuscated. Meanwhile, I am deciding that html obfuscation is more important than javascript support, especially given the fact that javascript support in the current version of cgi proxy is incredibly buggy to begin with.

Thank you all for visiting my site. I am very excited to be working on the free proxies project.

c40