Compressing data proxy

Wcool

For work I am in South Africa for a while. I still have access to my Ubuntu server in Ireland.

Data here is at a premium, around 30 euro for 5Gb.

I was wondering if could set up a proxy that compresses data between before it sends it on to me in SA. Something like Opera Mini, but preferably for all ports and data types. My computers in SA are mainly Windows. My server in Ireland is Ubuntu.

Just wonder if there is some software out there that compresses before sending and receiving? And it has to work for Windows clients.

Thanks

niallb

You could look into using putty to open a compressed connection to your ubuntu box and run it as a socks proxy. There's a lot of windows apps which can take advantage of that.

Do you have a machine at your end which is routing for the windows machines,
or does it need to be a simple windows app all the way?

Khannie

You are in luck. OpenSSH supports this.

ssh -C -D <port> <your server in Ireland>

will set up a socks proxy on localhost that will transfer your browsing over a compressed link. You would then set your browsers socks proxy to localhost:<port> and that should be it for browsing or anything else that can use a socks proxy (chat clients etc.).

If bandwidth is at a real premium you could add -o CompressionLevel=9 into the mix.

edit: There is no way to do it for all data types though. You can use the -C -o CompressionLevel=9 thing if you're using ssh or scp or any derivative though.

Wcool

Thanks for the tips, I actually do use Putty on Windows and have polipo installed on the server in Ireland. Never knew SSH can do SOCKS as well.

This is more for http(s) i guess, which is probably the biggest chunk anyway.
Now i need to find a process to make it wife friendly as well.

I am in the process of getting a 3G router on the SA side for the LAN but I don't think I can put openwrt or something similar on it to create a permanent compressed connection on it (that would solve every issue isn't it? I could have a permanent compressed data connection between here and there?)

niallb

I figured if you had an ubuntu server in Ireland, you'd have putty installed on the windows machines!
The command line options listed by Khannie are the same things you're looking for either way, and you should be able to set up a batch file on your desktop to launch the connection, and even add it to the machine's startup folder if necessary.

If you have a linux machine, or a spare old PC/laptop on your end, you can set it up as a router and have more options. Setting up a compressed VPN tunnel from that PC to the ubuntu server in Ireland would allow you take advantage of it for all data types by setting the local linux box as your router. That would then use your 3G router as it's default gateway.

Setting up a proxy on the local box would save even more bandwidth. Set up your local dns to resolve wpad to the proxy box, and it'll be easy to find. If you use it as your proxy, and tell it that the dublin polipo is its upstream proxy, you'll pull everything through in a compressed format iff it isn't already there on your LAN.

3G links will particularly benefit from DNS caching as well, so dnsmasq or pdnsd or similar is also a good ingredient to add to the mix on a local accelerating proxy.

Wcool

I knew i should have brought my sheeva plug computer but I did not
Almost worth buying one for this purpose, i normally use 5Gb a day!

yutta

I'd recommend switching off encryption to save bandwidth.

ssh -N -g -f -C -o CompressionLevel=9 -o Cipher=none Wcool@irish.ubuntu.server -L 6999:irish.ubuntu.server:80

You'll need one for https too:

ssh -N -g -f -C -o CompressionLevel=9 -o Cipher=none Wcool@irish.ubuntu.server -L 7000:irish.ubuntu.server:443

Now set your browser's proxy setting to http:irish.ubuntu.server:6999 and https:irish.ubuntu.server:7000

Khannie

yutta wrote: »

I'd recommend switching off encryption

Madness! I am far too paranoid to ever do such a thing.

You could switch it off and just have the socks proxy thing on the go, meaning you only have to set up one:

ssh -C -D <port> -o Cipher=none -o CompressionLevel=9 <remoteserver>

Wcool

In the end, i ticked the box Compression Enabled in Putty and setup a tunnel to Ireland. Not sure how much it saves though...
Will try to do some tests

Khannie

Probably around 50% for average data, give or take. For browsing probably not so much. Images are already compressed. Text data like css and html compresses incredibly well though (usually > 90% compression).

Capt'n Midnight

what data are you trying to transfer ?


on the offchance that you haven't already considered...

setup a squid proxy locally if you are getting a lot of repeat web traffic

look at rsync and diff if you only need to transmit small changes in files or could mirror content locally


reminded me of this
http://mg.co.za/article/2009-09-10-pigeon-carries-data-bundles-faster-than-telkom

Wcool

Mainly webpages, Facebook, and a little bit of NX.
But no movies, 5 Gb (for 2 people) per month is just not enough

Speeds are ok here actually, it's the cap that's killing. Until recently, SA was only connected by 1 undersea cable, now 3.

Still, data is at a premium here. even for ADSL.

I hope to have a dedicated linux box setup soon (my htpc running xbmc on ubuntu. I will add squid there). Good idea

h57xiucj2z946q

Installing OpenVPN might also be an option?

I know there is a setting toggle the client and server to enable compression.

Wcool

Ari CoolS Appendix wrote: »

Installing OpenVPN might also be an option?

I know there is a setting toggle the client and server to enable compression.

Quite frankly I don't dare to setup OpenVPN from here. I have tried a couple of times over the years and always my SSH connection conks out and I mess up my network. In the end I can only regain access by plugging a screen and keyboard into my server.
Scary stuff OpenVPN

yutta

Wcool wrote: »

Quite frankly I don't dare to setup OpenVPN from here. I have tried a couple of times over the years and always my SSH connection conks out and I mess up my network. In the end I can only regain access by plugging a screen and keyboard into my server.
Scary stuff OpenVPN

You set the keep alive variable to every 10 seconds if your ssh keeps conking out...

eth0

Wcool wrote: »

Mainly webpages, Facebook

Ah fer feck sake the trouble people go through to get their FB fix, just get your hands on a box of these and you're sorted

Standard Toaster

Might be worth your while to set your agent id on your browser to a mobile platform and have mobile versions of sites (that support them) load to save yet more bandwidth.

niallb

Any feedback on how this is going for you?

Wcool

Well, I don't really see much difference. Problem is that my wife does not really like to SSH before surfing or having this occasional Proxy error.

Another fact is that web pages less and less are made of text but binary blobs that can not be compressed anymore.

But I only have gone for the low tech ssh connections, I still need to setup Squid, but don't dare to setup a permanent VPN as i am afraid to lose my server in Ireland.

In the next few days I am moving to a more permanent home and I intended to pay more attention to this problem, I have a Acer revo with me with XBMC on it. I am going to wipe it and install full Ubuntu + XBMC on it and then try again.

But ideally the router would do this all and I don't have the right hardware here

niallb

Wcool wrote: »

Another fact is that web pages less and less are made of text but binary blobs that can not be compressed anymore.

But I only have gone for the low tech ssh connections, I still need to setup Squid, but don't dare to setup a permanent VPN as i am afraid to lose my server in Ireland.
...
But ideally the router would do this all and I don't have the right hardware here

squid will resend you repeated content, so save you bandwidth where compression won't help.

I'd suggest starting your revo with the xbmc-live cd. It's ubuntu underneath, and you can connect in and add squid.
I'll strongly suggest setting up a DHCP and DNS server in the revo and turning off the one on the router. Something simple like dnsmasq is plenty.
Search for auto configuring proxy with DHCP, and add entries in dns for wpad pointing to your proxy as well as option 252 in DHCP to tell systems a proxy configuration is available. That will take a bit if reading, but will end up working automatically with no config on the client end.
As the xbmc box is probably on most of the time, there's no extra power drain either.
Squid can of course benefit from a compressed pipe to Ireland, but a lot of benefit will still happen without it. Likewise
using dnsmasq on the revo as the default dns server for your network will speed up resolution by cacheing results and everything will seem a little snappier.

You could set the default route on the revo to use the ssh type connection to dublin by giving dublin as the gateway. Use dhcp to give the default route announced to your network as being via the revo rather the router and everything can take advantage of it.

Hope you get a chance to put some of this in place - it should make a big difference.
Good luck.

Wcool

Thanks a lot Niall,

What I wonder: at the moment I have 2 windows laptops connected via connection sharing on an adhoc network. It sucks as I need to have 1 laptop always on, and this is going to Ubuntu on the Acer.
BUT, what I noticed is that my Android phones, tablet and Kindle can not connect to an adhoc network.

If I setup the Acer as the router + squid etc, can i set it up as an AP? or do I need to plug in a separate AP/Wireless router

I have so many options at the moment. I bought this cheap wireless 3g router from Dealextreme. Supposedly you plugin your dongle and then it will create an AP LAN for you. Of course I could not get it to work Still trying.
But ideally that would be the AP and I could plug the Acer into it and install squid on that rather than the Acer be the AP.

niallb

Those 3G routers can be handy - make sure you've removed the PIN lock from the dongle before using it, and have the settings on the dongle set up so that the connection is brought up by telling it to dial the default connection (dial number should just be as simple as *99# or similar). If you're in South Africa, Vodacom's forums have good material on setting up these modems. (I used them a lot a few years ago as I was an early adopter of 3G by Irish standards!) Can you connect to the router at all to configure it?

Depending on the wireless card in the Revo, it may well be possible to set it up as an AP. That'd be the ideal solution. lspci will list out the devices, and lsmod should give you some idea of what wireless module it's using if you don't already know.
Which model Revo is it?

Ad Hoc is ... Get rid of it