My first Wireless Distribution System setup

The company I work for was asked to provide a Wifi solution for a little known hideaway place just under an hours drive away. The place in question is out near Sherwood Forest and has a reception which is basically a converted wood lodge and then a few wood lodges. That's really all there is to it!!

Anyway we were asked if it was possible to install an Access point on the reception and then install client devices inside the lodges themselves and housed away out of view. We were sceptical it would work and our scepticism proved to be correct! As I mentioned earlier it's at the edge of Sherwood forest which meant there was no shortage of trees around! At this time of year the trees were also in full bloom (which is good from a surveying point of view) and being Britain the trees were wet! Now Wifi uses RF and RF does not get on well with water!! So our initial survey doing it this way was a failure. We did hold the devices outside the cabin after confirming no signal inside the cabin and some that were close enough to the reception with a reasonable line of sight did manage to get one bar of signal strength but it was never going to work like this!

We suggested that as the park was going to expand that an ideal solution would be to build a lattice tower and mount an access point with a 360 degree high gain sector aerial that could be viewed from all the lodges (present and future). However customers being customers wanted it installing sooner which meant there was not enough time to build up the lattice tower. So it was decided that as a lot of the lodges did have line of sight to each other (including one near the reception that houses the Internet router) that we would mount Ubiquiti Nano locos on each lodge and set them up in Access Point WDS mode. Then internally we would have indoor TPlink access points that could all be mounted behind the TVs out of sight.

Before going to site I had a plan of the park and had to plan which Access Points would talk to which Access Point. I had 15 Ubiquiti Nano Locos to program so I drew on the map which loco would point where and how many MAC addresses would be needed for each Loco. For Access Point WDS mode on these devices you can list up to 6 MAC addresses of other devices that it will talk to so I had to plan this carefully.

The above diagram roughly shows how they're setup. As you can see this isn't ideal as some Locos are responsible for a lot of connections even if they aren't on their MAC list so a failure of one Loco could take down half the parks Internet access. But despite this it does work and works well.

Cabling worked very well as we used one power supply with a splitter that fed a PoE injector for the Nano Loco and another PoE injector for the internal access point. A small 1/2 meter Ethernet cable then joined each PoE injector via their respective LAN ports. All this is mounted inside the lodges behind the TVs so are out of sight and don't affect the internal aesthetics of the lodges.

Draytek Woes

For years I've used Draytek Vigor routers on and off and have loved them. User friendly and pretty much do as they're told! So when I started migrating customers over from private 4-wire circuits to VPNs, Draytek seemed the natural choice! The routers I chose had to support multiple VPN tunnels (well 2!) and be able to have 1 tunnel connected permanently with the second tunnel able to connect should the first tunnel fail! After looking at the models the Vigor 2930 and Vigor 3300 seemed ideal as they supported a feature called VPN trunking which did exactly what I wanted!

However initially setting this up was hard as the only documentation I could get hold of to set up these features were available on the US website! I only found that out by contacting UK support! NOT a good start! As we were bringing a few customers on board with this the idea was to use the more expensive (and much less user friendly) 3300 at our radio site which has a fibre-to-cab connection and the customers premises would use the 2930.

As we brought our second customer online (although first with these units), we setup them up with a VPN trunk which for a week or so seemed fine. But then they were reporting random loss of connectivity! The whole point of this setup was so they didn't have this problem! I checked the VPN uptimes and sure enough they were a lot lower than the WAN1 (ADSL) uptimes! Without actually needing to the VPN tunnel was failing over to the backup VPN tunnel and then going back to the primary VPN tunnel! This particular customer is literally down the road from the local BT exchange and had one of the best ADSL circuits I've seen! What's worse is that they had drop outs when the failover happened!

After talking with Draytek support and effectively getting nowhere I had to go back to conventional VPNs. I found that if I put the 'backup VPN' in VPN index 0 and the 'always on' primary VPN in index 1 then it seemed to work the way I wanted. However after a couple of months a new problem came to light! Each customer using this setup was using Voice Over IP units as a pair either side of the VPN to stream voice traffic. These units send a UDP stream to the other unit the other side of the VPN and vice versa. Simple and effective setup. However we were getting reports that the customer was getting one way audio traffic!! Further investigation showed that this only happened when the primary 'ADSL' VPN had dropped, failed over to the backup 'wireless' VPN and then returned to the primary VPN when the ADSL connection had been restored and stabilized! Rebooting the VOIP units didn't cure the problem but rebooting the routers did! Not good! Especially when the customer rings out of hours with this fault!!!

We didn't automatically blame the routers and set out to prove it. We replicated the customers setup in our workshop and replicated the failure straight away! Using Wireshark to monitor the ethernet packets going to and from the VOIP units we could see that they were sending their UDP streams to the router. But only one stream came out the other side! The manufacturer of the VOIP units also has software that replicates these units that can be ran on a PC (or anything else that runs Java). Again with a laptop each side of the VPN running this software and Wireshark we were able to prove that the stream was reaching one router but not coming out the other side until that router was rebooted! So there we had it, another Draytek issue!!

Around this time we had another customer come online and had no choice but to install this setup as at the time we had no alternative! We did get a firmware update (that Draytek said they had no intention of making it a readily available firmware upgrade!!!!) and it cured the fault! Just in time for our next customer to come on board!! Things seemed to be running smoothly except that our very first customer who was using a pair of Vigor 2820 routers also had this issue!! That's three different Vigor models all with the same fault!!! Draytek really dragged their heals in providing a firmware update for this model (in fact they didn't give us a firmware update).

We also noticed an issue with the 'fixed firmware' they sent us. The VPN tunnel wouldn't stay up for more than 7 hours without failing over for no reason at all (yup back to square 1 AGAIN!!). I logged on and actually watched the primary and secondary VPN tunnels 'fight' for control for a good 5 minutes before the primary tunnel won! This happened every 7 hours!!

All this happened in just under a year and to cut an already long waffle of a story short Draytek admitted they didn't have the resources to dedicate to fixing the issues and offered us a refund for all the equipment! My MD is also pushing for compensation for the shear man hours lost to this whole episode!!

We have now sucessfully removed all the Draytek routers and replaced them with Mikrotik routers. We can monitor these better and program them to do EXACTLY what we want. Plus since installing them we haven't as of yet had a single call about a fault which they'd caused! I'll write more about these another time :-)

Migrating a private 4 wire circuit to a VPN over ADSL

The company I work for is traditionally a radio communications company. We do so much more nowadays but radio communications is core to the business. As such we look after a lot of taxi companies. Most of the taxi companies we look after had several private 4 wire circuits between their premises and our aerial mast site. A private 4 wire circuit is similar to a telephone line except that all 4 wires are used and they are a dedicated line between two sites. As such they're maintained by either BT or Virgin Media and are expensive.

Each taxi company has a minimum of 2 radio channels (1 has 4). One channel is dedicated for voice and the other(s) for data. Diagram below shows the basic setup for one channel whether it be voice or data:

As already mentioned a dedicated 4 wire private circuit is expensive and most of our customers had at least 2! We came up with a more cost effective and modern solution that also included a backup! So how would we achieve this? ADSL and Wifi! The 4 wire circuit was replaced with a dual WAN router which would have a VPN tunnel configured per WAN port to another router at the aerial mast site. A single VPN connection could of course service more than one radio channel therefore saving the customer £1000s! To get the voice channel across the VPN we used identical VOIP units at either end. They convert line level audio to IP (the main advantage of this is there's no loss and you can literally stream anywhere in the world!) and stream to an identical unit which converts it back to line level audio.

As for the data channels, the operators enter jobs into a PC which then transmits it to an IP-to-serial unit. This then converts the serial data to audio using a modem and then connects to the radio base station. All this equipment is at the aerial mast site so the dispatcher PCs connect to the IP-to-serial unit over the VPN.

As specified earlier the primary VPN connection routes via ADSL. As ADSL fails now and again we had to come up with a backup plan. So we utilised another companies wireless network (whom we have an excellent working relationship with) and put up a wifi connection either by direct line of site to the aerial mast or via other nearby wifi devices which will then route to the aerial mast!

Below is a diagram of the improved setup.

Initially we used Draytek Vigor modems and routers which worked well to begin with but then we had no end of problems which were all VPN related! I'll go through all that in another post. We now use Mikrotik routers which are simply excellent and do exactly what they're programmed to do! The customer has saved £1000s a we have can monitor all IP equipment effectively and know of a fault before the customer does which is how it should be!