You may be frustrated at Microsoft Azure’s lack of ability to power off the Network gateways especially when they are chewing up resources and $. Unfortunately Azure provides no current way to power the gateways down so the only current solution is to delete them, however you need to delete them in the right order to remove the service pre-requisites.

Example:

Failed to delete virtual Network Gateway

Failed to delete virtual network gateway 'UKSouthGateway'.
Error: Gateway /subscriptions/xxxxx-xxxx-xxx-xxxx-xxxxxxx/resourceGroups/

Solution

The gateway devices must be deleted in a specific order

1. Connections (both sides)
2. Local Network Gateway (both sides)
3. Virtual Network Gateways (both sides)
4. IP Ranges (only if necessary)