Your own public development URL.
Tue, May 16, 2017How to setup a public URL to access your local machine? and why you would even need that.
There are a lof of use cases where a public URL is required: * you want to expose a webserver running on your local machine to the internet so that a colleague or a customer may have a look at it. * you are using a service, like twillio, which allows you to setup webhooks URL: the service will call those URLs to notifiy you of something. * you want to test an OAuth integration, with Facebook or Twitter, and you want to provide public callback URL.
It is always an option to install your code on a public server and have those requests hit this server. During development, it is much more convenient to have thos requests hit your local machine so that you can debug and/or see logs in real time.
There is already a great solution for that: ngrok. This is a wonderful tool, developed by Alan Shreve, very easy to use and perfect for webhooks.
You can also build your own custom redirecting solution using SSH, and more specifically its, SSH Reverse port forwarding feature.
A high level description of this redirecting solution would be:
Your local machine (a) is behind a firewall, either a corporate firewall or, if you are at home, behind your ISP router. In both case, you don’t want to poke a hole in your router to let public internet access your machine.
The idea is then to have a public machine (b) which would act as a gateway between the public internet and your local machine.
pre requisites
In order to implement this solution, you need:
- a domain name: thhis domain will be used to get public URL(s) for your machine
- a public DNS: as you want to have public URL to access your server, a DNS is required.
- a machine on the public internet: this machine will act as a bridge between the public interbet and your local machine
- a sshd daemon running on that server: SSH is the swiss army knife for that kind of setup, and you need to be able to fully configure the SSH server
- optional: SSL certificate. If you want to protect your connection, you can also setup SSL certificates so that the publci URL can be available only through https.
DNS server and a domain name
The DNS is required to resolve the adress of the URL to access your machine using your domain name.
You have a lot of options here:
- you can use the DNS can be the registrar which provided your domain name
- your server may also provide such a DNS,
- an external service, like a dynamic DNS such as DynDNS.
For this demo I will be using DynDNS because I have a free life account with them due to a domain I registered with them more than … 15 years ago!
This may not be the most convenient solution though and a simple hosted DNS server may be best for you.
Using subdomains like alfa.yourdomain.com or beta.yourdomain.com you will be able to have multiple private services or machines reachable using that redirecting solution.
a public server
This server has to be on the cloud so that it can be reached from the public internet.
You will use the public IP address of this machine to configure the DNS you have chosen.
An easy option is to use DigitalOcean - referral URL ( non referal URL ) and create a droplet to host the gateway.
a SSH server
This SSH server is where the magic happens. We will see the details below.
SSH server / gateway setup
The instructions below are specific to a DigitalOcean setup, but there may be easily applicable to any other hosting solution.
create a droplet
The first step is to create a droplet and setup a SSH key to access this machine. This article describes how to setup this ssh key.
For this example, I have used a 512 MB / 20 GB Disk / SFO1 - Ubuntu 16.04.2 x64
machine.
Using the SSH key, you can then login as root to that machine.
setup ssh server
You need to make sure that the sshd server is running and has the proper configuration.
This command will check the ssd configuration:
$ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
gatewayports no
allowtcpforwarding yes
What you want, is that boeth parameres are set to yes
.
To achieve this, you can open the file /etc/ssh/sshd_config
and add at the bottom of this file those 2 lines:
AllowTcpForwarding yes
GatewayPorts yes
Then, you need to restart the ssh daemon for those parameters to be taken into account:
service sshd restart
And then you can check that all is in order:
$ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
gatewayports yes
allowtcpforwarding yes
tests on your local machine
Before setting up the DNS, you can already check that the SSS setup is working properly.
In order to test that a loc al service can be reached from the public interbet, you need to start some sort of local server.
A smple HTTP server will do the trick:
You start from a directoty containing no sensitive data, a simple python server:
python -m SimpleHTTPServer 8000
You can check from your local browser from http:127.0.0.1:8000
that the server is working, and … not exposing sensitive data.
Not is time to invoke the SSH port forwarding voodoo incantation, from your local machine:
ssh root@a.b.c.d -N -R 8080:localhost:8000
where a.b.c.d is the IP address of your droplet. With DigitalOcean, this IP address will be displayed in your Droplets
screen.
You can check that the forwarding is working by pointing your server at: http://a.b.c.d:8080/
.
Let’s check this SSH command as it is much simpler than it looks like:
ssh root@a.b.c.d -N -R 8080:localhost:8000
root@a.b.c.d
: this command will connect to your remote machine (will use the SSH key used when creating the droplet)-N
: by defaukt, ssh will create a shell on the remote machine. We don’t need that here-R
: with this option you are asking ssh to answer on the remote side (your gateway)8080:localhost:8000
: the way you want to tunnel to answer is that any connection on port8080
will be tunneled to the the port8000
on your local machine (where the webserver we started previously is listening on).
We can represent that SSH tunnel like that:
More information about SSH tunnel can be founbd here.
DNS setup
The next step is that you want a nicer URL to access your service right?
You need to configure the DNS for your domain yourdomain.com and create a A record for alfa.yourdomain.com , with a TTL of 600, witha value of a.b.c.d which is the IP adress of your gateway.
I may take time for the DNS configuration to propagate, but once it is done, you can then access your local web server through the URL: http:alfa.yourdomain.com:88080
.
Better, but can still be improved: you may want to setup multiple subdomains which would allow you to host multiple local services, or have multiple machines using this tunnel (each using a specific subdomain).
To do that, you need a proxy on your gateway. Enter …
HAProxy
HAProxy is a reliable, High Performance TCP/HTTP Load Balancer. This can be used to configue multiple subdomains with different ports.
It’s easy to install and test haproxy:
$ apt-get install haproxy
$ haproxy -v
HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>
You can find more information in this DigitalOcean article.
You emable HAProxy by updating the file /etc/default/haproxy
with the line:
ENABLED=1
Then, you configure haproxy through its configuration file: /etc/haproxy/haproxy.cfg
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend alfa
bind *:80
mode http
acl host_alfa hdr(host) -i alfa.yourdomain.com
use_backend alfa if host_alfa
backend alfa
mode http
server node1 127.0.0.1:8001
With this setup, we tell haproxy to serve any request coming to the host alfa.yourdomain.com
with the localserver on port 8001.
So, if you create a tunnel (on your local machine) with:
ssh root@a.b.c.d -N -R 8001:localhost:8000
then, any request to https:alfa.yourdomain.com
will magically hit your local server on port 8000.
So, by configuring haproxy accordingly, you can:
- configure multiple domain alfa.yourdomain.com, beta.yourdomain.com, test.yourdomain.com, …
- each of these subdomains will hit a specific port on the gateway
- you can then setup a ssh tunnel, one one or multiple machine, to use a remote port … and then access the service from the public internet using the associated subdomain
IMPORTANT: each sudomain you create must also be configured on your DNS to access your gateway server.
https
A final step may be that you prefer if those public URL would be serve using SSL.
The good news is that it’s pretty easy to do that using letsencrypt: Let’s Encrypt is a free, automated, and open Certificate Authority.
The automated part is the interesting one.
There is a detailed DigitalOcean document describing the letsencrypt setup.
Here are the main steps.
You need first to install the cerbot client which is the tool to use to configure your server with the letsencrypt certificates.
$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot
Then you can use the certot client to configure the subdomains you want to have a certificate for.
The first step is to stop your haproxy server:
lets encrypt need to validate the fact that you pwn those domains so you need to make sure first that the DNS is propery configured for each of those subdomains as letsencrypt server will try to reach those URLs and access a standalone server started by certbot in the CLI command.
Thats why you must first stop haproxy to avoid any conflict with this standalone server:
$ service haproxy stop
Then, you can configure all your subdomains through the certbot CLI:
$ certbot certonly --standalone -d yourdomain.com -d blissed.yourdomain.com -d pcarion.yourdomain.com -d prod.yourdomain.com -d staging.yourdomain.com -d test01.yourdomain.com -d test02.yourdomain.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for yourdomain.com
tls-sni-01 challenge for blissed.yourdomain.com
tls-sni-01 challenge for pcarion.yourdomain.com
tls-sni-01 challenge for prod.yourdomain.com
tls-sni-01 challenge for staging.yourdomain.com
tls-sni-01 challenge for test01.yourdomain.com
tls-sni-01 challenge for test02.yourdomain.com
/usr/lib/python2.7/dist-packages/OpenSSL/rand.py:58: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct)
result_code = _lib.RAND_bytes(result_buffer, num_bytes)
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0000_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem
IMPORTANT NOTES:
- Congratulations! Your certificate and chain have been saved at
/etc/letsencrypt/live/yourdomain.com/fullchain.pem. Your cert will
expire on 2017-08-01. To obtain a new or tweaked version of this
certificate in the future, simply run certbot again. To
non-interactively renew *all* of your certificates, run "certbot
renew"
- If you like Certbot, please consider supporting our work by:
Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate
Donating to EFF: https://eff.org/donate-le
Once this is done, you can configure your haproxy with all those subdomains but also the SSL certificates:
As described in the letsencrypt documentation, there is a step to be done to allow the haproxy setup:
DOMAIN='yourdomain.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'
chmod -R go-rwx /etc/haproxy/certs
This .pem file needs to be provided in the haproxy configuration:
global
maxconn 2048
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend yourdomain-http
bind *:80
reqadd X-Forwarded-Proto:\ http
mode http
default_backend www-backend
frontend yourdomain-https
bind *:443 ssl crt /etc/haproxy/certs/yourdomain.com.pem
reqadd X-Forwarded-Proto:\ https
mode http
acl host_pcarion hdr(host) -i pcarion.yourdomain.com
acl letsencrypt-acl path_beg /.well-known/acme-challenge/
use_backend pcarion_node if host_pcarion
use_backend letsencrypt-backend if letsencrypt-acl
default_backend www-backend
backend www-backend
redirect scheme https if !{ ssl_fc }
server www-1 127.0.0.1:8000
backend pcarion_node
mode http
server node1 127.0.0.1:8001
backend letsencrypt-backend
server letsencrypt 127.0.0.1:54321
This haproxy setup:
- use the SSL certificate that we generated using
certbot
- redirects all the
http
traffic on port 80 to the port 443 forhttps
- it configures a backend for letsencrypt to allow autorenewal of the certificate.
You can check that your configuration is valid with:
check that haproxy properly configured:
status haproxy.service
letsencryp certificate autorenewal
The certificate generated by letsencrypt are short lived (90 days) and it is a good idea to setup your server to automatically renew those certificates as described in this document.
There is a very nice CLI script to simplify that process.
You can get that script with:
curl -L -o /usr/local/sbin/le-renew-haproxy https://gist.githubusercontent.com/thisismitch/7c91e9b2b63f837a0c4b/raw/700cfe953e5d5e71e528baf20337198195606630/le-renew-haproxy
chmod +x /usr/local/sbin/le-renew-haproxy
Then, to use it, you simply need to prepare a file at: /usr/local/etc/le-renew-haproxy.ini
where you configure the list of subdomains you have:
# This is an example of the kind of things you can do in a configuration file.
# All flags used by the client can be configured here. Run Certbot with
# "--help" to learn more about the available options.
#
# Note that these options apply automatically to all use of Certbot for
# obtaining or renewing certificates, so options specific to a single
# certificate on a system with several certificates should not be placed
# here.
# Use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096
# Uncomment and update to register with the specified e-mail address
email = pcarion@gmail.com
domains = yourdomain.com, blissed.yourdomain.com, pcarion.yourdomain.com, prod.yourdomain.com, staging.yourdomain.com, test01.yourdomain.com, test02.yourdomain.com
# Uncomment to use the standalone authenticator on port 443
# authenticator = standalone
standalone-supported-challenges = http-01
# Uncomment to use the webroot authenticator. Replace webroot-path with the
# path to the public_html / webroot folder being served by your web server.
# authenticator = webroot
# webroot-path = /usr/share/nginx/html
The last step is to add this /usr/local/sbin/le-renew-haproxy
in your crtontab so that it is called every 10 days for instance.
More information
If you want to know moe about all you can do with SSH , you should check that video:
The Black Magic Of SSH / SSH Can Do That?