Your own public development URL.

Tue, May 16, 2017

How to setup a public URL to access your local machine? and why you would even need that.

There are a lof of use cases where a public URL is required: * you want to expose a webserver running on your local machine to the internet so that a colleague or a customer may have a look at it. * you are using a service, like twillio, which allows you to setup webhooks URL: the service will call those URLs to notifiy you of something. * you want to test an OAuth integration, with Facebook or Twitter, and you want to provide public callback URL.

It is always an option to install your code on a public server and have those requests hit this server. During development, it is much more convenient to have thos requests hit your local machine so that you can debug and/or see logs in real time.

There is already a great solution for that: ngrok. This is a wonderful tool, developed by Alan Shreve, very easy to use and perfect for webhooks.

You can also build your own custom redirecting solution using SSH, and more specifically its, SSH Reverse port forwarding feature.

A high level description of this redirecting solution would be:

high level overview

Your local machine (a) is behind a firewall, either a corporate firewall or, if you are at home, behind your ISP router. In both case, you don’t want to poke a hole in your router to let public internet access your machine.

The idea is then to have a public machine (b) which would act as a gateway between the public internet and your local machine.

pre requisites

In order to implement this solution, you need:

a domain name: thhis domain will be used to get public URL(s) for your machine
a public DNS: as you want to have public URL to access your server, a DNS is required.
a machine on the public internet: this machine will act as a bridge between the public interbet and your local machine
a sshd daemon running on that server: SSH is the swiss army knife for that kind of setup, and you need to be able to fully configure the SSH server
optional: SSL certificate. If you want to protect your connection, you can also setup SSL certificates so that the publci URL can be available only through https.

DNS server and a domain name

The DNS is required to resolve the adress of the URL to access your machine using your domain name.

You have a lot of options here:

you can use the DNS can be the registrar which provided your domain name
your server may also provide such a DNS,
an external service, like a dynamic DNS such as DynDNS.

For this demo I will be using DynDNS because I have a free life account with them due to a domain I registered with them more than … 15 years ago!

This may not be the most convenient solution though and a simple hosted DNS server may be best for you.

Using subdomains like alfa.yourdomain.com or beta.yourdomain.com you will be able to have multiple private services or machines reachable using that redirecting solution.

a public server

This server has to be on the cloud so that it can be reached from the public internet.

You will use the public IP address of this machine to configure the DNS you have chosen.

An easy option is to use DigitalOcean - referral URL ( non referal URL ) and create a droplet to host the gateway.

a SSH server

This SSH server is where the magic happens. We will see the details below.

SSH server / gateway setup

The instructions below are specific to a DigitalOcean setup, but there may be easily applicable to any other hosting solution.

create a droplet

The first step is to create a droplet and setup a SSH key to access this machine. This article describes how to setup this ssh key.

For this example, I have used a 512 MB / 20 GB Disk / SFO1 - Ubuntu 16.04.2 x64 machine.

Using the SSH key, you can then login as root to that machine.

setup ssh server

You need to make sure that the sshd server is running and has the proper configuration.

This command will check the ssd configuration:

$ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
gatewayports no
allowtcpforwarding yes

What you want, is that boeth parameres are set to yes.

To achieve this, you can open the file /etc/ssh/sshd_config and add at the bottom of this file those 2 lines:

AllowTcpForwarding yes
GatewayPorts yes

Then, you need to restart the ssh daemon for those parameters to be taken into account:

service sshd restart

And then you can check that all is in order:

$ sshd -T | grep -E 'gatewayports|allowtcpforwarding'
gatewayports yes
allowtcpforwarding yes

tests on your local machine

Before setting up the DNS, you can already check that the SSS setup is working properly.

In order to test that a loc al service can be reached from the public interbet, you need to start some sort of local server.

A smple HTTP server will do the trick:

You start from a directoty containing no sensitive data, a simple python server:

python -m SimpleHTTPServer 8000

You can check from your local browser from http:127.0.0.1:8000 that the server is working, and … not exposing sensitive data.

Not is time to invoke the SSH port forwarding voodoo incantation, from your local machine:

ssh root@a.b.c.d -N -R 8080:localhost:8000

where a.b.c.d is the IP address of your droplet. With DigitalOcean, this IP address will be displayed in your Droplets screen.

You can check that the forwarding is working by pointing your server at: http://a.b.c.d:8080/.

Let’s check this SSH command as it is much simpler than it looks like:

ssh root@a.b.c.d -N -R 8080:localhost:8000

root@a.b.c.d : this command will connect to your remote machine (will use the SSH key used when creating the droplet)
-N: by defaukt, ssh will create a shell on the remote machine. We don’t need that here
-R: with this option you are asking ssh to answer on the remote side (your gateway)
8080:localhost:8000: the way you want to tunnel to answer is that any connection on port 8080 will be tunneled to the the port 8000 on your local machine (where the webserver we started previously is listening on).

We can represent that SSH tunnel like that:

SSH tunnel

More information about SSH tunnel can be founbd here.

DNS setup

The next step is that you want a nicer URL to access your service right?

You need to configure the DNS for your domain yourdomain.com and create a A record for alfa.yourdomain.com , with a TTL of 600, witha value of a.b.c.d which is the IP adress of your gateway.

I may take time for the DNS configuration to propagate, but once it is done, you can then access your local web server through the URL: http:alfa.yourdomain.com:88080.

Better, but can still be improved: you may want to setup multiple subdomains which would allow you to host multiple local services, or have multiple machines using this tunnel (each using a specific subdomain).

To do that, you need a proxy on your gateway. Enter …

HAProxy

HAProxy is a reliable, High Performance TCP/HTTP Load Balancer. This can be used to configue multiple subdomains with different ports.

It’s easy to install and test haproxy:

$ apt-get install haproxy
$ haproxy -v
HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>

You can find more information in this DigitalOcean article.

You emable HAProxy by updating the file /etc/default/haproxy with the line:

ENABLED=1

Then, you configure haproxy through its configuration file: /etc/haproxy/haproxy.cfg

global
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
  timeout connect 5000
  timeout client  50000
  timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend alfa
	bind *:80
	mode http
	acl host_alfa hdr(host) -i alfa.yourdomain.com
	use_backend alfa if host_alfa

backend alfa
	mode http
	server node1 127.0.0.1:8001

With this setup, we tell haproxy to serve any request coming to the host alfa.yourdomain.com with the localserver on port 8001.

So, if you create a tunnel (on your local machine) with:

ssh root@a.b.c.d -N -R 8001:localhost:8000

then, any request to https:alfa.yourdomain.com will magically hit your local server on port 8000.

So, by configuring haproxy accordingly, you can:

configure multiple domain alfa.yourdomain.com, beta.yourdomain.com, test.yourdomain.com, …
each of these subdomains will hit a specific port on the gateway
you can then setup a ssh tunnel, one one or multiple machine, to use a remote port … and then access the service from the public internet using the associated subdomain

IMPORTANT: each sudomain you create must also be configured on your DNS to access your gateway server.

https

A final step may be that you prefer if those public URL would be serve using SSL.

The good news is that it’s pretty easy to do that using letsencrypt: Let’s Encrypt is a free, automated, and open Certificate Authority.

The automated part is the interesting one.

There is a detailed DigitalOcean document describing the letsencrypt setup.

Here are the main steps.

You need first to install the cerbot client which is the tool to use to configure your server with the letsencrypt certificates.

$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt-get update
$ sudo apt-get install certbot

Then you can use the certot client to configure the subdomains you want to have a certificate for.

The first step is to stop your haproxy server:

lets encrypt need to validate the fact that you pwn those domains so you need to make sure first that the DNS is propery configured for each of those subdomains as letsencrypt server will try to reach those URLs and access a standalone server started by certbot in the CLI command.

Thats why you must first stop haproxy to avoid any conflict with this standalone server:

$ service haproxy stop

Then, you can configure all your subdomains through the certbot CLI:

$ certbot certonly --standalone -d yourdomain.com -d blissed.yourdomain.com -d pcarion.yourdomain.com -d prod.yourdomain.com -d staging.yourdomain.com -d test01.yourdomain.com -d test02.yourdomain.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for yourdomain.com
tls-sni-01 challenge for blissed.yourdomain.com
tls-sni-01 challenge for pcarion.yourdomain.com
tls-sni-01 challenge for prod.yourdomain.com
tls-sni-01 challenge for staging.yourdomain.com
tls-sni-01 challenge for test01.yourdomain.com
tls-sni-01 challenge for test02.yourdomain.com
/usr/lib/python2.7/dist-packages/OpenSSL/rand.py:58: UserWarning: implicit cast from 'char *' to a different pointer type: will be forbidden in the future (check that the types are as you expect; use an explicit ffi.cast() if they are correct)
  result_code = _lib.RAND_bytes(result_buffer, num_bytes)
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0000_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0000_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/yourdomain.com/fullchain.pem. Your cert will
   expire on 2017-08-01. To obtain a new or tweaked version of this
   certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

Once this is done, you can configure your haproxy with all those subdomains but also the SSL certificates:

As described in the letsencrypt documentation, there is a step to be done to allow the haproxy setup:

DOMAIN='yourdomain.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem /etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'
chmod -R go-rwx /etc/haproxy/certs

This .pem file needs to be provided in the haproxy configuration:

global
	maxconn	2048
	log /dev/log	local0
	log /dev/log	local1 notice
	chroot /var/lib/haproxy
	stats socket /run/haproxy/admin.sock mode 660 level admin
	stats timeout 30s
	user haproxy
	group haproxy
	daemon

	# Default SSL material locations
	ca-base /etc/ssl/certs
	crt-base /etc/ssl/private

	# Default ciphers to use on SSL-enabled listening sockets.
	# For more information, see ciphers(1SSL). This list is from:
	#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
	ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
	ssl-default-bind-options no-sslv3

defaults
	log	global
	mode	http
	option	httplog
	option	dontlognull
	option forwardfor
	option http-server-close
  timeout connect 5000
  timeout client  50000
  timeout server  50000
	errorfile 400 /etc/haproxy/errors/400.http
	errorfile 403 /etc/haproxy/errors/403.http
	errorfile 408 /etc/haproxy/errors/408.http
	errorfile 500 /etc/haproxy/errors/500.http
	errorfile 502 /etc/haproxy/errors/502.http
	errorfile 503 /etc/haproxy/errors/503.http
	errorfile 504 /etc/haproxy/errors/504.http

frontend yourdomain-http
	bind *:80
	reqadd X-Forwarded-Proto:\ http
	mode http
	default_backend www-backend

frontend yourdomain-https
	bind *:443 ssl crt /etc/haproxy/certs/yourdomain.com.pem
	reqadd X-Forwarded-Proto:\ https
	mode http
	acl host_pcarion hdr(host) -i pcarion.yourdomain.com
	acl letsencrypt-acl path_beg /.well-known/acme-challenge/
	use_backend pcarion_node if host_pcarion
	use_backend letsencrypt-backend if letsencrypt-acl
	default_backend www-backend

backend www-backend
   redirect scheme https if !{ ssl_fc }
   server www-1 127.0.0.1:8000

backend pcarion_node
	mode http
	server node1 127.0.0.1:8001

backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

This haproxy setup:

use the SSL certificate that we generated using certbot
redirects all the http traffic on port 80 to the port 443 for https
it configures a backend for letsencrypt to allow autorenewal of the certificate.

You can check that your configuration is valid with:

check that haproxy properly configured:

status haproxy.service

letsencryp certificate autorenewal

The certificate generated by letsencrypt are short lived (90 days) and it is a good idea to setup your server to automatically renew those certificates as described in this document.

There is a very nice CLI script to simplify that process.

You can get that script with:

curl -L -o /usr/local/sbin/le-renew-haproxy https://gist.githubusercontent.com/thisismitch/7c91e9b2b63f837a0c4b/raw/700cfe953e5d5e71e528baf20337198195606630/le-renew-haproxy
chmod +x /usr/local/sbin/le-renew-haproxy

Then, to use it, you simply need to prepare a file at: /usr/local/etc/le-renew-haproxy.ini where you configure the list of subdomains you have:

# This is an example of the kind of things you can do in a configuration file.
# All flags used by the client can be configured here. Run Certbot with
# "--help" to learn more about the available options.
#
# Note that these options apply automatically to all use of Certbot for
# obtaining or renewing certificates, so options specific to a single
# certificate on a system with several certificates should not be placed
# here.

# Use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096

# Uncomment and update to register with the specified e-mail address
email = pcarion@gmail.com
domains = yourdomain.com, blissed.yourdomain.com, pcarion.yourdomain.com, prod.yourdomain.com, staging.yourdomain.com, test01.yourdomain.com, test02.yourdomain.com

# Uncomment to use the standalone authenticator on port 443
# authenticator = standalone
standalone-supported-challenges = http-01

# Uncomment to use the webroot authenticator. Replace webroot-path with the
# path to the public_html / webroot folder being served by your web server.
# authenticator = webroot
# webroot-path = /usr/share/nginx/html

The last step is to add this /usr/local/sbin/le-renew-haproxy in your crtontab so that it is called every 10 days for instance.

More information

If you want to know moe about all you can do with SSH , you should check that video:

The Black Magic Of SSH / SSH Can Do That?

Pierre Carion