How to: LetsEncrypt in standalone mode for Unifi on Ubuntu 20.04 LTS

This is an update of my previous post, now that cert-manager is more mature, and i’ve rebuilt my server on Ubuntu 20.04 (from 18.04).

  1. install certbot
  2. install script to update unifi certificate
  3. Test
  4. Issue full certificate
  5. Install cron jobs to automate renewal

Install certbot

Certbot installation instructions are at online of course but here’s a summary:

  1. Update package list:
    sudo apt update
  2. install:
    sudo apt install -y certbot

Create a new certificate using LetsEncrypt

We’re going to use standalone mode, and first we’ll get a test certificate just to validate that everything’s working (so that we don’t trigger LetsEncrypt’s rate limits).

  1. open port 80 in ufw:
sudo ufw allow http
  1. Test certificate issuance:
sudo certbot certonly --standalone -d <hostname> -n --test-cert --agree-tos -m <email>

You should see something like this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for <hostname>
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/<hostname>/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/<hostname>/privkey.pem
   Your cert will expire on 2021-04-08. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.
  1. If that’s worked, close the firewall (sudo ufw deny http) and move on to the next step and install the certificate in unifi. Later, we’ll come back and get a ‘real’ (not staging) certificate.

Install certificate in unifi

I use an amazing certificate installation script from Steve Jenkins.

  1. Get the script:
wget https://raw.githubusercontent.com/stevejenkins/unifi-linux-utils/master/unifi_ssl_import.sh
  1. Edit the config settings in the script to add hostname, switch from Fedora/RedHat/CentOS to Debian/Ubuntu, enable LE_MODE, and disable key paths:
# CONFIGURATION OPTIONS
UNIFI_HOSTNAME=<hostname>   
UNIFI_SERVICE=unifi

# Uncomment following three lines for Fedora/RedHat/CentOS
# UNIFI_DIR=/opt/UniFi
# JAVA_DIR=${UNIFI_DIR}
# KEYSTORE=${UNIFI_DIR}/data/keystore

# Uncomment following three lines for Debian/Ubuntu
UNIFI_DIR=/var/lib/unifi
JAVA_DIR=/usr/lib/unifi
KEYSTORE=${UNIFI_DIR}/keystore

# Uncomment following three lines for CloudKey
#UNIFI_DIR=/var/lib/unifi
#JAVA_DIR=/usr/lib/unifi
#KEYSTORE=${JAVA_DIR}/data/keystore

# FOR LET'S ENCRYPT SSL CERTIFICATES ONLY
# Generate your Let's Encrtypt key & cert with certbot before running this script
LE_MODE=yes
LE_LIVE_DIR=/etc/letsencrypt/live

# THE FOLLOWING OPTIONS NOT REQUIRED IF LE_MODE IS ENABLED
# PRIV_KEY=/etc/ssl/private/hostname.example.com.key
# SIGNED_CRT=/etc/ssl/certs/hostname.example.com.crt
# CHAIN_FILE=/etc/ssl/certs/startssl-chain.crt
  1. copy to /usr/local/bin and make executable:
sudo cp unifi_ssl_import.sh /usr/local/bin/
sudo chmod +x /usr/local/bin/unifi_ssl_import.sh
  1. Run the script to import the certificate. Look for any errors:
sudo /usr/local/bin/unifi_ssl_import.sh
  1. Navigate to your server (https://<hostname>:8443). If it worked, you’ll see a warning that the certificate isnt trusted, but when you examine the cert, it’s issued by a ‘fake’ Lets Encrypt issuer, for example:

Get the real LetsEncrypt certificate

Simply run the same certbot command as before, but leave off the --test-cert flag, and add the --force-renew flag to force it to replace the (unexpired) test certificate:

sudo certbot certonly --standalone -d <hostname> -n --force-renew --agree-tos -m <email>

and rerun the installation script:

sudo /usr/local/bin/unifi_ssl_import.sh

Close the browser window and reopen it, then navigate to your server again. You should now see the valid certificate:

Automate renewal and issuance

Set up a crontab to renew the cert. Pick a randomish time. It should run every day – if the certificate is still valid, it’ll just skip

  1. load crontab – you may be asked to pick an editor – i suggest nano:
    sudo crontab -e
  2. add the schedule – use crontab guru if you arent familiar with crontab schedule expressions, and set up tasks to:
    1. request a new certificate, and
    2. install the updated certificate. I chose a time just over an hour after certificate issue.

It should look like this:

# renew any certificates due to expire soon at 05:20 each day
20 5 * * * /usr/bin/certbot renew --standalone -n --agree-tos -m <email> --pre-hook 'ufw allow http' --post-hook 'ufw deny http'
# install any updated certificates at 06:29 each day
29 6 * * * /usr/local/bin/unifi_ssl_import.sh

The --pre-hook and --post-hook commands tell UFW to open up port 80 and then close it again afterwards.

Can you do that new job?

Generally when evaluating someone for a role, I look for 5 things:

  1. Behaviours – how do they operate in a team? Do they admit to mistakes and learn from them? Do they help others? Communicate and live to their personal values? Are those values ones I want people in the team to live to?
  2. Accountability – can this person handle the magnitude of the role? Are they able to manage stakeholders of the right level of seniority?
  3. Domain – how deep is their knowledge of this business, industry, sector etc.? And how deep does it need to be?
  4. Function – what is their level of skill in this type of role? For example, if hiring a business analyst, how good a business analyst are they?
  5. Organisation – perhaps summarised as “knowing how things are done around here” – processes, culture etc. – does this person have the knowledge to make things happen?

Obviously, number 1 is a given – no one wants a brilliant jerk on the team. But most people have some of each of the others. The question is whether it’s enough to set them up for success in the new role. Usually, I’d expect someone to have strength in 1 or 2 of the others, and to have one or at most two which give headroom to grow as:

  • No headroom in role = boring job
  • Too many development areas = set up for failure

When a candidate is moving roles internally, they probably have number 5. So a step up to greater accountability, or moving to an entirely new business domain (if the company is big enough) might represent a solid plan. Doing both at once is probably too much for most people.

External candidates probably don’t possess organisational knowledge, so we should assume that’s a growth area. And that means they need to be fairly strong in two of the other areas. In my experience, people usually move company to step up. So i would expect external candidates to have strong domain knowledge and functional skill.

Fix: AccessToKeyVaultDenied when using KeyVault with Function App application setting

After following the instructions on the MS website to establish a KeyVault reference and place that in my App Settings, I set up a Managed Service Identity and grant that identity access to my KeyVault key. Next, wishing to follow Microsoft’s advice and secured a firewall around the KeyVault, ensuring that I checked the Allow trusted Microsoft services to bypass this firewall? setting, however, I was still receiving an AccessToKeyVaultDenied error:

I even checked and yes, App Service is supposed to be able to bypass the firewall – so what was going on? Well, on the KeyVault resolver reference page it has this text:

Key Vault references are not presently able to resolve secrets stored in a key vault with network restrictions.

That seemed ok when i first read it as after all, there’s an explicit setting to bypass the firewall. But when i disabled network firewall (allow access from all networks), everything suddenly worked, and the key status is Resolved with a nice green tick:

The 9 Ps

I often have conversations with friends and colleagues about their careers. And many times, i point people to a great blog post by my colleague Liz Aab, about the “7 Ps”. But i always find myself adding two to the list, so i thought i’d just post it here.

There are lots of factors which go in to choosing a job. You can’t have all of them, all of the time. At least, i think you can’t. But you can (and should) decide which are most important to you. Here are Liz’s 7 Ps (which she says were originally 5 Ps from some other source). I’ve added my two on the end, and i’ve reworded some of Liz’s original post:

  1. Place : Where geographically do you want to work? The city/country you are based in and your commute affect how you spend your time, and who you spend your time with, both inside and outside work.
  2. People : Who specifically would you work with on a daily basis? Do you like them? Does your boss care about you and want to see you succeed?
  3. Pay : Does the job or sector pay you enough to live the life you want? If not, will your pay will increase in a few years in this career path? Or, are you happy to change your lifestyle to accommodate a lower salary?
  4. Progression : Will you develop skills, knowledge, a network or a reputation that will help you move forward in your career? Does this job offer defined progression opportunities, or do you need to develop these for yourself? If so, are you comfortable with this?
  5. Perception : How do people react when you tell them what you do? Whose opinion do you really care about, and how important is that to you? Of course perceptions of jobs and industries change over time.
  6. Purpose : What is the company or organisation trying to achieve, and do you support that? It’s not just millennials that want to work on something they believe in.
  7. Procedures : In Liz’s list, this is how you do your job day to day. I’ve reworked it – for me, procedures is how the organisation operates. Do they expect a rigid 9-5, or are you trusted to deliver a result? Do decisions get made once and then implemented, or does it take a consensus to make change? Are you empowered to deliver, or do you need permission to take a bathroom break?
  8. Projects : While procedures might be how the work gets done, this is what you’re actually doing. Are you spending your day on the phone, or sitting reading stacks of paper, or crunching Excel, or standing on your feet in front of 25 teenagers? Is your work indoors or outdoors? And do you like doing those things?
  9. Pace : Is it frantic from the moment you wake to when you sleep? Or is there lots of space in the day for you to collect your thoughts or think things through? Are you expected to check your emails after hours, or do you ‘clock off’ when you’re done? What do you need to thrive?

Of course, as Liz points out, what you value today will differ to what’s important to you tomorrow. When you’re young and eager, you may want a role which is always on the go (high pace), and with a compelling purpose. If you start to plan a family, pay and progression move up the list.

Fix: This must be accepted explicitly before updates for this repository can be applied

Some repos, such a the one for the Unifi Controller, use different ‘field’ values to tie a release and require manual updates. For someone like me who has a standalone, automated controller setup designed mainly to keep the firmware up to date without much intervention, this is a hassle. It looks something like this:

robert@unifi:~$ sudo apt-get update
[sudo] password for robert: 
Hit:1 http://mirrors.linode.com/ubuntu bionic InRelease
Get:2 http://mirrors.linode.com/ubuntu bionic-updates InRelease [88.7 kB]          
Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]        
Get:4 http://mirrors.linode.com/ubuntu bionic-backports InRelease [74.6 kB]                                       
Ign:5 http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 InRelease                                         
Hit:6 http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 Release                                           
Get:7 https://dl.ubnt.com/unifi/debian stable InRelease [3,024 B]        
Reading package lists... Done                             
E: Repository 'https://dl.ubnt.com/unifi/debian stable InRelease' changed its 'Codename' value from 'unifi-5.12' to 'unifi-5.13'
N: This must be accepted explicitly before updates for this repository can be applied. See apt-secure(8) manpage for details.

It’s an easy fix. Just tell apt-get to ignore the codename field:

robert@unifi:~$ echo 'Acquire::AllowReleaseInfoChange::Codename "true";' | sudo tee    /etc/apt/apt.conf.d/99releaseinfochange
Acquire::AllowReleaseInfoChange::Codename "true";

and then it works!

robert@unifi:~$ sudo apt-get update
Ign:1 http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 InRelease
Hit:2 http://mirrors.linode.com/ubuntu bionic InRelease                                                           
Hit:3 http://mirrors.linode.com/ubuntu bionic-updates InRelease                                                   
Hit:4 http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.4 Release                                           
Hit:5 http://mirrors.linode.com/ubuntu bionic-backports InRelease                                                 
Hit:6 http://security.ubuntu.com/ubuntu bionic-security InRelease                                                 
Hit:7 https://dl.ubnt.com/unifi/debian stable InRelease                                                           
Reading package lists... Done

Fix pyodbc.Error: (‘01000’, “[01000] [unixODBC][Driver Manager]Can’t open lib ‘ODBC Driver 13 for SQL Server’ : file not found (0) (SQLDriverConnect)”)

I was connecting from my macbook to a SQL Azure Database when i hit the following error:

>>> environ.get('cloud_sql_conn_string')
'Driver={ODBC Driver 13 for SQL Server};Server=tcp:cynexia-sql.database.windows.net,1433;Database=cloud_scales;Uid=<username>;Pwd=<password;Encrypt=yes;TrustServerCertificate=no;Connection Timeout=30;Authentication=ActiveDirectoryPassword'
>>> import pyodbc
>>> cnxn = pyodbc.connect(environ.get('cloud_sql_conn_string'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
pyodbc.Error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 13 for SQL Server' : file not found (0) (SQLDriverConnect)")

The solution was to install the ODBC driver, following the instructions on the Microsoft website:

brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
HOMEBREW_NO_ENV_FILTERING=1 ACCEPT_EULA=Y brew install msodbcsql17 mssql-tools
ACCEPT_EULA=Y brew install msodbcsql@13.1.9.2 mssql-tools@14.0.6.0

Fix: unable to kmem_alloc enough memory for scatter/gather list in ZFS Solaris 10.5

The ZFS Pool on my server was showing degraded state. After checking the SMART status of the constituent drives and finding no problem, I discovered that there’s a bug in Solaris 10.5 where the system reports a growing number of errors and eventually fails the pool. dmesg shows an error unable to kmem_alloc enough memory for scatter/gather list, however, there is actually nothing wrong with the pool. Running zpool status shows degraded state:

root@fs:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM      CAP            Product /Disks     IOstat mess          SN/LUN
        rpool       ONLINE       0     0     0
          c1t0d0    ONLINE       0     0     0      32.2 GB        VMware Virtual S   S:5 H:25 T:0         000000000000000

errors: No known data errors

  pool: tank
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: scrub repaired 0 in 12h15m with 0 errors on Fri Dec 21 00:08:43 2020
config:

        NAME                       STATE     READ WRITE CKSUM      CAP            Product /Disks     IOstat mess          SN/LUN
        tank                       DEGRADED     0     0     0
          raidz1-0                 DEGRADED     0     0     0
            c0t50014EE20BF0750Dd0  ONLINE       0     0     0      4 TB           WDC WD40EFRX-68W   S:0 H:0 T:0          WDWCC4E6NAXVAS
            c0t50014EE263348A3Ed0  ONLINE       0     0     0      4 TB           WDC WD40EFRX-68W   S:0 H:0 T:0          WDWCC4E0FRRRRP
            c0t50014EE2B69D2D68d0  DEGRADED     0     0    20  too many errors      4 TB           WDC WD40EFRX-68W   S:0 H:0 T:0          WDWCC4E3AN2Y99

errors: No known data errors

Running zpool clear recovers the pool:

root@fs:~# zpool clear
root@fs:~# zpool status    
  pool: rpool
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c1t0d0    ONLINE       0     0     0

errors: No known data errors

  pool: tank
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://illumos.org/msg/ZFS-8000-9P
  scan: none requested
config:

        NAME                       STATE     READ WRITE CKSUM
        tank                       ONLINE       0     0     0
          raidz1-0                 ONLINE       0     0     0
            c0t50014EE20BF0750Dd0  ONLINE       0     0     2
            c0t50014EE263348A3Ed0  ONLINE       0     0     0
            c0t50014EE2B69D2D68d0  ONLINE       0     0     0

Wiring a Yale Keyless Connected Smart Lock to the mains

For various reasons, not least because I wanted to play with it, we have a Yale Keyless Connected Smart Door Lock with a Z-Wave module (we have the v1 module which works fine). This lock has a couple of key features that we liked:

  • You can grant or revoke access using RFID tags or cards, or by entering a 6-8 digit code on the keypad.
  • With the Z-Wave module (and a compatible Z-Wave controller), you can programatically add and remove codes so that you can enable codes at specific time or dates. For us, this meant we could create a code for the cleaner, but if they turned up at 2am on a Saturday, the door wouldn’t open for them.

It’s connected to our Samsung SmartThings hub, and i run the RBoy Apps custom device type and smart app to enable the scheduled key rotation etc. Overall, we’ve been fairly happy with it, but the thing really does eat up batteries, and I started to feel guilty about putting between 4 or 8 AA batteries in the bin each month. Of course I also got annoyed at constantly having to buy them and change them, so I decided to try rechargeables.

We bought some Panasonic Eneloop Pro batteries. I’d read a very interesting piece of research showing how high performance NiMH batteries actually outperform alkaline batteries – delivering a stable ~1.2v for far longer. As it turns out, this is a problem.

With a regular battery, as the charge drops, the device detects this and fires off an alert reminding you to change them. As the research showed, however, NiMH batteries provide a fairly constant 1.2v until the “power” in the batteries is pretty much depleted, and then they just die. This isn’t a problem for a radio controlled car. But of course if the batteries go flat on your front door lock, you can’t get in to your house as there’s no key override on it, and with no alerts, we wouldn’t know to change them. Although you can power the lock from the outside in an emergency using a 9v battery, after a particularly embarrassing situation where I discovered that the 9v battery I had stored in the glove box had expired 2 years ago, I decided to figure out how to wire the lock up to a permanent power supply. The main challenge here of course is that I would like to be able to unlock my house even when the power is out. After a bit of thinking, i decided that I probably needed a battery in there somewhere too.

Picture from ebay seller random-bargains2009

The first challenge is working out how to wire up the device. Ideally, didn’t want to be soldering connectors on. After a bit of research, I found a “4 X 6V AA MONEY SAVING BATTERY REPLACEMENT PLUG IN ADAPTER” on ebay (the item i bought is from “random-bargains2009” but there were three or four different ones from other sellers). This is basically an AA battery with a wire coming out of it connected to a mains adaptor, and 3 “dummy” battery blanks. I cut off the AC adaptor, and soldered on a USB A plug, and connected it to a Belkin USB battery pack, then plugged that in to a charger to keep it constantly topped up and … nothing. It turns out that Belkin battery packs can’t provide power and be charged at the same time. Doh!

I tried another battery pack, and all was fine until my wife tried to come in about 5 minutes later. Apparently the “smart” charge controller built in to the battery didn’t detect sufficient current, and so switched off the battery pack. Brilliant.

After a bit of research, I was able to find a 3,000mAh battery pack with a USB plug (from Amazon) that was designed to provide backup power for 12v CCTV cameras. Made by Chinese company TalentCell, it claims CE compliance for both the batteries and the charger. Mine arrived from Germany with an EU plug, but I already had some fused, screw fixed adaptor plugs, so not a problem.

Finally, the I ran the cable around the frame of the glass in the door to try and keep it discrete, and I’m now confident that we won’t ever be locked out again.

How 5G connectivity and new technology could pave the way for self-driving cars

Hybrid peer-to-peer/5G vehicle communication technology, C-V2CX ("cellular-vehicle-to-everything"), has evolved since it's 2016 debut, with recent demos showing how it helps vehicles "see" threats and obstacles out of sensor range (such as cars coming around corners, traffic lights and so on). But it's not the only protocol on the block – Toyota, the world's largest car manufacturer, and GM, prefer a competing protocol based on wifi. The winner should start to emerge later this year when 5G trials begin.
https://rob.al/2N03G6n
C-V2X enables vehicles to communicate, which should reduce accidents and aid autonomous driving.