Just what i find

Creating a VM to test Docker – migrating from Unraid to Napp-It

onwards…

downloaded the Ubuntu 16.04 Server LTS CD
Created new VM
Connected ISO to virtual DVD in console in VMWare vSphere client
booted to the ISO
pressed F4 and selected “minimal”
pressed F6 and selected minimal VM
walked through and added SSH at the right point
SSHd to the server
I installed docker using these instructions, including configuring docker to start on boot.
created /mnt/cups

ran a new docker container…

robert@ubd:/mnt$ sudo docker run -d --name="cups-google-print" --net="host" --privileged="true" -e TZ="UTC" -e HOST_OS="ubuntu" -e "CUPS_USER_ADMIN"="admin" -e "CUPS_USER_PASSWORD"="pass" -e "TCP_PORT_631"="631" -v "/mnt/cups":"/config":rw -v /dev:/dev -v /etc/avahi/services:/avahi -v /var/run/dbus:/var/run/dbus mnbf9rca/cups-google-print --restart=unless-stopped Unable to find image 'mnbf9rca/cups-google-print:latest' locally latest: Pulling from mnbf9rca/cups-google-print a3ed95caeb02: Pull complete 3b1d42cd9af9: Pull complete d2ff49536f4d: Pull complete f94adccdbb9c: Pull complete ae857e8dd13c: Pull complete 327565847940: Pull complete 83835dcb6373: Pull complete 78b26d55dd43: Pull complete 388ec0e358c7: Extracting [=======================================>           ]   190 MB/238.1 MB 388ec0e358c7: Pull complete 05dd908ba895: Pull complete 87c9e1d25f3b: Pull complete 75d49e6da022: Pull complete 4b8ca4d5d690: Pull complete Digest: sha256:3c231589c941288c4541016944658ee5915e4d8761648e1d88254c90dea9beca Status: Downloaded newer image for mnbf9rca/cups-google-print:latest f33434ebccaf2b5644260e664014d7364d4d5ead45bf4374e931a0acedd06015

checked it’s running with the docker ps -a command:

robert@ubd:/mnt$ sudo docker ps -a CONTAINER ID        IMAGE                        COMMAND             CREATED             STATUS              PORTS               NAMES f33434ebccaf        mnbf9rca/cups-google-print   "/sbin/my_init"     2 minutes ago       Up 2 minutes                            cups-google-print

Browsed to https://<server IP>:631/ to see that it’s running

bingo!

Fix: Flashing an M1015 – Error code = 64 Failed to flash the image. Please retry recovery

I purchased an IBM M1015 to use as an HBA in my server. As part of that, I wanted to flash it with the IT firmware, however, I was getting errors, even when I used the original IBM firmware from their website:

C:\> megarec -cleanflash 0 M1000FW.ROM

MegaRAID HWR Controller Recovery tool. Version 01.01-004 February 05, 2010
Copyright (c) 2006-2008 LSI Corp.
Supports 1079 controller and its Successors


Erasing Flash Chip (8MB)....
 Completed: 100%
Flashing Image. Please wait...


Currently flashing component = BIOS
Programming Flash....
 Completed: 100%
Verifying the Flashed Data...


Currently flashing component = HIIM
Programming Flash....
 Completed: 100%
Verifying the Flashed Data...


Currently flashing component = APP
Error in downloading the image.
Error code = 64
Failed to flash the image. Please retry recovery

I’d never seen an M1015 before, at least not close up. Closer inspection of the card, however, revealed a code: FRU 46C8927. I know FRU” means “Field Replaceable Unit” i.e. something you can order as a replacement part. So I googled that code, and discovered that this was an IBM M5015, not an M1015. The M5015 cannot be used in IT mode, so I had to send it back.

Installing VMWare – migrating from Unraid to Napp-It

get image
installed to internal 32gb microSD
steps: https://seiler.it/vmware-esxi-5-5-installation-hp-proliant-microserver-gen8/
Remember to use the v88 driver: http://homeservershow.com/forums/index.php?/topic/11159-esxi-6-cant-degrade-the-hpvsa-driver-to-v88/?p=116554

# To roll back to hpvsa build 88 (for MicroServer Gen8 compatibility)
esxcli system maintenanceMode set --enable true
esxcli network firewall ruleset set -e true -r httpClient
esxcli software vib remove -n scsi-hpvsa -f
esxcli software vib install -v http://vibsdepot.hp.com/hpq/nov2014/esxi-550-drv-vibs/hpvsa/scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib
esxcli system shutdown reboot -r "Rolled back to scsi-hpvsa build 88"

How to flash IBM ServeRaid M1015 to LSI9211-IT for ZFS in HP Gen8 Microserver

First things first – you do this at your own risk. I take no responsibility for anything going wrong – and it can go wrong. If you are in doubt – don’t do it. And if it goes wrong – don’t blame me…

Make a DOS USB boot stick
1. download the very useful Rufus from here.
2. select “Create a bootable disk using FreeDos”:
3. flash the USB.
Download the firwmare files from here: sas2008 (see footnote for original source)
Extract the files and place them on the root of the USB stick.
Download the latest LSI firmware from the Avago site. You’re looking for Firmware for an SAS 9211-8i Host Bus Adaptor. At the time of writing, this is version P20.
Extract the LSI firmware to a folder on your machine.
Create a subfolder on the USB called P20
From the extracted LSI firmware, copy the following to the P20 folder on the USB:
1. The 2118it.bin file from <zip>\\firmware\\HBA_9211_8i_IT folder
2. mptsas2.rom from sasbios_rel folder
3. sas2flsh.exe from sas2flash_dos_rel folder
Look at the back of the card and note down the SAS address – it’s something like 500605B0xxxxxxxx.
put the card in the machine, and switch it on.
Boot to the USB stick – press F11 during POST and select USB.
Flash the firmware:
1. Type the following:
  megarec -writesbr 0 sbrempty.bin megarec -cleanflash 0
2. Type the following:
  megarec -writesbr 0 sbrempty.bin megarec -cleanflash 0
3. Reboot, again booting from the USB stick
4. Next, install the P10 or P11 firmware – type the following:
  sas2flsh -o -f 2118it.bin -b mptsas2.rom sas2flsh -o -sasadd 500605bxxxxxxxxx (x= numbers for SAS address)
5. Reboot, again booting from the USB stick
6. Finally, upgrade to the P20 firmware – type the following to change to the folder and execute flash of the new firmware:
  cd p20 sas2flsh -o -f 2118it.bin -b mptsas2.rom
Remove the USB stick
Reboot.
Some people recommend to disable loading the Option ROM. On my machine, loading the option room caused an NMI, so i ignored it, but if you want to do it: Load the Option ROM (press CTRL-C on boot) and set “Boot Support” to “Disabled”

The original instructions for this task are here, with my additions to update to the P20 firmware – I’ve archived them here for my own reference.

Preparing for migration – migrating from Unraid to Napp-It

This post is the second in my series describing how i migrated from Unraid to Napp-It, and describes how I prepared for migration.

So – preparing for migration…

First, i wanted to capture the docker configuration for each of my existing containers. To do this, I forced the container to update, then when unraid presented me with the “success” screen, I captured the docker run command, like this (which captures the RUN command for my CUPS container with Google Print extensions):

root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name="cups-google-print" --net="host" --privileged="true" -e TZ="UTC" -e HOST_OS="unRAID" -e "CUPS_USER_ADMIN"="admin" -e "CUPS_USER_PASSWORD"="pass" -e "TCP_PORT_631"="631" -v "/mnt/user/appdata/cups-google-print":"/config":rw -v /dev:/dev -v /etc/avahi/services:/avahi -v /var/run/dbus:/var/run/dbus mnbf9rca/cups-google-print

Next, I found two old 2TB disks to copy my data on to – personally, I’m not concerned about losing the data, but if you are, this is going to be a blocker for you. Anyway, I added these to my server, and they were assigned device names sdd and sde.
i installed the unraid unassigned devices plugin.
I stopped the array then moved my 4TB device to “unassigned” and mounted it.

Then i connected via SSH and created two new folders called “SDD” and “SDE” (to match the device names) and moved the content so there was less than 2DB in each 2TB in each:

root@fs:~# cd /mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99
root@fs:/mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99# mkdir sdd
root@fs:/mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99# mkdir sde
root@fs:/mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99# mv  sdd
root@fs:/mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99# mv  sde

… and eventually i had them roughly balanced…

root@fs:/mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99# du -sh sd*
1.8T    sdd
1.7T    sde
root@fs:/mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99#

i used rsync to move the data from the old 4TB disk to the new one:

rsync -idIWpEAXogtlr --remove-source-files --numeric-ids --inplace /mnt/disks/WDC_WD40EFRX_68WT0N0_WD_WCC4E3AN2Y99/sdd/ /mnt/disks/WDC_WD20EARS_00MVWB0_WD_WCAZA4636167

physically removed the 2x2TB drives and put them somewhere safe, so that i didn’t accidentally erase them.

Now i was ready to move to the next step, installing my new OS host – VMWare.

Fix: Emby Docker fails to start when config on mounted share – SQLite “database is locked”

I have a clean VM running Ubuntu 16.04 on VWare ESXi 6.5. I have a CIFS share mounted at /mnt/appdata with the noperm flag. The share is writeable.

I installed Docker using the instructions here: https://hub.docker.c…mby/embyserver/

docker run -it --rm -v /usr/local/bin:/target \
     -e "APP_USER=robert" \
     -e "APP_CONFIG=/mnt/appdata/emby" \
     emby/embyserver instl

then

docker run -it --rm -v /etc/systemd/system:/target \
    emby/embyserver instl services

the next command, sudo systemctl enable emby-server.service, didnt work. Instead I had to do:

sudo systemctl enable emby-server@.service

then I ran emby-server and configured it with a path /mnt/video (Also a CIFS share mounted on my local machine). However, Emby doesnt work – and i see an error in the attached log (“svc.txt”):

?2016-12-04T09:49:04.572948238Z Error, Main, UnhandledException
52016-12-04T09:49:04.573027012Z  *** Error Report ***
42016-12-04T09:49:04.573039000Z  Version: 3.0.8500.0
2016-12-04T09:49:04.573049078Z  Command line: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe -programdata /config -ffmpeg /bin/ffmpeg -ffprobe /bin/ffprobe -restartpath /usr/lib/emby-server/restart.sh
@2016-12-04T09:49:04.573081031Z  Operating system: Unix 4.4.0.31
32016-12-04T09:49:04.573090300Z  Processor count: 4
02016-12-04T09:49:04.573097909Z  64-Bit OS: True
52016-12-04T09:49:04.573105143Z  64-Bit Process: True
;2016-12-04T09:49:04.573112588Z  Program data path: /config
b2016-12-04T09:49:04.573119889Z  Mono: 4.6.2 (Stable 4.6.2.7/08fd525 Mon Nov 21 15:56:40 UTC 2016)
h2016-12-04T09:49:04.573127634Z  Application Path: /usr/lib/emby-server/bin/MediaBrowser.Server.Mono.exe
=2016-12-04T09:49:04.573135097Z  One or more errors occurred.
:2016-12-04T09:49:04.573142348Z  System.AggregateException
2016-12-04T09:49:04.573151009Z    at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks, System.Int32 millisecondsTimeout, System.Threading.CancellationToken cancellationToken) [0x00242] in <8f2c484307284b51944a1a13a14c0266>:0 
2016-12-04T09:49:04.573161976Z    at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks, System.Int32 millisecondsTimeout) [0x00000] in <8f2c484307284b51944a1a13a14c0266>:0 
2016-12-04T09:49:04.573172331Z    at System.Threading.Tasks.Task.WaitAll (System.Threading.Tasks.Task[] tasks) [0x00000] in <8f2c484307284b51944a1a13a14c0266>:0 
>2016-12-04T09:49:04.573181859Z    at MediaBrowser.Server.Mono.MainClass.RunApplication (MediaBrowser.Server.Implementations.ServerApplicationPaths appPaths, MediaBrowser.Model.Logging.ILogManager logManager, MediaBrowser.Server.Startup.Common.StartupOptions options) [0x000cf] in <8385af0cf454438f8df15fa62f41afa4>:0 
2016-12-04T09:49:04.573191220Z    at MediaBrowser.Server.Mono.MainClass.Main (System.String[] args) [0x0008a] in <8385af0cf454438f8df15fa62f41afa4>:0 
S2016-12-04T09:49:04.573199399Z  InnerException: System.Data.SQLite.SQLiteException
32016-12-04T09:49:04.573206751Z  database is locked
32016-12-04T09:49:04.573213834Z  database is locked

I tried running the container directly:

docker run -d --name="EmbyServer" \
      --net="host" \
      -e TZ="UTC" \
      -e HOST_OS="ubuntu" \
      -e "TCP_PORT_8096"="8096" \
      -v "/mnt/appdata/emby/":"/config":rw \
      emby/embyserver

but i get the same error (“just run.txt”). I checked, and the /mnt/appdata/emby folder is being created:

robert@d:~$ ls /mnt/appdata/emby
abc config data localization logs
robert@d:~$ du -sh /mnt/appdata/emby
3.4M /mnt/appdata/emby

so clearly the share is writeable from within the container. If I run the container without using the mapped volume for the config:

docker run -d --name="EmbyServer" \
      --net="host" \
      -e TZ="UTC" \
      -e HOST_OS="ubuntu" \
      -e "TCP_PORT_8096"="8096" \
      emby/embyserver

it’s reachable at http://host:8096 and works fine (“no map.txt”) – but obviously the configuration isn’t persistent.

It turns out that the root of the problem is the way that CIFS handles byte-range locking, which is incompatible with SQLite. One way to fix this is to add the nobrl parameter to the mount, e.g.:

//fs.cynexia.net/appdata /mnt/appdata cifs iocharset=utf8,credentials=/root/.smbcredentials,nobrl,dir_mode=0775,nofail,gid=10,noperm 0 0

HP Gen8 Microserver error “Embedded media manager failed initialization” – how to get HPQLOCFG

During the process of installing VMWare on to my Gen8 Microserver, I had trouble writing data to the internal SD card – in fact, I couldn’t even see it. Looking in the ILO Event Logs I saw this:

Embedded Flash/SD-CARD: Embedded media manager failed initialization.

googling this didn’t get me much – just forum posts with people complaining about it, but then i found this HPE Customer Advisory, which lists out the steps needed to reset the error. Basically:

create an XML file with the following content:

<!-- RIBCL Sample Script for HP Lights-Out Products --> 
<!--Copyright (c) 2016 Hewlett-Packard Enterprise Development Company,L.P. --> 

<!-- Description: This is a sample XML script to force format ll --> 
<!-- the iLO partitions. --> 
<!-- iLO resets automatically for this operation to take effect --> 

<!-- Warning: This command erases all data on the partition(s) --> 
<!-- External providers will need to be re-configured if --> 
<!-- partition is formatted --> 

<!-- Input: VALUE tag: all - format all available partitions --> 

<!-- NOTE:You will need to replace the USER_LOGIN and PASSWORD values --> 
<!-- with values that are appropriate for your environment --> 

<!-- See "HP Integrated Lights-Out Management Processor Scripting --> 
<!-- and Command Line Resource Guide" for more information on --> 
<!-- scripting and the syntax of the RIBCL XML --> 

<!-- Firmware support information for this script: --> 
<!-- iLO 4 - Version 2.42 or later. --> 
<!-- iLO 3 - None. --> 
<!-- iLO 2 - None. -->

<RIBCL VERSION="2.0"> 
<LOGIN USER_LOGIN="Administrator" PASSWORD=""> 
<RIB_INFO MODE="write"> 
<FORCE_FORMAT VALUE="all" /> 
</RIB_INFO> 
</LOGIN> 
</RIBCL>

run that file against the server using HPQLOCFG.exe:

hpqlocfg -s <server IP> -l c:\hpqcfg.log -f c:\Force_Format.xml -v -t user=Administrator,password=<password>

some other steps to reinstall intelligent provisioning, if you use it.

All well and good – but where do you get HPQLOCFG from? If you follow the link in the article, it refuses to install because i don’t have the full PSP installed. So how can I apply the change?

Well, in my case, I installed VMWare to an internal USB stick and then ran the command from there – you could even do this with all of your other existing drives removed so that they don’t get erased. You could then restart the process. Problem solved!

Hardware required – migrating from Unraid to Napp-It

I’ve been a user of Unraid since 2012, when I had to find a solution to my home storage after Windows Home Server was abandoned by Microsoft. Unraid has been very good for me, and the introduction of a Docker engine with Unraid 6 was very welcome. That said, I’ve recently encountered issues with bitrot, and the fact that unraid can’t use ZFS as the disk format annoys me. LimeTech claim that their parity check process should detect bitrot – however, something doesn’t seem to be working, as using using the Dynamix File Integrity plugin i can see it happening. In any case, knowing it’s happened isnt the same as being able to correct it, which just isn’t possible on Unraid without using BTRFS but many people simply don’t trust BTRFS, and besides, I fancy a change. So – over to VMWare EXSi and ZFS on Napp-It.

This blog post describes the hardware i needed.

The HP Gen8 Microserver is certified for use with VMWare and you can even install it to an internal USB or MicroSD card. In my case, I want a RAIDZ array of 3 x 4TB drives, plus two or more SSDs for VM and Docker hosts.

For ZFS to work properly it needs access to the underlying host controller, not a virtualised verison. VMWare is capable of direct device passthrough, but only on processors which support VT-d. Additionally, if I’m running a VM (the Napp-IT guest) on a drive on the storage controller, I can’t pass that same storage controller through to the guest.

When i bought my microserver, i got the base model, and with cashback i think it cost me £120. But the base has a Celeron G1060 and although this has the VT-x extensions, it doesn’t support VT-d.

The B120i controller in the Gen8 has 2 6GBs SATA channels, and 2 3GBs. I wanted to ensure that all 4 drive bays ran at 6GBs, plus have 2 6GBs channels for my host SSDs.

My shopping list was therefore:

a new processor supporting VT-d. I used the chart maintained by users of the homeservershow forum to find a suitable processor for sale on eBay. I ordered an E3-1260L from China. Estimated delivery – a few weeks.
Some thermal paste
A second storage controller. I went for the IBM ServeRaid M1015 as it can be flashed to an LSI9211-81 in IT mode (meaning ZFS has direct access to the disks, without the controller being “smart” or “doing RAID” in the middle). See this post for instructions.
a Mini SAS (SFF-8087) to SATA cable, again from eBay.

some things i already had:

a molex splitter (I already had this)
a molex to two SATA HDD power splitter

error processing package apt-show-versions on Ubuntu 14.04 or Ubuntu 16.04

When installing Webmin, I’ve sometimes come across an error installing a dependency package, apt-show-versions:

Setting up apt-show-versions (0.22.7) ...
** initializing cache. This may take a while **
FATAL -> Failed to fork.
dpkg: error processing package apt-show-versions (--configure):
subprocess installed post-installation script returned error exit status 100
dpkg: dependency problems prevent configuration of webmin:FATAL -> Failed to fork.

This is caused by the fact that apt-show-versions can’t read compressed index files. Thankfully, the solution is quite simple:

First, we need to tell APT not to compress the index. To do this we create an entry in a file called /etc/apt/apt.conf.d/02compress-indexes:

sudo nano /etc/apt/apt.conf.d/02compress-indexes

If the file is empty (mine was), simply put this line in it:

Acquire::GzipIndexes "false";

if the file has some text, check if this parameter is in there as “true” and if so change to false. If it’s missing, just add it.

Then, we need to delete the existing indexes and re-download them:

sudo rm /var/lib/dpkg/info/apt-show*

followed by

sudo apt-get update

Finally, we just need to complete the installation:

sudo apt-get -f install webmin

And job done.

How to ensure you can revert changes to function apps

As I’ve been playing around with Azure Functions I’ve slowly outgrown the web-based editor. It’s not that it’s not useful, it’s just that I miss intellisense (I’ll come back to this in a later post), and I accidentally deployed a change which broke one of my functions. I’d made dozens of tiny changes, but I simply could not figure out which one it was. Not having a version history, I was kinda screwed.

I had seen the “Configure Continuous Integration” option before, but never really looked at it. I keep my source code in private GitHub repos, so it was relatively trivial to set up a new repo tied to this function app. After reading the setup instructions, however, I was a little confused by what exactly to do to put my existing functions in to repo, but it was actually much simpler than I thought. It turns out one of the best features is the ability to roll back to a previous commit with a single click:

First, I created a new private GitHub repo and cloned it to my local machine. I chose not to use branching – but I guess you could map different function apps to different branches to support a separation between “dev”, “test”, “production” etc. In the root of my repo, I created a folder for each of the functions I wished to deploy, named exactly the same as the existing functions (I assume they’re not case sensitive but I kept to the same case).

Then, I needed to put the actual code in there. Under the visual editor for each of the functions is a “view files” link: view-files . Clicking this, I was able to see the function.json and run.csx files within each function. I simply cut and pasted the code from there to a file of the same name in the relevant folder.

Next, I needed to find the host.json file. That’s a bit more tricky. In the end, I figured the easiest way was to use the Dev Console. Navigate to Function App Settings, and select “Open dev console”. After a few seconds, the dev console appears:

This appears to be a Linux shell. You should start in the d:\home\site\wwwroot folder – that’s where host.json lives. Just type cat host.json to see the contents. It turns out mine was empty (just an open and close curly brace):

D:\home\site\wwwroot

> ls
D:\home\site\wwwroot
QueueTriggerCSharp1
fetchDepartureBoardOrchestrator
host.json
postToTwitter
> cat host.json
D:\home\site\wwwroot
{}
>

I created this in the root of my repo, then committed the changes and pushed them back to GitHub. Within a few seconds, I was able to see the change by clicking “Configure continuous integrations” in Function App Settings. My changes deployed immediately. And when I next screw up, because I’m forced to push changes via GIT, I know I’ll be able to roll back to a known-good configuration.