Upgrading the disk space in my ZFS-based servers – pt 4

Part 1, Part 2, Part 3

Back on Deneb

I followed a similar procedure with the pools on deneb. The only change was that instead of running the second snapshot plus send/recv task with the system running normally, I ran it with the system running in no install/recovery mode. That way no services or zones were running.

After I had completed renaming, exporting and importing the pools, I rebooted as I had done with eridani. I immediately hit a problem: Smartos crashed at some point during the boot process. Unfortunately, the crash message scrolled off the screen before I could see what it was.

I rebooted and videoed the boot sequence on my ‘phone. There’s a kernel panic that causes the crash but it’s impossible to determine what the cause is.

On the basis that I can only really make progress with a running system, I decided to

  • reboot into recovery mode
  • destroy the the new pool
  • import the old pool as zzbackup
  • install SmartOS on to t a newly created pool
  • try and debug from there.

I removed dsk3 (containing the zzbackup pool) and then reinstalled smartos on to a newly created raidz1 pool.

When I rebooted without dsk3 the system was stable. When I then rebooted with dsk3 installed, the system panicked again!

I rebooted into recovery mode, imported zzbackup and destroyed it.

Now it reboots OK. Now I can import the destroyed zzbackup pool on to an alternate mount point.

[root@deneb ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
        logs
          c1t4d0    ONLINE       0     0     0

errors: No known data errors
[root@deneb ~]# zpool import -D
   pool: zzbackup
     id: 11000531473529046782
  state: ONLINE (DESTROYED)
 action: The pool can be imported using its name or numeric identifier.
 config:

        zzbackup    ONLINE
          c1t3d0    ONLINE
[root@deneb ~]# mkdir /alt
[root@deneb ~]# zpool import -D -R /alt zzbackup
[root@deneb ~]# zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c1t0d0  ONLINE       0     0     0
            c1t1d0  ONLINE       0     0     0
            c1t2d0  ONLINE       0     0     0
        logs
          c1t4d0    ONLINE       0     0     0

errors: No known data errors

  pool: zzbackup
 state: ONLINE
  scan: scrub repaired 0 in 4h47m with 0 errors on Tue Feb  6 22:32:20 2018
config:

        NAME        STATE     READ WRITE CKSUM
        zzbackup    ONLINE       0     0     0
          c1t3d0    ONLINE       0     0     0

errors: No known data errors
[root@deneb ~]# zfs mount
zones                           /zones
zones/archive                   /zones/archive
zones/cores/global              /zones/global/cores
zones/var                       /var
zones/config                    /etc/zones
zones/opt                       /opt
zones/usbkey                    /usbkey
zzbackup/opt/data               /alt/data
zzbackup/opt/data/backups       /alt/data/backups
zzbackup/opt/data/cfg-backups   /alt/data/cfg-backups
zzbackup/opt/data/dev_backups   /alt/data/dev_backups
zzbackup/opt/data/home          /alt/data/home
zzbackup/opt/data/home/git      /alt/data/home/git
zzbackup/opt/data/media         /alt/data/media
zzbackup/opt/data/public        /alt/data/public
zzbackup/opt/data/software      /alt/data/software
...
zzbackup/archive                /alt/zones/archive
...
zzbackup/cores/global           /alt/zones/global/cores
zzbackup                        /alt/zzbackup

Now I can rebuild deneb from the old system. A bit tedious though.

  1. Copied usbkey over and rebooted (had to destroy zzbackup first again though)
  2. Copied /opt over so that the custom services start up.
  3. Rebooted to be sure.

Before laboriously rebuilding, I decided to try booting with dsk3 as zones and the new pool as zznew.

It boots, but the mountpoints are screwed!

root@deneb ~ $ zfs list
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
zones                                             2.67T   863G   588K  /zones
zones/0246b0fe-771c-60ba-cbe6-92ea5795117b        1.21G  8.79G  1.27G  /zones/0246b0fe-771c-60ba-cbe6-92ea5795117b
zones/088b97b0-e1a1-11e5-b895-9baa2086eb33         528M   863G   527M  /zones/088b97b0-e1a1-11e5-b895-9baa2086eb33
zones/147f4eca-1783-4b80-d7e4-9a1d4420567a         294M  9.71G   432M  /zones/147f4eca-1783-4b80-d7e4-9a1d4420567a
zones/163cd9fe-0c90-11e6-bd05-afd50e5961b6         257M   863G   257M  /zones/163cd9fe-0c90-11e6-bd05-afd50e5961b6
zones/1870884c-780a-cb0b-fdc0-8e740afa4173         320M  9.69G   459M  /zones/1870884c-780a-cb0b-fdc0-8e740afa4173
zones/1bd84670-055a-11e5-aaa2-0346bb21d5a1        52.2M   863G  51.9M  /zones/1bd84670-055a-11e5-aaa2-0346bb21d5a1
zones/1ed69a26-f60b-401c-bde6-793df2d0547b        2.12G   498G  2.01G  /zones/1ed69a26-f60b-401c-bde6-793df2d0547b
zones/2a9bfaf4-ddf1-e146-ab80-e2f8723ec714         313M  9.69G   453M  /zones/2a9bfaf4-ddf1-e146-ab80-e2f8723ec714
zones/46c77656-5d22-cdaf-8056-88aaa11c1e58         790M  9.23G   868M  /zones/46c77656-5d22-cdaf-8056-88aaa11c1e58
zones/4bc5b510-2d5d-e47e-c3bc-d492dfeae320         813M  9.21G   813M  /zones/4bc5b510-2d5d-e47e-c3bc-d492dfeae320
zones/4bc5b510-2d5d-e47e-c3bc-d492dfeae320-disk0  53.9G   903G  11.1G  -
zones/5c7d0d24-3475-11e5-8e67-27953a8b237e         256M   863G   256M  /zones/5c7d0d24-3475-11e5-8e67-27953a8b237e
zones/7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b         241M   863G   240M  /zones/7b5981c4-1889-11e7-b4c5-3f3bdfc9b88b
zones/842e6fa6-6e9b-11e5-8402-1b490459e334         226M   863G   226M  /zones/842e6fa6-6e9b-11e5-8402-1b490459e334
zones/a21a64a0-0809-11e5-a64f-ff80e8e8086f         186M   863G   186M  /zones/a21a64a0-0809-11e5-a64f-ff80e8e8086f
zones/archive                                      152K   863G    88K  none
zones/b33d4dec-db27-4337-93b5-1f5e7c5b47ce         792M   863G   792M  -
zones/c8d68a9e-4682-11e5-9450-4f4fadd0936d         139M   863G   139M  /zones/c8d68a9e-4682-11e5-9450-4f4fadd0936d
zones/config                                       468K   863G   196K  legacy
zones/cores                                        250M   863G    88K  none
...
zones/cores/global                                 152K  10.0G    88K  /zones/global/cores
...
zones/dump                                         260K   863G   140K  -
...
zones/opt                                         2.50T   863G  1.20G  legacy
zones/opt/data                                    2.49T   863G   112K  /data
zones/opt/data/backups                             617G   863G   466G  /data/backups
zones/opt/data/cfg-backups                        57.2G   863G  47.8G  /data/cfg-backups
zones/opt/data/dev_backups                        2.61G   863G  2.61G  /data/dev_backups
zones/opt/data/home                                108G   863G   108G  /data/home
zones/opt/data/home/git                            152K   863G    88K  /data/home/git
zones/opt/data/media                              1.73T   863G  1.73T  /data/media
zones/opt/data/public                              172K   863G   108K  /data/public
zones/opt/data/software                            336K   863G   272K  /data/software
zones/swap                                        33.2G   896G   246M  -
zones/usbkey                                       196K   863G   132K  legacy
zones/var                                         1.05G   863G  1.03G  legacy
zznew                                             37.6G  3.47T  1018K  /zznew
zznew/archive                                      117K  3.47T   117K  /zznew/archive
zznew/config                                       139K  3.47T   139K  legacy
zznew/cores                                        234K  3.47T   117K  none
zznew/cores/global                                 117K  10.0G   117K  /zznew/global/cores
zznew/dump                                        1.84G  3.47T  1.84G  -
zznew/opt                                         2.88G  3.47T  2.88G  legacy
zznew/swap                                        32.9G  3.50T  74.6K  -
zznew/usbkey                                       261K  3.47T   261K  legacy
zznew/var                                         3.91M  3.47T  3.91M  /zznew/var

This may be the cause of the panic
The salient parts are:

root@deneb ~ $ zfs list
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
zones                                             2.67T   863G   588K  /zones
zones/archive                                      152K   863G    88K  none
…
zones/config                                       468K   863G   196K  legacy
zones/cores                                        250M   863G    88K  none
…
zones/cores/global                                 152K  10.0G    88K  /zones/global/cores
…
zones/dump                                         260K   863G   140K  -
…
zones/opt                                         2.50T   863G  1.20G  legacy
…
zones/swap                                        33.2G   896G   246M  -
zones/usbkey                                       196K   863G   132K  legacy
zones/var                                         1.05G   863G  1.03G  legacy
zznew                                             37.6G  3.47T  1018K  /zznew
zznew/archive                                      117K  3.47T   117K  /zznew/archive
zznew/config                                       139K  3.47T   139K  legacy
zznew/cores                                        234K  3.47T   117K  none
zznew/cores/global                                 117K  10.0G   117K  /zznew/global/cores
zznew/dump                                        1.84G  3.47T  1.84G  -
zznew/opt                                         2.88G  3.47T  2.88G  legacy
zznew/swap                                        32.9G  3.50T  74.6K  -
zznew/usbkey                                       261K  3.47T   261K  legacy
zznew/var                                         3.91M  3.47T  3.91M  /zznew/var

root@deneb ~ $ zfs mount
zones                           /zones
…
zznew                           /zznew
zznew/archive                   /zznew/archive
zznew/cores/global              /zznew/global/cores
zznew/var                       /zznew/var
zznew/config                    /etc/zones
zznew/opt                       /opt
zznew/usbkey                    /usbkey

As you can see, some of the legacy datasets on zznew are being mounted instead of the equivalents from zones. i.e. it seems to be mixing up the legacy mounts.

yet more to follow

Upgrading the disk space in my ZFS-based servers – pt 3

Part 1, Part 2
It was now time to try the same recipe on the main server: deneb

On Deneb

root@deneb ~ $ zpool status
pool: zones
state: ONLINE
scan: scrub repaired 0 in 7h21m with 0 errors on Fri May 19 18:22:48 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0
logs
c1t4d0 ONLINE 0 0 0

errors: No known data errors

Downgrade one of the mirrors to make room for a new 4TB disk that can be used as a temporary store for deneb‘s data.

root@deneb ~ $ zpool detach zones c1t3d0
root@deneb ~ $ zpool status
pool: zones
state: ONLINE
scan: scrub repaired 0 in 7h21m with 0 errors on Fri May 19 18:22:48 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
logs
c1t4d0 ONLINE 0 0 0

errors: No known data errors
root@deneb ~ $ poweroff
poweroff: Halting 9 zones.

I removed disk 4 and installed the third new 4TB disk in its place.

root@deneb ~ $ zpool status
pool: zones
state: ONLINE
scan: scrub repaired 0 in 7h21m with 0 errors on Fri May 19 18:22:48 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
logs
c1t4d0 ONLINE 0 0 0

errors: No known data errors
root@deneb ~ $ zpool create newzones c1t3d0
root@deneb ~ $ zpool status
pool: newzones
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
newzones ONLINE 0 0 0
c1t3d0 ONLINE 0 0 0

errors: No known data errors

pool: zones
state: ONLINE
scan: scrub repaired 0 in 7h21m with 0 errors on Fri May 19 18:22:48 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c1t0d0 ONLINE 0 0 0
c1t1d0 ONLINE 0 0 0
c1t2d0 ONLINE 0 0 0
logs
c1t4d0 ONLINE 0 0 0

errors: No known data errors

Now I can clone zones on to newzones

root@deneb ~ $ zfs snapshot -r zones@txfr1
root@deneb ~ $ zfs send -R zones@txfr1 | zfs recv -F newzones

This took a long time!

Upgrading the disk space in my ZFS-based servers – pt 2

See here for part 1

On Eridani

Once the initial send/recv had completed, I did another snapshot and sent the incremental data just in case anything had changed.

root@eridani ~ $ zfs snapshot -r zones@txfr2
root@eridani ~ $ zfs send -R -i txfr zones@txfr2 | pv | zfs recv tempzone

Lastly, I promoted the incremental snapshot to be the current version of tempzone

root@eridani ~ $ zfs rollback tempzone@txfr2

As a final (paranoid) check, I ran a dummy rsync task to check if /zones was the same as /tempzone

root@eridani ~ $ rsync -avn /zones/ /tempzone/ | less
sending incremental file list
global/cores/

sent 765011152 bytes  received 2764561 bytes  486087.82 bytes/sec
total size is 3454738169591  speedup is 4499.67 (DRY RUN)

Nothing had changed, so I could now swap the pools over

root@eridani ~ $ zpool export tempzone

GOTCHA!

I couldn’t export the root zones pool at this point because it had mounted filesystems that were in use by the running system. To get further I had to reboot the system into restore/recovery mode.

In this mode, no pools are imported, so I could execute the following commands

zpool status - to confirm that no pools were mounted
zpool import - to see what pools were available
zpool import -NR /t1 tempzone zones - -NR to avoid any datasets being mounted and use an alternate mount point
zpool import -NR /t2  zones oldzones
zpool export oldzones
zpool export zones
reboot

I was still getting errors even though the pools had been renamed. The problem turned out to be that when SmartOS boots, it seems to mount the pools in alphabetical order.

Probably more likely that SmartOS scans the disks in alphabetical order

Thus, oldzones was mounted before zones and it’s datasets were grabbing the mount points.
Rather than laboriously change the mountpoint property on all the datasets, I simply disconnected the disk.

Once I had completed this, eridani booted using the new pool.

root@eridani ~ $ 
root@eridani ~ $ zpool status
  pool: zones
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    zones       ONLINE       0     0     0
      raidz1-0  ONLINE       0     0     0
        c2d1    ONLINE       0     0     0
        c3d0    ONLINE       0     0     0
        c3d1    ONLINE       0     0     0

errors: No known data errors
root@eridani ~ $ zfs list
NAME                                                                  USED  AVAIL  REFER  MOUNTPOINT
zones                                                                2.52T  2.73T   318K  /zones
zones/archive                                                        29.3K  2.73T  29.3K  none
zones/backup                                                         2.50T  2.73T  29.3K  /zones/backup
zones/backup/deneb                                                   2.50T  2.73T  29.3K  /zones/backup/deneb
zones/backup/deneb/zones                                             2.50T  2.73T   324K  /zones/backup/deneb/zones
...
zones/config                                                         55.9K  2.73T  36.0K  legacy
zones/cores                                                          58.6K  2.73T  29.3K  none
zones/cores/global                                                   29.3K  10.0G  29.3K  /zones/global/cores
zones/dump                                                           1023M  2.73T  1023M  -
zones/opt                                                             423M  2.73T   422M  legacy
zones/swap                                                           17.9G  2.74T  1.44G  -
zones/usbkey                                                         38.6K  2.73T  38.6K  legacy
zones/var                                                            7.08M  2.73T  5.42M  legacy
root@eridani ~ $ zfs mount
zones                           /zones
zones/backup                    /zones/backup
zones/backup/deneb              /zones/backup/deneb
zones/backup/deneb/zones        /zones/backup/deneb/zones
...
zones/backup/deneb/zones/usbkey  /zones/backup/deneb/zones/usbkey
zones/backup/deneb/zones/var    /zones/backup/deneb/zones/var
zones/cores/global              /zones/global/cores
zones/var                       /var
zones/config                    /etc/zones
zones/opt                       /opt
zones/usbkey                    /usbkey
root@eridani ~ $ 

Upgrading the disk space in my ZFS-based servers – pt 1

Introduction

Back in 2016 I bought a Supermicro server and installed SMARTOS to create a very capable home server. I named this server Deneb. See my posts on my new ZFS home server for more details.
Some time later, I bought an old Dell T310 to act as a backup device for Deneb. I named this server Eridani and use ZnapZend2 to perform 4-hourly snapshots and nightly backups of Deneb to Eridani.
All has been working well, but I am now approaching the point where I am running out of space on Eridani and Deneb is not far off a similar position.

  • Eridani is close to full capacity (>90%)
  • Deneb is wasteful in its use of disk. It uses two dual-disk mirrors instead of raidz1
  • Deneb is also at 81% (3.62 total)

Objectives

After some consideration, I set the objectives as being to aim for 3-5 years of space on both servers, and to achieve this partly by moving to raidz1 to improve space utilisation.

Current Configurations

Deneb

  • Supermicro server, X10SDV-TLN4F motherboard, 32GB ECC RAM, single XEON
  • 4 x HGST Ultrastor 2TB 7200rpm SATA disks
  • 2 x mirrored pairs
  • 4TB in total

Eridani

  • Dell T301 server, single Xeon, 16GB ECC RAM
  • 2 x 3TB disks in mirror
  • 3TB in total

Strategy

  • Move Deneb and Eridani to raidz1
  • Increase Deneb from 4TB to 6TB (4 x 2TB disks in raidz1 = 6TB)
  • Increase Eridani to 9TB (3 x 4TB + 1 x 3TB disks in raidz1 = 9TB

At a later point, I may replace the 3TB disks to give 12TB.

In addition, I proposed to move a few zones from Deneb to Eridani. Deneb is currently used to backup a number of other computers in the house.

  • veeam is the backup target for the Windows computers in the house. They all run veeam nightly.
  • cfg-backups runs rsnapshot to backup a number of Raspberry Pi computers in the house.
  • Lastly, dev-backups contains dd images of the Raspberry Pi computers that can be loaded by NOOBS3
    Moving these to Eridani would free 700G on Deneb without increasing space usage on Eridani.

High Level Plan

Part 1 – Eridani

  1. Buy 3 new 4TB disks
  2. Split the mirror on Eridani to free one of the 3TB disks
  3. Create a new 6TB raidz1 pool with the freed 3TB disk and 2 of the new 4TB disks
  4. Migrate the existing pool to the new pool
  5. Destroy the old pool and add its 3TB to the new pool, making 9TB

Part 2 – Deneb

  1. Split one of the mirrors on Deneb and replace the detached disk with the third 4TB disk
  2. Create a new temporary pool containing just the 4TB disk
  3. Migrate the existing pool to the temporary pool
  4. Destroy the old pool and rebuild it as raidz1, giving 4TB
  5. Migrate the data back to the newly built pool
  6. Destroy the temporary pool and replace the 4TB disk with the original 2TB disk
  7. Add the 2TB disk into the raidz1 pool to make 6TB
  8. Keep the 4TB disk as a spare.

Implementation

Prepare Eridani

The first task was to downgrade the existing mirrored zpool to be a single disk so that I could re-use the second disk as part of the new raidz1 pool.

After logging in to Eridani…

  1. Check the status of the pool
root@eridani ~ $ zpool status
pool: zones
state: ONLINE
scan: scrub repaired 0 in 9h1m with 0 errors on Fri Sep 22 05:01:02 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c2d0 ONLINE 0 0 0
c3d0 ONLINE 0 0 0

errors: No known data errors
  1. Detach the second HD to make room for a new 4TB disk
root@eridani ~ $ zpool detach zones c3d0
root@eridani ~ $ zpool status
pool: zones
state: ONLINE
scan: scrub repaired 0 in 9h1m with 0 errors on Fri Sep 22 05:01:02 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
c2d0 ONLINE 0 0 0

errors: No known data errors
root@eridani ~ $ poweroff

Upgrade the disks

With the power off, I installed two new 4TB HGST Deskstar NAS disks into the empty slots in the T310. These will appear as c3d1 and c4d0.

Adjust the BIOS settings to recognise the new disks

The T310 originally came with a PERC 6/i RAID controller. Unfortunately, the 6/i only supports 2TB disks or smaller, so I had to ditch it and use the SATA ports on the motherboard itself. This was not a great loss, as I wasn’t going to make use of the RAID capabilities anyway, but it did throw me a curve ball when I originally got the T310.

SMARTOS runs from an external USB drive rather than from the internal disks. This is not the default behaviour and has to be forced by adjusting the BIOS settings. Adding new disks resets the boot sequence to the default.

I accessed the T310 via the iDRAC and intercepted the boot sequence with <F2>. I then went into SATA Settings to check that the new disks had been registered. I then went into Boot Sequence and modified it to reboot from the external USB.

Reconfigure Eridani’s disks

root@eridani ~ $ zpool status
pool: zones
state: ONLINE
scan: scrub repaired 0 in 9h1m with 0 errors on Fri Sep 22 05:01:02 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
c2d0 ONLINE 0 0 0

errors: No known data errors
root@eridani ~ $ zpool create tempzone raidz1 c2d1 c3d0 c3d1
root@eridani ~ $ zpool status
pool: tempzone
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
tempzone ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
c2d1 ONLINE 0 0 0
c3d0 ONLINE 0 0 0
c3d1 ONLINE 0 0 0

errors: No known data errors

pool: zones
state: ONLINE
scan: scrub repaired 0 in 9h1m with 0 errors on Fri Sep 22 05:01:02 2017
config:

NAME STATE READ WRITE CKSUM
zones ONLINE 0 0 0
c2d0 ONLINE 0 0 0

errors: No known data errors
root@eridani ~ $ zfs snapshot -r zones@txfr
root@eridani ~ $ zfs send -R zones@txfr | pv | zfs recv -F tempzone

Note: I used pv so I could see progress.

To be continued

New ZFS based NAS and VM Host – part 3

In part 1 of this series, I covered the requirements and hardware. In part 2 I covered the initial configuration of the new server. In this part I’ll cover setting the server up as a file server.

The main use case for this new server is to be the main file and media server for our home. To achieve this I needed NFS, SMB and AFP access to the imported datasets.

NFS access is available by default in ZFS, but to get SMB and AFP access requires software to be installed. As indicated in part 2, you are strongly discouraged from installing software in the global zone. The supported approach is to create a new zone and install the software in there.
Zones are sort-of virtual machines in that each thinks it has exclusive use to hardware and are separate security containers. Where SmartOS differs from VMWare is its hybrid approach to supporting virtual machines. It does this by supporting multiple “brands” of zones:

  • “joyent” branded zones appear to be running SmartOS itself. They have no kernel of their own, they re-use the global zone’s kernel and just provide resource and security isolation.
  • “lx” branded zones appear to be running a specific version of Linux. As with “joyent” zones, they re-use the global zone’s kernel but translate the “brand”‘s system calls into those supported by SmartOS. This gets you the benefits of the software that normally runs on the brand, but without the overhead of having two run the brand’s kernel on top of the SmartOS kernel. The result is near bare-metal speeds. Currently (Sept 2015), SmartOS supports Ubuntu, Centos, Debian, Fedora (maybe others).
  • “kvm” branded zones are more like any other KVM virtual machine: allowing just about any other operating to be installed.

First attempt using an Ubuntu branded zone

This failed, so I’m not going into the detail

You could install Ubuntu in a kvm branded zone, but using an Ubuntu version of a lx zone avoids running two kernels. Base images for many Ubuntu variants exist in Joyent’s public repository, so I simply followed the instructions in the SmartOS wiki to:

  1. import the base Ubuntu 14.04 LTS server image
  2. Create a json file that describes the new zone
  3. Create the new zone using the json file.
    At the end of this, I had an Ubuntu 14.04 virtual machine called capella, on the same IP address as the old server and with direct access to the ZFS datasets containing the files from the old server.

I now followed the guide at outcoldman.comcoldman to install SAMBA and the guide at … to install Netatalk.
At the end, I had a functioning Samba server, but I had trouble with netatalk. My Macbook Air running Mavericks couldn’t connect to Capella using AFP. Investigation showed that the 14.04 version of lx-ubuntu was missing the uams_dhx2.so security module that was needed to support Mavericks.
Note: SmartOS freely admit that branded zones are still being developed

Second attempt using a native SmartOS zone

Rather than spending too much time on this, I exploited one of the major advantages of using SmartOS. I simply deleted the zone, downloaded a basic joyent brand zone, created a new json file and created a new joyent branded zone. It took 5 minutes! I used the following json

{
"hostname": "capella.agdon.net",
"alias": "capella",
"brand": "joyent",
"max_physical_memory": 4096,
"image_uuid": "5c7d0d24-3475-11e5-8e67-27953a8b237e",
"resolvers": ["172.29.12.7","8.8.4.4"],
"nics": [
{
"nic_tag": "admin",
"ip": "172.29.12.11",
"netmask": "255.255.255.0",
"gateway": "172.29.12.1",
"primary": "1"
}
],
"filesystems": [
{
"type": "lofs",
"source": "/data/media",
"target": "/import/media"
},
{
"type": "lofs",
"source": "/data/home",
"target": "/import/home"
},
{
"type": "lofs",
"source": "/data/home/git",
"target": "/import/home/git"
},
{
"type": "lofs",
"source": "/data/public",
"target": "/import/public"
},
{
"type": "lofs",
"source": "/data/software",
"target": "/import/software"
}
]
}

I then installed Samba and Netatalk as before. This time all was well and I now had a functioning NFS, SMB and AFP file server.

I reconfigured the clients to access the new server and I was back where I was before I changed hardware. Simples!

Next Step, install Plex media server, SABNZBD, CouchPotato and Sickbeard to create a fully functioning media server.

New ZFS based NAS and VM Host – part 2

In part one I covered the motivation, requirements and hardware. In this post I will cover software installation and configuration.

Installing and configuring SmartOS

SmartOS differs from many other operating systems, though not FreeNAS, by not requiring installation. You simply copy a disk image to a USB thumb drive and boot from it. SmartOS then creates a RAMDisk, copies itself to the RAMDisk and runs from there. Alternatively you can boot across the network using PXE.

This exposes SmartOS’s primary use case as a data-centre operating system. By not requiring installation, upgrades are quickly deployed by copying a new image to the flash drive and rebooting.

This does have an important side effect however. SmartOS supports the notion of “zones”, first implemented in Solaris. When booted, SmartOS itself runs in the “global” zone. However, because the filesystem is on a RAMdisk, any changes you make do not persist across a reboot. There are ways to get around this so that (e.g.) you can ensure your SSH public key is an authorized_key and you can login without a password; but you are strongly discouraged from installing software in the global zone. More on that later.

Installation

I started by following the instructions in the SmartOS wiki to download the latest SmartOS image and copy it to a 2GB consumer grade USB thumb drive. It’s only 161MB so it didn’t take long.
I booted the server from the thumb drive and, because this was a clean system, I was presented with a wizard that asked for hostname, IP address (or DHCP) and the identities of the drives I wished to use for the “zones” pool; which is used to store all the datasets for the other zones.

Rather than risk getting it all wrong, I chose the first two HGST drives and put them in a mirror. After configuring the zpool, I was presented with the login prompt.

The default login is root/root, so I immediately changed the root password!

zpool status showed

# zpool status
pool: zones
state: ONLINE
scan: resilvered 1.98G in 0h0m with 0 errors on Fri Sep 11 13:22:01 2015
config:
NAME        STATE     READ WRITE CKSUM
zones       ONLINE       0     0     0
  mirror  ONLINE       0     0     0
    c1t0d0  ONLINE       0     0     0
    c1t1d0  ONLINE       0     0     0

After this, I added the other two drives in as a second mirrored vdev using

zpool add zones mirror c1t2do c1t3do

and then added the SSD as an slog

zpool add zones log c1t4do

at the end of this zpool status showed

# zpool status
  pool: zones
 state: ONLINE
  scan: resilvered 4.01G in 0h0m with 0 errors on Fri Sep 11 13:38:15 2015
config:

        NAME       STATE      READ WRITE CKSUM
        zones      ONLINE        0     0     0
          mirror-0 ONLINE        0     0     0
            c1t0d0 ONLINE        0     0     0
            c1t1d0 ONLINE        0     0     0
          mirror-2 ONLINE        0     0     0
            c1t2d0 ONLINE        0     0     0
            c1t3d0 ONLINE        0     0     0
        logs
          c1t4d0   ONLINE        0     0     0

errors: No known data errors

This whole process took about 30 minutes (including downloading and copying the SmartOS image)

Moving data from the old server

Now that I had the new server installed and ready to go, I needed to copy the data across from the old server. There are a number of ways to do this:

  1. Physically move the disks across
  2. Use ZFS Send/ZFS receive to copy the data across the network
  3. Use rsync to send files.

As the old server was still running, I didn’t want to move the disks, but experiment showed it was going to take days to copy the data across my network. So I compromised.

I split the mirror on the old server and moved one disk to the new server. I then imported it:

zpool import rdata

It showed up as a degraded mirror. To get the data across, I did the following:

# zfs create snapshot rdata@export
# zfs create zones/import -o mountpoint=/import
# zfs snapshot -r rdata@export
# for dset in "media public software home"; do zfs send -R rdata/${dset}@export | zfs recv data/${dset}; done

After checking the data was there I cleaned up:

zpool export rdata
poweroff

I’ll keep this as an archive disk and re-use the one in the old server.

Next Steps

The initial use for this server is as a file/print server and as a media server. In the next post, I’ll cover how I did this.

New ZFS based NAS and VM Host – part 1

For some time I’ve been using a re-purposed Acer desktop PC as a NAS. It runs OpenIndiana with 2 x 3TB disks in a ZFS mirror. It has Napp-IT installed for administration.

It’s been OK, but it’s not very powerful and only has limited throughout on its single 1000BaseT Ethernet connection. I’ve been considering an upgrade for ages, but finally bit the bullet this week. I’ll cover the requirements and spec in this post and then the build in a followup.

Requirements

The current server was just a file server. I did try to run another zone on it but it couldn’t hack it. I wanted to get back to a position where I could run Virtual Machines as well. The main driver was to be able to use the NAS as a Crashplan host. There is a version of Crashplan for Solaris, but at the last major upgrade they dropped support for being the destination of a Crashplan. I still had the cloud backup but it was nice to have a local replica as well. So, being able to have the Linux version running in a  VM would be good.

  • NFS access from a bunch of RaspberryPi devices in the house
  • CIFS access from PCs and the SONOS devices
  • NETATALK access from my Macbook Air and the Apple TV
  • Support for running multiple VMs: including
  • Windows Home Server V2 to back up the Windows PCs
  • Ubuntu to host Crashplan

Whilst noodling about these requirements I was also thinking about replacing the Thinkpad that runs my radio software in the shack. A lightbulb moment made me reconsider how the IT is structured here.

Currently, I have a CAT6 network throughout the house and down to the shack. The Acer sits in the garage attached to the house and there is a FreeNAS server in the shack who’s main purpose is to be a backup for the Acer. It also has a jail running some scripts to keep backups of the RaspberryPi devices. I then have a Thinkpad T60 as my shack computer.

The idea is to move the FreeNAS device to the garage and put the new server in the shack. If I ensured I could do IO virtualisation (so a VM could make best use of a video card and get isochronous access to USB devices) and ensured it had a good video card then I also use the new server as my shack computer.

Solution

Hardware

To get the virtualisation features means an Intel Xeon class processor or the AMD equivalent. I’ve been a fan of Supermicro for some time and saw that the X10SDV-TLN4F looked to be perfect. I did some research and came across this post by Benjamin Bryan on a Supermicro Datacentre in a box using a close relative of this board. Perfect.

In the end I opted to buy the SYS-5028D-TN4T barebones server which includes this motherboard in the stunning CSE-721TQ-250B case. This has four front access 3.5 drive slots and two internal positions for 2.5 drives. I also bought 32GB ECC memory from Crucial and four HGST 2TB drives from Hitachi.

This is an expensive build but I think it will be worth it. I haven’t bought the video card yet.

(Incidentally. Years ago (back in the 90’s), the rule of thumb was that the computer you really lusted after always cost £2000. For a while that hasn’t been true for desktops, but I reckon it’s still good for servers).

Software

In Ben’s build he opted for the Napp-IT in one approach of Illuminos/ESXi and then VMs. However, in the comments there were references to SmartOS. Having this would avoid the need for a PCI Host Bus Adapter and would be simpler.

SmartOS supports zones like Solaris and OpenIndiana but adds support for KVM virtual machines. SmartOS is run from a flash drive and builds ZFS zpools from the disks. Because you are starting from a flashdrive, SmartOS runs from a RAMDrive and isn’t persistent. This means the global zone should be kept simple: i.e. so you don’t install software in it. Instead, create another zone and install software in there.

The beauty of SmartOS zones is that they use the same kernel as the global zone: i.e. you only need space for new software and any data. What happens is that the new zone os created from a ZFS snapshot of the global zone. Elegant!

More on the build itself in part 2.