EON NAS Expansion and upgrade

EON NAS has served me well for a few years, but I am out of space. Also considering to upgrade EON, and perhaps ZPOOL and ZFS versions (options such as block-level deduplication are appealing, as it makes it easy to bulk backup tons of old disks which are largely older versions of my current repo, without worrying about wasting space by having the same file many times)

First order of business: expand from 9x2TB to 9x4TB drives.  Following refs such as:

http://www.itsacon.net/computers/unix/growing-a-zfs-pool/

I first check that all my drives are currently alive, the system is fully silvered, and then scrub to make doubly sure there are no errors.

My pool is raidz2, so in theory I could replace two disks at once, but obviously that leaves me vulnerable. Having dual parity allows me to swap one disk at a time, and still be protected again a disk failure mid-upgrade!

Start with "zpool status", which shows mediapool totally healthy.

"zpool scrub mediapool"

Says it'll take 50 hours to complete. Well, we wouldn't have gone out of our way to use ZFS unless we were paranoid about our data, so we'll suck it up and wait it out.

Okay, scrub completed successfully. Shut down, remove one of the 2TB drives and replace it with a 4TB drive. Verify at boot that the BIOS sees the 4TB drive.


media:1:~#zpool status
  pool: mediapool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-2Q
 scrub: none requested
config:

        NAME                    STATE     READ WRITE CKSUM
        mediapool               DEGRADED     0     0     0
          raidz2-0              DEGRADED     0     0     0
            c0t0d0              ONLINE       0     0     0
            c0t1d0              ONLINE       0     0     0
            c2t2d0              ONLINE       0     0     0
            c2t3d0              ONLINE       0     0     0
            c3t0d0              ONLINE       0     0     0
            c3t1d0              ONLINE       0     0     0
            c3t2d0              ONLINE       0     0     0
            c3t3d0              ONLINE       0     0     0
            390014080793031528  UNAVAIL      0     0     0  was /dev/dsk/c3t4d0s0

errors: No known data errors

#zpool replace mediapool c3t4d0s0

cannot open '/dev/dsk/c3t4d0s0': I/O error

cfgadm -s "select=type(disk)"

Ap_Id                          Type         Receptacle   Occupant     Condition

sata0/0::dsk/c3t0d0            disk         connected    configured   ok

sata0/1::dsk/c3t1d0            disk         connected    configured   ok

sata0/2::dsk/c3t2d0            disk         connected    configured   ok

sata0/3::dsk/c3t3d0            disk         connected    configured   ok

sata0/4::dsk/c3t4d0            disk         connected    configured   ok

sata1/0::dsk/c0t0d0            disk         connected    configured   ok

sata1/1::dsk/c0t1d0            disk         connected    configured   ok

sata2/2::dsk/c2t2d0            disk         connected    configured   ok

sata2/3::dsk/c2t3d0            disk         connected    configured   ok

So, the system sees c3t4d0. So then s0 does not exist probably. I can either replace with p0 to use the entire partition, or do a format to single slice and use s0. I don't recall why I used s0 before... probably so that I could upgrade some of the drives with larger ones with matching s0, then use s1 for extra space. Should be possible even with p0. Need to check my records.

Wait, that's not right. zpool replace should take a raw disk, format it, and resilver (according to http://docs.oracle.com/cd/E19082-01/817-2271/gbcet/index.html)

#zpool replace mediapool c3t4d0

cannot replace c3t4d0 with c3t4d0: device is too small

#format

Searching for disks...done

AVAILABLE DISK SELECTIONS:

       0. c0t0d0

          /pci@0,0/pci8086,3a46@1c,3/pci1458,b000@0/disk@0,0

       1. c0t1d0

          /pci@0,0/pci8086,3a46@1c,3/pci1458,b000@0/disk@1,0

       2. c1d0

          /pci@0,0/pci8086,244e@1e/pci-ide@0/ide@0/cmdk@0,0

       3. c2t2d0

          /pci@0,0/pci8086,244e@1e/pci1095,7124@1/disk@2,0

       4. c2t3d0

          /pci@0,0/pci8086,244e@1e/pci1095,7124@1/disk@3,0

       5. c3t0d0

          /pci@0,0/pci1458,b005@1f,2/disk@0,0

       6. c3t1d0

          /pci@0,0/pci1458,b005@1f,2/disk@1,0

       7. c3t2d0

          /pci@0,0/pci1458,b005@1f,2/disk@2,0

       8. c3t3d0

          /pci@0,0/pci1458,b005@1f,2/disk@3,0

       9. c3t4d0

          /pci@0,0/pci1458,b005@1f,2/disk@4,0

There's the problem; my drive, which shows up as 4000GB to BIOS, shows up as 1.64 TB within EON Solaris. Why? Poking around the net, it should be

It is an odd point that the perceived size is exactly 2TB less than the actual size. I'm running 64-bit solaris. The BIOS reports the drive as 4TB (need to double-check this is currently the case, as I tried a few different slots trying to get it to work)

Answer: although I was not able to find documentation of this, early versions of EON did not have support for 3TB+ drives completely in place. Need to update to recent version.

After updating to EON version 1.0b,


media:1:~#format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c0t0d0
          /pci@0,0/pci8086,3a46@1c,3/pci1458,b000@0/disk@0,0

The drive's correct size is seen.

Check pool status


media:2:~#zpool status
  pool: mediapool
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://www.sun.com/msg/ZFS-8000-4J
  scan: none requested
config:

        NAME                      STATE     READ WRITE CKSUM
        mediapool                 DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            14963257236115845187  UNAVAIL      0     0     0  was /dev/dsk/c0t0d0s0
            c0t1d0                ONLINE       0     0     0
            c4t2d0                ONLINE       0     0     0
            c4t3d0                ONLINE       0     0     0
            c5t0d0                ONLINE       0     0     0
            c5t1d0                ONLINE       0     0     0
            c5t2d0                ONLINE       0     0     0
            c5t3d0                ONLINE       0     0     0
            c5t4d0                ONLINE       0     0     0

and we're ready to replace

media:4:~#zpool replace mediapool c0t0d0

cannot replace c0t0d0 with c0t0d0: devices have different sector alignment

Crap.

According to this thread it is completely impossible to reaplce a 512k-sector drive with a 4k-sector drive. Oddly, it seems that ZFS will actually tolerate some level of mixage between aligments (and sector size?) at time of pool creation, or adding drives to a pool, but not when it comes time to replace a drive. The *only* option appears to be to back all the data up and create a fresh pool. A very unappealing prospect. I also means I will need 7x2TB drives.

Although the above thread seems authoritative, need to sleep on it and understand the situation a little better before taking action.

I have enough 2TB drives to make a complete backup, but some of them are in use in other ways. I might have a few which are already backups from an older state of the EON pool. I can overwrite those with the current pool contents. After that I have some matched master / backup pairs of data which is queued to enter pool.

Also worth noting: at time of writing, the cheapest 4tb drives ($210) are the same value as 2TB drives ($105) but could be saved for spares in the new pool. So any new drives purchased should be 4TB.

...

After backing all media up to UFS-formatted drives, I created a new pool from the 9 4TB drives and coping all the files back. I failed to upgrade the pool in-place, but did end up with all my data safe and sound in the new system.

Update EON from snv_125

According to andre, best is to fresh install. Failing that:

I have a working windows -> USB install but not yet ready for release. USB install/image coming ...

For now what you can do is

Insert 2nd key(new) and run install.sh. Install to new key.

After it is done you will have a clone of the first key.

Then, use transporter with the new iso to overwrite the 2nd key that you just installed/cloned.

Now you have the new image installed on the new key.

so, let's try that.

First, list all the drives:

media:5:~#echo | format -e
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c0t0d0
          /pci@0,0/pci8086,3a46@1c,3/pci1458,b000@0/disk@0,0
...
       9. c3t4d0
          /pci@0,0/pci1458,b005@1f,2/disk@4,0

then plug in the USB key and try again:

media:6:~#echo | format -e
Searching for disks...
The current rpm value 0 is invalid, adjusting it to 3600
done

c4t0d0: configured with capacity of 14.80GB

AVAILABLE DISK SELECTIONS:
       0. c0t0d0
          /pci@0,0/pci8086,3a46@1c,3/pci1458,b000@0/disk@0,0

      10. c4t0d0
          /pci@0,0/pci1458,5006@1a,7/storage@5/disk@0,0
Specify disk (enter its number): Specify disk (enter its number):

I can clearly see that the 16GB USB key is c4t0d0

media:7:~#install.sh

This process installs EON ZFS Storage to a destination listed below:

[1] c0t0d0 (Unknown MB)

[2] c0t1d0 (Unknown MB)

[3] c1d0 (3816MB)

[4] c2t2d0 (Unknown MB)

[5] c2t3d0 (Unknown MB)

[6] c3t0d0 (Unknown MB)

[7] c3t1d0 (Unknown MB)

[8] c3t2d0 (Unknown MB)

[9] c3t3d0 (Unknown MB)

[10] c3t4d0 (Unknown MB)

[11] c4t0d0 (Unknown MB)

Enter destination choice[?]: 11

 ... lots of stuff, have to answer 'y' a few times

/mnt/install/boot/x86.eon:       OK

EON ZFS Storage install complete on /dev/dsk/c4t0d0s0

rebooting... as before, my system hangs when a bootable solaris system is present on USB. It doesn't even finish listing all the PCI cards. So have to try again, targeting a second CF card.

media:4:~#install.sh

This process installs EON ZFS Storage to a destination listed below:

[1] c0t0d0 (Unknown MB)

[2] c0t1d0 (Unknown MB)

[3] c1d0 (3816MB)

[4] c1d1 (Unknown MB)

[5] c2t2d0 (Unknown MB)

[6] c2t3d0 (Unknown MB)

[7] c3t0d0 (Unknown MB)

[8] c3t1d0 (Unknown MB)

[9] c3t2d0 (Unknown MB)

[10] c3t3d0 (Unknown MB)

[11] c3t4d0 (Unknown MB)

Enter destination choice[?]: 4

After it succeeds, I shut down, remove the old boot system from slot 0 of the CF card, and move the newly created one from slot 1 to slot 0. It boots okay, I am able  to log in with my previously created account, and verify that the pool is healthy and ip addresses are correct. Looks like the new install worked. Now I have a playground to update, and the safety of the old system on the other CF card.

Now it is time to upgrade to the latest and greatest. For versions of EON slightly newer than mine, the upgrade tool is/was transporter.sh - see this post - and elsewhere in comments Andre advises those of us using older versions without transporter.sh to simply download the script from the downloads page.

I download transporter.sh and scp it into /tmp/ on my eon box.

Download the latest CIFS ISO and scp it into /tmp/ as well

Then as root on my eon box:

#chmod +x /tmp/transporter.sh

#/tmp/transporter.sh -i /tmp/eon-1.0b-151-64-cifs-min.iso -d /mnt/eon0

 OK: lofiadm -a /tmp/eon-1.0b-151-64-cifs-min.iso /dev/lofi/1

 OK: mount -F hsfs /dev/lofi/1 /tmp/upgrade

 removing /mnt/eon0/boot

 copying /tmp/upgrade -> /mnt/eon0

  ...

 unmounting /dev/lofi/1

 OK: umount /dev/lofi/1

 releasing /dev/lofi/1

 OK: lofiadm -d /dev/lofi/1

Great, complete with no problems. But we're not done quite yet. The new system just installed contains none of the setup I made to the original system - ip addresses, accounts, and other config. Normally we run updimg.sh to write changes we've made to the running system into the stored image so that they persist across a reboot. But in the comments here Andre cautions that by default, that script will overwrite parts of the new system with the old one. So according to his advice, edit /mnt/eon0/.backup and comment out the three entries in /usr/bin/ to avoid reverting them. Then

#updimg.sh /mnt/eon0/boot/x86.eon 

Updating files in /mnt/eon0/.backup to x86.eon

backup in /mnt/eon0/boot/x86.eon.1

 /mnt/eon0/.backup:     OK

gzcat /mnt/eon0/boot/x86.eon > /tmp/x86.1336

lofiadm -a /tmp/x86.1336 /dev/lofi/1

mounting ... /dev/lofi/1 /mnt/upd

 copying /etc/svc/repository.db

umounting ... /mnt/upd

lofiadm -d /dev/lofi/1

mv -f /mnt/eon0/boot/x86.eon /mnt/eon0/boot/x86.eon.oem

gzip -f -9 -c /tmp/x86.1336 > /mnt/eon0/boot/x86.eon

/mnt/eon0/boot/x86.eon:  OK

At this point we should be ready to reboot and enjoy our new system!

Rebooting, GRUB displays the new EON logo as well as the version number 0.97

Before the login prompt there are some errors related to the crypto service

I am not able to SSH in. I can log in using one of the accounts I previously created. zpool shows no pool. ifconfig -a shows that the ip address settings were also lost.

Unfortunate. I still have the disk with the original system, so I could go through the whole process again, but it is probably easier to move forward and fix the config.

first, 

#zpool import

resurrects my pool. Everything looks healthy, although "zpool status" notes that I am using an older version of on-disk format (meaning that there is now a new version available but I have yet to upgrade).

Actually, the docs on the eonstorage google site don't mention such pedestrian configs as network addresses. Referring to the notes from my original install... I also glossed over those bits.

ifconfig fails with an "Object not found" error

svcs -xv shows:

svc:/network/physical:default (physical network interfaces)

 State: maintenance since (time of boot)

Reason: Start method exited with $SMF_EXIT_ERR_CONFIG.

Impact: 21 dependent services are not running:

  nfs

  smb

  etc

svc:/network/physical:nwam (physical network interface autoconfiguration)

 State: disabled since (time of boot)

Reason: Disabled by an administrator

Impact: 21 dependent services are running

svc:/system/idmap/default (Native Identity Mapping Service)

State: maintenance since (time of boot)

Reason: Start method failed repeatedly, last exited with status 1

Impact: 5 dependent services

  smb

  nfs

  etc

svc:/system/cryptosvc:default (cryptographic services)

State: maintenance since (time of boot)

Reason: restarting too quickly

Impact: 2 dependent services

  ssh

  milestone

Okay, that's a bit of a mess, but it seems that if I could get the network working the rest might fall into place. ifconfig -a does show both of my cards, so they seem to be working at the hardware and driver level. What is the recommended way to configure stuff in EON?

Well, /etc/hostname.rge0 does contain the IP address that I had previously configured. Looking at /usr/bin/setup, I try to reproduce it... all the relevant files seem to have the correct info from previous setup. The only missing step is to start the net service:

#/lib/svc/method/net-physical

/lib/svc/method/net-physical[73]: /sbin/ibd_upgrade: not found [No such file or directory]

ifconfig: cannot plumb rge0: Interface already exists

ifconfig: cannot plumb rge1: Interface already exists

configuring IPv4 interfaces: ifconfig: could not create address:Object not found

 rge0ifconfig: could not create address:Object not found

 rge1

Failed to configure IPv4 interface(s): rge0 rge1

it is true that /sbin/ibd_upgrade does not exist. Googling only turns up two results for "/sbin/ibd_upgrade: not found" neither of which offers a clear explanation

Reboot again and transcribe onscreen errors:

WARNING: No major number for driver mega_sas in class scsi
WARNING: failed to resolve 'scsa.probe' driver alias, defaulting to 'nulldriver'
WARNING: no randomness provider enabled for /dev/random. Use cryptoadm(1M) to enable a provider
Configuring devices
Failed to configure IPv4 interface(s): rge0 rge1
svc.startd[8]: svc:/network/physical:default: Method "/lib/svc/method/net-physical" failed with exit status 96.
svc.startd[8]: network/physical:default misconfigured: transitioned to maintenance (see svcs -xv for details)
... may have missed some, output too fast for video recording to keep up with
svc.startd[8]: svc:/system/idmap:default: Method "/usr/lib/idmapd" failed with exit status 1
... cryposvc
... failed to abandon contract 37: Permission denied


Okay, put that aside for now and continue with setup: edit /mnt/eon0/.exec and enter correct pool name.
zpool import -> success
updimg.sh /mnt/eon0/boot/x86.eon
init 6 -> reboot

my zpool is still not imported on boot, even though .exec contains my correct pool name. "zpool import" in the command line succeeds. It seems that .exec is not being run? Or it is run, but "zpool import -f -a" fails because SMB etc services can't be started.

So 2 main issues: networking fails, zpool doesn't load on boot. Need to seek help, and read through blog comments for any similar troubleshooting.

Andre suggests I just start over, so backup my upgraded img and copy the OEM one to /mnt/eon0/boot/x86.eon

So now I'm following the "new setups" instructions from here

cd /mnt/eon0/bin
./slinky r
./slinky c
setup

check poolname in .exec

add group and user for media per my notes from original setup

updimg.sh

skip the nfs setup (I don't think I ever actually used it)
skip the permissions/owner/acl stuff for zfs and zpool - should be set correctly on disk from before, and I made sure to use same group and user IDs
reboot - zpool is online with no errors

okay i lied, I use NFS from my media server. Try to follow the instructions in the faq but the paths don't work anymore. Perhaps they moved, or perhaps they're part of binary kit now?

Found the answer in the forums:

"With the move to Illumos /var/svc/manifest/*.xml changed to /lib/svc/manifest/*.xml
Please redo the steps with /lib/svc/manifest"
Repeat the steps in under /lib/ and all the imports succeed, but:


#svcadm enable -r nfs/server
svcadm: svc:/milestone/network depends on svc:/network/physical, which has multiple instances.

nevertheless, despite the warning I am able to connect via NFS

For now, this seems to be complete and successful.