Suppose you have a Solaris system already set up with a ZFS root volume and you wish to clone it to another system by replicating the disks. This used to be easy with UFS volumes as you could simply use ufsdump piped to ufsrestore onto a new target disk, install boot blocks then move the new disk over to the target system and boot as normal. With ZFS there are a few extra hurdles incurred by the meta-data but it also saves us from problems like mixing up which disk was the primary mirror and which was the shadow copy. Since we can only replicate from a snapshot, we guarantee that all datasets in the pool contain a consistent set of data from a particular point in time.
Steps to clone a system with ZFS root volumes
Source System
Verify status before starting
# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0
errors: No known data errors
# zpool get bootfs rpool
NAME PROPERTY VALUE SOURCE
rpool bootfs rpool/ROOT/5.10 local
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 4.55G 62.4G 95K /rpool
rpool/ROOT 1.54G 62.4G 18K legacy
rpool/ROOT/5.10 1.54G 62.4G 1.12G /
rpool/ROOT/5.10/var 434M 7.58G 434M /var
rpool/dump 1.00G 62.4G 1.00G -
rpool/home 31.5K 8.00G 31.5K /home
rpool/srv 9.60M 62.4G 9.60M /srv
rpool/swap 2G 64.4G 12.2M -
Create recursive snapshot
# zfs snapshot -r rpool@20091116
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 4.56G 62.4G 95K /rpool
rpool@20091116 0 - 95K -
rpool/ROOT 1.54G 62.4G 18K legacy
rpool/ROOT@20091116 0 - 18K -
rpool/ROOT/5.10 1.54G 62.4G 1.12G /
rpool/ROOT/5.10@20091116 133K - 1.12G -
rpool/ROOT/5.10/var 434M 7.58G 434M /var
rpool/ROOT/5.10/var@20091116 0 - 434M -
rpool/dump 1.00G 62.4G 1.00G -
rpool/dump@20091116 0 - 1.00G -
rpool/home 31.5K 8.00G 31.5K /home
rpool/home@20091116 0 - 31.5K -
rpool/srv 9.60M 62.4G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 2.01G 64.4G 12.2M -
rpool/swap@20091116 0 - 12.2M -
Insert and label new disks
# format </dev/null
Searching for disks...done
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@0,0
1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@1,0
Specify disk (enter its number):
# devfsadm -c disk
# format
Searching for disks...done
c0t2d0: configured with capacity of 68.35GB
c0t3d0: configured with capacity of 68.35GB
AVAILABLE DISK SELECTIONS:
0. c0t0d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@0,0
1. c0t1d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@1,0
2. c0t2d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@2,0
3. c0t3d0 <SUN72G cyl 14087 alt 2 hd 24 sec 424>
/pci@1c,600000/scsi@2/sd@3,0
Specify disk (enter its number): 2
selecting c0t2d0
[disk formatted]
Disk not labeled. Label it now? y
FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> disk 3
selecting c0t3d0
[disk formatted]
Disk not labeled. Label it now? y
format> ^D
Create partition tables
# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t2d0s2
fmthard: New volume table of contents now in place.
# prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t3d0s2
fmthard: New volume table of contents now in place.
Create new zpool
# zpool create newpool mirror c0t2d0s0 c0t3d0s0
# zpool status
pool: newpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
newpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t2d0s0 ONLINE 0 0 0
c0t3d0s0 ONLINE 0 0 0
errors: No known data errors
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0
errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
newpool 106K 66.9G 18K /newpool
rpool 4.56G 62.4G 95K /rpool
rpool@20091116 0 - 95K -
rpool/ROOT 1.54G 62.4G 18K legacy
rpool/ROOT@20091116 0 - 18K -
rpool/ROOT/5.10 1.54G 62.4G 1.12G /
rpool/ROOT/5.10@20091116 244K - 1.12G -
rpool/ROOT/5.10/var 434M 7.58G 434M /var
rpool/ROOT/5.10/var@20091116 620K - 434M -
rpool/dump 1.00G 62.4G 1.00G -
rpool/dump@20091116 0 - 1.00G -
rpool/home 31.5K 8.00G 31.5K /home
rpool/home@20091116 0 - 31.5K -
rpool/srv 9.60M 62.4G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 2.01G 64.4G 12.2M -
rpool/swap@20091116 0 - 12.2M -
Replicate pool’s dataset
The base dataset contains the /boot/ directory which includes the boot_archive and kernel for booting.
This can also be done over the network using a transport such as ssh or rsh but you must have a target system already running.
# zfs send rpool@20091116 | zfs receive -F newpool
receiving full stream of rpool@20091116 into newpool@20091116
cannot mount '/rpool': directory is not empty
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
newpool 194K 66.9G 95K /rpool
newpool@20091116 0 - 95K -
rpool 4.56G 62.4G 95K /rpool
rpool@20091116 0 - 95K -
rpool/ROOT 1.54G 62.4G 18K legacy
rpool/ROOT@20091116 0 - 18K -
rpool/ROOT/5.10 1.54G 62.4G 1.12G /
rpool/ROOT/5.10@20091116 244K - 1.12G -
rpool/ROOT/5.10/var 434M 7.58G 434M /var
rpool/ROOT/5.10/var@20091116 620K - 434M -
rpool/dump 1.00G 62.4G 1.00G -
rpool/dump@20091116 0 - 1.00G -
rpool/home 31.5K 8.00G 31.5K /home
rpool/home@20091116 0 - 31.5K -
rpool/srv 9.60M 62.4G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 2.01G 64.4G 12.2M -
rpool/swap@20091116 0 - 12.2M -
Replicate remaining datasets
# for dataset in ROOT ROOT/5.10 ROOT/5.10/var dump home srv swap; do
> zfs send rpool/$dataset@20091116 | zfs receive newpool/$dataset
> done
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
newpool 2.14G 64.8G 95K /rpool
newpool@20091116 0 - 95K -
newpool/ROOT 1.12G 64.8G 19K /rpool/ROOT
newpool/ROOT@20091116 16K - 18K -
newpool/ROOT/5.10 1.12G 64.8G 1.12G /rpool/ROOT/5.10
newpool/ROOT/5.10@20091116 0 - 1.12G -
newpool/ROOT/5.10/var 434M 64.4G 434M /var
newpool/ROOT/5.10/var@20091116 0 - 434M -
newpool/dump 1.00G 64.8G 1.00G -
newpool/dump@20091116 0 - 1.00G -
newpool/home 31.5K 64.8G 31.5K /rpool/home
newpool/home@20091116 0 - 31.5K -
newpool/srv 9.60M 64.8G 9.60M /rpool/srv
newpool/srv@20091116 0 - 9.60M -
newpool/swap 12.2M 64.8G 12.2M -
newpool/swap@20091116 0 - 12.2M -
rpool 4.56G 62.4G 97K /rpool
rpool@20091116 18K - 95K -
rpool/ROOT 1.54G 62.4G 18K legacy
rpool/ROOT@20091116 0 - 18K -
rpool/ROOT/5.10 1.54G 62.4G 1.12G /
rpool/ROOT/5.10@20091116 246K - 1.12G -
rpool/ROOT/5.10/var 434M 7.58G 434M /var
rpool/ROOT/5.10/var@20091116 620K - 434M -
rpool/dump 1.00G 62.4G 1.00G -
rpool/dump@20091116 0 - 1.00G -
rpool/home 31.5K 8.00G 31.5K /home
rpool/home@20091116 0 - 31.5K -
rpool/srv 9.60M 62.4G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 2.01G 64.4G 12.2M -
rpool/swap@20091116 0 - 12.2M -
Set mountpoints
# zfs set mountpoint=legacy newpool/ROOT
# zfs set mountpoint=/ newpool/ROOT/5.10
cannot mount '/': directory is not empty
property may be set but unable to remount filesystem
# zfs set mountpoint=/home newpool/home
cannot mount '/home': directory is not empty
property may be set but unable to remount filesystem
# zfs set mountpoint=/srv newpool/srv
cannot mount '/srv': directory is not empty
property may be set but unable to remount filesystem
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
newpool 2.14G 64.8G 95K /rpool
newpool@20091116 0 - 95K -
newpool/ROOT 1.12G 64.8G 19K legacy
newpool/ROOT@20091116 16K - 18K -
newpool/ROOT/5.10 1.12G 64.8G 1.12G /
newpool/ROOT/5.10@20091116 0 - 1.12G -
newpool/ROOT/5.10/var 434M 64.4G 434M /var
newpool/ROOT/5.10/var@20091116 0 - 434M -
newpool/dump 1.00G 64.8G 1.00G -
newpool/dump@20091116 0 - 1.00G -
newpool/home 31.5K 64.8G 31.5K /home
newpool/home@20091116 0 - 31.5K -
newpool/srv 9.60M 64.8G 9.60M /srv
newpool/srv@20091116 0 - 9.60M -
newpool/swap 12.2M 64.8G 12.2M -
newpool/swap@20091116 0 - 12.2M -
rpool 4.56G 62.4G 96K /rpool
rpool@20091116 20K - 95K -
rpool/ROOT 1.54G 62.4G 18K legacy
rpool/ROOT@20091116 0 - 18K -
rpool/ROOT/5.10 1.54G 62.4G 1.12G /
rpool/ROOT/5.10@20091116 246K - 1.12G -
rpool/ROOT/5.10/var 434M 7.58G 434M /var
rpool/ROOT/5.10/var@20091116 659K - 434M -
rpool/dump 1.00G 62.4G 1.00G -
rpool/dump@20091116 0 - 1.00G -
rpool/home 31.5K 8.00G 31.5K /home
rpool/home@20091116 0 - 31.5K -
rpool/srv 9.60M 62.4G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 2.01G 64.4G 12.2M -
rpool/swap@20091116 0 - 12.2M -
Set bootfs property
Failure to complete this step will produce a ‘no pool_props’ error message when the target system boots.
# zpool get bootfs rpool
NAME PROPERTY VALUE SOURCE
rpool bootfs rpool/ROOT/5.10 local
# zpool get bootfs newpool
NAME PROPERTY VALUE SOURCE
newpool bootfs - default
# zpool set bootfs=newpool/ROOT/5.10 newpool
# zpool get bootfs newpool
NAME PROPERTY VALUE SOURCE
newpool bootfs newpool/ROOT/5.10 local
Export the new pool
# zpool export newpool
# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0
errors: No known data errors
Install boot blocks to both disks
# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t2d0s0
# installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c0t3d0s0
#
On x86 hardware we use grub as a boot loader so the command is different:
# installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t2d0s0
stage1 written to partition 0 sector 0 (abs 4096)
stage2 written to partition 0, 272 sectors starting at 50 (abs 4146)
Target System
Transfer disks and boot failsafe
Make sure you are on the system console (ALOM) of the target system.
SC Alert: DISK @ HDD0 has been inserted.
SC Alert: DISK @ HDD1 has been inserted.
{0} ok boot -F failsafe -Z newpool/ROOT/5.10
SC Alert: Host System has Reset
Probing system devices
Probing memory
Probing I/O buses
rsc not found.
rsc not found.
Probing system devices
Probing memory
Probing I/O buses
Rebooting with command: boot -F failsafe -Z newpool/ROOT/5.10
Boot device: /pci@1c,600000/scsi@2/disk@0,0:a File and args: -F failsafe -Z newpool/ROOT/5.10
krtld: Ignoring invalid kernel option -Z.
krtld: Unused kernel arguments: `newpool/ROOT/5.10'.
SunOS Release 5.10 Version Generic_139555-08 64-bit
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hardware watchdog enabled
Configuring devices.
Searching for installed OS instances...
ROOT/5.10 was found on newpool.
Do you wish to have it mounted read-write on /a? [y,n,?] n
Starting shell.
#
On x86 hardware, you should be presented with the grub splash screen where you can select to boot in failsafe mode.
Import zpool as rpool
This step is necessary otherwise the system will complain that the zpool was last mounted on a different system and refuse to import it. Renaming is not strictly necessary but keeps consistency among various systems.
# zpool import -f newpool rpool
cannot mount '/': directory is not empty
cannot mount '/home': failed to create mountpoint
cannot mount '/rpool': failed to create mountpoint
cannot mount '/srv': failed to create mountpoint
cannot mount '/var': directory is not empty
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.57G 64.4G 95K /rpool
rpool@20091116 0 - 95K -
rpool/ROOT 1.54G 64.4G 19K legacy
rpool/ROOT@20091116 16K - 18K -
rpool/ROOT/5.10 1.54G 64.4G 1.12G /
rpool/ROOT/5.10@20091116 3.98M - 1.12G -
rpool/ROOT/5.10/var 434M 64.4G 434M /var
rpool/ROOT/5.10/var@20091116 0 - 434M -
rpool/dump 1.00G 64.4G 1.00G -
rpool/dump@20091116 16K - 1.00G -
rpool/home 31.5K 64.4G 31.5K /home
rpool/home@20091116 0 - 31.5K -
rpool/srv 9.60M 64.4G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 12.2M 64.4G 12.2M -
rpool/swap@20091116 0 - 12.2M -
# zpool get bootfs rpool
NAME PROPERTY VALUE SOURCE
rpool bootfs rpool/ROOT/5.10 local
Export the zpool
Now that we have updated the zpool’s meta-data we can export it again and boot it without the system complaining.
# zpool export rpool
# init 0
syncing file systems... done
Program terminated
{0} ok
Boot to single user mode
If you see messages like this when booting, you have probably failed to replicate the /var dataset. Return the disks to the source system, import the zpool and try zfs send rpool/ROOT/5.10/var again.
ERROR: svc:/system/filesystem/minimal:default failed to mount /var/run (see 'svcs -x' for details)
Nov 16 17:52:47 svc.startd[7]: svc:/system/filesystem/minimal:default: Method "/lib/svc/method/fs-minimal" failed with exit status 95.
Nov 16 17:52:47 svc.startd[7]: system/filesystem/minimal:default failed fatally: transitioned to maintenance (see 'svcs -xv' for details)
Requesting System Maintenance Mode
(See /lib/svc/share/README for more information.)
Console login service(s) cannot run
Root password for system maintenance (control-d to bypass):
This is what you should see:
{0} ok boot -s
SC Alert: Host System has Reset
Probing system devices
Probing memory
Probing I/O buses
rsc not found.
rsc not found.
Probing system devices
Probing memory
Probing I/O buses
Rebooting with command: boot -s
Boot device: /pci@1c,600000/scsi@2/disk@0,0:a File and args: -s
SunOS Release 5.10 Version Generic_141414-01 64-bit
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hardware watchdog enabled
Booting to milestone "milestone/single-user:default".
Hostname: host01
WARNING: bge0 has duplicate address 192.168.105.104 (in use by 00:03:ba:bf:72:c1); disabled
Requesting System Maintenance Mode
SINGLE USER MODE
Root password for system maintenance (control-d to bypass):
single-user privilege assigned to /dev/console.
Entering System Maintenance Mode
Nov 17 12:03:38 su: 'su root' succeeded for root on /dev/console
Sun Microsystems Inc. SunOS 5.10 Generic January 2005
host01:/#
On x86 hardware, you may need to edit the kernel line in grub’s menu to boot to single user mode.
- Press 'e' to edit the menu item
- Press down arrow then 'e' again to edit the kernel line.
- Append '-s' to the end of the line
- Press Enter to accept the line, then 'b' to boot it.
Unconfigure the system
host01:/# sys-unconfig
WARNING
This program will unconfigure your system. It will cause it
to revert to a "blank" system - it will not have a name or know
about other systems or networks.
This program will also halt the system.
Do you want to continue (y/n) ? y
stopping NetWorker daemons:
nsr_shutdown -q
svc.startd: The system is coming down. Please wait.
svc.startd: 22 system services are now being stopped.
svc.startd: The system is down.
syncing file systems... done
Program terminated
{0} ok
Reconfigure the system
{0} ok boot
SC Alert: Host System has Reset
Probing system devices
Probing memory
Probing I/O buses
rsc not found.
rsc not found.
Probing system devices
Probing memory
Probing I/O buses
Rebooting with command: boot
Boot device: bootdisk File and args:
SunOS Release 5.10 Version Generic_141414-01 64-bit
Copyright 1983-2009 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.
Hardware watchdog enabled
Hostname: unknown
Configuring devices.
Reading ZFS config: done.
Mounting ZFS filesystems: (6/6)
Select a Language
0. English
1. es
2. fr
Please make a choice (0 - 2), or press h or ? for help: 0
Note: If you connected to the ALOM remotely, you should choose vt100 when prompted for your terminal type.
At the end of the reconfiguration, the system will reboot automatically. It may be necessary to add host entries back into /etc/hosts after the system has booted.
Cleanup
Once you are satisfied the system is running properly, destroy the source system’s snapshot which is now on the target system. This can also be done on the source system when you are finished replicating disks.
# zpool status
pool: rpool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
mirror ONLINE 0 0 0
c0t0d0s0 ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0
errors: No known data errors
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.67G 64.3G 95K /rpool
rpool@20091116 0 - 95K -
rpool/ROOT 1.65G 64.3G 19K legacy
rpool/ROOT@20091116 16K - 18K -
rpool/ROOT/5.10 1.65G 64.3G 1.12G /
rpool/ROOT/5.10@20091116 93.3M - 1.12G -
rpool/ROOT/5.10/var 446M 64.3G 445M /var
rpool/ROOT/5.10/var@20091116 1.32M - 434M -
rpool/dump 1.00G 64.3G 1.00G -
rpool/dump@20091116 16K - 1.00G -
rpool/home 52K 64.3G 31.5K /home
rpool/home@20091116 20.5K - 31.5K -
rpool/srv 9.60M 64.3G 9.60M /srv
rpool/srv@20091116 0 - 9.60M -
rpool/swap 12.2M 64.3G 16K -
rpool/swap@20091116 12.2M - 12.2M -
# zfs destroy -r rpool@20091116
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
rpool 2.57G 64.4G 95K /rpool
rpool/ROOT 1.56G 64.4G 19K legacy
rpool/ROOT/5.10 1.56G 64.4G 1.12G /
rpool/ROOT/5.10/var 445M 64.4G 445M /var
rpool/dump 1.00G 64.4G 1.00G -
rpool/home 31.5K 64.4G 31.5K /home
rpool/srv 9.60M 64.4G 9.60M /srv
rpool/swap 16K 64.4G 16K -