Article for Message ID:   ZFS-8000-ER



ZFS Errata #1

Type

Compatibility

Severity

Moderate

Description

The ZFS pool contains an on-disk format incompatibility.

Automated Response

No automated response will be taken.

Impact

Until the pool is scrubbed using ZoL version 0.6.3 or newer the pool may not be imported by older versions of ZoL or other non-Linux ZFS implementations.

Suggested Action for System Administrator

The pool contains an on-disk format incompatibility. Affected pools must be imported and scrubbed using the current version of ZFS. This will return the pool to a state in which it may be imported by other implementations. This errata only impacts compatibility between ZFS versions, no user data is at risk as result of this erratum. 
# zpool status -x
  pool: test
 state: ONLINE
status: Errata #1 detected.
action: To correct the issue run 'zpool scrub'.
   see: https://zfsonlinux.org/msg/ZFS-8000-ER
  scan: none requested
config:

	NAME            STATE     READ WRITE CKSUM
	test            ONLINE    0    0     0
	  raidz1-0      ONLINE    0    0     0
	    vdev0       ONLINE    0    0     0
	    vdev1       ONLINE    0    0     0
	    vdev2       ONLINE    0    0     0
	    vdev3       ONLINE    0    0     0

errors: No known data errors

# zpool scrub test

# zpool status -x
all pools are healthy




ZFS Errata #2

Type

Compatibility

Severity

Moderate

Description

The ZFS packages were updated while an asynchronous destroy was in progress and the pool contains an on-disk format incompatibility.

Automated Response

No automated response will be taken.

Impact

The pool cannot be imported until the issue is corrected.

Suggested Action for System Administrator

Affected pools must be reverted to the previous ZFS version where they can be correctly imported. Once imported all asynchronous destroy operations must be allowed to complete. The ZFS packages may then be updated and the pool can be imported cleanly by the newer software. 
# zpool import
  pool: test
    id: 1165955789558693437
 state: ONLINE
status: Errata #2 detected.
action: The pool cannot be imported with this version of ZFS due to
	an active asynchronous destroy.  Revert to an earlier version
	and allow the destroy to complete before updating.
   see: https://zfsonlinux.org/msg/ZFS-8000-ER
config:

	test           ONLINE
	  raidz1-0     ONLINE
	    vdev0      ONLINE
	    vdev1      ONLINE
	    vdev2      ONLINE
	    vdev3      ONLINE
Revert to previous ZFS version, import the pool, then wait for the freeing property to drop to zero. This indicates that all outstanding asynchronous destroys have completed. 
# zpool get freeing
NAME  PROPERTY  VALUE    SOURCE
test  freeing   0        default
The ZFS packages may be now be updated and the pool imported. The on-disk format incompatibility can now be corrected online as described in Errata #1.




ZFS Errata #3

Type

Compatibility

Severity

Moderate

Description

An encrypted dataset contains an on-disk format incompatibility.

Automated Response

No automated response will be taken.

Impact

Encrypted datasets created before the ZFS packages were updated cannot be mounted or opened for write. The errata impacts the ability of ZFS to correctly perform raw sends, so this functionality has been disabled for these datasets.

Suggested Action for System Administrator

System administrators with affected pools will need to recreate any encrypted datasets created before the new version of ZFS was used. This can be accomplished by using zfs send and zfs receive. Note, however, that backups can NOT be done with a raw zfs send -w, since this would preserve the on-disk incompatibility. Alternatively, system administrators can use conventional tools to back up data to new encrypted datasets. The new version of ZFS will prevent new data from being written to the impacted datasets, but they can still be mounted read-only. 
# zpool status
  pool: test
    id: 1165955789558693437
 state: ONLINE
status: Errata #3 detected.
action: To correct the issue backup existing encrypted datasets to new
	encrypted datasets and destroy the old ones.
   see: https://zfsonlinux.org/msg/ZFS-8000-ER
config:

	test           ONLINE
	  raidz1-0     ONLINE
	    vdev0      ONLINE
	    vdev1      ONLINE
	    vdev2      ONLINE
	    vdev3      ONLINE
Import the pool and backup any existing encrypted datasets to new datasets. To ensure the new datasets are re-encrypted, be sure to receive them below an encryption root or use zfs receive -o encryption=on, then destroy the source dataset. 
# zfs send test/crypt1@snap1 | zfs receive -o encryption=on -o keyformat=passphrase -o keylocation=file:///path/to/keyfile test/newcrypt1
# zfs send -I test/crypt1@snap1 test/crypt1@snap5 | zfs receive test/newcrypt1
# zfs destroy -R test/crypt1
New datasets can be mounted read-write and used normally. The errata will be cleared upon reimporting the pool and the alert will only be shown again if another dataset is found with the errata. To ensure that all datasets are on the new version reimport the pool, load all keys, mount all encrypted datasets, and check zpool status. 
# zpool export test
# zpool import test
# zfs load-key -a
Enter passphrase for 'test/crypt1':
1 / 1 key(s) successfully loaded
# zfs mount -a
# zpool status -x
all pools are healthy




ZFS Errata #4

Type

Compatibility

Severity

Moderate

Description

An encrypted dataset contains an on-disk format incompatibility.

Automated Response

No automated response will be taken.

Impact

Encrypted datasets created before the ZFS packages were updated cannot be backed up via a raw send to an updated system. These datasets also cannot receive additional snapshots. New encrypted datasets cannot be created until the bookmark_v2 feature has been enabled.

Suggested Action for System Administrator

First, system administrators with affected pools will need to enable the bookmark_v2 feature on their pools. Enabling this feature will prevent this pool from being imported by previous versions of the ZFS software after any new bookmarks are created (including read-only imports). If the pool contains no encrypted datasets, this is the only step required. If there are existing encrypted datasets, administrators will then need to back these datasets up. This can be done in several ways. Non-raw zfs send and zfs receive can be used as per usual, as can traditional backup tools. Raw receives of existing encrypted datasets and raw receives into existing encrypted datasets are currently disabled because ZFS is not able to guarantee that the stream and the existing dataset came from a consistent source. This check can be disabled which will allow ZFS to receive these streams anyway. Note that this can result in datasets with data that cannot be accessed due to authentication errors if raw and non-raw receives are mixed over the course of several incremental backups. To disable this restriction, set the zfs_disable_ivset_guid_check module parameter to 1. Streams received this way (as well as any received before the upgrade) will need to be manually checked by reading the data to ensure they are not corrupted. Note that zpool scrub cannot be used for this purpose because the scrub does not check the cryptographic authentication codes. For more information on this issue, please refer to the zfs man page section on zfs receive which describes the restrictions on raw sends. 
# zpool status
  pool: test
 state: ONLINE
status: Errata #4 detected.
        Existing encrypted datasets contain an on-disk incompatibility
        which needs to be corrected.
action: To correct the issue enable the bookmark_v2 feature and backup
        any existing encrypted datasets to new encrypted datasets and
        destroy the old ones. If this pool does not contain any
        encrypted datasets, simply enable the bookmark_v2 feature.
   see: http://zfsonlinux.org/msg/ZFS-8000-ER
  scan: none requested
config:

        NAME           STATE     READ WRITE CKSUM
        test           ONLINE       0     0     0
          /root/vdev0  ONLINE       0     0     0

errors: No known data errors
Import the pool and enable the bookmark_v2 feature. Then backup any existing encrypted datasets to new datasets. This can be done with traditional tools or via zfs send. Raw sends will require that the zfs_disable_ivset_guid_check is set to 1 on the receive side. Once this is done, the original datasets should be destroyed. 
# zpool set feature@bookmark_v2=enabled test
# echo 1 > /sys/module/zfs/parameters/zfs_disable_ivset_guid_check
# zfs send -Rw test/crypt1@snap1 | zfs receive test/newcrypt1
# zfs send -I test/crypt1@snap1 test/crypt1@snap5 | zfs receive test/newcrypt1
# zfs destroy -R test/crypt1
# echo 0 > /sys/module/zfs/parameters/zfs_disable_ivset_guid_check
The errata will be cleared upon reimporting the pool and the alert will only be shown again if another dataset is found with the errata. To check that all datasets are fixed, perform a zfs list -t all, and check zpool status once it is completed. 
# zpool export test
# zpool import test
# zpool scrub # wait for completion
# zpool status -x
all pools are healthy