This chapter examines problems that, while universal for file systems, have unique solutions when your system is configured with AdvFS.

The chapter includes the following sections:

• Checking Free Space and Disk Usage
• Display Block Allocation Information
• Display Fileset Disk Space Usage
• Display File Domain Disk Space Usage
• Limiting Disk Space Usage
• Fileset Quotas
• User and Group Quotas
• Running into Limits

• Checking Disk Activity
• Defragmenting Files
• Balancing File Distribution
• Striping Files
• Migrating Files
• Changing System Resources
• Adding Volumes
• Removing Volumes
• Exchanging Volumes

• Disk Failure
• Domain Panic

• Restoring the /etc/fdmns Directory
• Reconstructing the /etc/fdmns Directory Using advscan
• Reconstructing the /etc/fdmns Directory Manually
• Recovering from Failure of the Root Domain
• Restoring a Multivolume usr Domain

• Verifying File System Consistency
• Displaying On-Disk Structures
• Moving an AdvFS Disk to an Undamaged Machine

The first step to managing excessive disk space consumption is to request that users delete unnecessary files. There are a number of utilities that look at file usage so that you can monitor storage allocation. You can also limit disk space consumption by imposing quotas on users and groups or on the filesets set up on the system.

Table 11 lists commands that are useful for examining disk space usage:

Table 11 Disk Space Usage Information Commands

Use the du command to display information about block allocation for files in specific directories. By specifying the -a flag, the du command displays the number of blocks in use by individual files. Refer to the du(1) reference page for details on all of the options available for this command.

To display disk space usage information for individual files in a directory, use the following du command format:

du -a directory

Use the df command to display disk space usage for filesets. Included in the display is the size, in blocks, of the file domain. This is the maximum amount of space that a fileset can occupy. If a fileset quota is set, the quota limit is displayed as the size, in blocks, because this is the maximum amount of space that the fileset can use. The display also includes used space which is the amount of space that a fileset consumes.

To display disk space usage information for all AdvFS filesets on the system, use the following df command format:

df -t advfs

The df command has several other display options. Refer to the df(1) reference page for information on all of the available options.

Use the showfdmn command to display the attributes and block usage for each volume in an active file domain. For multivolume domains, the showfdmn command also displays the total volume size, the total number of free blocks, and the total percentage of volume space currently allocated.

To display disk space usage information for individual file domains, use the following showfdmn command format:

showfdmn domain_name

Refer to the showfdmn(8) reference page for information on all of the available options.

To display information about all file domains on a system, run the following commands:

# cd /etc/fdmns
showfdmn *

If your system has been running without any limits on resource usage, you can add quotas to your system to limit the amount of disk space your users can access. AdvFS quotas provide a layer of control beyond that available with UFS. You can limit the number of files or blocks used by a fileset as well as the resources used by individual users and by groups. See Chapter 3: Managing Quotas for complete information.

You can set two types of quotas: hard limits that cannot be exceeded and soft limits that can be exceeded for a period of time called the grace period. You can turn quota enforcement on and off.

Fileset quotas restrain a fileset from grabbing all of the available space in a file domain. Without them, any fileset can use all of the available space in a file domain. Table 12 lists all of the commands that you use to set up and manage fileset quotas.

Table 12 Fileset Quota Commands

User and group quotas limit the amount of space a user or group can allocate for a fileset. Table 13 lists the commands that operate on user and group quotas.

Table 13 User and Group Quotas Commands

If you are working in an editor and realize that the information you need to save will put you over your quota limit, do not abort the editor or write the file because data may be lost. Instead, remove files to make room for the edited file prior to writing it. You can also write the file to another fileset, such as tmp, remove files from the fileset whose quota you exceeded, and then move the file back to that fileset.

AdvFS will impose quota limits in the rare case that a file is less than 8 kilobytes below its quota limit and less than 8 kilobytes are to be added to it. This is because AdvFS is structured to allocate storage in pages of 8 kilobytes each time a file is created or extended. When less storage is needed, the system accesses the frag file for the fileset to obtain smaller pieces of storage (1 through 7 kilobytes). However, quota limits are tested before the decision is made to allocate a fragment. AdvFS assumes that 8 kilobytes will be added, which would put the file over the quota limit.

The performance of a disk depends upon the I/O demands upon it. If your file domain is structured so that heavy access is focused on one volume, it is likely that system performance will degrade. Once you have determined the load balance on your system, there are a number of ways to equalize the activity and increase throughput. See Chapter 5: Performance Tuning.

The first step in determining the cause of poor performance is to examine disk activity. Use the iostat utility to display the number of transfers per second (tps) and kilobytes transferred per second (bps). From this you can determine where I/O bottlenecks are occurring. That is, if one device shows sustained large numbers in a column, this device is being hit more than others. Then you can decide what action might increase throughput: moving files, obtaining faster volumes, striping files, etc.

The following example of the output from the iostat command displays CPU, terminal, and disk statistics for four disks on a system. The example displays five reports at 1-second intervals:

# iostat 1 5
           tty     rz1      rz2      rz3      rz4     cpu
      tin tout bps tps  bps tps  bps tps  bps tps  us ni sy id
        1   52   2   0    1   0   13   1    4   1   8  0  9 83
        1   16   7   1    2   0    5   2    2   0   3  0 10 87
        0    0   0   0    0   0    0   0    0   0   0  0  1 98
        2    2   2   1    0   0   50   6    0   0   9  0  9 82
        1  191   2   1    0   0   47   6    0   0   8  0  9 83

As files grow, contiguous space on disk is not available to accommodate new data, so files become fragmented. File fragmentation can reduce system performance because more I/O is required to read or write a file. Run the defragment utility using the -v and -n flags to display fragmentation statistics. Use the following format:

defragment -v -n domain_name

From the output you can determine how fragmented your file domain is and if it is a possible cause of poor system performance.

You can improve system performance if you distribute files evenly over all your volumes. Files that are distributed unevenly can degrade system performance.

You can use the showfdmn command to display the percent of used space on each volume in a multivolume file domain. Issue the showfdmn command using the following format:

showfdmn domain_name

When the percent of used space is uneven among the volumes, you can use the balance utility to redistribute the files among the volumes. Issue the balance command using the following format:

balance domain_name

When a volume is added to a domain with the addvol command, all the files of the file domain remain on the previously existing volume(s) and the new one is empty. Run the balance utility to even the file distribution.

AdvFS allows you to choose individual files to stripe across multiple volumes. If your system has very large files with heavy I/O requirements, consider striping these files across volumes so that I/O will be directed to more than one disk.

To stripe a file, you first create a new, zero-length (empty) file, issue the stripe command, and then write your file to the striped file. You can choose the number of volumes on which to stripe a file. Issue the stripe command using the following format:

stripe -n volume_count file_name

You can use the migrate utility to move a heavily accessed file or selected pages of a file to another volume in the file domain. You can move the file to a specific volume or you can let the system choose.

To move an entire file to a specific volume, issue the migrate command using the following format:

migrate -s source_volume_index -d destination_volume_index file_name

You can change your file-system size in the following ways:

• Increase the size of a file domain by adding a volume with the addvol utility.
• Shrink a file domain by removing a volume with the rmvol command.
• Exchange volumes by first adding a new one, moving your files to it, and then removing the old.

These operations can take place on line while all filesets remain mounted. The addvol command completes in a few seconds. The rmvol command moves the data off the volume to be removed. The time it takes for the command to complete depends upon the amount of data stored on the volume and the load on the system.

When your file domain runs out of space, you can add a volume to the domain quickly and without interrupting your users. Use the addvol utility to increase the number of volumes within an existing file domain. For optimum performance, each volume you add should consist of the entire disk (typically, partition c). Do not add a volume containing any data you want to keep. When you run the addvol command, existing data on the added disk is destroyed.

If you are adding volumes because you plan to add a large number of files, see Creating a Domain for a Large Number of Files before you add the volumes.

Adding volumes to a file domain does not affect the logical structure of the filesets within a file domain. You can add a volume to an active file domain while its filesets are mounted and in use. Run the balance utility after a new volume has been added to distribute files to it.

When you run the rmvol utility, the system automatically migrates the contents of the old volume to another volume in the domain. The logical structure of the filesets in a file domain is unaffected.

If you remove a volume that contains a stripe segment, the rmvol utility moves the segment to another volume that does not already contain a stripe segment of the same file. If all remaining volumes contain stripe segments, the system requests confirmation before the segment is moved to a volume that already contains a stripe segment of the file.

You can interrupt the rmvol process without damaging your file domain. Files already removed from the volume will remain in their new location. If the volume that has had the files removed does not allow new file allocations after the aborted rmvol operation, use the chvol command with the -A flag to reactivate the volume.

Without taking your system off line, you can replace a smaller volume with a larger one to provide more disk space, and you can exchange a slower device with a faster one to improve throughput. The procedure is as follows:

1. Add the new volume to the file domain using the addvol utility.
2. Remove the old volume with the rmvol utility. The system automatically migrates the contents of the old volume to another volume in the domain. The logical structure of the filesets in a file domain is unaffected.
3. If you want a specific file on the new volume, for example if it is heavily accessed and your new volume is fast, use the migrate utility to move the file to the new volume. You can also run the balance utility to distribute your files when the exchange is complete.

Back up your data regularly and frequently and watch for signs of impending disk failure. Removing files from a problem disk before it fails can prevent a lot of trouble.

There is no particular message that will tell you that your disk is about to fail, but some warning messages may indicate potential problems. Run the uerf utility to print out the hardware-detected events. This report provides information that may help you identify some hardware-related problems.

Hardware problems cannot be repaired by your file system. If you start seeing unexplained errors for a volume, remove that volume from the file domain as soon as possible. If you can read data from your disk, you can remove the volume with the rmvol utility. If you wait and there is a disk failure, your metadata will be inaccessible, and it will be extremely difficult to access your data.

When a log or metadata write error occurs, AdvFS will initiate a domain panic, rather than a system panic, on any non-root file domain. A domain panic prevents further access to the file domain but allows the filesets in the file domain to be unmounted.When a domain panic occurs, a message is displayed in the following format:

AdvFS Domain Panic; Domain name Id domain_Id

For example:

AdvFS Domain Panic; Domain cybase_domain  Id  2dad7c28.0000dfbb

After a domain panic, use the mount command with the -p flag to list all mounted filesets. Then use the umount command to unmount all filesets in the file domain specified in the domain panic message. You can then take the necessary steps to correct the hardware problem.

After you have corrected the hardware problem, run the verify utility (the file domain structure checker) on the file domain before remounting any filesets. Running this utility will show you whether a log or metadata write error left you with any inconsistent files.

You can restore your files with the restore or vrestore command. You must use the command that corresponds to the dump utility that you used, either the dump or vdump command. You cannot mix UFS and AdvFS backup utilities.

AdvFS must have a current /etc/fdmns directory in order to mount filesets. A missing or damaged /etc/fdmns directory prevents access to a file domain, but the data within the file domain remains intact. You can restore the /etc/fdmns directory from backup or you can recreate it.

If you have a current backup copy of the directory, it is preferable to restore the /etc/fdmns directory from backup. Any standard backup facility (vdump, dump, tar, or cpio) can back up the /etc/fdmns directory. To restore the directory, use the recovery procedure that is compatible with your backup process.

You can reconstruct the /etc/fdmns directory manually or with the advscan command. The procedure for reconstructing the /etc/fdmns directory is similar for both single-volume and multivolume file domains.

If you choose to reconstruct the directory manually, you must know the name of each file domain on your system and its associated volumes.

You can use the advscan command to determine which partitions on a disk or Logical Storage Manager (LSM) disk group are part of an AdvFS file domain. Then you can use the command to rebuild all or part of your /etc/fdmns domain.

The advscan command can:

• List partitions in the order they are found on disk.
• Scan all disks found in any /etc/fdmns domain.
• Recreate missing domain directories. The domain name is created from the device name.
• Fix the domain count and links if you specify a domain.
• Operate on LSM disk groups.

For each domain there are three numbers that must match for the AdvFS file system to operate properly:

• The number of physical partitions found by the advscan command that have the same domain ID
• The domain volume count (the number stored in the domain attributes table that specifies how many partitions the domain has)
• The number of /etc/fdmns links to the partitions, because each partition must be represented by a link

Inconsistencies can occur in these numbers in a number of ways and for a number of reasons. In general, the advscan command treats the domain volume count as more reliable than the number of partitions or /etc/fdmns links. Table 14 lists anomalies, possible causes, and suggested corrections. In the table, a letter N represents the value that is expected to be consistent for the number of partitions, domain volume count, and number of links.

Table 14 Fileset Anomalies and Corrections

In the following example there are no missing file domains. The advscan command scans devices rz0 and rz5 for AdvFS partitions and finds nothing amiss. There are two partitions found (rz0c and rz5c), the domain volume count reports two, and there are two links entered in the /etc/fdmns directory.

# advscan rz0 rz5

Scanning disks  rz0 rz5
Found domains:

usr_domain

                Domain Id       2e09be37.0002eb40

                Created         Thu Jun 23 09:54:15 1994

                Domain volumes          2

                /etc/fdmns links        2

                Actual partitions found:

                                        rz0c

                                        rz5c

In the following example, directories that define the file domains that include rz6 were removed from the /etc/fdmns directory. This means that the number of /etc/fdmns links, the number of partitions, and the domain volume counts are no longer equal.

The advscan command scans device rz6 and recreates the missing file domains as follows:

1. A partition is found containing an AdvFS file domain. The domain volume count reports one, but there is no file-domain directory in the /etc/fdmns directory that contains this partition.
2. Another partition is found containing a different AdvFS file domain. The file domain volume count is also one. There is no file-domain directory that contains this partition.
3. No other AdvFS partitions are found. The domain volume counts and the number of partitions found match for the two discovered domains.
4. The advscan command creates directories for the two file domains in the /etc/fdmns directory.
5. The advscan command creates symbolic links for the devices in the /etc/fdmns file-domain directories.

The command and output are as follows:

# advscan -r rz6

Scanning disks  rz6

Found domains:

*unknown*

                Domain Id       2f2421ba.0008c1c0

                Created         Mon Jan 23 13:38:02 1995



                Domain volumes          1

                /etc/fdmns links        0



                Actual partitions found:

                                        rz6a*

*unknown*

                Domain Id       2f535f8c.000b6860

                Created         Tue Feb 28 09:38:20 1995



               Domain volumes          1

                /etc/fdmns links       0



                Actual partitions found:

                                        rz6b*



Creating /etc/fdmns/domain_rz6a/

        linking rz6a



Creating /etc/fdmns/domain_rz6b/

        linking rz6b

If you accidentally lose all or part of your /etc/fdmns directory, and you know which file domains and links are missing, you can reconstruct it manually.

The following example reconstructs the /etc/fdmns directory and two file domains where the names of the file domains are known. Each contains a single volume (or special device). Note that the order of creating the links in these examples does not matter. The file domains are:

domain1 on /dev/rz1c
domain2 on /dev/rz2c

To reconstruct the two single-volume file domains, enter:

# mkdir /etc/fdmns

# mkdir /etc/fdmns/domain1

# cd /etc/fdmns/domain1

# ln -s /dev/rz1c

# mkdir /etc/fdmns/domain2

# cd /etc/fdmns/domain2

# ln -s /dev/rz2c

The following example reconstructs one multivolume file domain. The domain1 file domain contains the following three volumes:

/dev/rz1c
/dev/rz2c
/dev/rz3c

To reconstruct the multivolume file domain, issue the following:

# mkdir /etc/fdmns

# mkdir /etc/fdmns/domain1

# cd /etc/fdmns/domain1

# ln -s /dev/rz1c

# ln -s /dev/rz2c 

# ln -s /dev/rz3c

A catastrophic failure of the disk containing your AdvFS root file domain requires that you recreate your root file domain and then restore the root file domain contents from your backup media.

The following example assumes that you are booting from the CD-ROM device DKA400, which is the installation Stand Alone System (SAS). The tape drive is tz5. Typing the show device command from the boot prompt shows A/5/0 for device TLZ06. The root is being restored to device rz1, which is an RZ25 disk:

1. Boot your system as stand-alone:

   b DKA400

2. Pick option:

   3) UNIX Shell

   You will now be at the # prompt in single-user mode.

3. Make the device special files for the tape and the disk:

   # MAKEDEV rz1
   
   # MAKEDEV tz5
   

4. Make the disk label:

   #  disklabel -rw -t advfs rrz1a rz25

5. Create the root file domain and fileset. Note that if you have changed the root file domain name or fileset name, use the new names:

   # mkfdmn -r /dev/rz1a root_domain
   
   # mkfset root_domain root

6. Mount the newly created root domain and restore from tape using a restore utility compatible with your dump utility:

   # mount root_domain#root /mnt
   
   # cd /mnt
   
   # vrestore -x -D .

You can now boot your restored root domain.

To restore a multivolume /usr file system, the usr_domain file domain must first be reconstructed with all of its volumes before you restore the files. However, creating a multivolume file domain requires the addvol utility, and the addvol command will not run unless the License Management Facility (LMF) database, which resides in the /usr/sbin directory, is available. See the lmf(8) reference page for information.

On some systems the /var directory, where the LMF database resides, and the /usr directory are both located in the usr fileset. So the directory containing the license database must be recovered from usr fileset before the addvol command can be accessed. On some systems the /var directory is in a separate fileset. If this is the case, the addvol command can be recovered first and then can be used to add the volumes.

The following example restores a multivolume file domain where the /var directory and the /usr directory are both in the usr fileset in the usr_domain file domain consisting of the rz1g, rz2c, and rz3c volumes. The procedure assumes that the root file system has already been restored.

1. Mount the root fileset as read/write:

   # mount -u /

2. Create usr_domain using the initial volume:

   # rm -rf /etc/fdmns/usr_domain
   
   # mkfdmn /dev/rz1g usr_domain
   

3. Create and mount the /usr and /var filesets:

   # mkfset usr_domain usr
   
   # mount -t advfs usr_domain#usr /usr

4. Create a soft link in /usr because that is where the lmf command looks for its database:

   # ln -s /var /usr/var

5. Insert the /usr backup tape:

   # cd /usr
   
   # vrestore -vi
   
   (/)  add sbin/addvol 
   
   (/)  add sbin/lmf
   
   (/)  add var/adm/lmf
   
   (/)  extract
   
   (/)  quit

6. Reset the license database:

   # /usr/sbin/lmf reset

7. Add the extra volumes to usr_domain:

   # /usr/sbin/addvol /dev/rz2c usr_domain
   
   # /usr/sbin/addvol /dev/rz3c usr_domain

8. Do a full restore of the /usr backup:

   # cd /usr
   
   # vrestore -xv

The following example restores a multivolume file domain where the /usr and /var directories are in separate filesets in the same multivolume domain, usr_domain, consisting of rz1g, rz2c, and rz3c. This means that you must mount both the /var and the /usr backup tapes. The procedure assumes that the root file system has already been restored.

1. Mount the root fileset as read/write:

   # mount -u /

2. Create usr_domain using the initial volume:

   # rm -rf /etc/fdmns/usr_domain
   
   # mkfdmn /dev/rz1g usr_domain

3. Create and mount the /usr and /var filesets:

   
   # mkfset usr_domain usr
   
   # mkfset usr_domain var
   
   # mount -t advfs usr_domain#usr  /usr
   
   # mount -t advfs usr_domain#var  /var

4. Insert the /var backup tape and restore from it:

   # cd /var
   
   # vrestore -vi
   
   (/) add adm/lmf
   
   (/) extract
   
   (/) quit

5. Insert the /usr backup tape:

   # cd /usr
   
   # vrestore -vi
   
   (/) add sbin/addvol
   
   (/) add sbin/lmf
   
   (/) extract
   
   (/) quit

6. Reset the license database:

   # /usr/sbin/lmf reset

7. Add the extra volumes to usr_domain:

   # /usr/sbin/addvol /dev/rz2c usr_domain
   
   # /usr/sbin/addvol /dev/rz3c usr_domain

8. Do a full restore of /usr backup:

   # cd /usr
   
   # vrestore -xv

9. Insert the /var backup tape and do a full restore of /var backup:

   # cd /var
   
   # vrestore -xv

As each domain is mounted after a crash, it automatically runs recovery code that checks through the transaction log to ensure that any file-system operations that were occurring when the system crashed are either completed or backed out. This ensures that AdvFS metadata is in a consistent state after a crash.

If you want to be sure that the metadata is consistent, you can run the verify command verify to verify the file-system structure. This utility checks on-disk structures such as the bitfile metadata table (BMT), the storage bitmaps, the tag directory, and the frag file for each fileset. It verifies that the directory structure is correct and that all directory entries reference a valid file and that all files have a directory entry.

Note The verify command replaces the msfsck command of earlier releases.

If the verify command is unable to mount a fileset due to the failure of a file domain, as a last resort run the command with the -F flag. This will cause the fileset to be mounted using the -d option of the mount command, which mounts the fileset without running recovery on the file domain. This will cause your file domain to be inconsistent because the file structure will not have been checked and made consistent.

The following example verifies the domainx file domain, which contains the filesets setx and sety:

#  verify domainx

+++Domain verification+++

Domain Id 2f03b70a.000f1db0

Checking disks ...

Checking storage allocated on disk /dev/rz10g

Checking storage allocated on disk /dev/rz10a

Checking mcell list ...

Checking mcell position field ...

Checking tag directories ...

+++ Fileset verification +++

+++ Fileset setx +++

Checking frag file headers ...

Checking frag file type lists ...

Scanning directories and files ...

     1100

Scanning tags ...

     1100

Searching for lost files ...

     1100

+++ Fileset sety +++

Checking frag file headers ...

Checking frag file type lists ...

Scanning directories and files ...

     5100
Scanning tags ...

     5100
Searching for lost files ...

     5100

Table 15 lists the on-disk structure dumping utilities that enable you to examine a file domain with suspected metadata corruption. The commands display raw data from the disk in a number of formats.

Table 15 On-Disk Structure Dumping Utilities

If a machine has failed, it is possible to move disks containing AdvFS file domains to another computer running AdvFS. As explained in this section, you connect the disk(s) to the new machine and modify the /etc/fdmns directory so the new system will recognize the transferred volume(s).

Caution Do not use either the addvol command or the mkfdmn command to add the volumes to the new machine. Doing so will delete all data on your disk.

If you do not know what partitions your domains were on, you can add the disks on the new machine and run the advscan command, which may be able to recreate this information. You can also look at the disk label on the disk to see which partitions in the past have been made into AdvFS partitions. This will not tell you which partitions belong to which file domain.

For example, assume a system has a file domain, testing_domain, on two disks, rz3 and rz4. This domain contains two filesets: sample1_fset and sample2_fset. These filesets are mounted on /data/sample1 and /data/sample2. If the motherboard of the machine fails, you need to move the disks to another system. Assume you also want disks to use different SCSI ID numbers, for example: rz6 and rz8, because rz3 and rz4 are already in use on the second computer.

Assume you already know that the file domain that you are moving had partitions rz3c, rz4a, rz4b, and rz4g. You would then take the following steps:

1. Shut down the working machine to which you are moving the disks.
2. Connect the disks from the bad machine to the good one. Configure the disk that was rz3 on the old machine as rz6 on the new one. Configure the old rz4 as rz8.
3. Reboot. You do not need to reboot to SAS; multiuser mode works because you can complete the following steps while the system is running.
4. You may have to make special device nodes for the two new disks, rz6 and rz8. For example, if /dev/rz6c doesn't exist, then as root user do the following:

   # cd /dev
   
   # /dev/MAKEDEV rz6

   If necessary, do the same for rz8.

5. Modify your /etc/fdmns directory to include the information from the transferred domains:

   # mkdir -p /etc/fdmns/testing_domain
   
   # cd /etc/fdmns/testing_domain
   
   # ln -s /dev/rz6c rz6c
   
   # ln -s /dev/rz8a rz8a
   
   # ln -s /dev/rz8b rz8b
   
   # ln -s /dev/rz8g rz8g
   
   # mkdir /data/sample1
   
   # mkdir /data/sample2

6. Edit the /etc/fstab file and add the fileset mount-point information:

   testing_domain#sample1_fset  /data/sample1
   
   testing_domain#sample2_fset  /data/sample2
   

7. Mount the volumes:

   # mount /data/sample1
   
   # mount /data/sample2
   

Note that if you run the mkfdmn command or the addvol command on partition rz6c, rz8a, rz8b, or rz8g, or an overlapping partition, you will destroy the data on the disk.