Most recent update:
$Date$
$Author$

Admin Tools for Auto-Destination Mapping


Contents:

  1. Executive Summary
  2. Map Maintenance: Overview
  3. Utilities for Maintaining the Autodestination Map
  4. Utilities for Auditing the Autodestination Destination Directories
  5. Database Entities involved in Autodest Mapping
  6. Update history

Note to developers/maintainers: The documentation for the individual sam_admin commands is also maintained as a part of the individual autodest commands. Any changes made in this document should also be made there as well (and vise-versa). Sorry for the redundancy but haven't found a way around it yet.


I. Executive Summary:

This document describes the detailed use of the autodestination mapping tools, including the most useful maintenance commands:
   $ samadmin autodest get   mapfile --mapfile=myMap.py  # obtain the master
   $ samadmin autodest synch mapfile --mapfile=myMap.py  # synch reality with specified mapfile
   $ samadmin autodest load  mapfile --mapfile=myMap.py  [--verify=all]
#
#  OBSOLETE and very very slow:
#  $ samadmin autodest verify mapfile --mapfile=myMap.py   # verify a mapfile
#  $ samadmin autodest synch  mapfile --mapfile=myMap.py   # bring reality into synch
#  $ samadmin autodest load   mapfile --mapfile=myMap.py   # synch AND upload map
#


II. Map Maintenance: Overview

A set of utilities has been provided for maintaining the Autodestination Map, creating and verifying the /pnfs paths and their attributes, as well as checking the sam locations in the database.

Before using any of the following commands, you should:

	$ setup [-q dev|int|prd] sam

The autodest maintenance commands all use the following syntax:

	$ samadmin autodest [subcommand] [--options] [-flags]
and all adhere to the following conventions/rules:
  1. -d will provide additional debugging information
  2. -v will provide a more verbose output report (when applicable)

The following procedure should generally be followed when updating or maintaining the map:

  1. Obtain a copy of the Master Map from the database: $ samadmin autodest get master mapfile --mapfile=myMapFile.py
  2. Edit the mapfile $ xemacs myMapFile.py        # or your personal favorite editor
  3. Verify the mapfile: $ samadmin autodest verify mapfile --mapfile=myMapFile.py [--verify=all]
  4. Synchronize reality with your new mapfile: $ samadmin autodest synch mapfile --mapfile=myMapFile.py [--synch=all]
  5. Load the new mapfile: $ samadmin autodest load mapfile --mapfile=myMapFile.py [--verify=all]

Other commands that may be useful to know are:

  6. Verify the Master Map: $ samadmin autodest verify master mapfile [options]
  7. Renumber mapfile: $ samadmin autodest renumber mapfile --mapfile=myMapFile.py [options]
  8. Create a new pnfs location: $ samadmin autodest create location --path=/pnfs/new/path [options]
  9. Modify an existing pnfs location: $ samadmin autodest modify location --path=/pnfs/path [options]
10. Verify a location: $ samadmin autodest verify location --path=/pnfs/path [options]
11. Test an autodest string: $ samadmin autodest teststring --path=stringToTest [options]


III. Utilities for Maintaining the Autodestination Map

  1. Obtain a copy of the Master Map from the database
  2. $ samadmin autodest get master mapfile --mapfile=<myMapFile.py>
    
    The database contains the master of the autodestination mappings. Whenever you wish to add new entries, you should first get the master map. Make your modifications to this file.

    Items to Note:

    1. You may run this command from the sam account, but it isn't necessary.

  3. Edit the mapfile

  4. The mapfile will look similar to:
    destList = [
    {	# map entry number 0:
    	'pathPattern' : '(/pnfs/sam/mammoth/copy1/monte_carlo/)([^/]+)(/mcpc_in2p3/)([^/]+)(/)([^/]+)',
        'destinationPath' : '/pnfs/sam/mammoth/mcc99_2/in2p3',
    	    'library' : 'sammam',
    	'file_family' : 'in2p3',
    'file_family_wrapper' : 'cpio_odc',
          'storage_group' : 'D0',
      'file_family_width' : '1',
    	'permissions' : 'rwxrwxr-x',
    },
    {	# map entry number 1:
    	'pathPattern' : '(/pnfs/sam/mammoth/copy2/monte_carlo/)([^/]+)(/mcpc_in2p3/)([^/]+)(/)([^/]+)',
        'destinationPath' : '/pnfs/sam/mammoth/dev/mcc99_2/in2p3',
    	    'library' : 'sammam',
    	'file_family' : 'in2p3_dev',
    'file_family_wrapper' : 'cpio_odc',
          'storage_group' : 'D0',
      'file_family_width' : '1',
    	'permissions' : 'rwxrwxr-x',
    },
    ]
    
    Items to Note::
    1. The order of map entries is important; this is how the database will determine priority in matching patterns.
    2. The "map entry number" comments are there to guide you, but they are only comments. If you add/remove entries and/or change the order of entries, the comments may very well be wrong. Use the renumber command to assist you.
    3. This is a python module. Punctuation, dictionary keywords, etc., all matter.

  5. Verify the mapfile
  6. $ samadmin autodest verify mapfile --mapfile=<myMapFile.py> [--verify=all]
    
    After you have made your changes to the mapfile, you should verify it. This will compare the entries in your mapfile to "reality" and report on any differences. By default, only NEW entries will be verified; entries which already exist in the master map will not be re-verified.

    Note, to see the "real" values of pnfs tags, you can

     $ setup encp -q d0en    # or appropriate qualifier
     $ cd /pnfs/sam/path/of/interest
     $ enstore pnfs --tags
     

    A typical report would look like:

    Verifying map entry number 0 (path = /pnfs/sam/mammoth/mcc99_2/in2p3):
    SUMMARY: Invalid pathPattern in map entry; Attribute mismatch
     pathPattern: INV. EXPR.      mapfile = (/pnfs/sam/mammoth/copy1/monte_carlo/)[^/]+)(/mcpc_in2p3/)([^/]+)(/)([^/]+) 
    			      actual  = <N/A>
         library: MISMATCH        mapfile = samnull              actual  = sammam              
     file_family: NEW TAG VALUE   mapfile = in2p3_def            actual  = in2p3               
     permissions: MISMATCH        mapfile = rwxrwx---            actual  = rwxrwxr-x           
    
    Verifying map entry number 1 (path = /pnfs/sam/mammoth/dev/mcc99_2/in2p3_dev):
    SUMMARY: Path does not exist; Path is not a valid SAM location; Attribute mismatch
    destinationPath: NO SUCH PATH    mapfile = /pnfs/sam/mammoth/dev/mcc99_2/in2p3_dev 
    			      actual  = DOES NOT EXIST      
         library:                 mapfile = samnull              actual  = <unknown>           
     file_family:                 mapfile = in2p3_dev            actual  = <unknown>           
    file_family_wrapper: NEW TAG VALUE   mapfile = cpio_odx             actual  = <unknown>           
    storage_group:                 mapfile = D0                   actual  = <unknown>           
    file_family_width:                 mapfile = 1                    actual  = <unknown>           
     permissions:                 mapfile = rwxr-xr-x            actual  = <unknown>           
    
    Verifying map entry number 2 (path = /pnfs/sam/mammoth/dev/mcc99_2/in2p3):
    SUMMARY: Attribute mismatch
         library: MISMATCH        mapfile = samnull              actual  = sammam           
    
    Verifying map entry number 3 (path = /pnfs/sam/NULL/test/mcc99):
    SUMMARY: Verified.
    
    Verifying map entry number 4 (path = d0mino.fnal.gov:/sam/cache22/bogus/boo):
    SUMMARY: Path is not a valid SAM location
     pathPattern: Ok.             mapfile = (/pnfs/lauri)(.*)    actual  = <n/a>
    destinationPath: NOT IN SAM      mapfile = d0mino.fnal.gov:/sam/cache22/bogus/boo 
    			      actual  = <not a /pnfs path>
    

    How to Read the Verification Report
    The SUMMARY line contains a summary of problems found with each map entry. If there are inconsistencies, the summary may contain one or more of the following:

    Summary statement Interpretation
    Path does not exist The specified destinationPath does not exist (or cannot be stat'ed on the current machine). If destinationPath is in /pnfs space (i.e., begins with "/pnfs", it will be created if you synchronize or load the mapfile; or you may create the directory manually with the create location command. (The destinationPath directory will not be created for you if it is not in /pnfs space).
    Path is not a valid SAM location The specified destinationPath is not a recognized location in the SAM database. If destinationPath is in /pnfs space, then it will be entered into the SAM database as a valid tape location if you synchronize or load the mapfile; or you may create the tape location manually with the create location command. (The destinationPath will not be registered in the SAM database for you if it is not in /pnfs space).
    Missing required element(s) in map entry The map entry is missing one or more required elements. This must be corrected before you can successfully synchronize or load the mapfile. (Not applicable when destinationPath is not in /pnfs space).
    Invalid pathPattern in map entry The pathPattern is not a valid regular expression; most likely, it is missing some required punctuation, etc. This must be corrected before you can successfully synchronize or load the mapfile
    Attribute mismatch The attribute specified in the mapfile does not match the actual value. Reality will be altered to match the mapfile if you synchronize or load the map. (Not applicable when destinationPath is not in /pnfs space).

    Following the SUMMARY are the details of any inconsistencies found, in the format:

      entityName: ERROR FLAG      mapfile = mapFile-value        actual  = real-value
    
    where:

    Detail statement Interpretation
    entityName pathPattern, destinationPath, file_family, etc.
    ERROR FLAG Any of the following:
    NO SUCH PATH destinationPath does not exist in /pnfs space
    INV. EXPR. pathPattern is an invalid regular expression
    NOT IN MAPFILE this element is missing from the mapfile entry
    MISMATCH the value specified in the mapfile is inconsistent with reality
    NO TAG FOUND The directory exists, but the enstore pnfs --tags command could not determine the value of this tag. (This should never occur).
    NOT IN SAM destinationPath is not in /pnfs space, and the location is not a valid SAM location.
    mapFile-value Value specified in the mapfile
    real-value Value found by actual examination of the destinationPath in /pnfs space

    Items to Note::

    1. You should run this command from the sam account.
    2. You must run this command from a node which has mounted /pnfs space.
    3. The verification pass will check the following items:
      • pathPattern is a legal regular expression
      • destinationPath exists in the sam database (i.e., is a valid sam location)
      • destinationPath exists in pnfs space
      • enstore pnfs --tags: library, file_family, file_family_wrapper, storage_group, file_family_width for destinationPath match the mapfile entry
      • permissions for destinationPath match the mapfile entry

  7. Synchronize reality with your new mapfile
  8. $ samadmin autodest synch mapfile --mapfile=<myMapFile.py> [] [--synch=all]
    

    You have made changes to the mapfile. You have verified the new mapfile, and are satisfied that any differences between the mapfile and "reality" should be corrected, using the mapfile as the "master" information source. You are now ready to alter pnfs space (and, if necessary, the sam data_storage_locations) to match the new mapfile.

    The synch mapfile command will read through the mapfile and create any new pnfs directories that are listed but do not yet exist; modify pnfs directory tags and permissions so that they match what is in the map file; ensure that all destinationPath entries are valid locations in the SAM database. By default, only new entries will be synchronized; entries which already exist in the master map will not be re-synched.

    Items to Note::

    1. You should run this command from the sam account.
    2. You must run this command from a node which has mounted /pnfs space.
    3. This command will fail if it is not able to perform any of the required steps.
    4. If the mapfile contains encp tag values which are not known to Enstore, you must use the option on the command line. (See http://d0ensrv2.fnal.gov/enstore/tape_inventory/VOLUMES_DEFINED" for the source of "known Enstore tags").

  9. Load the new mapfile
  10. $ samadmin autodest load mapfile --mapfile=<myMapFile.py> [] [--nocommit] [--verify=all]
    

    When you load the new mapfile, any new entries will be verified and synched before the records are uploaded. If, for some reason, the autodest script cannot make reality match the new mapfile, it will not be uploaded to the database.

    Items to Note::

    1. You should run this command from the sam account.
    2. You must run this command from a node which has mounted /pnfs space.
    3. If the mapfile contains encp tag values which are not known to Enstore, you must use the option on the command line. (See http://d0ensrv2.fnal.gov/enstore/tape_inventory/VOLUMES_DEFINED" for the source of "known Enstore tags").
    4. The --nocommit option may be used for testing; this will turn off the "commit" flag when you upload to the database. In essence, you will have performed a synch, and made sure that the information could be uploaded, without actually doing the database upload.

  11. Verify the Master Map
  12. $ samadmin autodest verify master map
    

    This command will compare the database map to reality and report on any inconsistencies.

  13. Renumber mapfile
  14. $ samadmin autodest renumber mapfile --mapfile=<myMapFile.py>
    

    This command will read a file and rewrite it, changing only the comments indicating map entry number. It is provided to make it easier to determine which entries have problems when running samadmin autodest verify mapfile.

  15. Create a new location
  16. $ samadmin autodest create location --path=/pnfs/new/path \
    	[--permissions=<perm>] [--library=<lib>] \
    	[--file_family=<ffam>] [--file_family_wrapper=<ffw>] \
    	[--storage_group=<sg>] [--file_family_width=<ffwid>] \
    	[]
    

    This command will allow you to manually create a new pnfs directory (and enter it into the sam database if necessary) with the specified attributes.

    Items to Note::

    1. You should run this command from the sam account.
    2. You must run this command from a node which has mounted /pnfs space.
    3. Permission string must be 9 characters long, as in rwxr-x--r.
    4. If any of the encp tag values specified are not known to Enstore, you must use the option on the command line. (See http://d0ensrv2.fnal.gov/enstore/tape_inventory/VOLUMES_DEFINED" for the source of "known Enstore tags").

  17. Modify an existing location
  18. $ samadmin autodest modify location --path=/pnfs/new/path \
    	[--permissions=<perm>] [--library=<lib>] \
    	[--file_family=<ffam>] [--file_family_wrapper=<ffw>] \
    	[--storage_group=<sg>] [--file_family_width=<ffwid>] \
    	[]
    

    This command will modify the attributes of an existing pnfs directory.

    Items to Note::

    1. You should run this command from the sam account.
    2. You must run this command from a node which has mounted /pnfs space.
    3. Permission string must be 9 characters long, as in rwxr-x--r.
    4. If any of the encp tag values specified are not known to Enstore, you must use the option on the command line. (See http://d0ensrv2.fnal.gov/enstore/tape_inventory/VOLUMES_DEFINED" for the source of "known Enstore tags").

  19. Verify a location
  20. $ samadmin autodest verify location --path=/pnfs/path
    

    This command will verify that the specified location is a valid SAM location, and that the directory exists. It will print out the current attributes. A typical report would look like:

    $ samadmin autodest verify --path=/pnfs/sam/NULL
    /pnfs/sam/NULL is a valid SAM location.
    Location /pnfs/sam/NULL exists with properties:
    file_family = jon
    file_family_width = 1
    file_family_wrapper = cpio_odc
    library = samnull
    permissions = rwxrwxr-x
    storage_group = D0
    
    Items to Note:
    1. You should run this command from the sam account.
    2. You must run this command from a node which has mounted /pnfs space.

  21. Test an autodest string
  22. $ samadmin autodest teststring --path=/path/to/test \
      [--pattern=<testPattern> | --mapfile=<myMapFile.py>]
    

    This command will compare a given path and determine if it matches the pattern provided, or any of the patterns in the specified mapfile. If neither a pattern nor a mapfile is provided on the command line, the path will be compared to the master mapfile.

    A sample command looks like the following:

    $ samadmin autodest teststring \
    	--path=/pnfs/sam/mammoth/copy1/monte_carlo/field1/mcpc_uta/field2/field3   
    Testpath /pnfs/sam/mammoth/copy1/monte_carlo/field1/mcpc_uta/field2/field3
    	matches map entry number 4: (/pnfs/sam/mammoth/copy1/monte_carlo/)([^/]+)(/mcpc_uta/)([^/]+)(/)([^/]+)
    	would be mapped to path /pnfs/sam/mammoth/mcc99_2/uta
    Testpath /pnfs/sam/mammoth/copy1/monte_carlo/field1/mcpc_uta/field2/field3
    	matches map entry number 22: /.*/
    	would be mapped to path /pnfs/sam/NULL
    


IV. Utilities for Auditing the Auto-Destination PNFS Locations

There are 2 types of auditing (monitoring) being performed on the pnfs locations (destination paths):
  1. Maximum number of files in a specific directory ( command usage)

  2. Enstore datebase size

    There is no automated tool at this time. Information is available on the web at http://d0ensrv2.fnal.gov/enstore/pnfsRate.html

    The relevant info is:

    609141 -rw------- 2 root root 628187948 Mar 12 10:30 db2
    40 -rw------- 2 root root 39196 Mar 11 17:33 db3
    40 -rw------- 2 root root 39256 Mar 11 17:33 db4

    Where in the example above the db2 database shows 600+Mb used.

    The individual files can grow up to 2 GB. We should switch to a new database when the file systems gets between 1.2 - 1.7 Gb in size.


V. Database Entities involved in Autodest Mapping

In order to keep track of all pnfs locations and their various attributes, a database table (auto_destinations) is maintained with following columns:
DB Entity Description
PNFS_PATTERN a regular expression for string matching
PATH_PRIORITY a number representing the order of patterns to be matched (first match 'wins'); this number is generated automatically when the database table is loaded
AUTO_DEST_FAMILY encp file_family for this pnfs directory
LOCATION_ID an index into the DATA_STORAGE_LOCATIONS table (e.g., a mapping to the file name)
LIBRARY encp library for this pnfs directory
FILE_FAMILY_WRAPPER encp file_family_wrapper for this pnfs directory
STORAGE_GROUP encp storage_group for this pnfs directory
FILE_FAMILY_WIDTH encp file_family_width for this pnfs directory
PERMISSIONS the unix file permissions for this pnfs directory

=============================================================================
Project : SAM
Package : sam_admin_pyapi
$Id$

This work is part of a development project, called SAM, which consists of a
number of coordinated packages each named sam_xxxx .

Notice of authorship, copyright status, and terms and conditions, should
the software eventually become available for use outside Fermilab, can be
found in the README and LICENCE files in the top level directory
of the main sam package.
==============================================================================


Update history:
Date Author Description
2004/03/24 llc cleanup, brushup, for sam_admin_pyapi re-implementation.
2003/06/06 weigand Added 'audit directories' command.
Added 'Contents' section
Added 'Utilities for Auditing the Auto-Destination PNFS Locations' section