Getting started

Unpacking and compiling

Untar the distribution package. A subdirectory like pnfs3.1.<n> will be created.

cd pnfs3.1.<n>
make clean

The 'make clean' command dermines the OS type and version and creates a soft link of ./version to the corresponding ./version.<os-type> file. Now you may customize the ./version file to your needs. It mainly needs information about the compiler, the lex and where to find the gdbm library. If there is no gdbm on our host installed yet, ./gdb/gdbm-1.7.3 contains the gdbm source distribution which has to be compiled prior to running make to create the pnfs part. To create the binaries run :

make
make install

The exeutables will end up in tools/<os-type>.

Creating the pnfs file system

Step by step :

You need to setup the /usr/etc/pnfsSetup file. (see below).

add the pnfs tools to your path. Something like :

    . /usr/etc/pnfsSetup
    PATH=$PATH:$pnfs/tools

Try to find some space on the local disk to hold the first database. Each file or directory entry will occupy about a KBytes.
Create the first ( admin ) database. Each database has a name, an ID and the actual gdbm database file. The name of the first database has to be admin.
```
    mdb create admin /needSpace/admin
    
```
All subsequent databases may have any name ( except some pnfs keywords ) and can be created in different directories.
e.g. :
```
    mdb create user1 /moreSpace/user-db-1
    
```
I would recommend not to store user data on the admin database. So at least one additional database is required.
At DESY, we are currently running 38 databases of up to 150MB size without observing any performance penalties. Each pnfsd can access each database without interfering with other communications. So, using multiple database will share the load among the db processes.
The process of creating a link from the admin database to a different one is described in The Basics.

Start the pnfs processes.

    root@watphrakeo:/home/patrick > pnfs.server start
    Shmcom : Installed 8 Clients and 8 Servers
    Starting database server for admin (/uss/db/info.x/admin) ... O.K.
    Starting database server for user1 (/uss/db/info.x/user-db-1) ... O.K.
     Waiting for dbservers to register ... Ready
    Starting Mountd : pmountd
    Starting nfsd : pnfsd

Check if the processes has been created.
```
    root@watphrakeo:/home/patrick > mdb show
      ID   Name         Type    Status       Path
    ----------------------------------------------
      0    admin         r     enabled (r)   /uss/db/info.x/admin
      1    user1         r     enabled (r)   /uss/db/info.x/user-db-1
    
```
The (r) states that the corresponding database server is running. To create new databases use the 'mdb create ...' command and simply run mdb update to start the corresponding database server. There is no need to restart the pnfs processes for creating additional databases.
Mount the pnfs filesystem.
```
    mkdir -p /pnfs/fs
    mount -o intr,hard,rw localhost:/fs   /pnfs/fs
    
```
Make sure you are using the 'localhost:/fs' mountpoint and not the '<hostName>:/fs' mountpoint.
This entry into pnfs allows you to configure the export tables a.s.o. It can't be used as the frontend to the HSM because it allows I/O to the filesystem which you certainly don't want. See No Io for further information about I/O and no I/O.
From here on the system is configured and can be used following the description in export. But for convenience there is a tool to modify the export table : pmount This tool assumes that the 'localhost:/fs' is mounted on '/pnfs/fs'. If you need to change this path you have to modify the exports=/pnfs.... line with in pmount as well.
Prepare the regular mountpoints. (Example only)
```
    cd /pnfs/fs/admin/etc/exports
    mkdir mountpoints
    cd mountpoints
    cat >Generic
    /admin   /0/root/fs/admin   0   nooptions
    /usr     /0/root/fs/usr    30   nooptions
    ^D   
    
```
/admin and /usr can now be exported. /usr is a HSM aware mountpoint. No I/O is allowed. Any number of mountpoints may be added here. Note that the regular clients must only mount filesystems exported with the 30 option( NOIO).

Declare a host to be a regular pnfs client. EXPORT( HSM usage only, I/O is not allowed )

    root@watphrakeo:/pnfs/fs/admin/etc/exports > pmount add host /usr
     Info : Host knossos not yet in export table
     Info : Adding knossos (131.169.87.223) to export table

From now on knossos may mount <pnfsHostName>:/usr to whatever it likes.

REMARK NFS-2 isn't something which can be called a secure protocol. So, to reduce the possible misuse pnfs disallows remote 'root' access by default. This behaviour can be changed on host basis creating a file within the already existing /pnfs/fs/admin/etc/export/trusted directory. The filename must be the decimal dotted representation of the ip number of the host you want to grant root access. The file must contain a 15. e.g. :
```
    cd /pnfs/fs/admin/etc/exports/trusted
    cat 15 >131.169.1.221
    
```
131.169.1.221 can now become root on the pnfs filesystem. Usually there should be no need to do this.

The /usr/etc/pnfsSetup File

When starting pnfs, the pnfsd and the pmountd assume to find the configuration information in /usr/etc/pnfsSetup. A sample setupfile can be found in the root directory of the distribution. The file contains one key value pair per line, separated by an equals sign. The meaning of most of the keys is discussed below. Keys which are not mentioned in this list can be taken from the example file and need not to be modified.

Key meaning

shmkey The key of the shared master shared memory area. You may take any, as long as no other application wants to make use of the same key. (The default is set to '1122' ).
shmclients The maximum number of clients which needs access to the shared memory area. The rule of thumb 'number of pnfsd processes' + 1 ( for the pmountd ) + 'number of admin tools running simultaniously'. So if pnfscopies=4, shmclients=8 should be fine.
shmservers Must be larger then the largest database ID. If only the pnfs admin tools are used to add databases, this is essentially the number of databases. To avoid unnesseccary restarts of the pnfs system, the number should be sufficiently large to cover the near futur plans concerning database groth.
pnfs Is the full path of the pnfs root directory. $pnfs/tools needs to contain the pnfs tools and the links to the binaries. This path usually looks like /x...x/pnfs3.1.<n>
database Is the full path of an existing directory where pnfs stores information about its databases. It doesn't need much space and only contains ascii files which point to the actual gdbm database.
trash Is the full path of a directory where pnfs provides information about files which have been deleted from pnfs. The connected HSMs need those information to remove the files from the HSM database ( or from tape). Pnfs assumes to find subdirectories within $trash where it can copy the different file levels into. The size of the directory highly depends on the number of remove operations and the time interval this directory is scanned to synchronize the HSM. ( see remove )
pnfscopies Number of copies of the pnfsd deamon to be started. shmclients must exceed this number by at least 'two'.
hardlinks Can be yes or no depending on wether pnfs should support hardlinks or not. The disadvantage of hardlinks is that there is no defined path back to the root. So, disabling hardlinks allows you to get the filepath from an object only knowing its pnfsid(=inode).
netmask The ip netmask of you site. ( class B = 16 , class C = 8 ).
levelmask A sequence of 8 0 or -1 separated by ':'. The first number has to be a 0. Each number corresponds to a pnfs file level and determines if the specified level should belong to the user ( uid, gid .. ) or should belong to 'root'. The way it is configured highly depends on the HSM system you intend to use.
pmountdLog
dbserverLog
pnfsdLog Points to the full pathname of the corresponding log files. dbserverLog should point to /dev/null in an production environment because it still produces large amounts of debug output which can't be switched off. pmountdLog contains the mounts and dismounts. pnfsdLog should be configured with pnfsdLevel=5 to report the modifications only. See logformat for more information.
pnfsdLevel
pmountdLevel
dbserverLevel These numbers determine the output level for the three deamons. Currently only pnfsdLevel is recognized and should be set to 5 to report the modifications only. The smaller the number the more info you will get.

Key	meaning
shmkey	The key of the shared master shared memory area. You may take any, as long as no other application wants to make use of the same key. (The default is set to '1122' ).
shmclients	The maximum number of clients which needs access to the shared memory area. The rule of thumb 'number of pnfsd processes' + 1 ( for the pmountd ) + 'number of admin tools running simultaniously'. So if pnfscopies=4, shmclients=8 should be fine.
shmservers	Must be larger then the largest database ID. If only the pnfs admin tools are used to add databases, this is essentially the number of databases. To avoid unnesseccary restarts of the pnfs system, the number should be sufficiently large to cover the near futur plans concerning database groth.
pnfs	Is the full path of the pnfs root directory. $pnfs/tools needs to contain the pnfs tools and the links to the binaries. This path usually looks like /x...x/pnfs3.1.<n>
database	Is the full path of an existing directory where pnfs stores information about its databases. It doesn't need much space and only contains ascii files which point to the actual gdbm database.
trash	Is the full path of a directory where pnfs provides information about files which have been deleted from pnfs. The connected HSMs need those information to remove the files from the HSM database ( or from tape). Pnfs assumes to find subdirectories within $trash where it can copy the different file levels into. The size of the directory highly depends on the number of remove operations and the time interval this directory is scanned to synchronize the HSM. ( see remove )
pnfscopies	Number of copies of the pnfsd deamon to be started. shmclients must exceed this number by at least 'two'.
hardlinks	Can be yes or no depending on wether pnfs should support hardlinks or not. The disadvantage of hardlinks is that there is no defined path back to the root. So, disabling hardlinks allows you to get the filepath from an object only knowing its pnfsid(=inode).
netmask	The ip netmask of you site. ( class B = 16 , class C = 8 ).
levelmask	A sequence of 8 0 or -1 separated by ':'. The first number has to be a 0. Each number corresponds to a pnfs file level and determines if the specified level should belong to the user ( uid, gid .. ) or should belong to 'root'. The way it is configured highly depends on the HSM system you intend to use.
pmountdLog dbserverLog pnfsdLog	Points to the full pathname of the corresponding log files. dbserverLog should point to /dev/null in an production environment because it still produces large amounts of debug output which can't be switched off. pmountdLog contains the mounts and dismounts. pnfsdLog should be configured with pnfsdLevel=5 to report the modifications only. See logformat for more information.
pnfsdLevel pmountdLevel dbserverLevel	These numbers determine the output level for the three deamons. Currently only pnfsdLevel is recognized and should be set to 5 to report the modifications only. The smaller the number the more info you will get.