File Hopping on arrival is a term, denoting the possibility of initiating a pool to pool transfer as the result of a file successfully arriving on a pool from some external client. Files restored from HSM or arriving on a pool as the result of a pool to pool transfer will not yet be forwarded.
Forwarding of incoming files can be enabled by setting the pool.destination.replicate property in the /etc/dcache/dcache.conf file or per pool in the layout file. It can be set to on, PoolManager or HoppingManager, where on does the same as PoolManager.
The pool is requested to send a replicateFile message to either the PoolManager or to the HoppingManager, if available. The different approaches are briefly described below and in more detail in the subsequent sections.
The
replicateFilemessage is sent to thePoolManager. This happens for all files arriving at that pool from outside (no restore or p2p). No intermediateHoppingManageris needed. The restrictions areAll files are replicated. No pre-selection, e.g. on the storage class can be done.
The mode of the replicated file is determined by the destination pool and cannot be overwritten. See the section called “File mode of replicated files”
The
replicateFilemessage is sent to theHoppingManager. TheHoppingManagercan be configured to replicate certain storage classes only and to set the mode of the replicated file according to rules. The file mode of the source file cannot be modified.
The mode of a replicated file can either be determined by
settings in the destination pool or by the HoppingManager. It
can be cached or precious.
If the
PoolManageris used for replication, the mode of the replicated file is determined by the destination pool. The default setting iscached.If a
HoppingManageris used for file replication, the mode of the replicated file is determined by theHoppingManagerrule responsible for this particular replication. If the destination mode is set tokeepin the rule, the mode of the destination pool determines the final mode of the replicated file.
To enable replication on arrival by the PoolManager set the
property pool.destination.replicate to
PoolManager for the particular pool
[<exampleDomain>] [<exampleDomain>/pool] ... pool.destination.replicate=PoolManager
or for several pools in the /etc/dcache/dcache.conf file.
... pool.destination.replicate=PoolManager
File hopping configuration instructs a pool to send a
replicateFile request to the
PoolManager as the result of a file arriving on that pool
from some external client. All arriving files will be treated
the same. The PoolManager will process this transfer
request by trying to find a matching link (Please find
detailed information at Chapter 7, The poolmanager Service.
It is possible to configure the PoolManager such that files
are replicated from this pool to a special set of destination
pools.
Example:
Assume that we want to have all files, arriving on pool
ocean to be immediately replicated to a
subset of read pools. This subset of pools is described by the
poolgroup ocean-copies. No other pool is
member of the poolgroup ocean-copies.
Other than that, files arriving at the pool
mountain should be replicated to all read
pools from which farm nodes on the 131.169.10.0/24 subnet are
allowed to read.
The layout file defining the pools ocean
and mountain should read like this:
[exampleDomain]
[exampleDomain/pool]
name=ocean
path=/path/to/pool-ocean
pool.wait-for-files=${path}/data
pool.destination.replicate=PoolManager
name=mountain
path=/path/to/pool-mountain
pool.wait-for-files=${path}/data
pool.destination.replicate=PoolManager
In the layout file it is defined that all files arriving on
the pools ocean or
mountain should be replicated
immediately. The following
PoolManager.conf file contains
instructions for the PoolManager how to replicate these
files. Files arriving at the ocean pool
will be replicated to the ocean-copies
subset of the read pools and files arriving at the pool
mountain will be replicated to all read
pools from which farm nodes on the 131.169.10.0/24 subnet are
allowed to read.
# # define the units # psu create unit -protocol */* psu create unit -net 0.0.0.0/0.0.0.0 psu create unit -net 131.169.10.0/255.255.255.0 # create the faked net unit psu create unit -net 192.1.1.1/255.255.255.255 psu create unit -store *@* psu create unit -store ocean:raw@osm # # # define unit groups # psu create ugroup any-protocol psu create ugroup any-store psu create ugroup ocean-copy-store psu create ugroup farm-network psu create ugroup ocean-copy-network # psu addto ugroup any-protocol */* psu addto ugroup any-store *@* psu addto ugroup ocean-copy-store ocean:raw@osm psu addto ugroup farm-network 131.169.10.0/255.255.255.0 psu addto ugroup ocean-copy-network 192.1.1.1/255.255.255.255 psu addto ugroup allnet-cond 0.0.0.0/0.0.0.0 psu addto ugroup allnet-cond 131.169.10.0/255.255.255.0 psu addto ugroup allnet-cond 192.1.1.1/255.255.255.255 # # # define the write-pools # psu create pool ocean psu create pool mountain # # # define the write-pools poolgroup # psu create pgroup write-pools psu addto pgroup write-pools ocean psu addto pgroup write-pools mountain # # # define the write-pools-link, add write pools and set transfer preferences # psu create link write-pools-link any-store any-protocol allnet-cond psu addto link write-pools-link write-pools psu set link farm-read-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 # # # define the read-pools # psu create pool read-pool-1 psu create pool read-pool-2 psu create pool read-pool-3 psu create pool read-pool-4 # # # define the farm-read-pools poolgroup and add pool members # psu create pgroup farm-read-pools psu addto pgroup farm-read-pools read-pool-1 psu addto pgroup farm-read-pools read-pool-2 psu addto pgroup farm-read-pools read-pool-3 psu addto pgroup farm-read-pools read-pool-4 # # # define the ocean-copy-pools poolgroup and add a pool # psu create pgroup ocean-copy-pools psu addto pgroup ocean-copy-pools read-pool-1 # # # define the farm-read-link, add farm-read-pools and set transfer preferences # psu create link farm-read-link any-store any-protocol farm-network psu addto link farm-read-link farm-read-pools psu set link farm-read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 # # # define the ocean-copy-link, add ocean-copy-pools and set transfer preferences # psu create link ocean-copy-link ocean-copy-store any-protocol ocean-copy-network psu addto link ocean-copy-link ocean-copy-pools psu set link ocean-copy-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 # #
While 131.169.10.1
is a legal IP address e.g. of one of your farm nodes, the
192.1.1.1 IP
address must not exist anywhere at your site.
With the HoppingManager you have several configuration options
for file hopping on arrival, e.g.:
-
With the
HoppingManageryou can define a rule such that only the files with a specific storage class should be replicated. - You can specify the protocol the replicated files can be accessed with.
- You can specify from which ip-adresses it should be possible to access the files.
Add the hoppingmanager service to a domain in your layout
file and restart the domain.
[<DomainName>] [<DomainName>/hoppingmanager]
Initially no rules are configured for the
HoppingManager. You may add rules by either edit the file
/var/lib/dcache/config/HoppingManager.conf
and restart the hoppingmanager service, or use the admin
interface and save the modifications by the
save command into the
HoppingManager.conf
To enable replication on arrival by the HoppingManager set
the property pool.destination.replicate to
HoppingManager for the particular pool
[<exampleDomain>] [<exampleDomain>/pool] ... pool.destination.replicate=HoppingManager
or for several pools in the /etc/dcache/dcache.conf file.
... pool.destination.replicate=HoppingManager
The
HoppingManageressentially receivesreplicateFilemessages from pools, configured to support file hopping, and either discards or modifies and forwards them to thePoolManager, depending on rules described below.The
HoppingManagerdecides on the action to perform, based on a set of configurable rules. Each rule has a name. Rules are checked in alphabetic order concerning their names.A rule it triggered if the storage class matches the storage class pattern assigned to that rule. If a rule is triggered, it is processed and no further rule checking is performed. If no rule is found for this request the file is not replicated.
If for whatever reason, a file cannot be replicated, NO RETRY is being performed.
Processing a triggered rule can be :
The message is discarded. No replication is done for this particular storage class.
The rule modifies the
replicateFilemessage, before it is forwarded to thePoolManager.An ip-number of a farm-node of the farm that should be allowed to read the file can be added to the
replicateFilemessage.The mode of the replicated file can be specified. This can either be
precious,cachedorkeep.keepmeans that the pool mode of the source pool determines the replicated file mode.The requested protocol can be specified.
define hop OPTIONS <name> <pattern> precious|cached|keep
OPTIONS
-destination=<cellDestination> # default : PoolManager
-overwrite
-continue
-source=write|restore|* # !!!! for experts only StorageInfoOptions
-host=<destinationHostIp>
-protType=dCap|ftp...
-protMinor=<minorProtocolVersion>
-protMajor=<majorProtocolVersion> - name
- This is the name of the hopping rule. Rules are checked in alphabetic order concerning their names.
- pattern
patternis a storage class pattern. If the incoming storage class matches this pattern, this rule is processed.- precious|cached|keep
precious|cached|keepdetermines the mode of the replicated file. Withkeepthe mode of the file will be determined by the mode of the destination pool.- -destination
This defines which
cellto use for the pool to pool transfer. By default this is thePoolManagerand this should not be changed.- -overwrite
In case, a rule with the same name already exists, it is overwritten.
- -continue
If a rule has been triggered and the corresponding action has been performed, no other rules are checked. If the
continueoption is specified, rule checking continues. This is for debugging purposes only.- -source
-sourcedefines the event on the pool which has triggered the hopping. Possible values arerestoreandwrite.restoremeans that the rule should be triggered if the file was restored from a tape andwritemeans that it should be triggered if the file was written by a client.- -host
Choose the id of a node of the farm of worker-nodes that should be allowed to access the file. Configure the
poolmanagerrespectively.- -protType, -protMajor, -protMinor
Specify the protocol which should be used to access the replicated files.
In order to instruct a particular pool to send a
replicateFile message to the
hoppingmanager service, you need to add the line
pool.destination.replicate=HoppingManager to the
layout file.
Example:
[exampleDomain]
[exampleDomain/pool]
name=write-pool
path=/path/to/write-pool-exp-a
pool.wait-for-files=${path}/data
pool.destination.replicate=HoppingManager
...
Assume that all files of experiment-a will be written to an
expensive write pool and subsequently flushed to tape. Now
some of these files need to be accessed without delay. The
files that need fast acceess possibility will be given the
storage class exp-a:need-fast-access@osm.
In this example we will configure the file hopping such that a
user who wants to access a file that has the above storage
info with the NFSv4.1 protocol will be able to do so.
Define a rule for hopping in the
/var/lib/dcache/config/HoppingManager.conf
file.
define hop nfs-hop exp-a:need-fast-access@osm cached -protType=nfs -protMajor=4 -protMinor=1
This assumes that the storage class of the file is
exp-a:nfs@osm. The mode of the file, which
was precious on the write pool will have to
be changed to cached on the read pool.
The corresponding
/var/lib/dcache/config/poolmanager.conf
file could read like this:
# # define the units # psu create unit -protocol */* psu create unit -net 0.0.0.0/0.0.0.0 psu create unit -store exp-a:need-fast-access@osm # # # define unit groups # psu create ugroup any-protocol psu create ugroup exp-a-copy-store psu create ugroup allnet-cond # psu addto ugroup any-protocol */* psu addto ugroup exp-a-copy-store exp-a:need-fast-access@osm psu addto ugroup allnet-cond 0.0.0.0/0.0.0.0 # # # define the write-pool # psu create pool write-pool # # # define the read-pool # psu create pool read-pool # # # define the exp-a-read-pools poolgroup and add a pool # psu create pgroup exp-a-read-pools psu addto pgroup exp-a-read-pools read-pool # # # define the exp-a-write-pools poolgroup and add a pool # psu create pgroup exp-a-write-pools psu addto pgroup exp-a-write-pools write-pool # # # define the exp-a-read-link, add exp-a-read-pools and set transfer preferences # psu create link exp-a-read-link exp-a-copy-store any-protocol allnet-cond psu addto link exp-a-read-link exp-a-read-pools psu set link exp-a-read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 # # # define the exp-a-write-link, add exp-a-write-pools and set transfer preferences # psu create link exp-a-write-link exp-a-copy-store any-protocol allnet-cond psu addto link exp-a-write-link exp-a-write-pools psu set link exp-a-write-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 # # #