File Hopping on arrival is a term, denoting the possibility of initiating a pool to pool transfer as the result of a file successfully arriving on a pool from some external client. Files restored from HSM or arriving on a pool as the result of a pool to pool transfer will not yet be forwarded.
Forwarding of incoming files can be enabled by setting the pool.destination.replicate property in the /opt/d-cache/etc/dcache.conf file or per pool in the layout file. It can be set to on, PoolManager or HoppingManager, where on does the same as PoolManager.
The pool is requested to send a replicateFile message to either the PoolManager or to the HoppingManager, if available. The different approaches are briefly described below and in more detail in the subsequent sections.
The replicateFile message is sent to the
PoolManager. This happens for all files arriving at that
pool from outside (no restore or p2p). No intermediate
HoppingManager is needed. The restrictions are
All files are replicated. No pre-selection, e.g. on the storage class can be done.
The mode of the replicated file is determined by the destination pool and cannot be overwritten. See the section called “File mode of replicated files”
The replicateFile message is sent to the
HoppingManager. The HoppingManager can be configured to
replicate certain storage classes only and to set the mode
of the replicated file according to rules. The file mode of
the source file cannot be modified.
The mode of a replicated file can either be determined by
settings in the destination pool or by the HoppingManager. It
can be cached or precious.
If the PoolManager is used for replication, the mode
of the replicated file is determined by the destination
pool. The default setting is cached.
If a HoppingManager is used for file replication, the mode
of the replicated file is determined by the HoppingManager
rule responsible for this particular replication. If the
destination mode is set to keep in the
rule, the mode of the destination pool determines the
final mode of the replicated file.
To enable replication on arrival by the PoolManager set the
property pool.destination.replicate to
PoolManager for the particular pool
[<exampleDomain>] [<exampleDomain>/pool] ... pool.destination.replicate=PoolManager
or for several pools in the /opt/d-cache/etc/dcache.conf file.
... pool.destination.replicate=PoolManager
File hopping configuration instructs a pool to send a
replicateFile request to the
PoolManager as the result of a file arriving on that pool
from some external client. All arriving files will be treated
the same. The PoolManager will process this transfer
request by trying to find a matching link (Please find
detailed information at Chapter 7, The poolmanager Service.
It is possible to configure the PoolManager such that files
are replicated from this pool to a special set of destination
pools.
Example:
Assume that we want to have all files, arriving on pool
ocean to be immediately replicated to a
subset of read pools. This subset of pools is described by the
poolgroup ocean-copies. No other pool is
member of the poolgroup ocean-copies.
Other than that, files arriving at the pool
mountain should be replicated to all read
pools from which farm nodes on the 131.169.10.0/24 subnet are
allowed to read.
The layout file defining the pools ocean
and mountain should read like this:
[exampleDomain]
[exampleDomain/pool]
name=ocean
path=/path/to/pool-ocean
pool.wait-for-files=${path}/data
pool.destination.replicate=PoolManager
name=mountain
path=/path/to/pool-mountain
pool.wait-for-files=${path}/data
pool.destination.replicate=PoolManager
In the layout file it is defined that all files arriving on
the pools ocean or
mountain should be replicated
immediately. The following
PoolManager.conf file contains
instructions for the PoolManager how to replicate these
files. Files arriving at the ocean pool
will be replicated to the ocean-copies
subset of the read pools and files arriving at the pool
mountain will be replicated to all read
pools from which farm nodes on the 131.169.10.0/24 subnet are
allowed to read.
# # define the units # psu create unit -protocol */* psu create unit -net 0.0.0.0/0.0.0.0 psu create unit -net 131.169.10.0/255.255.255.0 # create the faked net unit psu create unit -net 192.1.1.1/255.255.255.255 psu create unit -store *@* psu create unit -store ocean:raw@osm # # # define unit groups # psu create ugroup any-protocol psu create ugroup any-store psu create ugroup ocean-copy-store psu create ugroup farm-network psu create ugroup ocean-copy-network # psu addto ugroup any-protocol */* psu addto ugroup any-store *@* psu addto ugroup ocean-copy-store ocean:raw@osm psu addto ugroup farm-network 131.169.10.0/255.255.255.0 psu addto ugroup ocean-copy-network 192.1.1.1/255.255.255.255 psu addto ugroup allnet-cond 0.0.0.0/0.0.0.0 psu addto ugroup allnet-cond 131.169.10.0/255.255.255.0 psu addto ugroup allnet-cond 192.1.1.1/255.255.255.255 # # # define the write-pools # psu create pool ocean psu create pool mountain # # # define the write-pools poolgroup # psu create pgroup write-pools psu addto pgroup write-pools ocean psu addto pgroup write-pools mountain # # # define the write-pools-link, add write pools and set transfer preferences # psu create link write-pools-link any-store any-protocol allnet-cond psu addto link write-pools-link write-pools psu set link farm-read-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 # # # define the read-pools # psu create pool read-pool-1 psu create pool read-pool-2 psu create pool read-pool-3 psu create pool read-pool-4 # # # define the farm-read-pools poolgroup and add pool members # psu create pgroup farm-read-pools psu addto pgroup farm-read-pools read-pool-1 psu addto pgroup farm-read-pools read-pool-2 psu addto pgroup farm-read-pools read-pool-3 psu addto pgroup farm-read-pools read-pool-4 # # # define the ocean-copy-pools poolgroup and add a pool # psu create pgroup ocean-copy-pools psu addto pgroup ocean-copy-pools read-pool-1 # # # define the farm-read-link, add farm-read-pools and set transfer preferences # psu create link farm-read-link any-store any-protocol farm-network psu addto link farm-read-link farm-read-pools psu set link farm-read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 # # # define the ocean-copy-link, add ocean-copy-pools and set transfer preferences # psu create link ocean-copy-link ocean-copy-store any-protocol ocean-copy-network psu addto link ocean-copy-link ocean-copy-pools psu set link ocean-copy-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 # #
While 131.169.10.1
is a legal IP address e.g. of one of your farm nodes, the
192.1.1.1 IP
address must not exist anywhere at your site.
With the HoppingManager you have several configuration options
for file hopping on arrival, e.g.:
HoppingManager you can define a rule such that only
the files with a specific storage class should be
replicated.
Add the hoppingmanager service to a domain in your layout
file and restart the domain.
[<DomainName>] [<DomainName>/hoppingmanager]
Initially no rules are configured for the
HoppingManager. You may add rules by either edit the file
/opt/d-cache/config/HoppingManager.conf
and restart the hoppingmanager service, or use the admin
interface and save the modifications by the
save command into the
HoppingManager.conf
To enable replication on arrival by the HoppingManager set
the property pool.destination.replicate to
HoppingManager for the particular pool
[<exampleDomain>] [<exampleDomain>/pool] ... pool.destination.replicate=HoppingManager
or for several pools in the /opt/d-cache/etc/dcache.conf file.
... pool.destination.replicate=HoppingManager
The HoppingManager essentially receives
replicateFile messages from pools,
configured to support file hopping, and either discards
or modifies and forwards them to the PoolManager,
depending on rules described below.
The HoppingManager decides on the action to perform,
based on a set of configurable rules. Each rule has a
name. Rules are checked in alphabetic order concerning
their names.
A rule it triggered if the storage class matches the storage class pattern assigned to that rule. If a rule is triggered, it is processed and no further rule checking is performed. If no rule is found for this request the file is not replicated.
If for whatever reason, a file cannot be replicated, NO RETRY is being performed.
Processing a triggered rule can be :
The message is discarded. No replication is done for this particular storage class.
The rule modifies the
replicateFile message, before it
is forwarded to the PoolManager.
An ip-number of a farm-node of the farm that should
be allowed to read the file can be added to the
replicateFile message.
The mode of the replicated file can be
specified. This can either be
precious,
cached or
keep. keep
means that the pool mode of the source pool
determines the replicated file mode.
The requested protocol can be specified.
define hop OPTIONS <name> <pattern> precious|cached|keep
OPTIONS
-destination=<cellDestination> # default : PoolManager
-overwrite
-continue
-source=write|restore|* # !!!! for experts only StorageInfoOptions
-host=<destinationHostIp>
-protType=dCap|ftp...
-protMinor=<minorProtocolVersion>
-protMajor=<majorProtocolVersion>
pattern is a storage class
pattern. If the incoming storage class matches this
pattern, this rule is processed.
precious|cached|keep determines the
mode of the replicated file. With
keep the mode of the file will be
determined by the mode of the destination pool.
This defines which cell to use for
the pool to pool transfer. By default this is the
PoolManager and this should not be changed.
In case, a rule with the same name already exists, it is overwritten.
If a rule has been triggered and the corresponding
action has been performed, no other rules are
checked. If the continue option is
specified, rule checking continues. This is for
debugging purposes only.
-source defines the event on the pool
which has triggered the hopping. Possible values are
restore and
write. restore
means that the rule should be triggered if the file was
restored from a tape and write means
that it should be triggered if the file was written by a
client.
Choose the id of a node of the farm of worker-nodes that
should be allowed to access the file. Configure the
poolmanager respectively.
Specify the protocol which should be used to access the replicated files.
In order to instruct a particular pool to send a
replicateFile message to the
hoppingmanager service, you need to add the line
pool.destination.replicate=HoppingManager to the
layout file.
Example:
[exampleDomain]
[exampleDomain/pool]
name=write-pool
path=/path/to/write-pool-exp-a
pool.wait-for-files=${path}/data
pool.destination.replicate=HoppingManager
...
Assume that all files of experiment-a will be written to an
expensive write pool and subsequently flushed to tape. Now
some of these files need to be accessed without delay. The
files that need fast acceess possibility will be given the
storage class exp-a:need-fast-access@osm.
In this example we will configure the file hopping such that a
user who wants to access a file that has the above storage
info with the NFSv4.1 protocol will be able to do so.
Define a rule for hopping in the
/opt/d-cache/config/HoppingManager.conf
file.
define hop nfs-hop exp-a:need-fast-access@osm cached -protType=nfs -protMajor=4 -protMinor=1
This assumes that the storage class of the file is
exp-a:nfs@osm. The mode of the file, which
was precious on the write pool will have to
be changed to cached on the read pool.
The corresponding
/opt/d-cache/config/poolmanager.conf
file could read like this:
# # define the units # psu create unit -protocol */* psu create unit -net 0.0.0.0/0.0.0.0 psu create unit -store exp-a:need-fast-access@osm # # # define unit groups # psu create ugroup any-protocol psu create ugroup exp-a-copy-store psu create ugroup allnet-cond # psu addto ugroup any-protocol */* psu addto ugroup exp-a-copy-store exp-a:need-fast-access@osm psu addto ugroup allnet-cond 0.0.0.0/0.0.0.0 # # # define the write-pool # psu create pool write-pool # # # define the read-pool # psu create pool read-pool # # # define the exp-a-read-pools poolgroup and add a pool # psu create pgroup exp-a-read-pools psu addto pgroup exp-a-read-pools read-pool # # # define the exp-a-write-pools poolgroup and add a pool # psu create pgroup exp-a-write-pools psu addto pgroup exp-a-write-pools write-pool # # # define the exp-a-read-link, add exp-a-read-pools and set transfer preferences # psu create link exp-a-read-link exp-a-copy-store any-protocol allnet-cond psu addto link exp-a-read-link exp-a-read-pools psu set link exp-a-read-link -readpref=10 -writepref=0 -cachepref=10 -p2ppref=-1 # # # define the exp-a-write-link, add exp-a-write-pools and set transfer preferences # psu create link exp-a-write-link exp-a-copy-store any-protocol allnet-cond psu addto link exp-a-write-link exp-a-write-pools psu set link exp-a-write-link -readpref=0 -writepref=10 -cachepref=0 -p2ppref=-1 # # #