Client requests to a dCache system may have rather diverse behaviour. Sometimes it is possible to classify them into several typical usage patterns. An example are the following two concurrent usage patterns:
Example:
Data is copied with a high transfer rate to the dCache
system from an external source. This is done via the
GridFTP protocol. At the same time batch jobs on a local
farm process data. Since they only need a small part of each
file, they use the dCap protocol via the dCap library
and seek to the position in the file they are interested in,
read a few bytes, do a few hours of calculations, and
finally read some more data.
As long as the number of active requests does not exceed the
maximum number of allowed active requests, the two types of
requests are processed concurrently. The GridFTP transfers
complete at a high rate while the processing jobs take hours
to finish. This maximum number of allowed requests is set with
mover set max active and
should be tuned according to capabilities of the pool host.
However, if requests are queued, the slow processing jobs
might clog up the queue and not let the fast GridFTP request
through, even though the pool just sits there waiting for the
processing jobs to request more data. While this could be
temporarily remedied by setting the maximum active requests to
a higher value, then in turn GridFTP request would put a
very high load on the pool host.
The above example is pretty realistic: As a rule of thumb,
GridFTP requests are fastest, dCap requests with the
dccp program are a little slower and dCap requests
with the dCap library are very slow. However, the usage
patterns might be different at other sites and also might
change over time.
Use separate queues for the movers, depending on the door initiating them. This easily allows for a separation of requests of separate protocols. (Transfers from and to a tape backend and pool-to-pool transfers are handled by separate queues, one for each of these transfers.)
A finer grained queue selection mechanism based on, e.g. the
IP address of the client or the file which has been
requested, is not possible with this mechanism. However, the
pool selection unit
(PSU) may provide a separation onto separate pools
using those criteria.
In the above example, two separate queues for fast GridFTP
transfers and slow dCap library access would solve the
problem. The maximum number of active movers for the GridFTP
queue should be set to a lower value compared to the dCap
queue since the fast GridFTP transfers will put a high load
on the system while the dCap requests will be mostly idle.
For a multi mover queue setup, the pools have to be told to
start several queues and the doors have to be configured to
use one of these. It makes sense to create the same queues on
all pools. This is done by the following change to the file
/opt/d-cache/etc/dcache.conf:
poolIoQueue=queueA,queueB
Each door may be configured to use a particular mover queue. The pool, selected for this request, does not depend on the selected mover queue. So a request may go to a pool which does not have the particular mover queue configured and will consequently end up in the default mover queue of that pool.
gsiftpIoQueue=queueA dcapIoQueue=queueB
All requests send from this kind of door will ask to be scheduled to the given mover queue. The selection of the pool is not affected.
The doors are configured to use a particular mover queue as in the following example:
Example:
Create the queues queueA and
queueB, where queueA
shall be the queue for the GridFTP transfers and
queueB for dCap.
poolIoQueue=queueA,queueB gsiftpIoQueue=queueA dcapIoQueue=queueB
If the pools should not all have the same queues you can define queues for pools in the layout file. Here you might as well define that a specific door is using a specific queue.
Example:
In this example queueCis defined for
pool1 and queueD is
defined for pool2. The GridFTP door
running in the domain myDoors is using the
queue queueB.
[myPools] [myPools/pool1] poolIoQueue=queueC [myPools/pool2] poolIoQueue=queueD [myDoors] [myDoors/dcap] dcapIoQueue=queueC [myDoors/gridftp] gsiftpIoQueue=queueD
There is always a default queue called
regular. Transfers not requesting a
particular mover queue or requesting a mover queue not
existing on the selected pool, are handled by the
regular queue.
The pool cell commands mover ls and
mover set max active have a
-queue option to select the mover queue to
operate on. Without this option, mover set max active will act on the default
queue while mover ls will list all
active and waiting client transfer requests.
For the dCap protocol, it is possible to allow the client to
choose another queue name than the one defined in the file
dcache.conf. To achieve this
the property dcapIoQueueOverwrite needs to
be set to allowed.
Example:
Create the queues queueA and
queue_dccp, where queueA
shall be the queue for dCap.
poolIoQueue=queueA,queue_dccp dcapIoQueue=queueA dcapIoQueueOverwrite=allowed
With the dccp command the queue can now be specified as follows:
[user] $dccp -X-io-queue=queue_dccp <source> <destination>
Since dccp requests may be quite different from other
requests with the dCap protocol, this feature may be used to
use separate queues for dccp requests and other dCap
library requests. Therefore, the dccp command may be
changed in future releases to request a special
dccp-queue by default.
| Property | Default Value | Description |
|---|---|---|
| poolIoQueue | Not set | I/O queue name |
| dcapIoQueue | Not set | Insecure dCap I/O queue name |
| gsidcapIoQueue | Not set | GSIdCap I/O queue name |
| dcapIoQueueOverwrite | denied | Controls whether an application is allowed to overwrite a queue name |
| gsidcapIoQueueOverwrite | denied | Controls whether an application is allowed to overwrite a queue name |
| kerberosdcapIoQueueOverwrite | denied | Controls whether an application is allowed to overwrite a queue name |
| gsiftpIoQueue | Not set | GSI-FTP I/O queue name |
| nfsIoQueue | Not set | NFS I/O queue name |
| remoteGsiftpIoQueue | Not set | queue used for SRM third-party transfers (i.e. the srmCopy command) |
| webdavIoQueue | Not set | WebDAV and HTTP I/O queue name |
| xrootdIoQueue | Not set | xrootd I/O queue name |