release notes | Book: 1.9.5, 1.9.12 (opt, FHS), 2.11 (FHS), 2.12 (FHS), 2.13 (FHS), 2.14 (FHS), | Wiki | Q&A black_bg
Web: Multi-page, Single page | PDF: A4-size, Letter-size | eBook: epub black_bg

Multiple Queues for Movers in each Pool

[return to top]


Client requests to a dCache system may have rather diverse bahaviour. Sometimes it is possible to classify them into several typical usage patterns. An example are the following two usage patterns:

Example 26.1. To Concurrent Usage Patterns

Data is copied with a high transfer rate to the dCache system from an external source. This is done via the GridFTP protocol. At the same time batch jobs on a local farm process data. Since they only need a small part of each file, they use the dCap protocol via the dCap library and seek to the position in the file they are interested in, read a few bytes, do a few hours of calculations, and finally read some more data.

As long as the number of active requests do not exceed the maximum number of allowed active requests, the two types of requests are processed concurrently. The GridFTP transfers complete at a high rate while the processing jobs take hours to finish. This maximum number of allowed requests is set with mover set max active and should be tuned according to capabilities of the pool host.

However, if requests are queued, the slow processing jobs might clog up the queue and not let the fast GridFTP request through, even though the pool just sits there waiting for the processing jobs to request more data. While this could be temporarily remedied by setting the maximum active requests to a higher value, then in turn GridFTP request would put a very high load on the pool host.

The above example is pretty realistic: As a rule of thumb, GridFTP requests are fastest, dCap requests with the dccp program are a little slower and dCap requests with the dCap library are very slow. However, the usage patterns might be different at other sites and also might change over time.

[return to top]


Use separate queues for the movers, depending on the door initiating them. This easily allows for a separation of requests of separate protocols. Up to 10 mover queues for client transfers are available since dCache version 1.6.6. Earlier versions support only one queue. (Transfers from an to a tape backend and pool-to-pool transfers are handled by separate queues, one for each of these transfers.)

A finer grained queue selection mechanism based on, e.g. the IP address of the client or the file which has been requested, is not possible with this mechanism. However, the pool selection unit (PSU) may provide a separation onto separate pools using those criteria.

In the above example, two separate queues for fast GridFTP transfers and slow dCap library access would solve the problem. The maximum number of active movers for the GridFTP queue should be set to a lower value compared to the dCap queue since the fast GridFTP transfers will put a high load on the system while the dCap requests will be mostly idle.

[return to top]


For a multi mover queue setup, the pools have to be told to start several queues and the doors have to be configured to use one of these. It makes sense to create the same queues on all pools. This is done by the following change to the config/pool.batch file:

Example 26.2. Modified config/pool.batch file for multiple mover queues

define context startPools endDefine
  create diskCacheV111.pools.MultiProtocolPool2 ${0} \
         "!MoverMap \
         ${1} \
         -io-queues=<queueName-1>[,<queueName-2>[,...,<queueName-10>]] \
         -recover-control=yes \
         -version=4 \
         -sticky=allowed \
         -sendHitInfoMessages=yes \
         -${2} -${3} -${4} -${5} -${6} -${7} -${8} \

The same can be achived by appending -io-queues=<queueName-1>,...,<queueName-n> to each line in the poollist file. However, this only makes sense if the pools should not all have the same queues.

The first in this list of queues (<queueName-1>) is the default mover queue. Transfers not requesting a particular mover queue or requesting a mover queue not existing on the selected pool, are handled by this default queue.

The pool cell commands mover ls and mover set max active have an -queue option to select the mover queue to operate on. Without this option, mover set max active will act on the default queue while mover ls will list the requests of all pools for backward compatibility.

Each door may be configured to use a particular mover queue. The pool, selected for this request, doesn’t depend on the selected mover queue. So a request may go to a pool which doesn’t have the particular mover queue configured and will consequently end up in the default mover queue of that pool.

The doors are configured to use a particular mover queue as in the following example:

Example 26.3. Batch file for a GridFTP door using a mover queue

create GFTP \
       "<portName> \
       diskCacheV111.doors.GsiFtpDoorV1 \
       -io-queue=<queueName> \
       ... \

All requests send from this door will ask to be scheduled to the given mover queue. The selection of the pool is not affected.

For the dCap protocol, the corresponding door may be configured to allow the client to determine the mover queue name. In that case the client may use the extra option facility to specify a mover queue. Whether the the dCap door allows the client to request a particular mover queue or not is configured with the -io-queue={allowed|denied} option as in the following example:

Example 26.4. Batch file for a dCap door for allowing the client to select the mover queue

create DCap \
       "${dCapPort} \
       diskCacheV111.doors.DCapDoor \
       -io-queue=<queueName> \
       -io-queue-overwrite=allowed \
       ... \

With the dccp command the queue can now be specified as follows:

[user] $ dccp -X-io-queue=queueName <source> <destination>

Since dccp requests may be quite different from other requests with the dCap protocol, this feature may be used to use separate queues for dccp requests and other dCap library requests. Therefore, the dccp command may be changed in future releases to request a special dccp-queue by default.