Checksums

Checksums
Prev	Chapter 23. Pool Operations	Next

In dCache the storage of a checksum is part of a successful transfer.

For an incoming transfer a checksum can be sent by the client (Client Checksum, it can be calculated during the transfer (Transfer Checksum) or it can be calculated on the server after the file has been written to disk (Server File Checksum).
For a pool to pool transfer a Transfer Checksum or a Server File Checksum can be calculated.
For data that is flushed to or restored from tape a checksum can be calculated before flushed to tape or after restored from tape, respectively.

Client Checksum

The client calculates the checksum before or while the data is sent to dCache. The checksum value, depending on when it has been calculated, may be sent together with the open request to the door and stored into Chimera before the data transfer begins or it may be sent with the close operation after the data has been transferred.

The dCap protocol provides both methods, but the dCap clients use the latter by default.

The FTP protocol does not provide a mechanism to send a checksum. Nevertheless, some FTP clients can (mis-)use the “site” command to send the checksum prior to the actual data transfer.

Transfer Checksum

While data is coming in, the server data mover may calculate the checksum on the fly.

Server File Checksum

After all the file data has been received by the dCache server and the file has been fully written to disk, the server may calculate the checksum, based on the disk file.

The default configuration is that a checksum is calculated on write, i.e. a Server File Checksum.

[return to top]

How to configure checksum calculation

Configure the calculation of checksums in the admin interface. The configuration has to be done for each pool separately.

(local) admin > cd <poolname>
(<poolname>) admin > csm set policy -<option>=<on/off>
(<poolname>) admin > save

The configuration will be saved in the file <path/to/pool>/<nameOfPooldirectory>/setup.

Use the command csm info to see the checksum policy of the pool.

(<poolname>) admin > csm info
 Policies :
        on read : false
       on write : true
       on flush : false
     on restore : false
    on transfer : false
    enforce crc : true
  getcrcfromhsm : false
          scrub : false

The default configuration is to check checksums on write.

Use the command help csm set policy to see the configuration options.

The syntax of the command csm set policy is

csm set policy [-<option>=on [|off]]

where <option> can be replaced by

OPTIONS

ontransfer: If supported by the protocol, the checksum is calculated during file transfer.
onwrite: The checksum is calculated after the file has been written to disk.
onrestore: The checksum is calculated after data has been restored from tape.
onflush: The checksum is calculated before data is flushed to tape.
getcrcfromhsm: If the HSM script supports it, the <pnfsid>.crcval file is read and stored in Chimera.
scrub: Pool data will periodically be veryfied against checksums. Use the command help csm set policy to see the configuration options.
enforcecrc: If no checksum has been calculated after or during the transfer, this option ensures that a checksum is calculated and stored in Chimera.

The option onread has not yet been implemented.

If an option is enabled a checksum is calculated as described. If there is already another checksum, the checksums are compared and if they match stored in Chimera.

Important

Do not change the default configuration for the option enforcecrc. This option should always be enabled as this ensures that there will always be a checksum stored with a file.