release notes | Book: 1.9.5, 1.9.12 (opt, FHS), 2.6 (FHS), 2.7 (FHS), 2.8 (FHS), 2.9 (FHS), 2.10 (FHS), 2.11 (FHS), | Wiki | Q&A black_bg
Web: Multi-page, Single page | PDF: A4-size, Letter-size | eBook: epub black_bg

migration copy

migration copy — Copies files to other pools.

SYNOPSIS

migration copy [<options>] <target>...

DESCRIPTION

Copies files to other pools. Unless filter options are specified, all files on the source pool are copied.

The operation is idempotent, that is, it can safely be repeated without creating extra copies of the files. If the replica exists on any of the target pools, then it is not copied again. If the target pool with the existing replica fails to respond, then the operation is retried indefinitely, unless the job is marked as eager.

Please notice that a job is only idempotent as long as the set of target pools does not change. If pools go offline or are excluded as a result of an exclude or include expression then the job may stop being idempotent.

Both the state of the local replica and that of the target replica can be specified. If the target replica already exists, the state is updated to be at least as strong as the specified target state, that is, the lifetime of sticky bits is extended, but never reduced, and cached can be changed to precious, but never the opposite.

Transfers are subject to the checksum computiton policy of the target pool. Thus checksums are verified if and only if the target pool is configured to do so. For existing replicas, the checksum is only verified if the verify option was specified on the migration job.

Jobs can be marked permanent. Permanent jobs never terminate and are stored in the pool setup file with the save command. Permanent jobs watch the repository for state changes and copy any replicas that match the selection criteria, even replicas added after the job was created. Notice that any state change will cause a replica to be reconsidered and enqueued if it matches the selection criteria — also replicas that have been copied before.

Several options allow an expression to be specified. The following operators are recognized: <, <=, ==, !=, >=, >, lt, le, eq, ne, ge, gt, ~=, !~, +, -, *, /, %, div, mod, |, &, ^, ~, &&, ||, !, and, or, not, ?:, =. Literals may be integer literals, floating point literals, single or double quoted string literals, and boolean true and false. Depending on the context, the expression may refer to constants.

Please notice that the list of supported operators may change in future releases. For permanent jobs we recommend to limit expressions to the basic operators <, <=, ==, !=, >=, >, +, -, *, /, &&, || and !.

Options

-accessed=<n>|[<n>]..[<m>]

Only copy replicas accessed <n> seconds ago, or accessed within the given, possibly open-ended, interval; e.g. -accessed=0..60 matches files accessed within the last minute; -accesed=60.. matches files accessed one minute or more ago.

-al=ONLINE|NEARLINE

Only copy replicas with the given access latency.

-pnfsid=<pnfsid>[,<pnfsid>] ...

Only copy replicas with one of the given PNFS IDs.

-rp=CUSTODIAL|REPLICA|OUTPUT

Only copy replicas with the given retention policy.

-size=<n>|[<n>]..[<m>]

Only copy replicas with size <n>, or a size within the given, possibly open-ended, interval.

-state=cached|precious

Only copy replicas in the given state.

-sticky[=<owner>[,<owner>...]]

Only copy sticky replicas. Can optionally be limited to the list of owners. A sticky flag for each owner must be present for the replica to be selected.

-storage=<class>

Only copy replicas with the given storage class.

-concurrency=<concurrency>

Specifies how many concurrent transfers to perform. Defaults to 1.

-order=[-]size|[-]lru

Sort transfer queue. By default transfers are placed in ascending order, that is, smallest and least recently used first. Transfers are placed in descending order if the key is prefixed by a minus sign. Failed transfers are placed at the end of the queue for retry regardless of the order. This option cannot be used for permanent jobs. Notice that for pools with a large number of files, sorting significantly increases the initialization time of the migration job.

size

Sort according to file size.

lru

Sort according to last access time.

-pins=move|keep

Controls how sticky flags owned by the PinManager are handled:

move

Ask PinManager to move pins to the target pool.

keep

Keep pins on the source pool.

-smode=same|cached|precious|removable|delete[+<owner>[(<lifetime>)] ...]

Update the local replica to the given mode after transfer:

same

does not change the local state (this is the default).

cached

marks it cached.

precious

marks it precious.

removable

marks it cached and strips all existing sticky flags exluding pins.

delete

deletes the replica unless it is pinned.

An optional list of sticky flags can be specified. The lifetime is in seconds. A lifetime of 0 causes the flag to immediately expire. Notice that existing sticky flags of the same owner are overwritten.

-tmode=same|cached|precious[+<owner>[(<lifetime>)]...]

Set the mode of the target replica:

same

applies the state and sticky bits excluding pins of the local replica (this is the default).

cached

marks it cached.

precious

marks it precious.

An optional list of sticky flags can be specified. The lifetime is in seconds.

-verify
Force checksum computation when an existing target is updated.
-eager

Copy replicas rather than retrying when pools with existing replicas fail to respond.

-exclude=<pool>[,<pool>...]

Exclude target pools. Single character (?) and multi character (*) wildcards may be used.

-exclude-when=<expression>

Exclude target pools for which the expression evaluates to true. The expression may refer to the following constants:

source.name or target.name

pool name

source.spaceCost or target.spaceCost

space cost

source.cpuCost or target.cpuCost

cpu cost

source.free or target.free

free space in bytes

source.total or target.total

total space in bytes

source.removable or target.removable

removable space in bytes

source.used or target.used

used space in bytes

-include=<pool>[,<pool>...]

Only include target pools matching any of the patterns. Single character (?) and multi character (*) wildcards may be used.

-include-when=<expression>

Only include target pools for which the expression evaluates to true. See the description of -exclude-when for the list of allowed constants.

-refresh=<time>

Specifies the period in seconds of when target pool information is queried from the pool manager. The default is 300 seconds.

-select=proportional|best|random

Determines how a pool is selected from the set of target pools:

proportional

selects a pool with a probability inversely proportional to the cost of the pool.

best

selects the pool with the lowest cost.

random

selects a pool randomly.

The default is proportional.

-target=pool|pgroup|link

Determines the interpretation of the target names. The default is ’pool’.

-pause-when=<expression>

Pauses the job when the expression becomes true. The job continues when the expression once again evaluates to false. The following constants are defined for this pool:

queue.files

The number of files remaining to be transferred.

queue.bytes

The number of bytes remaining to be transferred.

source.name

Pool name.

source.spaceCost

Space cost.

source.cpuCost

CPU cost.

source.free

Free space in bytes.

source.total

Total space in bytes.

source.removable

Removable space in bytes.

source.used

Used space in bytes.

targets

The number of target pools.

-permanent

Mark job as permanent.

-stop-when=<expression>

Terminates the job when the expression becomes true. This option cannot be used for permanent jobs. See the description of -pause-when for the list of constants allowed in the expression.