release notes | UserGuide: 4.2, 5.0, 5.1, 5.2, 6.0, 6.1, 6.2, 7.0, 7.1, 7.2 (unreleased), 8.0, 8.1, 8.2, 9.0, 9.1, 9.2, 10.0, 10.1, 10.2, 7.2 (unreleased) | Book: 4.2, 5.0, 5.1, 5.2, 6.0, 6.1, 6.2, 7.0, 7.1, 7.2 (unreleased), 8.0, 8.1, 8.2, 9.0, 9.1, 9.2, 10.0, 10.1, 10.2, 7.2 (unreleased) | Wiki | Q&A black_bg
Edit me on GitHub
Back to the contents page.

dCache as NFSv4.1 Server

This chapter explains how to configure dCache in order to access it via the NFSv4.1 protocol, allowing clients to mount dCache and perform POSIX IO using standard NFSv4.1 clients.

Important

The pNFS mentioned in this chapter is the protocol NFSv4.1/pNFS and not the namespace pnfs.

Setting up

To allow file transfers in and out of dCache using NFSv4.1/pNFS, a new NFSv4.1 door must be started. This door acts then as the mount point for NFS clients.

To enable the NFSv4.1 door, you have to change the layout file corresponding to your dCache-instance. Enable the NFS within the domain that you want to run it by adding the following line

[<domainName>/nfs]
nfs.version = 4.1

Example: You can just add the following lines to the layout file:

[nfs-${host.name}Domain]
[nfs-${host.name}Domain/nfs]
nfs.version = 4.1

Exporting file system

In addition to run an NFSv4.1 door you need to add exports to the exports file. The location of exports file is controlled by nfs.export.file property and defaults to /etc/exports.

After reading exports file dCache will read the content of the directory with additional export tables. The location of directory defined by nfs.export.dir property and default to /etc/exports.d. Only files ending with .exports are considered. Files starting with a dot are ignored. The format of the export tables is similar to the one which is provided by Linux:

#
<path> [host [(options)]]

Where is a comma separated combination of:

  • ro matching clients can access this export only in read-only mode. This is the default value.

  • rw matching clients can access this export only in read-write mode

  • noacl dCache ACLs will be ignored; only posix access permissions will be considered. This is the default value.

  • acl dCache ACLs will be respected; if present, they override posix permissions. To view the ACLs at the NFS client side, use nfs4_getfacl which is in EL7 package nfs4-acl-tools.

  • sec=sys matching clients must access NFS using RPCSEC_SYS authentication, where client’s local uid and gid is used. This is the default value.

  • sec=krb5 matching clients must access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is NONE, e.g., the data is neither encrypted nor signed when sent over the network. Nevertheless the RPC packets header still protected by checksum.

  • sec=krb5i matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is INTEGRITY. The RPC requests and response are protected by checksum.

  • sec=krb5p matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is PRIVACY. The RPC requests and response are protected by encryption.

  • all_squash Map all uis and gids to the anonymous user.

  • no_root_squash Do not map requests from the root to anonymous user. This is useful for clients with administrative rights.

  • all_root the requests from matching clients will be handles as from user root. Additionally, when a new file or directory is created, then parent directory’s ownership is inherited.

  • anonuid and anongid specify which uid and gid should be used for anonymous user.

  • dcap Expose the fact, that nfs server is a dCache instance through special dot file. This option is useful when dcap client are used over mounted file system and preferred IO protocol is DCAP. This is the default value.

  • no_dcap The opposite of dcap option. Hide the fact, that nfs server is a dCache instance. This option is useful when dcap client are used over mounted file system but nfs protocol still have to be used for IO.

  • pnfs Enable NFSv4.1/pNFS to redirect pnfs capable clients to the dCache pools. This is the default value.

  • no_pnfs and nopnfs Opposite of pnfs option. Enforce clients to send all IO requests to NFS door event independent from client’s pNFS capabilities.

  • secure Specifying whether clients are required to use a privileged port (< 1024). This option typically is used to disallow user-space NFS clients, as they might spoof request credentials.

  • insecure The opposite of secure option. The use of privileged port by a client is not required. This is the default value.

  • lt= A colon separated ordered list of pnfs layouts that offered to the client. The valid values are:

  • flex_files

  • nfsv4_1_files

For modern clients (linux kernel >= 4.10 and RHEL7.4) flex_files is recommended. Default is nfsv4_1_files.

For example:

/pnfs/dcache.org/data *.dcache.org (rw,sec=krb5i)

Notice, that security flavour used at mount time will be used for client - pool communication as well.

Multiple specifications can be declared like this:

/pnfs/dcache.org/data *.dcache.org(rw) externalhost.example.org(ro)

In this example, hosts in the dcache.org may read and write, while host externalhost.example.org may only read.

If there are multiple path specifications, the shortest matching path wins. If there are multiple host/subnet specifications, the most precise specification wins.

Configuring NFSv4.1 door with GSS-API support

Adding sec=krb5 into /etc/exports is not sufficient to get kerberos authentication to work.

All clients, pool nodes and node running DOOR-NFS4 must have a valid kerberos configuration. Each clients, pool node and node running DOOR-NFS4 must have a /etc/krb5.keytab with nfs service principal:

nfs/host.domain@<YOUR.REALM>

The /etc/dcache/dcache.conf on pool nodes and node running NFSv4.1 door must enable kerberos and RPCSEC_GSS:

nfs.rpcsec_gss=true
dcache.authn.kerberos.realm=<YOUR.REALM>
dcache.authn.jaas.config=/etc/dcache/gss.conf
dcache.authn.kerberos.key-distribution-center-list=your.kdc.server

The /etc/dcache/gss.conf on pool nodes and node running NFSv4.1 door must configure Java’s security module:

com.sun.security.jgss.accept {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useKeyTab=true
keyTab="${/}etc${/}krb5.keytab"
debug=false
storeKey=true
principal="nfs/host.domain@<YOUR.REALM>";
};

Now your NFS client can securely access dCache.

Configuring NFSv4.1 with RPC-over-TLS

dCache NFS door supports RPC-over-TLS protocol extension for data-in-transit protection. To work with pNFS, host certificates must be installed on NFS doors and pools. As pNFS uses multipath addresses for client redirection, the host certificates must include the IPv4/IPv6 records in Subject Alternative Name.

To enable, the RPC-over-TLS capability on pools and nfs doors the following properties must be enabled:

pool.mover.nfs.enable.rpc-over-tls=true
nfs.enable.rpc-over-tls=true

or

dcache.enable.rpc-over-tls=true

NOTE: the TLS layer is only used for in-flight data protection and have no influence on user mapping or authentication.

Client side configuration

To support certificate handling by the Linux kernel the tlshd service must be installed and running (provided by ktls-utils package on RHEL and clones). If dCache certificates are not signed by OS-wide trusted CA, then /etc/tlshd.conf should be updated such, that x509.truststore points server the trust certificate chain in PEM format.

The mount option xprtsec controls the transport security for NFS:

mount -o xprtsec=tls nfs-door:/data /mnt/

Configuring principal-id mapping for NFS access

The NFSv4.1 uses utf8 based strings to represent user and group names:

user@DOMAIN.NAME

This is the case even for non-kerberos based accesses. Nevertheless UNIX based clients as well as dCache internally use numbers to represent uid and gids. A special service, called idmapd, takes care for principal-id mapping. On the client nodes the file /etc/idmapd.conf is usually responsible for consistent mapping on the client side. On the server side, in case of dCache mapping done through gplazma2. The identity type of plug-in required by id-mapping service. Please refer to Chapter 10, Authorization in dCache for instructions about how to configure `gPlazma.

For correct user id mapping nfs4 requires that server and client use the same naming scope, called nfs4domain. This implies a consistent configuration on both sides. To reduce deployment overhead a special auto-discovery mechanism was introduced by SUN Microsystems ( now Oracle) - a DNS TXT record. dCache supports this discovery mechanism. When nfs.domain property is set, it gets used. If it’s left unset, then DNS TXT record for _nfsv4idmapdomain is taken or the default localdomain is used when DNS record is absent.

To avoid big latencies and avoiding multiple queries for the same information, like ownership of a files in a big directory, the results from gPlazma are cached within NFSv4.1 door. The default values for cache size and life time are good enough for typical installation. Nevertheless they can be overriden in dcache.conf or layoutfile:

# maximal number of entries in the cache
nfs.idmap.cache.size = 512

# cache entry maximal lifetime
nfs.idmap.cache.timeout = 30

# time unit used for timeout. Valid values are:
# SECONDS, MINUTES, HOURS and DAYS
nfs.idmap.cache.timeout.unit = SECONDS

Managing group ids

The remote procedure call v2 (RPC) that used by nfs protocol defines different mechanisms for user authentication. One of them is AUTH_SYS, which majority of deployments are using. Dispire it’s popularity, the one downside of using AUTH_SYS is the sixteen groups limit. To overcome this limitation dCache nfs server can query gplazma to discover additional groups that the client might have. To enable it the nfs.idmap.manage-gids property can be used.

nfs.idmap.manage-gids = true

As enabling it might have negative effects on the performance, the default value it false.

NOTICE: currently only the ldap plugin supports group mapping by uid when gplazma.ldap.try-uid-mapping option is enabled.

Accessing directory tags

There are two ways to access dCache directory tags over NFS mount: the legacy one as via magic files and as extended attributes.

Directory tags as magic files

Each directory tag can be accessed via special file name, for example to read or write a tag with the name foo, the magic file is .(tag)(foo):

echo bar > '.(tag)(foo)'

or

cat '.(tag)(foo)'
|bar

To list all existing file use .(tags)() magic file:

 cat '.(tags)()'
|.(tag)(OSMTemplate)
|.(tag)(foo)
|.(tag)(sGroup)

NOTICE: the directory tags can’t be remove via magic files

Directory tags as extended attributes

The dCache’s NFSv4 door implements extended attributes over NFS which is specified in RFC8276. With compatible NFS client (for the moment only Linux kernel 5.10 and later), the directory tags can be access as extended attributes prefixed with dcache.tag.. For example:

 attr -l .
|Attribute "dcache.tag.OSMTemplate" has a 15 byte value for .
|Attribute "dcache.tag.sGroup" has a 8 byte value for .

or

getfattr  .
|# file: .
|user.dcache.tag.OSMTemplate
|user.dcache.tag.sGroup

NOTICE: the getfattr and setfattr commands adds an extra user. prefix.