![black_bg](/images/black_bg.png)
dCache as NFSv4.1 Server
This chapter explains how to configure dCache in order to access it via the NFSv4.1
protocol, allowing clients to mount dCache and perform POSIX IO using standard NFSv4.1
clients.
Important
The
pNFS
mentioned in this chapter is the protocolNFSv4.1/pNFS
and not the namespace pnfs.
- Setting up
- Configuring NFSv4.1 door with GSS-API support
- Configuring NFSv4.1 with RPC-over-TLS
- Configuring principal-id mapping for NFS access
- Managing group ids
- Accessing directory tags
Setting up
To allow file transfers in and out of dCache using NFSv4.1/pNFS, a new NFSv4.1 door must be started. This door acts then as the mount point for NFS clients.
To enable the NFSv4.1 door, you have to change the layout file corresponding to your dCache-instance. Enable the NFS within the domain that you want to run it by adding the following line
[<domainName>/nfs]
nfs.version = 4.1
Example: You can just add the following lines to the layout file:
[nfs-${host.name}Domain]
[nfs-${host.name}Domain/nfs]
nfs.version = 4.1
Exporting file system
In addition to run an NFSv4.1 door you need to add exports to the exports file. The location of exports file is controlled by nfs.export.file property and defaults to /etc/exports
.
After reading exports file dCache will read the content of the directory with additional export tables. The location of directory defined by nfs.export.dir property and default to /etc/exports.d
. Only files ending with .exports are considered. Files starting with a dot are ignored. The format of the export tables is similar to the one which is provided by Linux:
#
<path> [host [(options)]]
Where
-
ro matching clients can access this export only in read-only mode. This is the default value.
-
rw matching clients can access this export only in read-write mode
-
noacl dCache ACLs will be ignored; only posix access permissions will be considered. This is the default value.
-
acl dCache ACLs will be respected; if present, they override posix permissions. To view the ACLs at the NFS client side, use
nfs4_getfacl
which is in EL7 packagenfs4-acl-tools
. -
sec=sys matching clients must access NFS using RPCSEC_SYS authentication, where client’s local uid and gid is used. This is the default value.
-
sec=krb5 matching clients must access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is NONE, e.g., the data is neither encrypted nor signed when sent over the network. Nevertheless the RPC packets header still protected by checksum.
-
sec=krb5i matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is INTEGRITY. The RPC requests and response are protected by checksum.
-
sec=krb5p matching clients have to access NFS using RPCSEC_GSS authentication. The Quality of Protection (QOP) is PRIVACY. The RPC requests and response are protected by encryption.
-
all_squash Map all uis and gids to the anonymous user.
-
no_root_squash Do not map requests from the root to anonymous user. This is useful for clients with administrative rights.
-
all_root the requests from matching clients will be handles as from user root. Additionally, when a new file or directory is created, then parent directory’s ownership is inherited.
-
anonuid and anongid specify which uid and gid should be used for anonymous user.
-
dcap Expose the fact, that nfs server is a dCache instance through special dot file. This option is useful when dcap client are used over mounted file system and preferred IO protocol is DCAP. This is the default value.
-
no_dcap The opposite of dcap option. Hide the fact, that nfs server is a dCache instance. This option is useful when dcap client are used over mounted file system but nfs protocol still have to be used for IO.
-
pnfs Enable NFSv4.1/pNFS to redirect pnfs capable clients to the dCache pools. This is the default value.
-
no_pnfs and nopnfs Opposite of pnfs option. Enforce clients to send all IO requests to NFS door event independent from client’s pNFS capabilities.
-
secure Specifying whether clients are required to use a privileged port (< 1024). This option typically is used to disallow user-space NFS clients, as they might spoof request credentials.
-
insecure The opposite of secure option. The use of privileged port by a client is not required. This is the default value.
-
lt= A colon separated ordered list of pnfs layouts that offered to the client. The valid values are:
-
flex_files
- nfsv4_1_files
For modern clients (linux kernel >= 4.10 and RHEL7.4) flex_files is recommended. Default is nfsv4_1_files.
For example:
/pnfs/dcache.org/data *.dcache.org (rw,sec=krb5i)
Notice, that security flavour used at mount time will be used for client - pool communication as well.
Multiple specifications can be declared like this:
/pnfs/dcache.org/data *.dcache.org(rw) externalhost.example.org(ro)
In this example, hosts in the dcache.org may read and write, while host externalhost.example.org may only read.
If there are multiple path specifications, the shortest matching path wins. If there are multiple host/subnet specifications, the most precise specification wins.
Configuring NFSv4.1 door with GSS-API support
Adding sec=krb5
into /etc/exports
is not sufficient to get kerberos authentication to work.
All clients, pool nodes and node running DOOR-NFS4 must have a valid kerberos configuration. Each clients, pool node and node running DOOR-NFS4 must have a /etc/krb5.keytab
with nfs
service principal:
nfs/host.domain@<YOUR.REALM>
The /etc/dcache/dcache.conf
on pool nodes and node running NFSv4.1 door
must enable kerberos and RPCSEC_GSS:
nfs.rpcsec_gss=true
dcache.authn.kerberos.realm=<YOUR.REALM>
dcache.authn.jaas.config=/etc/dcache/gss.conf
dcache.authn.kerberos.key-distribution-center-list=your.kdc.server
The /etc/dcache/gss.conf
on pool nodes and node running NFSv4.1
door must configure Java’s security module:
com.sun.security.jgss.accept {
com.sun.security.auth.module.Krb5LoginModule required
doNotPrompt=true
useKeyTab=true
keyTab="${/}etc${/}krb5.keytab"
debug=false
storeKey=true
principal="nfs/host.domain@<YOUR.REALM>";
};
Now your NFS client can securely access dCache.
Configuring NFSv4.1 with RPC-over-TLS
dCache NFS door supports RPC-over-TLS protocol extension for data-in-transit protection. To work with pNFS, host certificates must be installed on NFS doors and pools. As pNFS uses multipath addresses for client redirection, the host certificates must include the IPv4/IPv6 records in Subject Alternative Name
.
To enable, the RPC-over-TLS capability on pools and nfs doors the following properties must be enabled:
pool.mover.nfs.enable.rpc-over-tls=true
nfs.enable.rpc-over-tls=true
or
dcache.enable.rpc-over-tls=true
NOTE: the TLS layer is only used for in-flight data protection and have no influence on user mapping or authentication.
Client side configuration
To support certificate handling by the Linux kernel the tlshd
service must be installed and running (provided by ktls-utils package on RHEL and clones). If dCache certificates are not signed by OS-wide trusted CA, then /etc/tlshd.conf
should be updated such, that x509.truststore
points server the trust certificate chain in PEM format.
The mount option xprtsec
controls the transport security for NFS:
mount -o xprtsec=tls nfs-door:/data /mnt/
Configuring principal-id mapping for NFS access
The NFSv4.1
uses utf8 based strings to represent user and group names:
This is the case even for non-kerberos based accesses. Nevertheless UNIX based clients as well as dCache internally use numbers to represent uid and gids. A special service, called idmapd
, takes care for principal-id mapping. On the client nodes the file /etc/idmapd.conf
is usually responsible for consistent mapping on the client side. On the server side, in case of dCache mapping done through gplazma2. The identity
type of plug-in required by id-mapping service. Please refer to Chapter 10, Authorization in dCache for instructions about how to configure `gPlazma.
For correct user id mapping nfs4 requires that server and client use the same naming scope, called nfs4domain. This implies a consistent configuration on both sides. To reduce deployment overhead a special auto-discovery mechanism was introduced by SUN Microsystems ( now Oracle) - a DNS TXT record. dCache supports this discovery mechanism. When nfs.domain
property is set, it gets used. If it’s left unset, then DNS TXT record for _nfsv4idmapdomain
is taken or the default localdomain
is used when DNS record is absent.
To avoid big latencies and avoiding multiple queries for the same information, like ownership of a files in a big directory, the results from gPlazma
are cached within NFSv4.1 door
. The default values for cache size and life time are good enough for typical installation. Nevertheless they can be overriden in dcache.conf
or layoutfile:
# maximal number of entries in the cache
nfs.idmap.cache.size = 512
# cache entry maximal lifetime
nfs.idmap.cache.timeout = 30
# time unit used for timeout. Valid values are:
# SECONDS, MINUTES, HOURS and DAYS
nfs.idmap.cache.timeout.unit = SECONDS
Managing group ids
The remote procedure call v2 (RPC) that used by nfs protocol defines different mechanisms for user authentication. One of them is AUTH_SYS, which majority of deployments are using. Dispire it’s popularity, the one downside of using AUTH_SYS is the sixteen groups limit. To overcome this limitation dCache nfs server can query gplazma to discover additional groups that the client might have. To enable it the nfs.idmap.manage-gids
property can be used.
nfs.idmap.manage-gids = true
As enabling it might have negative effects on the performance, the default value it false.
NOTICE: currently only the ldap plugin supports group mapping by uid when
gplazma.ldap.try-uid-mapping
option is enabled.
Accessing directory tags
There are two ways to access dCache directory tags over NFS mount: the legacy one as via magic files and as extended attributes.
Directory tags as magic files
Each directory tag can be accessed via special file name, for example to read or write a tag with the name foo
, the magic file is .(tag)(foo)
:
echo bar > '.(tag)(foo)'
or
cat '.(tag)(foo)'
|bar
To list all existing file use .(tags)()
magic file:
cat '.(tags)()'
|.(tag)(OSMTemplate)
|.(tag)(foo)
|.(tag)(sGroup)
NOTICE: the directory tags can’t be remove via magic files
Directory tags as extended attributes
The dCache’s NFSv4 door implements extended attributes over NFS which is specified in RFC8276. With compatible NFS client (for the moment only Linux kernel 5.10 and later), the directory tags can be access as extended attributes prefixed with dcache.tag.
. For example:
attr -l .
|Attribute "dcache.tag.OSMTemplate" has a 15 byte value for .
|Attribute "dcache.tag.sGroup" has a 8 byte value for .
or
getfattr .
|# file: .
|user.dcache.tag.OSMTemplate
|user.dcache.tag.sGroup
NOTICE: the
getfattr
andsetfattr
commands adds an extrauser.
prefix.