Accessing CephFS over Samba

CephFS is a very versatile, scaleable, distributed file system. I’ve gradually been scaling out my Ceph cluster and now have 65TB across 11 nodes, which in fairness is quite small as Ceph clusters go! One downside of CephFS however is that support for Windows clients is very lacking at the moment. There is currently some very early work ongoing to address this, but it is some way off being an easily deployed, stable, solution.

There is however an alternative method – serve CephFS over Samba. This can then be accessed by Windows clients without any extra software or special configuration, as well as other clients that lack native Ceph support.

The ‘traditional’ way of achieving this would be to mount the CephFS filesystem on the Samba host, and then have Samba share out that path. However, the vfs_ceph module for Samba provides a far better solution – with Samba accessing the Ceph cluster directly. No local mounts need to be made on the Samba host, and all configuration is done via Samba.

Prerequisities

In order to follow this guide and serve a CephFS filesystem over Samba you’ll need;

• A functional Ceph cluster with a CephFS filesystem. This won’t work for serving RBD devices or RADOS objects.
• A Linux machine that can access the Ceph cluster.

Installing Samba

Samba packages are available for almost all Linux distributions. However, you’ll also need to install Samba’s collection of vfs modules. These are typically not installed by default and are contained in a separate package. On Ubuntu-based systems, the following will install the required packages;

apt install samba samba-vfs-modules

Warning: Ubuntu 18.04’s samba-vfs-modules package contains an older version of the vfs_ceph module which is incompatible with the new Messenger v2 syntax for Ceph Monitors. If you are setting up this Samba server on an existing Ceph node and intending on using an existing ceph.conf file which has Messenger v2 syntax, you will need to do one of the following;

1. Stop using Messenger v2 syntax.
2. Make a copy of the ceph.conf file for Samba and modify it to not use the Messenger v2 syntax.
3. Use a newer version of Samba, either via a newer version of Ubuntu or via a PPA.

Configuration files with Messenger v2 syntax will have a mon_host configuration entry similar to the one below;

mon_host = [v2:10.0.0.1:3300/0,v1:10.0.0.1:6789/0]

If Messenger v2 syntax is not being used, then the mon_host configuration entry will look similar to the one below;

mon_host = 10.0.0.1:6789

Docker

Samba can also be installed in a Docker container rather than direct on a host. However, very few pre-built container images for Samba include the VFS modules needed to natively serve CephFS filesystems. I have built one based off Ubuntu 19.10 (to avoid the Messenger v2 syntax issue mentioned above) and published it on Docker Hub. The Dockerfile is also available should you wish to modify and add in additional packages (e.g. LDAP support).

Configuring Samba

Configuration of Samba to serve content from a CephFS filesystem is surprisingly easy with the vfs_ceph module. A typical [share] entry using vfs_ceph would look as follows;

[myshare]
vfs objects = ceph
path = /documents/
kernel share modes = no
ceph:config_file = /etc/ceph/ceph.conf

Breaking this down line by line;

[myshare]

This is the name of the Samba share, as visible and used by Samba clients when connecting.

vfs objects = ceph

This instructs Samba to use the vfs_ceph module for this share.

path = /documents/

This is the path on the CephFS filesystem which should be accessible from this share. The full path will be hidden from a client accessing this share – if path is set to /very/long/complicated/path, all a Samba client will see is \\server\myshare.

kernel share modes = no

Although Samba’s documentation notes that it is very desirable to leave this enabled, the documentation for the vfs_ceph module states that it must be disabled for file serving to work properly.

ceph:config_file = /etc/ceph/ceph.conf

This points Samba’s vfs_ceph module towards a Ceph configuration file. This is essential for the module to be able to identify the Ceph Monitors for the cluster, and read a suitable authentication token.

Complete Configuration Example

If you do not already have a complete Samba configuration file, the following should allow you to serve out a CephFS filesystem with read-only anonymous guest support.

[global]
security = user
map to guest = Bad User
username map = /etc/samba/smbusers

[myshare]
vfs objects = ceph
path = /documents/
kernel share modes = no
public = yes
read only = yes
guest only = yes
ceph:config_file = /etc/ceph/ceph.conf

nobody = guest

Start Samba

Start Samba using the standard process control for the system (systemctl etc) and then try connecting with a Samba client, such as a Windows machine. You should find that you can browse the CephFS filesystem sub-path configured for the share and access the files. That’s it, you’ve got the basics configured!

Permissions and ACLs

One caveat of this method is that permissions become multilayered – Samba’s own permissions are applied on top of Ceph’s permissions. For anything more than read-only guest access, this can become a bit of a headache!

If authentication is used for Samba, this is not passed through to CephFS. Instead, Ceph’s permissions are based on the optional ceph:user_id configuration option for the share. This can be set to a user ID and the CephFS permission model will then apply.

High Availability

It’s not much use having a clustered highly-available Ceph cluster if access via Samba is done via a single host! The vfs_ceph module can be used with CTDB to create a highly available Samba server, however as noted by vfs_ceph’s documentation CTDB_SAMBA_SKIP_SHARE_CHECK=yes must be set, otherwise CTDB will not reach a healthy state.

Conclusion

Hopefully this has given you an insight in how to make a CephFS filesystem available to clients without native CephFS support, such as Windows machines, through the usage of Samba. For the purposes of sharing out parts of my own CephFS filesystem without sensitive data, this was a surprisingly easy system to set up – compared to the typical complexity of Ceph that is! For anyone that needs stricter access controls, the multilayered permissions model may cause some complications but is not insurmountable.

Although it’s exciting to see work progressing on a native CephFS client for Windows, access via Samba is still a great solution that works in the interim and without requiring any software installation on the clients themselves.