ATOL: Lustre - the new generation file system

9.7.2009 19:27 | Přečteno: 983× | Linux

Autor: Petr Motejlek

Nasledovný príspevok je študentskou prácou, ktorá vznikla v rámci predmetu Advanced Topics of Linux Administration. Predmet je vypisovaný na Fakulte informatiky MU v spolupráci so spoločnosťou Red Hat Czech. Vyučovacím jazykom je angličtina a preto je v nej aj nasledovný príspevok.

For starters

Lustre is a new generation object based distributed file system. It's heavily used in the world's super computers (entire half of the top 30 super computers uses Lustre for its file storage); mainly for its enormous speed and scalability. The term Lustre is a mixture of the words Linux and cluster. Lustre is known to sustain thousands of nodes and petabytes of disk space while maintaining the speed, security and high availability. Oh, and did I forgot to say it's developed under the GNU GPL? ;).

History of Lustre

Lustre started with the Cluster File Systems, Inc. company founded by Dr. Peter Braam in the year 2001. Cluster File Systems, Inc. held its offices in the United States of America, Canada and even China. Cluster File Systems, Inc.'s clients include such famous companies as Hewlett-Packard or Cray and famous super computer laboratories as Oak Ridge National Laboratory or Los Alamos National Laboratory. On September 12, 2007 Sun Microsystems, Inc. and Cluster File Systems, Inc. signed an acquisition agreement (Sun Microsystems, Inc. were to acquire Cluster File Systems, Inc.). The acquisition was completed on October 1, 2007.

Architecture of Lustre

Lustre composes of three main units:

• Meta data target (MDT) per file system. It stores meta data about the storage. It is located on the meta data server (MDS).
• Object storage targets (OST). These targets store data on the object storage servers (OSSes). An OSS can (depending on its hardware) typically handle from two to eight OSTs, each target up to 8 terabytes big. The total capacity of the whole Lustre system is sum of all the OST's capacity.
• Lustre clients that connect to the Lustre system and use the provided data. Lustre system allows the clients to use POSIX read and write methods and it also supports POSIX ACLs and for example the root squashing feature.

Theoretically it is possible to have all the three units on one machine, but who would want something like this? ;). Typically all the units are spread on different nodes using from two to four OSTs per OSS in the Lustre system; and all the nodes should be dedicated. Lustre system can run on various network types —  including TCP/IP, Infiniband and other proprietary systems. Lustre system can also use remote direct memory access (RDMA) transfers to improve the bandwidth and to reduce the CPU usage.

The OSSs' storage is usually partitioned and organized by Logical Volume Manager and/or backed by RAID. The storage is formatted as a Lustre file system which is used by the clients. Internally Lustre still uses EXT3 (and plans on using ZFS in the future) to store meta data.

Access to a file from Lustre client is handled by the Lustre system in these steps:

1. Lustre system looks up the file name on the MDS.
2. The file is then created on behalf of the Lustre client or the layout of an existing file is returned to the Lustre client.
3. For read and write operations the Lustre client passes the received file layout to a logical object volume (LOV) which maps the offset and size to one or more objects which are each situated on a different OST.
4. Lustre client then locks the range of the objects it operates on and writes or reads directly to or from the OSTs.

Thanks to this feature bottlenecks of client-to-OST communications are eliminated. Therefore the speed of read and write operations is higher according to the amount of OSTs the Lustre client communicates with — it is almost linear.

Accidentally the Lustre clients do not modify the data directly by themselves, but delegate these tasks to the OSSes. Thanks to this the scalability is ensured and the security and reliability is improved as well.

How is Lustre implemented in Linux?

On a Linux client Lustre can be either a user-space library or a kernel module. In the beginning there used to be only the kernel module and a typical Lustre installation would use it to mount a Lustre file system as any other file system. The client applications see the file system as a single unified file system (hence there may be thousands of nodes comprising this one).

From the year 2008 Lustre system may also use the user-space library liblustre that allows not only the same features as the kernel module, but also enables the node to see the Lustre file system, even though the node is not properly configured as a Lustre client. Liblustre allows to modify or read data directly from the OSSes. This approach does not require the data copy to go through the kernel (providing low latency and high bandwidth).

Data objects and file striping in Lustre

Normally in a Linux file system, an inode contains all the basic information about each file (e. g. where its data is contained). The Lustre file system uses inodes as well, but inodes on MDSes point to the objects associated with the file (not to the data blocks). These objects are backed by single files on the OSSes. When a Lustre client tries to open a file, the file open operation does a transfer of set of object pointers from the MDS to the Lustre client, thus the Lustre client can interact with the OSS node where the object is stored directly.

There is a special case in which where only one object is associated with and MDS inode. In that case that object contains all the data. When more objects are associated with a file, the data in the file is striped across all the objects (this feature is similar to the RAID 0 and in Lustre it is possible to stripe a file in up to 160 parts). The striping brings high performance and when the striping is used the maximum file size is not limited by the size of a single target, but is aggregated by the number of OSTs.

Networking of Lustre

The network between the Lustre nodes is implemented over the Lustre Networking (aka LNET) API that provides the infrastructure for the Lustre file system.

Lustre Networking API supports many network types including TCP/IP and Infiniband. It can contain such features as transparent recovery with failover servers.

Lustre Networking API provides end-to-end bandwidth over 100 MBps on 1 Gigabit ethernet links, up to 1.5 GBps on Infiniband and over 1 GBps across 10 Gigabit ethernet links.

High availability in Lustre

Lustre's high availability features include failover and recovery mechanisms allowing server failures and reboots or shutdowns transparent. Version compatibility between Lustre minor versions allows the system administrator to shutdown Lustre on one server, upgrade or repair it, and then restart it.

Summary

While current network file systems like the Global File System need to deliver the same storage to all the participating server nodes, Lustre allows one to aggregate the storage available to their servers with high performance gain and availability. On the other hand if they do not have enough failover mechanisms like redundant power supply or use RAID the outcomes could be catastrophic.

The quick example on using Lustre

Administering Lustre system is really simple. The first thing you need to do is to install the Lustre packages. These can be downloaded from Sun Microsystems' website or in the case of GNU/Debian distribution they are part of the official repositories.

For this example I am using five machines —  MDS (named mds.example.com), OSS (named oss1.example.com), OSS (named oss2.example.com) and Lustre Client (named client.example.com). For the connection I am using ethernet.

On the MDS and OSS we need to install the

• kernel-lustre-smp-VERSION,
• kernel-lustre-bigsmp-VERSION,
• lustre-modules-VERSION,
• lustre-VERSION,
• lustre-ldiskfs-VERSION and
• e2fsprogs-VERSION packages.

Do not forget to boot the server with the newly installed patched kernel.

On the Lustre Client we need to install the

• lustre-client-modules-VERSION and
• lustre-client-VERSION packages.

Add this line to /etc/modprobe.conf on all the machines:
options lnet networks=tcp
That tells Lustre to use the TCP/IP networking.

On the mds.example.com create the MDT device (I use RAID device for that.):
mkfs -t lustre --fsname=MyFirstLustreFS --mgs --mdt /dev/md0
And now mount it, so that it can be used:
mount -t lustre /dev/md0 /mnt/mdt

On the oss1.example.com create let's say two OST devices (I also use RAID here for those.):
mkfs -t lustre --ost --fsname=MyFirstLustreFS --mgsnode=mds.example.com@tcp0 /dev/md0
mkfs -t lustre --ost --fsname=MyFirstLustreFS --mgsnode=mds.example.com@tcp0 /dev/md1
And mount it, so that it can be used:
mount -t lustre /dev/md0 /mnt/ost1
mount -t lustre /dev/md1 /mnt/ost2
And on the oss2.example.com create let's say one OST device (I also use RAID here for that.):
mkfs -t lustre --ost --fsname=MyFirstLustreFS --mgsnode=mds.example.com@tcp0 /dev/md0
And mount it, so that it can be used:
mount -t lustre /dev/md0 /mnt/ost3

Mount the Lustre file system from the client.example.com:
mount -t lustre mgs.example.com:/MyFirstLustreFS /dev/MyFirstLustreFS
If you want to have more clients or OSTs, just use the same procedures again.

       

Anketa

Komentáře

[petr_p]

Domníval jsem se, že objektový souborový systém umožňuje věci, jako je strukturovat obsah „souboru“ a proces pak může číst nebo zapisovat jednotlivé části struktury, může vyhledávat části všech souborů, které odpovídají kritériu apod. (Něco jako dnes umí relační/objektové databázové systémy.)

Končí schopnosti Lustre rozhodit obsah souboru do více geograficky vzdálených bloků – objektů, nebo kromě posixového rozhraní nabízí Lustre skutečně objektové služby?