Portál AbcLinuxu, 30. dubna 2025 11:21
Autor: Ladislav Tkáč
Nasledovný príspevok je študentskou prácou, ktorá vznikla v rámci predmetu Advanced Topics of Linux Administration. Predmet je vypisovaný na Fakulte informatiky MU v spolupráci so spoločnosťou Red Hat Czech. Vyučovacím jazykom je angličtina a preto je v nej aj nasledovný príspevok.
In this report I will compare GFS/iSCSI with NFS and point out their main benefits. I also provide the list of the tools which could be used for performance testing and their description.
There are lot of options for maintaining remote access to files. From basic file sharing to distributed file systems, where some of them are multi-platform and wide used. Choosing the right one depends on the environment, where the file system will be used. For enterprise cluster we definitively need another approach than for small office file server. There is a plenty of options like NFSv3 NFSv4, CIFS aka Samba, Coda, AFS, GFS, DAFS and choosing the one that's fitting could be difficult.
If we look on the problem closely, we have to divide approaches to block I/O and file I/O. File I/O is used in network file sharing protocols like CIFS or NFS. However, there is number of applications that do not require file sharing and even whose performance is significantly degraded when accessing data using this approach. For instance databases or other transaction-oriented applications. However, those applications do require IP connectivity and redundant fail over solutions when communicating between servers and clients. And there come block I/O protocols like SCSI.
iSCSI is based on IP protocol. It sends SCSI commands over IP networks and uses TCP flow control, congestion control and segmentation mechanisms. ISCSI is often used in SAN. The protocol is build up on the client server model and is bi-directional (can also be used to return data in response to the original request). Client is called initiator and sends SCSI commands to server part called target which is representing remote SCSI storage device. Targets and initiators could be software or hardware based. Security is based on CHAP authentication and can use Ipsec. VLAN is also good option limiting use by unauthorized clients, but single host compromised could be used for attack on other storage resources on the network.
One of them main benefits of the iSCSI is the ability to run over existing Ethernet networks and FC (fibre channel) as other SAN approach FCIP(Fiber Channel over IP) can run only on FC. This will reduce the cost of the equipment on the places, where use of FC is not necessary. iSCSI is also more scalable with respect to the number of storage devices and has practically no distance limitations. Another big benefit of iSCSI is that it works across routed IP networks as opposite to FCoE (Fiber Channel over Ethernet) which is not routable at the IP layer. ISCSI is included as an option on most server systems (even Windows) and requires minimal or no configuration on these operating systems.
NFS is using RPC (Remote procedure call) developed by SUN. Portmap maintains binding RPC services with specific UDP and TCP ports. NFS formerly used UDP but because of no congestion control use on large networks was ineffective it uses now TCP protocol. It is stateless protocol. NFS is based on client server model. Computer running daemon process (nfsd) is acting as server. Exported directories and their sharing parameters are set in /etc/exports configuration file. Valid client can then mount exported directories and use them as local. NFS can use quotas as do GFS. NFS is easy to configure and easier to use but there is also a few weaknesses arising from it's simplicity. In particular, NFS does not support the same file synchronization semantics known from Linux/Unix processes support. If one process writes to a file, another process reading that file at a later time is guaranteed to see the write. There are no such guarantees in NFS unless particular write caching settings are used. These setting however have negative impact on the performance. There is small number of windows clients implementation and the protocol (NFSv3) is causing caching problems. Security is also weak as it is based on Unix user IDs which could be spoofed. But there are additional way how to secure NFS exports, for instance kerberos.
GFS is GNU/GPL file system developed by Red Hat as a part of Red Hat Cluster Suite. It is used in computer clusters based on Linux, where access to a common storage pool is needed. It allows all computers to have direct access to same block storage concurrently but could be also used as local file system. All nodes work as peers and there is no centralized entity acting as server. For functionality GFS needs to load module providing locking mechanism. For local file system "nolock" module can be used and for clusters Distributed Lock Manager aka DLM. GFS can be deployed in a cluster that is connected to a LAN with servers that use GNBD (Global Network Block Device) or to iSCSI devices. Use of GFS enables easier patching as there is need to patch only once for entire cluster, allows concurrent read/write access by many clients, simplifies backup and disaster recovery. It also eliminates need for redundant copies of application data. Use of GFS avoids using partitions and allows to manage storage as a whole.
GFS/iSCSI is scalable for 300+ clients but for NFS and lot of heavy traffic workloads about 10-20 clients. There are no problems with scalability and complexity of GFS/iSCSI infrastructure because iSCSI provides three options for unique name formats: iSCSI Qualified Name (IQN), Extended Unique Identifier (EUI) and T11 Network Address Authority (NAA). But with NFS at large scale, file systems must be spread across several distinct volumes, increasing management complexity. It's impossible to add more NFS servers to increase the processing power available to serve up a particular file system. In contrast, adding iSCSI servers connected to shared storage via a SAN provide more processing capacity for the clusters using GFS. New storage or servers can be even added on the fly. When used on the small scale basis (5-10 clients in low performance environments) both NFS and GFS/iSCSI perform similarly.
I don't have the resources to make complex benchmarking by myself, so I will just try to describe tools what can be used. Tools emulate different storage access patterns, such as streaming writes and reads (Bonnie), database access (IOgen) small file access (PostMark) and I/O performance (IOzone).
Iozone is useful for performing a broad file system analysis of a vendor's computer platform. The benchmark tests file I/O performance for the following operations: Read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread ,mmap, aio_read, aio_write. It has recently added tests for NFS, CIFS, and distributed/cluster systems.
PostMark benchmark measures the performance of applications that extensively work with small files. Those applications are the Internet related ones such as electronic mail, net news, and web-based commerce. PostMark initially generates a large pool of files ranging in size and then performs a specified number of transactions, where each transaction consists of a pair of small transactions: file creation or deletion and file read or append. PostMark provides a set of results, from which the transaction rate (in transactions per second), and read and write rates (in kilobytes per second) are reported. It can be compiled under Solaris, Digital Unix, and Win32 environments.
IOgen benchmarks measures I/O rates for random reads and writes performed by multiple processes, thus emulating database type access. When IOgen is configured, it is given a file for performing I/O. The ratio of read and write operations can be also specified, as well as a number of processes to perform I/O and a block size for I/O transactions. At the beginning of each run, IOgen creates a specified number of child processes, each of which computes random offset on a given file and then do seek and I/O operation at that offset. At the end of a run, IOgen reports average I/O service and response time, number of I/O per second, and I/O rate.
Bonnie benchmark measures sequential I/O access to a file system as well as random seek rate. The sequential I/O is measured per character and per block. Bonnie also reports CPU utilization for all its measurements. Bonnie initially creates a file of predefined size and then performs sequential write first per character and then per block. After that, Bonnie does sequential read also per character and per block. At last, the benchmark performs random seek operations.
Packets which are going through the network looks quite differently for iSCSI and for NFS. RPC part of NFS packet is replaced with SCSI commands, but performance of both iSCSI and NFS is affected mainly by TCP/IP protocol. Processing needed by CPU thus lower final throughput. Use of iSCSI is the best option in the situations where small file transactions are performed because in this scenario iSCSI highly outperforms NFS. While iSCSI cannot provide shared file access, it is suitable for applications that do not need sharing. This limitation could be avoided when GFS is used on the top of iSCSI. When we look on the comparison of GFS and NFS on clusters GFS provides better scalability and easier maintenance and also significant performance gain especially when running Active/Active cluster.
Tiskni
Sdílej:
10.9.2009 20:03
Salamek | skóre: 22
| blog: salamovo
10.9.2009 20:17
hikikomori82 | skóre: 18
| blog: foobar
| Košice
..Predmet je vypisovaný na Fakulte informatiky MU ..
pro ty, kterym to tak rychle nemysli jedno upozorneni: - ta univerzita je na Morave.
11.9.2009 15:53
belisarivs | skóre: 22
| blog: Psychobláboly
A sup blboune do blokace.
Hmm hadam ze je ten clanek tyka pouze umplementace NFS serveru na linuxu. Pokud by autor srovnaval implementaci iSCSI a NFS treba na NetAppu tak by ty rozdily nebyla az tak markantni.
Ivan
PS: jasne ze je NFS zastaraly dinosaurus, ale je tezke hodnotit technologii podle jedine implementace.
). Docela rad bych videl nasazeni GFS/iSCSI pro hostovani home adresaru v nadnarodni firme o par tisich zamestnanich, kde zamestnanci potrebuji pristup odkudkoliv. Skoda, ze tu vubec neni popsano, odkud si autor vycucal ty cisla 300+ a 10-20, tohle spise vypada na marketingovou prezentaci na tema jak je GFS/iSCSI super.
GFS/iSCSI v podobnej situácii neuvidiš, ale riešenia kde máš diskové pole s GFS pripojené cez FC do 50 serverov (300 by možno, šlo teoreticky, ale imho to nie je oficiálne podporované), ktoré to exportujú cez NFS príp. sambu je dostatok.
ISSN 1214-1267, (c) 1999-2007 Stickfish s.r.o.