ATOL: Resource Agents

7.6.2009 19:17 | Přečteno: 1063× | Linux | poslední úprava: 7.6.2009 19:57

Author: Petr Bartel
Clustering is technique to interconnect two and more computers (devices, nodes) to be able to provide added value. This added value depends on what type of cluster do you want to deploy.

There is a few types. Let's have a look at them. High-availability cluster is the first of them. This type of cluster takes care about availability of services or storage for users (other devices or services). Another example of cluster solution is a load-balancing cluster. We have some service or resource and we have large community of users using it. So one machine would be little and we have to deploy more machines. This type of cluster ensures splitting of the load among available nodes in the most fair way. We can name computing cluster, which focus on difficult and time consuming computing problems or GRIDs.

As I wrote a few lines higher, availability of resources are very important in high-availability clusters. Resource can be part of hardware, such a hard drive or partition, IP address or whole service, such as apache web server or postfix mail server.

In this type of cluster needn't be all the nodes same. We can deploy different hardware for every node and run different services on each of them. There has to be some mechanism how to stop using of resource on one node and start it on another one. Of course, it can be done automatically in the case of failure of service or even of hardware (node itself). That is the main responsibility of Resource agent.

So, resource agent is an information about specific resource and can work with it. Although the resource itself is very specific, so the resource agent input (or interface between resource and resource agent) is specific and tailored too. On the other hand, the output (interface between resource agent and resource manager) should be generic enough. Resource agents are the layer between resource and resource manager, so that resource manager can use some generic routines to control various resources.

It is very important to keep some rules in creating resource agents. LSB is one of the standard matters. It determines some path and placement of important parts of ordinary Linux distribution. The usual resource agent is written in shell code and has structure similar to LSB init scripts. (Most of the nowadays Linux distributions use System V init scripts)

There is a definition of resource agent by OCF (Open cluster framework). This definition is used by Redhat Cluster Suite, Pacemaker and others. Although, the placement of resource agents differs, the important part and structure remain the same. Let's introduce OCF resource agent.

OCF resource agents stands on LSB resource agent. The original LSB resource agent (something like init script) provides functionality of

start	The service isn't running. Check whether service did start or not. The service is running, so ensure yourself service is still running.
stop	The service is running. Check whether service did stop or not. The service isn't running, so ensure yourself service is still stopped.
status	Should accept this action and return state of script (stopped or running).

All commands given to the script can be tested automatically and all of them print their result after finish their work. All scripts should be placed in /etc/init.d/ and return specified return value (exit code). This is definition on LSB resource agent v.1 in short.

OCF resource agent specifies more actions

monitor	this function replaces status from LSB v.1. It should test state of the service at the localhost and return 0 (in case the service is running), 7 for stopped service or whatever else for another state.
meta-data	XML snippet is returned as an output and this action ends with 0 return value.

Actions start, stop, monitor and meta-data are required. There is one more (optional) action. It's name is validate-all

validate-all	is an action which can check given configuration parameters and ends with exit code according to it.
reload	usually not implemented, reload configuration and doesn't stop the service
recover	usually not implemented, similar to start action

Some resource agents can take additional actions not included in OCF specs. Among these actions belong promote, demote (to change state of service from/to master/slave state on local machine) and notify (to send notifications to services).

Beside action OCF resource agent can react to parameters. These parameters serves to distinguish specific instance of resource. Each of them has to have prefix OCF_RESKEY_ . We can call these instance parameters. We should try to minimize set of them. There is an option to mark some of them as unique. Each of them has its own name and value (which has to be according to POSIX definition and it is case sensitive).

There is a set of other global OCF parameters. OCF_RA_VERSION_MAJOR and OCF_RA_VERSION_MINOR (versioning major.minor). Another parametr -– OCF_ROOT -- is for specification of root of OCF hierarchy. OCF_RESOURCE_INSTANCE and OCF_RESOURCE_TYPE refer to name and type of service. You can specify behaviour of action monitor by its parameter OCF_CHECK_LEVEL.

As I said earlier, every action has got appropriate reaction. The reaction is an exit code. In most cases, 0 means everything was OK and others values indicate some troubles. We can introduce some most important or usual.

0	see the previous paragraph
2	wrong parameters
3	indicates not implemented feature
4	tells us about insufficient rights
5	not installed
6	not configured
7	not running

most of the rest is reserved for future use.

Finally, if you want to test your resource agent for its correctness and fulfilling OCF specs, you can you use osf-tester script (you can find it in heartbeat package). That is all for defining generic OCF resource agent. Now we can have a look at the way of implementing and maintaining it in Redhat Cluster Suite.

Redhat Cluster Suite provides a resource manager. Its name is rgmanager (rg probably as resource group). Its function is to take care about running services (resources) and theirs state (health). So if some node dies, rgmanager move the service to another healthy node. In this way, users usually aren't able to notice a failure of service. If the service doesn't run, rgmanager tries to restart it first. We can define group of nodes which are suitable to run that service (for instance a web server) and in the case of failure, one of these nodes is chosen (this group is called failover domain). Of course, we are able to set more than one service group. At least one half of nodes in service group have to be alive (active, running).

It is possible to group services and resources. Parameter called 'depth' has got similar purpose as OCF_CHECK_LEVEL in OCF specs. If you want to find where is your service running, you can use clustat or GUI application. There is a tool for validation of resource collision, its name is rg_test and you can invoke it by rg_test name /etc/cluster/cluster.conf (which is main configuration file for Redhat Cluster Suite).

Examples and templates of resource agents can be found at /usr/share/cluster/doc/services/examples. Now let's go through one of the real resource agent script in /usr/share/cluster/. Let's choose tomcat-5.sh and its metadata in tomcat-5.metadata.

It's shell script and we can notice hash bang of Bash. Script declare some local variables and set language (locale) setting to standard C. Script supports start, stop, monitor (and status), restart, metadata and verify-all actions. That should mean it is compatible with LSB. Every action starts with verify_all function. With command clog_service_verify checks setting of all variables and theirs placement. If anything goes wrong, the resource agent ends with specific exit code.

Start action create config and its directory, PID, IP addresses and a few other parameters. Finally it will start the tomcat service with the defined parameters and check if it was successful. Stop action tries to stop it as a generic service and wait for its shutdown. Action status just check for its PID file and if it doesn't find it will exit with some code. Remaining actions meta-data, just print .metadata file to the output) and restart calls stop and start in succession.

New service can be added with "cluadmin service add". You will need a service script, information such as service name, IP address, type of relocation policy, script location, shared disk partition and so on. After its creation you can display resource configuration with "cluadmin show config relocate to another node and delete resource (remove the script and config). Redhat Cluster Suite (also cluadmin and clustat) is available as package group "Clustering".

For further information please visit the links bellow. For advanced users, experts and developers: I am just a newbie and user of Redhat Cluster Suite and this is my point view, which I got from materials mentioned in following links. So be indulgent and if there is some mistakes or inaccuracy don't hesitate to correct me.

Resources:

• DRAFT:Open Clustering Framework Resource Agent API
• Pacemaker
• Linux-HA
• Red Hat Cluster Manager / Service Configuration

       

Komentáře

[Michal Fecko]

OMFG

[AsciiWolf]

?

[vogo]

OMN :)

[D-Evil]

Obsahově dobrý, ale ta angličtina nic moc...

[petr_p]

Opravdu POSIX definuje System V init skripty? Zajímavější by bylo popsat, jak funguje rgmanager. Tohle povídání mi přišlo takové dlouhé a nudné. Akorát mě pobavila věta o tom, že agenti jsou bashové skripty a skutečně začínají #! :D