ProCKSI

h1. Job Management

The queuing system (resource manager) is the heart of the distributed computing on a cluster. It consists of three parts: the server, the scheduler, and the machine-oriented mini-server (MOM) executing the jobs.

We are assuming the following configuration:

 ||PBS TORQUE|| version 2.1.8           ||server, basic scheduler, mom  ||[source:Externals/Cluster/torque-2.1.8.tgz  download from repository]

 ||MAUI      || version 3.2.6.p18	||scheduler                     ||[source:Externals/Cluster/maui-3.2.6p18.tgz download from repository]

Please check the distributors website's for newer versions:

 ||PBS TORQUE ||http://www.clusterresources.com/pages/products/torque-resource-manager.php

 ||MAUI       ||http://www.clusterresources.com/pages/products/maui-cluster-scheduler.php

The install directories for _TORQUE_ and _MAUI_ will be:

 ||PBS TORQUE ||_/var/spool/torque_

 ||MAUI       ||_/var/spool/maui_

h2. TORQUE

h3. Register new services

Edit _/etc/services_ and add at the end:

<pre>

# PBS/Torque services

pbs           15001/tcp    # pbs_server

pbs           15001/udp    # pbs_server

pbs_mom       15002/tcp    # pbs_mom <-> pbs_server

pbs_mom       15002/udp    # pbs_mom <-> pbs_server

pbs_resmom    15003/tcp    # pbs_mom resource management

pbs_resmom    15003/udp    # pbs_mom resource management

pbs_sched     15004/tcp    # pbs scheduler (pbs_sched)

pbs_sched     15004/udp    # pbs scheduler (pbs_sched)

</pre>

h3. Setup and Configuration on the Master Node

Extract and build the distribution TORQUE on the master node. Configure server, monitor and clients to use secure file transfer (scp).

<pre>

export TORQUECFG=/var/spool/torque

tar -xzvf TORQUE.tar.gz

cd TORQUE

</pre>

Configuration for a 64bit machine with the following compiler options:

<pre>

FFLAGS   = "-m64 -march=[Add Architecture]  -O3 -fPIC"

CFLAGS   = "-m64 -march=[Add Architecture]  -O3 -fPIC"

CXXFLAGS = "-m64 -march=[Add Architecture]  -O3 -fPIC"

LDFLAGS  = "-L/usr/local/lib -L/usr/local/lib64"

</pre>

*Attention*: For Intel Xenon processors use _-march=nocona_, for AMD Opteron processors use _-march=opteron_.

Configure, build, and install:

<pre>

./configure  --prefix=/usr/local --with-spooldir=$TORQUECFG

make

make install 

</pre>

If not configures otherwise, binaries are installed in _/usr/local/bin_ and _/usr/local/sbin_. 

Initialise/configure the queuing system's server daemon (_pbs_server_):

<pre>

pbs_server -t create

</pre>

Set the PBS operator and manager (must be a valid user name). 

<pre>

qmgr

> set server_name = master01.procksi.local

> set server scheduling = true

> set server operators  = "root@master01.procksi.local,procksi@master01.procksi.local"

> set server managers  = "root@master01.procksi.local,procksi@master01.procksi.local"

</pre>

Allow only _procksi_ and _root_ to submit jobs into the queue:

<pre>

> set server acl_users = "root,procksi" 

> set server acl_user_enable = true

</pre>

Set email address for email that is sent by PBS:

<pre>

> set server mail_from = pbs@procksi.net	

</pre>

Allow submissions from slave hosts (only):

*ATTENTION: NEEDS TO BE CHECKED. DOES NOT WORK PROPERLY YET!! *

<pre>

<pre>

> set server allow_node_submit = true

> set server submit_hosts = master01.procksi.local

                            slave01.procksi.local

                            slave02.procksi.local

                            slave03.procksi.local

                            slave04.procksi.local

</pre>

Restrict nodes that can access the PBS server:

<pre>

> set server acl_hosts = master01.procksi.local

                         slave01.procksi.local

                         slave02.procksi.local

                         slave03.procksi.local

                         slave04.procksi.local

> set acl_host_enable = true

</pre>

And set in _torque.cfg_ in order 

to use the internal interface:

<pre>

SERVERHOST              master01.procksi.local

ALLOWCOMPUTEHOSTSUBMIT  true

</pre>

</pre>

Configure default node to be used (see below):

<pre>

> set server default_node = slave

</pre>

Set the default queue to _batch_

<pre>

> set server default_queue=batch

</pre>

Configure the main queue _batch_:

<pre>

> create queue batch queue_type=execution

> set queue batch started=true

> set queue batch enabled=true

> set queue batch resources_default.nodes=1

</pre>

Configure queue _test _accordingly_. 

Specify all compute nodes to be used by creating/editing _$TORQUECFG/server_priv/nodes._ This may include the same machine where pbs_server will run. If the compute nodes have more than one processor, just add np=X after the name with X being the number of processors. Add node attributes so that a subset of nodes can be requested during the submission stage.

<pre>

master01.procksi.local  np=2  procksi  master  xeon

slave01.procksi.local   np=2  procksi  slave   xeon

slave02.procksi.local   np=2  procksi  slave   xeon

slave03.procksi.local   np=4  procksi  slave   opteron

slave04.procksi.local   np=4  procksi  slave   opteron

</pre>

Although the master node (_master01_) has two processors as well, we only allow one processor to be used for the queueing system as the other processor will be used for handling all frontend communication and I/O. (Make sure that hyperthreading technology is disabled on the head node and all compute nodes!)

Request job to be run on specific nodes (on submission):

* Run on any compute node: 	

<pre>

 qsub -q batch -l nodes=1:procksi

</pre>

* Run on any slave node:	

<pre>

 qsub -q batch -l nodes=1:slave

</pre>

* Run on master node:		

<pre>

 qsub -q batch -l nodes=1:master

</pre>

h3. Setup and Configuration on the Slave Nodes

Extract and build the distribution TORQUE on each slave node. Configure monitor and clients to use secure file transfer (scp).

<pre>

export TORQUECFG=/var/spool/torque

tar -xzvf TORQUE.tar.gz

cd TORQUE

</pre>

Configuration for a 64bit machine with the following compiler options:

<pre>

FFLAGS   = "-m64 -march=[Add Architecture] -O3 -fPIC"

CFLAGS   = "-m64 -march=[Add Architecture] -O3 -fPIC"

CXXFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"

LDFLAGS  = "-L/usr/local/lib -L/usr/local/lib64"

</pre>

Attention: For Intel Xenon processors use _-march=nocona_, for AMD Opteron processors use _-march=opteron_.

Configure, build, and install:

<pre>

./configure  --prefix=/usr/local --with-spooldir=$TORQUECFG --disable-server --enable-mom --enable-clients --with-default-server=master01.procksi.local

make

make install 

</pre>

Configure the compute nodes by creating/editing _$TORQUECFG/mom_priv/config_. The first line specifies the PBS server, the second line specifies hosts which can be trusted to access mom services as non-root, and the last line allows copying data via NFS without using SCP.

<pre>

$pbsserver   master01.procksi.local

$loglevel    255

$restricted  master01.procksi.local

$usecp       master01.procksi.local:/home/procksi   /home/procksi

</pre>

Start the queueing system (manually) in the correct order:

* Start the mom:

<pre>

 /usr/local/sbin/pbs_mom

</pre>

* Kill the server:

<pre>

 /usr/local/sbin/qterm -t quick

</pre>

* Start the server:

<pre>

 /usr/local/sbin/pbs_server

</pre>

* Start the scheduler:		

<pre>

 /usr/local/sbin/pbs_sched

</pre>

If you want to use MAUI as the final scheduler, keep in mind to kill _pbs_sched_ after testing the TORQURE installation.

Check that all nodes are properly configured and correctly reporting

<pre>

qstat  -q

pbsnodes -a

</pre>

h3. Prologue and Epilogue Scripts

Get [repos:Externals/procksi_pbs.tgz] from the repository and untar it:

<pre>

untar –xvzf procksi_pbs.tgz

</pre>

The _prologue_ script is executed just before the submitted job starts. Here, it generates a unique temp directory for each job in _/scratch_. 

It must be installed on each NODE (master, slave):

<pre>

cp ./pbs/NODE/var/spool/torque/mom/priv/prologue $TORQUECFG/mom_priv

chmod 500 $TORQUECFG/mom_priv/prologue

</pre>

The _epilogue_ script is executed right after the submitted job has ended. Here, it deletes the job's temp directory from _/scratch._ It must be installed on each NODE (master, slave)

<pre>

cp ./pbs/NODE/var/spool/torque/mom/priv/epilogue $TORQUECFG/mom_priv

chmod 500 $TORQUECFG/mom_priv/epilogue

</pre>

h2. MAUI

h3. Register new services

Edit _/etc/services_ and add at the end:

<pre>

# PBS/MAUI services

pbs_maui  42559/tcp    # pbs scheduler (maui)

pbs_maui  42559/udp    # pbs scheduler (maui)

</pre>

h3. Setup and Configuration on the Head Node

Extract and build the distribution MAUI.

<pre>

export MAUIDIR=/var/spool/maui

tar -xzvf MAUI.tar.gz

cd TORQUE

</pre>

Configuration for a 64bit machine with the following compiler options:

<pre>

FFLAGS   = “-m64 -march=[Add Architecture] -O3 -fPIC"

CFLAGS   = “-m64 -march=[Add Architecture] -O3 -fPIC"

CXXFLAGS = “-m64 -march=[Add Architecture] -O3 -fPIC"

LDFLAGS	 = “-L/usr/local/lib -L/usr/local/lib64"

</pre>

*Attention*: For Intel Xenon processors use _-march=nocona_, for AMD Opteron processors use _-march=opteron_.

Configure, build, and install:

<pre>

./configure --with-pbs=$TORQUECFG --with-spooldir=$MAUIDIR

make

make install 

</pre>

Fine-tune MAUI in $_MAUIDIR/maui.cfg_:

<pre>

SERVERHOST            master01.procksi.local

# primary admin must be first in list

ADMIN1                procksi

ADMIN1                root

# Resource Manager Definition

RMCFG[MASTER01.PROCKSI.LOCAL]

]		

TYPE=PBS@RMNHOST@ 			

PORT=15001

EPORT=15004	[CAN BE ALTERNATIVELY: 15017 - TRY!!!]

SERVERPORT  42559

SERVERMODE  NORMAL

# Node Allocation:

# JOBCOUNT  number of jobs currently running on node

# LOAD      current 1 minute load average

# AMEM      real memory currently available to batch jobs

# APROCS    processors currently available to batch jobs

# PREF      node meets job specific resource preferences

NODEALLOCATIONPOLICY  PRIORITY

NODECFG[DEFAULT] PRIORITYF='-JOBCOUNT - 2*LOAD + 0.5*AMEM + 0.25*APROCS + PREF'

</pre>

Start the MAUI scheduler manually. Make sure that pbs_sched is not running any longer.

* Start the scheduler:

<pre>

 /usr/local/sbin/maui

</pre>

Get [repos:Externals/Cluster/procksi_pbs.tgz] from the repository and untar it:

<pre>

untar –xvzf procksi_pbs.tgz

</pre>

Make the entire queuing system (Torque + Maui) start at bootup:

<pre>

cp ./pbs/master/etc/init.d/pbs_* /etc/init.d/

/sbin/chkconfig  --add pbs_mom

/sbin/chkconfig  --add pbs_maui

/sbin/chkconfig  --add pbs_server

/sbin/chkconfig  pbs_mom  on

/sbin/chkconfig  pbs_maui  on

/sbin/chkconfig  pbs_server  on

</pre>

If you want to use the simple scheduler that comes with PBS Torque, then substitute _pbs_maui_ with _pbs_sched_.

h3. Setup and Configuration on the Slave Nodes

Get [repos:Externals/Cluster/procksi_pbs.tgz] from the repository and untar it:

<pre>

untar –xvzf procksi_pbs.tgz

</pre>

Make the entire queuing system start at bootup:

<pre>

cp ./pbs/slave/etc/init.d/pbs_mom /etc/init.d/

/sbin/chkconfig  --add pbs_mom

/sbin/chkconfig  pbs_mom  on

</pre>

h3. Monitoring Grid Status

* display queue information (active/idle jobs)

<pre>

 showq

</pre>

* current and historical scheduling statistics

<pre>

 showstats -v

</pre>

* display job state and resources information

<pre>

 checkjob <JOB_ID>

</pre>

* display node state and resources information

<pre>

 checknode <NODE_NAME>

</pre>