ProCKSI

= Job Management =
The queuing system (resource manager) is the heart of the distributed computing on a cluster. It consists of three parts: the server, the scheduler, and the machine-oriented mini-server (MOM) executing the jobs.

We are assuming the following configuration:

||PBS TORQUE|| version 2.1.8 ||server, basic scheduler, mom ||[source:Externals/Cluster/torque-2.1.8.tgz download from repository]
||MAUI || version 3.2.6.p18 ||scheduler ||[source:Externals/Cluster/maui-3.2.6p18.tgz download from repository]

Please check the distributors website's for newer versions:

||PBS TORQUE ||http://www.clusterresources.com/pages/products/torque-resource-manager.php
||MAUI ||http://www.clusterresources.com/pages/products/maui-cluster-scheduler.php


The install directories for ''TORQUE'' and ''MAUI'' will be:

||PBS TORQUE ||''/var/spool/torque''
||MAUI ||''/var/spool/maui''


== TORQUE ==

=== Register new services ===
Edit ''/etc/services'' and add at the end:
{{{
# PBS/Torque services
pbs 15001/tcp # pbs_server
pbs 15001/udp # pbs_server
pbs_mom 15002/tcp # pbs_mom <-> pbs_server
pbs_mom 15002/udp # pbs_mom <-> pbs_server
pbs_resmom 15003/tcp # pbs_mom resource management
pbs_resmom 15003/udp # pbs_mom resource management
pbs_sched 15004/tcp # pbs scheduler (pbs_sched)
pbs_sched 15004/udp # pbs scheduler (pbs_sched)
}}}


=== Setup and Configuration on the Master Node ===
Extract and build the distribution TORQUE on the master node. Configure server, monitor and clients to use secure file transfer (scp).
{{{
export TORQUECFG=/var/spool/torque
tar -xzvf TORQUE.tar.gz
cd TORQUE
}}}

Configuration for a 64bit machine with the following compiler options:
{{{
FFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"
CFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"
CXXFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"
LDFLAGS = "-L/usr/local/lib -L/usr/local/lib64"
}}}
'''Attention''': For Intel Xenon processors use ''-march=nocona'', for AMD Opteron processors use ''-march=opteron''.

Configure, build, and install:
{{{
./configure --prefix=/usr/local --with-spooldir=$TORQUECFG
make
make install
}}}
If not configures otherwise, binaries are installed in ''/usr/local/bin'' and ''/usr/local/sbin''.

Initialise/configure the queuing system's server daemon (''pbs_server''):
{{{
pbs_server -t create
}}}

Set the PBS operator and manager (must be a valid user name).
{{{
qmgr
> set server_name = master01.procksi.local
> set server scheduling = true
> set server operators = "root@master01.procksi.local,procksi@master01.procksi.local"
> set server managers = "root@master01.procksi.local,procksi@master01.procksi.local"
}}}

Allow only ''procksi'' and ''root'' to submit jobs into the queue:
{{{
> set server acl_users = "root,procksi"
> set server acl_user_enable = true
}}}

Set email address for email that is sent by PBS:
{{{
> set server mail_from = pbs@procksi.net
}}}

Allow submissions from slave hosts (only):
'''ATTENTION: NEEDS TO BE CHECKED. DOES NOT WORK PROPERLY YET!! '''
{{{
{{{
> set server allow_node_submit = true
> set server submit_hosts = master01.procksi.local
slave01.procksi.local
slave02.procksi.local
slave03.procksi.local
slave04.procksi.local
}}}


Restrict nodes that can access the PBS server:
{{{
> set server acl_hosts = master01.procksi.local
slave01.procksi.local
slave02.procksi.local
slave03.procksi.local
slave04.procksi.local
> set acl_host_enable = true
}}}

And set in ''torque.cfg'' in order
to use the internal interface:
{{{
SERVERHOST master01.procksi.local
ALLOWCOMPUTEHOSTSUBMIT true
}}}
}}}

Configure default node to be used (see below):
{{{
> set server default_node = slave
}}}

Set the default queue to ''batch''
{{{
> set server default_queue=batch
}}}

Configure the main queue ''batch'':
{{{
> create queue batch queue_type=execution
> set queue batch started=true
> set queue batch enabled=true
> set queue batch resources_default.nodes=1
}}}

Configure queue ''test ''accordingly''.

Specify all compute nodes to be used by creating/editing ''$TORQUECFG/server_priv/nodes.'' This may include the same machine where pbs_server will run. If the compute nodes have more than one processor, just add np=X after the name with X being the number of processors. Add node attributes so that a subset of nodes can be requested during the submission stage.
{{{
master01.procksi.local np=2 procksi master xeon
slave01.procksi.local np=2 procksi slave xeon
slave02.procksi.local np=2 procksi slave xeon
slave03.procksi.local np=4 procksi slave opteron
slave04.procksi.local np=4 procksi slave opteron
}}}

Although the master node (''master01'') has two processors as well, we only allow one processor to be used for the queueing system as the other processor will be used for handling all frontend communication and I/O. (Make sure that hyperthreading technology is disabled on the head node and all compute nodes!)

Request job to be run on specific nodes (on submission):
* Run on any compute node:
{{{
qsub -q batch -l nodes=1:procksi
}}}
* Run on any slave node:
{{{
qsub -q batch -l nodes=1:slave
}}}
* Run on master node:
{{{
qsub -q batch -l nodes=1:master
}}}



=== Setup and Configuration on the Slave Nodes ===
Extract and build the distribution TORQUE on each slave node. Configure monitor and clients to use secure file transfer (scp).
{{{
export TORQUECFG=/var/spool/torque
tar -xzvf TORQUE.tar.gz
cd TORQUE
}}}

Configuration for a 64bit machine with the following compiler options:
{{{
FFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"
CFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"
CXXFLAGS = "-m64 -march=[Add Architecture] -O3 -fPIC"
LDFLAGS = "-L/usr/local/lib -L/usr/local/lib64"
}}}
Attention: For Intel Xenon processors use ''-march=nocona'', for AMD Opteron processors use ''-march=opteron''.

Configure, build, and install:
{{{
./configure --prefix=/usr/local --with-spooldir=$TORQUECFG --disable-server --enable-mom --enable-clients --with-default-server=master01.procksi.local
make
make install
}}}

Configure the compute nodes by creating/editing ''$TORQUECFG/mom_priv/config''. The first line specifies the PBS server, the second line specifies hosts which can be trusted to access mom services as non-root, and the last line allows copying data via NFS without using SCP.
{{{
$pbsserver master01.procksi.local
$loglevel 255
$restricted master01.procksi.local
$usecp master01.procksi.local:/home/procksi /home/procksi
}}}

Start the queueing system (manually) in the correct order:
* Start the mom:
{{{
/usr/local/sbin/pbs_mom
}}}
* Kill the server:
{{{
/usr/local/sbin/qterm -t quick
}}}
* Start the server:
{{{
/usr/local/sbin/pbs_server
}}}
* Start the scheduler:
{{{
/usr/local/sbin/pbs_sched
}}}

If you want to use MAUI as the final scheduler, keep in mind to kill ''pbs_sched'' after testing the TORQURE installation.

Check that all nodes are properly configured and correctly reporting
{{{
qstat -q
pbsnodes -a
}}}

=== Prologue and Epilogue Scripts ===
Get [repos:Externals/procksi_pbs.tgz] from the repository and untar it:
{{{
untar –xvzf procksi_pbs.tgz
}}}

The ''prologue'' script is executed just before the submitted job starts. Here, it generates a unique temp directory for each job in ''/scratch''.
It must be installed on each NODE (master, slave):
{{{
cp ./pbs/NODE/var/spool/torque/mom/priv/prologue $TORQUECFG/mom_priv
chmod 500 $TORQUECFG/mom_priv/prologue
}}}

The ''epilogue'' script is executed right after the submitted job has ended. Here, it deletes the job's temp directory from ''/scratch.'' It must be installed on each NODE (master, slave)
{{{
cp ./pbs/NODE/var/spool/torque/mom/priv/epilogue $TORQUECFG/mom_priv
chmod 500 $TORQUECFG/mom_priv/epilogue
}}}

== MAUI ==

=== Register new services ===
Edit ''/etc/services'' and add at the end:
{{{
# PBS/MAUI services
pbs_maui 42559/tcp # pbs scheduler (maui)
pbs_maui 42559/udp # pbs scheduler (maui)
}}}

=== Setup and Configuration on the Head Node ===
Extract and build the distribution MAUI.
{{{
export MAUIDIR=/var/spool/maui
tar -xzvf MAUI.tar.gz
cd TORQUE
}}}

Configuration for a 64bit machine with the following compiler options:
{{{
FFLAGS = “-m64 -march=[Add Architecture] -O3 -fPIC"
CFLAGS = “-m64 -march=[Add Architecture] -O3 -fPIC"
CXXFLAGS = “-m64 -march=[Add Architecture] -O3 -fPIC"
LDFLAGS = “-L/usr/local/lib -L/usr/local/lib64"
}}}
'''Attention''': For Intel Xenon processors use ''-march=nocona'', for AMD Opteron processors use ''-march=opteron''.

Configure, build, and install:
{{{
./configure --with-pbs=$TORQUECFG --with-spooldir=$MAUIDIR
make
make install
}}}

Fine-tune MAUI in $''MAUIDIR/maui.cfg'':
{{{
SERVERHOST master01.procksi.local

# primary admin must be first in list
ADMIN1 procksi
ADMIN1 root

# Resource Manager Definition
RMCFG[MASTER01.PROCKSI.LOCAL]
]
TYPE=PBS@RMNHOST@
PORT=15001
EPORT=15004 [CAN BE ALTERNATIVELY: 15017 - TRY!!!]

SERVERPORT 42559
SERVERMODE NORMAL

# Node Allocation:
# JOBCOUNT number of jobs currently running on node
# LOAD current 1 minute load average
# AMEM real memory currently available to batch jobs
# APROCS processors currently available to batch jobs
# PREF node meets job specific resource preferences

NODEALLOCATIONPOLICY PRIORITY
NODECFG[DEFAULT] PRIORITYF='-JOBCOUNT - 2*LOAD + 0.5*AMEM + 0.25*APROCS + PREF' PRIORITY=&#x27;- JOBCOUNT&#x27;
}}}

Start the MAUI scheduler manually. Make sure that pbs_sched is not running any longer.

* Start the scheduler:
{{{
/usr/local/sbin/maui
}}}


Get [repos:Externals/Cluster/procksi_pbs.tgz] from the repository and untar it:
{{{
untar –xvzf procksi_pbs.tgz
}}}

Make the entire queuing system (Torque + Maui) start at bootup:
{{{
cp ./pbs/master/etc/init.d/pbs_* /etc/init.d/
/sbin/chkconfig --add pbs_mom
/sbin/chkconfig --add pbs_maui
/sbin/chkconfig --add pbs_server
/sbin/chkconfig pbs_mom on
/sbin/chkconfig pbs_maui on
/sbin/chkconfig pbs_server on
}}}

If you want to use the simple scheduler that comes with PBS Torque, then substitute ''pbs_maui'' with ''pbs_sched''.

=== Setup and Configuration on the Slave Nodes ===
Get [repos:Externals/Cluster/procksi_pbs.tgz] from the repository and untar it:
{{{
untar –xvzf procksi_pbs.tgz
}}}

Make the entire queuing system start at bootup:
{{{
cp ./pbs/slave/etc/init.d/pbs_mom /etc/init.d/
/sbin/chkconfig --add pbs_mom
/sbin/chkconfig pbs_mom on
}}}