ProCKSI

h1. Data Storage

This page describes the design of the database that is/will be used in order to store all necessary pieces information that are obtained from the "stand-alone" ProCKSI/Comparison and ProCKSI/Consensus applications (see [[DataStandardisation]]).

h2. Database Design for the (static) Protein Multiverse

The database stores results from _Transformations_, _Comparisons_ and _Compositions_:

* A _Transformation_ is a process that derives ONE (main) _Result_ from ONE single input file.

  +Example+: The transformation of _Structure_, _Tree_, _SimilarityMatrix_, etc., using a certain _Method_ with a certain _ParameterSet_, produces a contact map, a tree, ...

* A _Comparison_ is a process that derives ONE (main) _Result_ from TWO input files. 

  +Example+: The comparison of _Structures_, _Trees_, etc., using a _Method_ with a certain _ParameterSet_, produces a similarity value and an alignment

* An _Aggregation_ is a process that derive ONE (main) _Result_ from SEVERAL input files that are grouped together into  _DataSets_. 

  +Example+: The aggregation of _SimilarityMatrices_, _Trees_, using a _Method_ with a certain _ParameterSet_, produces a consensus similarity matrix, a consensus tree, ...

!ProteinMultiverseDataBase-6.png!

* There are multiple (similarity comparison) _Methods_: e.g. USM, MaxCMO, DaliLite, ...

* Each _Method_ is executed with a specific _ParameterSet_, which is a combination of different _Parameters_ with its values: e.g. MaxCMO/restarts/10, USM/compressor/bzip2, ... 

* If a _Method_ does not accept any _Parameters_, the _ParameterSet_ does exist but is empty; e.g. DaliLite, CE, ...

* Each _Method_ procudes multiple similarity _Measures_: e.g. DaliLite/Z, FAST/Z, MaxCMO/Overlap, ...

* Each _Structure_ is uniquely determined by its PDB code, model and chain. (Domains are not taken into accout yet.) The location of the PDB file is given and a link to a further _Container_ file that holds further information in XML format: e.g. sequence, secondary structure, experimental resolution, ...

* Each _Structure_ is extended by further classifiction information from _CATH_ and _SCOP_ in separtate relations.

* Multiple _Structures_ can be grouped together into _DataSets_, which are needed for _Aggregations_.

* The location of the _Containers_ in which results are stored can be found in the _Transformations_, _Comparisons_, and _Aggregations_ relations, respectively. 

* Additionally, similarity values from _Comparisons_ are stored directly in the database for quicker access. Alignments could be accessed in the same way, as soon as a standardised format has been defined.

Note that this design does not allow _Datasets_ to comprise other files than _Structures_ although some of the _Results_ need to be grouped into a _DataSet_, too.

+Example+: Contact maps that have been produces by a _Transformation_ of _Structures_ and that are available from within the _Containers_ need to form a _DataSet_ in order to act as input for the _Comparisons_ with the USM or MaxCMO _Methods_.

h3. Storing Further Information and Results externally

Similarity values are stored directly in the relational database. All further information regarding one structure (e.g. sequence, resolution, ...) or regarding a pair of structures (e.g. alignment, rotation/translation matrices, ...) are stored in external files.

For storing further information for _single structures_, there are several approaches:

* All information in one file: file too big

* All information in separate files grouped by the protein structure

For storing further information for _pairs of structures_, there are several approaches:

* All information in separate files grouped by methods: files too big

* All information in separate files grouped by pairs: too many files

* All information in separate files grouped by the the first structure: files with unbalanced sizes

* All information in separate files with fixed size:

  "Bin-packing" algorithm decides where to put new information, and opens a new "bin" if necessary. "Bins" must be balanced from time to time in order to provide a fast retrieval of information.

h2. Extended Database Design for (dynamic) Management of Experiments (ProCKSI)

This has not been modelled yet, but the database for the (static) Protein Multiverse was designed with the ProCKSI integration in mind.

Some remarks:

* _Experiments_ (formerly _Requests_) apply _Methods_ to_DataSet_ with a certain _ParameterSet_.

* _Packages_ (formerly _Jobs_) deal with a subset of a _DataSet_ and a subset of the requested _Methods_, partitioning the the 3D problem space, and are calculated using the ProCKSI's "stand-alone" core application "in one go". If they are sent to a queuing system, they become a _Job_ there.

* It has to be discussed if there is still the need of a _Tasks_ relation in the database, which have always been rather _!RequestMethods_.