ProCKSI

= Data Storage =

This page describes the design of the database that is/will be used in order to store all necessary pieces information that are obtained from the "stand-alone" ProCKSI/Comparison and ProCKSI/Consensus applications ProCKSI ''core'' application (see [wiki:DataStandardisation]).

== Database Design for the (static) Protein Multiverse ==
The database stores results from ''Transformations'', ''Comparisons'' and ''Compositions'':
* A ''Transformation'' is a process that derives ONE (main) ''Result'' from ONE single input file.[[br]]
__Example__: The transformation of ''Structure'', ''Tree'', ''!SimilarityMatrix'', etc., using a certain ''Method'' with a certain ''!ParameterSet'', produces a contact map, a tree, ...
* A ''Comparison'' is a process that derives ONE (main) ''Result'' from TWO input files. [[br]]
__Example__ The comparison of ''Structures'', ''Trees'', etc., using a ''Method'' with a certain ''!ParameterSet'', produces a similarity value and an alignment
* An ''Aggregation'' is a process that derive ONE (main) ''Result'' from SEVERAL input files that are grouped together into ''!DataSets''. [[br]]
__Example__ The aggregation of ''!SimilarityMatrices'', ''Trees'', using a ''Method'' with a certain ''!ParameterSet'', produces a consensus similarity matrix, a consensus tree, ...

[[Image(ProteinMultiverseDataBase6.png)]]

* There are multiple (similarity comparison) ''Methods'': e.g. USM, MaxCMO, !DaliLite, ...
* Each ''Method'' is executed with a specific ''!ParameterSet'', which is a combination of different ''Parameters'' with its values: e.g. MaxCMO/restarts/10, USM/compressor/bzip2, ...
* If a ''Method'' does not accept any ''Parameters'', the ''!ParameterSet'' does exist but is empty; e.g. !DaliLite, CE, ...
* Each ''Method'' procudes multiple similarity ''Measures'': e.g. !DaliLite/Z, FAST/Z, MaxCMO/Overlap, ...

* Each ''Structure'' is uniquely determined by its PDB code, model and chain. (Domains are not taken into accout yet.) The location of the PDB file is given and a link to a further ''Container'' file that holds further information in XML format: e.g. sequence, secondary structure, experimental resolution, ...
* Each ''Structure'' is extended by further classifiction information from ''CATH'' and ''SCOP'' in separtate relations.
* Multiple ''Structures'' can be grouped together into ''!DataSets'', which are needed for ''Aggregations''.

* The location of the ''Containers'' in which results are stored can be found in the ''Transformations'', ''Comparisons'', and ''Aggregations'' relations, respectively.
* Additionally, similarity values from ''Comparisons'' are stored directly in the database for quicker access. Alignments could be accessed in the same way, as soon as a standardised format has been defined.

Note that this design does not allow ''Datasets'' to comprise other files than ''Structures'' although some of the ''Results'' need to be grouped into a ''!DataSet'', too.[[br]]
__Example__ Contact maps that have been produces by a ''Transformation'' of ''Structures'' and that are available from within the ''Containers'' need to form a ''!DataSet'' in order to act as input for the ''Comparisons'' with the USM or MaxCMO ''Methods''.

=== Storing Further Information and Results externally ===
Similarity values are stored directly in the relational database. All further information regarding one structure (e.g. sequence, resolution, ...) or regarding a pair of structures (e.g. alignment, rotation/translation matrices, ...) are stored in external files.[[br]]
For storing further information for ''single structures'', there are several approaches:
* All information in one file: file too big
* All information in separate files grouped by the protein structure

For storing further information for ''pairs of structures'', there are several approaches:
* All information in separate files grouped by methods: files too big
* All information in separate files grouped by pairs: too many files
* All information in separate files grouped by the the first structure: files with unbalanced sizes
* All information in separate files with fixed size:[[br]]
"Bin-packing" algorithm decides where to put new information, and opens a new "bin" if necessary. "Bins" must be balanced from time to time in order to provide a fast retrieval of information.

== Extended Database Design for (dynamic) Management of Experiments (ProCKSI) ==

This has not been modelled yet, but the database for the (static) Protein Multiverse was designed with the ProCKSI integration in mind.

Some remarks:
* ''Experiments'' (formerly ''Requests'') apply ''Methods'' to''!DataSet'' with a certain ''!ParameterSet''.
* ''Packages'' (formerly ''Jobs'') deal with a subset of a ''!DataSet'' and a subset of the requested ''Methods'', partitioning the the 3D problem space, and are calculated using the ProCKSI's "stand-alone" core application "in one go". If they are sent to a queuing system, they become a ''Job'' there.
* It has to be discussed if there is still the need of a ''Tasks'' relation in the database, which have always been rather ''!RequestMethods''.