ProCKSI

= Data Storage =

This page describes the design of the database that is/will be used in order to store all necessary pieces information that are obtained from the "stand-alone" ProCKSI ''core'' application (see [wiki:DataStandardisation]).

== Database Design for the (static) Protein Multiverse ==
The [[Image(ProteinMultiverseDataBase.png)]]

'''Explanation of the database stores results from ''Transformations'', ''Comparisons'' and ''Compositions'': design''':
* A ''Transformation'' is a process that derives ONE (main) ''Result'' from ONE single input file.[[br]]
__Example__: The transformation of ''Structure'', ''Tree'', ''!SimilarityMatrix'', etc., using a certain ''Method'' with a certain ''!ParameterSet'', produces a contact map, a tree, ...
* A ''Comparison'' is a process that derives ONE (main) ''Result'' from TWO input files. [[br]]
__Example__ The comparison of ''Structures'', ''Trees'', etc., using a ''Method'' with a certain ''!ParameterSet'', produces a similarity value and an alignment
* An ''Aggregation'' is a process that derive ONE (main) ''Result'' from SEVERAL input files that are grouped together into ''!DataSets''. [[br]]
__Example__ The aggregation of ''!SimilarityMatrices'', ''Trees'', using a ''Method'' with a certain ''!ParameterSet'', produces a consensus similarity matrix, a consensus tree, ...

[[Image(ProteinMultiverseDataBase6.png)]]

* There are multiple (similarity comparison) similarity comparison ''Methods'': e.g. USM, MaxCMO, !DaliLite, ...
* Each ''Method'' is executed with a specific ''!ParameterSet'', which is a combination There are multiple similarity ''Measures'': e.g. Z-score, TM-score, Number of Alignments, ...
* Some different ''Parameters'' ''Methods'' produce ''Measures'' with its values: the same name, but not necessarily the same meaning: e.g. MaxCMO/restarts/10, USM/compressor/bzip2, ...
!DaliLite/Z, TMalign/Z, ...[[br]]
Thus, a ''!MethodMeasures'' relation is necessary.

* If a Each ''Method'' does not accept any ''Parameters'', the ''!ParameterSet'' does exist but is empty; can have multiple (different) ''Parameters'': e.g. !DaliLite, CE, USM/Compressor, USM/Equation, ...
* Each ''Method'' procudes can have multiple similarity ''Measures'': e.g. !DaliLite/Z, FAST/Z, MaxCMO/Overlap, (different) ''!ParameterOptions'': USM/Compressor/bzip, USM/Compressor/gzip, ...
* A "!ParameterSet" is used to calculate the ''Similarity'' of ''!StructurePairs''. It is a collection of specific ''!ParameterSetOptions''. [[br]]

* The ''!StructurePairs'' relation holds all possible combinations of ''Structures'', and a link to a further ''Results'' file in XML format. This file may contain results for multiple ''!StructurePairs'', e.g. alignments, matrices, etc.
* Each ''Structure'' is uniquely determined by its PDB code, model and chain. (Domains are not taken into accout yet.) The location of the PDB file is given and a link to a further ''Container'' ''Results'' file that holds further information in XML format: format. This file may contain additional information for multiple ''Structures'', e.g. sequence, secondary structure, experimental resolution, ...


* Each ''Structure'' is extended by further classifiction information from ''CATH'' and ''SCOP'' in separtate relations.
* Multiple ''Structures'' can be grouped together into ''!DataSets'', which are needed for ''Aggregations''.

* The location of the ''Containers'' in which results are stored can be found in the ''Transformations'', ''Comparisons'', and ''Aggregations'' relations, respectively.
* Additionally, similarity values from ''Comparisons'' are stored directly in the database for quicker access. Alignments could be accessed in the same way, as soon as a standardised format has been defined.

Note that this design does not allow ''Datasets'' to comprise other files than ''Structures'' although some of the ''Results'' need to be grouped into a ''!DataSet'', too.[[br]]
__Example__ Contact maps that have been produces by a ''Transformation'' of ''Structures'' and that are available from within the ''Containers'' need to form a ''!DataSet'' in order to act as input for the ''Comparisons'' with the USM or MaxCMO ''Methods''.

''SCOP''.

=== Storing Further Information and Results externally ===
Similarity values are stored directly in the relational database. All further information regarding one structure (e.g. sequence, resolution, ...) or regarding a pair of structures (e.g. alignment, rotation/translation matrices, ...) are stored in external files.[[br]]
For storing further information for ''single structures'', there are several approaches:
* All information in one file: file too big
* All information in separate files grouped by the protein structure

For storing further information for ''pairs of structures'', there are several approaches:
* All information in separate files grouped by methods: files too big
* All information in separate files grouped by pairs: too many files
* All information in separate files grouped by the the first structure: files with unbalanced sizes
* All information in separate files with fixed size:[[br]]
"Bin-packing" algorithm decides where to put new information, and opens a new "bin" if necessary. "Bins" must be balanced from time to time in order to provide a fast retrieval of information.

=== Comments dxb: ===
* If a ''Method'' does not use any parameters, it is not included in the ''!ParameterSet'', but accessible via the ''!MethodMeasure'' relation. [[br]]
Alternatively, such methods could have a ''Parameter'' "none" with an ''!ParameterOption'' "none" so that the ''!ParameterSet'' would always include all possible ''Methods''. [[br]]
* It can be argued that there is another strong entity ''Options'' needed that holds only possible values and their description, e.g. "CoM" and "Centre of Mass". Similar to "!MethodMeasures", the ''!ParameterOption'' would only hold combinations of ''Parameters'' with ''Options''.
* It can be further argued that there is no list that describes what further information can be found in the external files.

== Extended Database Design for the (static) Protein Multiverse ==

[[Image(ProteinMultiverseDataBaseExt4.png)]]

This proposal for an extended database design for the (static) Protein Multiverse aims to include not only ''Comparisons'' but also ''Transformations'' and ''Compositions'' (following the latest development of the I/O specificaitions for the ProCKSI "stand-alone" ''core'' application):
* A ''Transformation'' is a process that derives ONE (main) ''Result'' from ONE single input file.[[br]]
__Example__: The transformation of ''Structure'', ''Tree'', ''!SimilarityMatrix'', etc., using a certain ''Method'' with a certain ''!ParameterSet'', produces a contact map, a tree, ...
* A ''Comparison'' is a process that derives ONE (main) ''Result'' from TWO input files. [[br]]
__Example__ The comparison of ''Structures'', ''Trees'', etc., using a ''Method'' with a certain ''!ParameterSet'', produces a similarity value and an alignment
* A ''Composition'' is a process that derive ONE (main) ''Result'' from SEVERAL input files that are grouped together into ''DataSets''. [[br]]
__Example__ The composition of ''!SimilarityMatrices'', ''Trees'', using a ''Method'' with a certain ''!ParameterSet'', produces a consensus similarity matrix, a consensus tree, ...

This design does not allow ''Datasets'' to comprise other files than ''Structures'' although some of the ''Results'' need to be grouped into a ''Dataset'', too.[[br]]
__Example__ Contact maps that have been produces by a ''Transformation'' of ''Structures'' and that are available from within the ''Results'' need to form a ''Dataset'' in order to act as input for the ''Comparisons'' with the USM or MaxCMO ''Methods''.

== Extended Database Design for (dynamic) Management of Experiments (ProCKSI) ==

This has not been modelled yet, but the database for the (static) Protein Multiverse was designed with the ProCKSI integration in mind.

Some remarks:
* ''Experiments'' (formerly ''Requests'') apply ''Methods'' to''!DataSet'' "Methods" to "!DataSets" with a certain ''!ParameterSet''. "!ParameterSet''.
* ''Packages'' (formerly ''Jobs'') deal with a subset of a ''!DataSet'' "!DataSet" and a subset of the requested ''Methods'', partitioning the the 3D problem space, and are calculated using the ProCKSI's "stand-alone" core application "in one go". If they are sent to a queuing system, they become a ''Job'' there.
* It has to be discussed if there is still the need of a ''Tasks'' relation in the database, which have always been rather ''!RequestMethods''.