Bioinformatics Software Development Course

Apr 18-20 2006

Details of Talks

Talk 1 - Introducing EMBOSS

Background information for Practical 1

1. Introduction to EMBOSS

There is a short overview of EMBOSS and its key features which you should read now if you aren't already familiar with the project. EMBOSS is a big project and has much to offer to a diverse group including systems administrators, application programmers, bioinformaticians, training and education experts and, of course, the biologist end-user. This course is primarily for software developers.

You might need to use all the following pages to find background information, detailed information in a specific area or to solve a particular problem:

• EMBOSS Documentation For Users
• EMBOSS Documentation for Developers
• EMBOSS Documentation For Administrators

The user documentation is well worth a visit because it summarises some of the major themes in EMBOSS.

Practical 1 should give all the information on installing and compiling EMBOSS that you'll need for this course. There are, however, some notes below on know how to keep your installation up-to-date and configuration options, to read once you've done the practical. Further information is in the administrator documentation, especially our very comprehensive Administrators Guide.

2. Keeping your EMBOSS CVS copy up to date

To update, first cd to the emboss directory containing the ajax and nucleus subdirectories. For example, if you checked-out EMBOSS in the directory /home/fred then you should "cd" into the directory /home/fred/emboss/emboss. Then type the following commands:

cvs -d :pserver:cvs@cvs.open-bio.org:/home/repository/emboss login
  [password "cvs"]
cvs -d :pserver:cvs@cvs.open-bio.org:/home/repository/emboss update -d
cvs -d :pserver:cvs@cvs.open-bio.org:/home/repository/emboss update -P
cvs -d :pserver:cvs@cvs.open-bio.org:/home/repository/emboss logout

3. Associated libraries and compilation

    --with-pngdriver=/usr/local

Talk 2 - Navigating EMBOSS

Background information for Practical 2

1. Meaning of AjSStr, AjOStr, AjPStr

AjOStr could, in principle, be used in code to create an instance of the object in memory, but in practice AjPStr is used (memory is allocated to the pointer - more on this later). You'll notice in SRS that "AjPStr" is given after "Name" and that's because AjSStr or AJOStr are never really used in the code (other than in the object definition). For the sake of brevity, we often say "object" (to refer to an AjPStr for example) when what we really mean is "object pointer". Watch out for that in the course so that you're clear about what we're referring to.

Talk 3 - Your first EMBOSS application

Background information for Practical 3

1. Programming guide

For your future reference, there is a guide to programming EMBOSS application. However, during this course (or the first day at least) you should stick to the material in the talks and practicals.

2. helloworld in C

#include < stdio.h >   /* Omit the space after '<' and before '>' */
int main(void)
{
  printf("Hello, World!\n");
  return 0;
}

Unix % gcc helloworld.c -o helloworld        (using gcc, an ANSI C compiler)

Unix % cc  helloworld.c -o helloworld    (using cc, the default C compiler)

Unix % helloworld

Hello, World!

Unix % 

3. helloworld in EMBOSS

It's good practice to write your ACD file before the source code because this forces you to think closely about the application inputs and outputs and exactly what's required from the user. You should then test the ACD file by using an EMBOSS application called acdc (more on this shortly). Finally, the application is added to EMBOSS, compiled and ran.

So, the basic steps involved in writing your first EMBOSS application are as follows:

1. Decide on inputs and outputs
2. Write ACD file
3. Test ACD file
4. Write source code
5. Add application to EMBOSS
6. Compile

• Design (think about problem, design software)
• Debugging (getting it working)
• Testing (ensuring it works correctly under all conditions)
• Documentation (describing how it works)

1) Decide on inputs and outputs
To begin, the input and output is trivial. All the program has to do is print "Hello, World!" to the screen and so nothing is required from the user.

2) Write ACD file
It's no surprise then that the ACD file is pretty sparse:

application: helloworld
[
    documentation: "My first EMBOSS program"
]

Every ACD file must contain an application definition, and this should come first in the file. The definition consists of the application: token, followed by the application name and a block of attributes between square brackets. The definition above contains a single documentation: attribute. The text should be a succinct description of the program and will be printed to screen when the program is run.

If the documentation attribute is missing, a warning will be issued when you run the program. ACD files will be covered in great detail later on.

3) Test ACD file
Testing the ACD file is easy. You simply run acdc with your application name as a parameter on the command line:

Unix % acdc helloworld

My first EMBOSS program

Unix % 

4) Write source code
Happy in the knowledge we have a working ACD file we can turn to the C source code itself, which looks something like this:

/* @source helloworld application
**
** @author: Copyright (C) Arthur Geek (ageek@ebi.ac.uk)
**                        
** @@
**
** This program is free software; you can redistribute it and/or
** modify it under the terms of the GNU General Public License
** as published by the Free Software Foundation; either version 2
** of the License, or (at your option) any later version.
** 
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
** GNU General Public License for more details.
** 
** You should have received a copy of the GNU General Public License
** along with this program; if not, write to the Free Software
** Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
******************************************************************************/

#include "emboss.h"

/* @prog helloworld **********************************************************
**
** Prints "Hello, World!" to the screen.
**
******************************************************************************/
int main(int argc, char **argv)
{  
      embInit("helloworld", argc, argv);

      ajFmtPrint("Hello, World!\n");

      ajExit();
      return 0;
}

The source begins with a comment block including the copyright notice, disclaimer and authors contact details. EMBOSS applications are licensed under the GNU General Public License so its essential that these comments are included in the source.
Next we have the preprocessor directive #include "emboss.h". In contrast to to #include < stdio.h >, this imports the entire EMBOSS interface i.e. makes all the EMBOSS library calls available to you. This must be included at the start of every EMBOSS program. If we look inside the file we see eventually that stdio.h is itself included:

Unix % more nucleus/emboss.h 

#ifndef emboss_h
#define emboss_h

#include "ajax.h"
#include "ajgraph.h"
#include "embaln.h"
#include "embcom.h"
#include "embcons.h"
#include "embdbi.h"
.
.


Unix % more ajax/ajax.h

#ifdef __cplusplus
extern "C"
{
#endif
#ifndef ajax_h
#define ajax_h

#include "ajarch.h"
#include < stdarg.h >
#include < stdio.h >
#include < stdlib.h >
#include < string.h >
#include "ajassert.h"
#include "ajdefine.h"
#include "ajstr.h"
#include "ajtime.h"
#include "ajfile.h"
.
.

Every function, including main(), in an EMBOSS application must be documented. Undocumented code often has little value, with the explanation of code that is self-explanatory. Even then it's helpful to provide at least basic documentation.
EMBOSS uses a standard format for function documentation which you'll learn about later. For now, you need to know that the @prog token is used for documenting the main() function and @source is used for the application name in header documentation. These tokens are read by a program that parses the source code and automatically generates documentation that goes on the web and in SRS.

The source code proper begins with the main() function. The command line must be available therefore main must include it. This is done in the parameter list using int main(int argc, char **argv).

Three calls to the EMBOSS libraries are made. All EMBOSS applications must contain a call to embInit. This is the function that handles all of the user input processing and so must be called right at the start of the application. embInit reads in local database definitions, finds the right ACD file to use (the first argument is "helloworld" so it looks for "helloworld.acd" in the ACD directory) and processes the command line (it uses argc and argv from main). If our ACD file was more complicated, and required a sequence as input and a file as output for example, then by the time the call returned it would have read in the sequence and put it somewhere in memory and also opened the output file. ajFmtPrint is used to print text to the screen. ajExit calls some internal clean-up routines before calling exit with a successful code (zero).

5) Add application to EMBOSS
Once we have our C source code and an ACD file, we must add our application to EMBOSS before being able to compile it.
EMBOSS includes two files both called 'Makefile.am' which together contain information about each C source file and ACD file known to EMBOSS. To add helloworld to EMBOSS you must therefore edit these files. Assuming you checked EMBOSS out into /home/fred/emboss you have the following directories for the .c and .acd files respectively:

  /home/fubar/emboss/emboss/emboss      /* The 'executables directory'  (for C source files and executables) */
  /home/fubar/emboss/emboss/emboss/acd  /* The 'acd directory' (for ACD files)                               */

The files we have to edit are :

  /home/fred/emboss/emboss/emboss/Makefile.am
  /home/fred/emboss/emboss/emboss/acd/Makefile.am

The Makefile.am in the executables directory contains information about each C source file. First, you must add your program name to the "bin_PROGRAMS" list. This is usually done in alphabetical order. The before and after editing stages will look something like this:

i) Before editing

bin_PROGRAMS = aaindexextract abiview acdc antigenic \
...
garnier geecee getorf helixturnhelix hmoment \
...

ii) After editing

bin_PROGRAMS = aaindexextract abiview acdc antigenic \
...
garnier geecee getorf helixturnhelix helloworld hmoment \
...

N.B. Line continuation characters ('\') must be explicitly added (it's bad practice to have lines longer than 80 characters).

Secondly you must add the name of your source file to a SOURCES line. Again this is usually done alphabetically. The before and after appearance of the file would be as follows, given the above example.

iii) Before editing

...
geecee_SOURCES = geecee.c
getorf_SOURCES = getorf.c
helixturnhelix_SOURCES = helixturnhelix.c
hmoment_SOURCES = hmoment.c
iep_SOURCES = iep.c
infoalign_SOURCES = infoalign.c
...

iv) After editing

...
geecee_SOURCES = geecee.c
getorf_SOURCES = getorf.c
helixturnhelix_SOURCES = helixturnhelix.c
helloworld_SOURCES = helloworld.c
hmoment_SOURCES = hmoment.c
iep_SOURCES = iep.c
infoalign_SOURCES = infoalign.c

The Makefile.am in the ACD directory contains information about each ACD file. All that needs to be done for this file is to add the name of the new ACD file. Again, it is usual to do this alphabetically. Here are the before and after positions:

i) Before editing

pkgdata_DATA = codes.english \
        aaindexextract.acd abiview.acd ajbad.acd ajfeatest.acd ajtest.acd \
...
        garnier.acd geecee.acd getorf.acd helixturnhelix.acd hmoment.acd \
        histogramtest.acd iep.acd infoalign.acd infoseq.acd isochore.acd \
        lindna.acd listor.acd \
        marscan.acd maskfeat.acd maskseq.acd \
        matcher.acd

i) After editing

pkgdata_DATA = codes.english \
        aaindexextract.acd abiview.acd ajbad.acd ajfeatest.acd ajtest.acd \
...
        garnier.acd geecee.acd getorf.acd helixturnhelix.acd helloworld.acd \
        hmoment.acd histogramtest.acd iep.acd infoalign.acd infoseq.acd \
        isochore.acd lindna.acd listor.acd \
        marscan.acd maskfeat.acd maskseq.acd \
        matcher.acd

Again, line continuation characters ('\') must be added explicitly.

6) Compile

After the two Makefile.am files have been correctly edited, you can compile the application by typing "make helloworld" from the executables directory (quickest way) or "make" from either the executables or second emboss directory (slower as this will compile everything).

The GNU tools will recognise whether the Makefile.am files have been edited and reconstruct the Makefile files when a "make" command is given. It is bad practice to edit the Makefile files themselves.

Unix % pwd 
/home/fred/emboss/emboss/emboss/

Unix % make helloworld
/bin/sh ../libtool --tag=CC --mode=link gcc  -O2 -Wall -fno-strict-aliasing   -o helloworld  helloworld.o 
../nucleus/libnucleus.la ../ajax/libajaxg.la ../ajax/libajax.la ../plplot/libplplot.la -L/usr/X11R6/lib 
-lX11  -lm -lgd -lpng -lz -lm

gcc -O2 -Wall -fno-strict-aliasing -o .libs/helloworld helloworld.o  ../nucleus/.libs/libnucleus.so 
../ajax/.libs/libajaxg.so ../ajax/.libs/libajax.so ../plplot/.libs/libplplot.so -L/usr/X11R6/lib -lX11 
-lgd -lpng -lz -lm -Wl,--rpath -Wl,/home/jison/emboss_test_installation_for_course/emboss/lib

creating helloworld

And at last:

Unix % helloworld
My first EMBOSS program
Hello, World!
Unix % 

Talk 4 - Introduction to Objects Using Strings

Background information for Practical 4

1. EMBOSS objects

EMBOSS borrows the concept of objects from C++. In C++, an object can be thought of as a 'black box' which takes a defined input and produces a defined output. An object store its own data (member data) and knows how to perform certain actions (via member functions). From the perspective of the user of the object, it doesn't matter what is going on inside so long as the interface, i.e. the inputs and outputs, remains the same. The use of objects allows the programmer to model their program on the problem, breaking it down into smaller, easily managed pieces.

It's important to make the distinction between an object and an instance of an object. Strictly speaking, an object is a definition, or a template for instances of that object. The instance is the actual thing that can be manipulated. If you want to do anything you must create an instance, i.e. instantiate the object.

The objects in EMBOSS are C data structure definitions. Elements in the structure are the member data. There are no member functions as such, however, all the functions that use an object are documented above the structure definitions themselves, and are accessible from the web and SRS. Therefore EMBOSS maintains the link between the data and functions that act upon it. You can see that for yourself by inspecting a library header file, e.g. ajstr.h.

EMBOSS objects should be considered as black boxes, for instance, you should never access the elements of an instance directly; that is what the library functions are there for. The string object is one of the simplest so lets have a look at it now:

typedef struct AjSStr 
{
  ajint Res;
  ajint Len;
  ajint Use;
  char *Ptr;
} AjOStr;
#define AjPStr AjOStr*
typedef AjPStr* AjPPStr;

C programmers will see that we have defined a structure called AjSStr of 4 elements (Len, Res, Use and Ptr) and have created three new datatype names, AjOStr for the object itself, AjPStr for the object pointer, and AjPPStr for a pointer to an AjPStr.

Don't worry about the elements of the structure, we'll come onto them in a bit.

2. Managing objects

#include "emboss.h"
int main(void)
{
    AjOStr  my_structure;
}

The standard way to instantiate an object is to dynamically allocate memory to the object pointer. Its for this reason (and because AjOStr is almost never used) that we often say 'object' instead of the more cumbersome 'object pointer' when referring to an AjPStr etc - so make sure you understand the difference. All AJAX pointers start with "AjP" so its easy to guess that a sequence object is an AjPSeq.

All objects should be allocated dynamically and free'd once you're done with them. This is easy because a constructor function (for memory allocation) and destructor function (for freeing memory) are provided for every type of object. Here is a code snippet illustrating this:

#include "emboss.h"
int main(int argc, char **argv)
{
    AjPStr  my_pointer=NULL;
  
    embInit("helloworld", argc, argv);

    my_pointer = ajStrNew();

    ajStrAssC(&my_pointer, "Hello, World!\n");
    ajFmtPrint("%S", my_pointer);
  
    ajStrDel(&my_pointer);

    ajExit();
    return 0;
}

3. The golden rule

For consistency, all functions in EMBOSS use the following rule:

If the function is to change the object or the data pointed to in any way, the address of the object pointer must be passed.

If the function merely reads the data pointed to, just pass in the object pointer.

Armed with this rule, we can see that ajStrAssC, which copies text into an AjPStr, must receive the address of my_pointer, whereas ajFmtPrint which merely prints an AjPStr only requires my_pointer.

The above rule is one of the most important keys to coding in EMBOSS. There'll be more on pointers and objects later in the course, so don't worry if you don't fully understand what's going on yet - just remember and apply the rule you'll be fine. The library functions are well documented as to whether an object or address of an object is required so you can always find out what's needed if you're not sure.

4. Elements of the string object

typedef struct AjSStr {
  ajint Res;
  ajint Len;
  ajint Use;
  char *Ptr;
} AjOStr;
#define AjPStr AjOStr*
typedef AjPStr* AjPPStr;

If you intend altering the contents of an object then safety is guaranteed if you use the available library functions.

Talk 5 - ACD Files : Basic Skills

The notes below should describe everything you need to do this practical. If you need further information on ACD files, more comprehensive documentation is available, and will serve as a reference for the rest of the course. Please note there might be some slight differences in the nomenclature used.

1. Introduction to ACD files

2. General ACD syntax

datatype: label 
[ 
attributes 
] 

The available datatypes are pre-defined and include sequence, integer, float and many other types. There is a complete list of supported datatypes.

The label can be anything you like within reason. This is the name by which the data item is referred to both from within your C source code and on the command line. The programmer must have a handle on each ACD data item from within the C source code and the label is used for this purpose, as you'll see later. On the command line, the value of a data item can be set if the label is specified. For this reason the label can be referred to as the qualifier name.

The attributes allow you to specify such things as informative text, default values, maxima and minima and so on. Global attributes apply to all datatypes whereas others are datatype-specific, and are assigned values (calculated in some cases) after the data item is validated. For example, once a sequence has been read in from file. Here is a complete list of the supported global and datatype-specific attributes.

Comment lines can be added to an ACD file and you begin the line of comments with '#'. Any whitespace in an ACD file is ignored.

3. Application group

4. Retrieving ACD values

The ajAcdGet family of functions is used to retrieve values for ACD data items. These functions are in the library file ajacd.c that is summarised here.

5. Qualifiers and parameters

• Parameter: "Y"
• Standard: "Y" (this used to be 'required')
• Additional: "Y" (this used to be 'optional')

Parameter: "Y" means that the data item is a parameter, i.e. you do not have to use the data label to specify a value for it on the command line. e.g. "myprog 10".

Standard: "Y" and Additional: "Y" mean that the data item is a qualifier, i.e. you DO have to use the data label to specify a value for it on the command line. e.g. "myprog -somevalue 10".

Values for parameters and standard qualifiers are always prompted for (with their default value) if not specified on the command line.
Values for additional qualifiers are not prompted for (a default value will be used) unless '-option' is given on the command line. A default value for additional qualifiers should always be given in the ACD file!

The standard and additional qualifiers also differ in where the information from the built-in "-help" qualifier is shown. Help info. from 'standard' qualifiers appears in the 'required' section of the help display. Help info. from the 'additional' qualifiers appears in the 'advanced' section of the help display.

If none of Parameter: "Y", Standard: "Y" or Additional: "Y" are specified then the data item defaults to an Advanced qualifier. An advanced qualifier is never prompted for and would appear in the "advanced" section of the documentation if the program was run with the '-help' qualifier. To specify a value for an advanced qualifier on the command line you must use the data label.

You wouldn't normally specify Paramter: "N", Standard: "N" or Additional: "N". The "Y" in the previous definitions is given for consistency because every ACD attribute, being a label:value pair, has to have a value. You'll see later, however, that the "N" can in exceptional circumstances be used to override the default behaviour of these attributes.

Talk 6 - EMBOSS String Handling

Background information for Practical 6

All the information you need is given in the practical.

Talk 7 (Part I) - The AJAX Library & The Software Cycle

Background information for Practical 7 (Part I)

1. Overview of the AJAX library

You can't hope to cover everything, but you'll need to become familiar with several other library files to do the practicals today. The main libraries you'll need are:

• ajacd - controls all aspects of processing the AJAX command definition syntax (ACD files), command line handling and prompting of the user.
• ajfile - input and output data types are provided for handling files and directories of files. Includes convenience functions for reading data files (which are typically kept in specific directories), functions for parsing file names, paths and extensions from a full file name and buffering of lines so that files can be read again from memory; useful for files which cannot be rewound like stdin.
• ajarr - these functions control all aspects of AJAX array handling. This includes dynamic 1, 2 and 3-dimensional arrays of long,double, float, int and short datatypes. There are convenience functions for reading arrays from formatted strings.
• ajfmt - string formatting functions. These are similar to the C functions printf, fprintf, vprintf etc. AJAX formatting is a superset of ANSI C, with additional types for strings, booleans and dates. Output can be to a string with automatic adjustment of memory allocation. Null pointers are checked for and reported rather than causing failures (for example with %s).
• ajlist - These functions create and control linked lists, which can be tricky to manage because of the ownership of the data items. Functions are provided for all manner of list manipulations: creating, destroying, appending to, iterating through, popping, peeking, sorting etc.
• ajseq - Functions for managing individual sequences and database of sequences (which are processed a sequence at a time). A comprehensive library that provides many useful utilities, e.g. for probing sequence attributes and for handling substitution tables.
• ajseqread - More sequence handling functions - for reading sequences from formatted input files.
• ajseqwrite - Yet more sequence handling functions - for formatted output.

2. The Software Development Cycle

1) Suggested implementation steps under EMBOSS

There's much more to software than just coding; especially when programming for EMBOSS which is very widely distributed and deployed in production environments. You should familiarise yourself with the basic steps now, so that you know what's involved before starting a project and so can plan your work. This should help you develop your software efficiently and deliver your projects on time.

As mentioned before, the basic steps to write an EMBOSS application are:

1. Design (think about problem, design software)
2. Decide on inputs and outputs
3. Write application
4. Write ACD file
5. Test ACD file
6. Write source code
7. Add application to EMBOSS
8. Compile
9. Debugging (get it working)
10. Testing (ensure it works correctly under all conditions)
11. Documentation (describe how it works)

Each of these steps is essential and takes time, so the first thing that should be obvious is that a software project will take much more time than that required to simply write the code.

2) Consider your users

The steps above would lead to a finished product, but that's only the start of the lifetime of your software. In practice, you must consider your users and there are four fundamental processes in this regard:

1. Software specification : What are the requirements? What functionality is required? What are the constraints?
2. Software development : Design the software and implement it, both in consultation with users where necessary.
3. Software validation : Ensure that the software meets the users needs.
4. Software evolution : Your software must evolve to keep pace with your users, whose needs are often evolving.

You should now appreciate that writing the first version of the code is the beginning rather than the end of the project. The four steps above have many implications, especially in terms of additional requirements (e.g. how to survey user feedback) and strategy (e.g. how best to implement for evolving requirements). These issues are beyond the scope of this course, but the take home message is:

3) Models for developing and releasing software

In terms of developing and releasing software, there is no universal model but the following stages are typical:

1. Planning
   • Specification of requirements.
   • Design
2. Pre-Alpha : Software still under development or design, e.g. prototype software. Useful to get a gist of what's to come and for early feedback from users.
3. Alpha : Software that's quite well developed with major features implemented and major bugs removed. Possibly redesigned from the initial prototype. Usually with very limited deployment, if any.
4. Beta : The first significant release of the software, used for testing by users and to get feedback. Beta software might not be perfect, e.g. it might not work under all circumstances, but it should be relatively bug-free - the majority (if not all) known bugs should be fixed.
5. Release Candidate : An early release if no significant bugs were reported from the Beta version. Usually features are "frozen" and only bugfixes are allowed from this point.
6. Release (e.g. Version 1.0) : Fully tested software that's ready to be used in anger by your users.

Of course in practice early releases often contain many bugs and for this reason people are usually wary of software until it has matured over a period of months or even years.

In some cases, usually to adapt the software for evolving requirements, it may be necessary to move the software back into beta or even to start this entire cycle from scratch in cases where a complete redesign is necessary.

4) Software engineering and project management

There is a well-defined standard (IEEE 1074) developed by the Institute of Electrical and Electronics Engineers for creating a software life cycle process. The standard is intended for use by process architects (e.g. project managers) but should be useful for managing & performing software projects. You'll certainly not need it for this course but you might benefit from a quick look now:

IEEE 1074: Standard for Developing Life Cycle Processes

Process group	Processes	Clause	Activities
Life Cycle Modeling	Selection of a Life Cycle Model
Project Management	Project Initiation	3.1.3 3.1.4 3.1.5 3.1.6	3 Map Activities to Software Life Cycle Model 4 Allocate Project Resources 5 Establish Project Environment 6 Plan Project Management
	Project Monitoring and Control	3.2.3 3.2.4 3.2.5 3.2.6 3.2.7	Analyze Risks Perform Contingency Planning Manage the Project Retain Records Implement Problem Reporting Model
	Software Quality Management	3.3.3 3.3.4 3.3.5 3.3.6	Plan Software Quality Management Define Metrics Manage Software Quality Identify Quality Improvement Needs
Pre-development	Concept Exploration	4.1.3 4.1.4 4.1.5 4.1.6 4.1.7	Identify Ideas or Needs Formulate Potential Approaches Conduct Feasibility Studies Plan System Transition (If Applicable) Refine and Finalize the Idea or Need
	System Allocation	4.2.3 4.2.4 4.2.5	Analyze Functions Develop System Architecture Decompose System Requirements
Development	Requirements	5.1.3 5.1.4 5.1.5	Define and Develop Software Requirements Define Interface Requirements Prioritize and Integrate Software Requirements
	Design	5.2.3 5.2.4 5.2.5 5.2.6 5.2.7	Perform Architectural Design Design Data Base (If Applicable) Design Interfaces Select or Develop Algorithms (If Applicable) Perform Detailed Design
	Implementation	5.3.3 5.3.4 5.3.5 5.3.6 5.3.7 5.3.8	Create Test Data Create Source Generate Object Code Create Operating Documentation Plan Integration Perform Integration
Post-development	Installation	6.1.3 6.1.4 6.1.5 6.1.7	Plan Installation Distribution of Software Installation of Software Accept Software in Operational Environment
	Operation and Support	6.2.3 6.2.4 6.2.5	Operate the System Provide Technical Assistance and Consulting Maintain Support Request Log
	Maintenance	6.3.3	Reapply Software Life Cycle
	Retirement	6.4.3 6.4.4 6.4.5	Notify Users Conduct Parallel Operations (If Applicable) Retire System
Integral Processes	Verification and Validation	7.1.3 7.1.4 7.1.5 7.1.6 7.1.7 7.1.8	Plan Verification and Validation Execute Verification and validation Tasks Collect and Analyze Metric Data Plan Testing Develop Test Requirements Execute Tests
	Software Configuration Management	7.2.3 7.2.4 7.2.5 7.2.6	Plan Configuration Management Develop Configuration Identification Perform Configuration Control Perform Status Accounting
	Document Development	7.3.3 7.3.4 7.3.5	Plan Documentation Implement Documentation Produce and Distribute Documentation
	Training	7.4.3 7.4.4 7.4.5 7.4.6	Plan the Training Program Develop Training Materials Validate the Training Programs Implement the Training Program

If you're interested in learning more, there's a nice introduction to software engineering. Here are a few useful definitions of the term:

• As the usual contemporary term for the broad range of activities that was formerly called programming and systems analysis.
• As the broad term for the technical analysis of all aspects of the practice, as opposed to the theory of computer programming.
• As the term embodying the advocacy of a specific approach to computer programming, one that urges that it be treated as an engineering profession rather than an art or a craft, and advocates the codification of recommended practices in the form of software engineering methodologies.
• Software engineering is "(1) the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, that is, the application of engineering to software," and "(2) the study of approaches as in (1)." -- IEEE Standard 610.12.
• Software engineers model parts of the real world in the software. As the real world changes, software must also change, and software engineering is concerned with the evolution of these models and how they meet changing requirements.

But really the take-home message for today is ...

Talk 7 (Part II) - ACD files : Intermediate Skills

Further background information for Practical 7 (Part II)

1. Parameters and Qualifiers

Parameter:  "Y"
Standard:   "Y"   (used to be 'required')
Additional: "Y"   (used to be 'optional')

Their behaviour is as follows:

Type	Prompt	Flag	Help info.
parameter	Yes	No	Required section
standard	Yes	Yes	Required section
additional	Yes (-option)* or No (default needed)	Yes	Advanced section
advanced (default)	No	Yes	Advanced section

Prompt - whether a value will be prompted for if one is not specified on the command line.
Flag - whether the qualifier name must be specified on the command line.
Help info - where the information from the built-in "-help" qualifier is shown.
* Additional qualifiers will only be prompted for if -option is given on the command line.

2. Summary

Values for parameters and standard qualifiers are always prompted for (with their default value) if not specified on the command line.
Values for additional qualifiers are not prompted for (a default value will be used) unless '-option' is given on the command line. A default value should always be given in the ACD file.

The standard and additional qualifiers also differ in where the information from the built-in "-help" qualifier is shown. Help info. from 'standard' attributes appears in the 'required' section of the help display. Help info. from the 'additional' attribute appears in the 'advanced' section of the help display.

If none of Parameter: "Y", Standard: "Y" or Additional: "Y" are specified then the data item defaults to an Advanced qualifier.
An advanced qualifier is never prompted for and would appear in the "advanced" section of the documentation if the program was run with the '-help' qualifier. To specify a value for an advanced qualifier on the command line you must use the data label.

You'd never normally specify Paramter: "N", Standard: "N" or Additional: "N".

Talk 7 (Part III) - Sequence Handling

Further background information for Practical 7 (Part III)

1. Uniform Sequence Addresses

The USA syntax has the following types:

• "file"
• "format::file"
• "format::file:entry"
• "dbname:entry"
• "@listfile"

Where "format" is the database format of a file ("file") you have provided and "entry" is the database entry code. Alternatively an entry can be retrieved from an installed database of format "dnmame". "listfile" is the name of a file which itself contains a list of file names.

The "::" and ":" syntax is to allow, for example, "embl" and "pir" to be both database names and sequence formats.

2. Sequence formats

embl::myfile.seq

The format is not required, however. When reading in a sequence, EMBOSS will guess the sequence format by trying all known formats until one succeeds.

When writing out a sequence, EMBOSS will use fasta format by default. The output format can be changed by adding "-osformat format" on the command line or by giving it in the USA (Uniform Sequence Address) of the output filename, e.g.

gcg::myresults.seq

3. Input sequence command-line qualifiers

  -sbegin	integer		first base used
  -send		integer		last base used, default=seq length
  -sreverse	boolean		reverse (if DNA)
  -sask		boolean		ask for begin/end/reverse
  -snucleotide	boolean		sequence is nucleotide
  -sprotein	boolean		sequence is protein
  -slower	boolean		make lower case
  -supper	boolean		make upper case
  -sformat	string		input sequence format
  -sopenfile	string		input filename
  -sdbname	string		database name
  -sid		string		entryname
  -ufo		string		UFO features
  -fformat	string		features format
  -fopenfile	string		features file name

4. Output sequence command-line qualifiers

  -osformat           string     output sequence file format
  -osextension        string     file name extension
  -osname             string     base file name
  -osdirectory        bool       output sequence file directory
  -osdbname           string     database name to add
  -ossingle           bool       create a separate output file for each entry
  -oufo               string     feature file to create
  -offormat           string     features format
  -ofname             string     features file name
  -ofdirectory        string     features output directory

5. Supported output format

• "gcg"
• "gcg8"
• "embl"
• "em"
• "swiss"
• "sw"
• "fasta"
• "pearson"
• "ncbi"
• "nbrf"
• "pir"
• "genbank"
• "gb"
• "gff"
• "ig"
• "codata"
• "strider"
• "acedb"
• "experiment"
• "staden"
• "text"
• "plain"
• "raw"
• "fitch"
• "msf"
• "clustal"
• "selex"
• "aln"
• "phylip"
• "phylip3"
• "asn1"
• "hennig86"
• "mega"
• "meganon"
• "nexus"
• "nexusnon"
• "paup"
• "paupnon"
• "jackknifer"
• "jackknifernon"
• "treecon"

6. AJAX library files for sequence handling

Talk 7 (Part IV) - ACD Files : Advanced Skills

Background information for Practical 7 (Part IV)

If you need further information on ACD files, very comprehensive documentation is available.

Talk 7 (Part V) - Software Consolidation

Background information for Practical 7 (Part V)

1. Coding standards

Currently, you will probably find exceptions to these standards but in the future they will be enforced (or at least exceptions raised) by software which automatically checks submitted code against the standards.

You should at least familiarise yourself with the standards, most of which concern the layout of code and are summarised below.

Ease of reading

a) Line length limit

Style

a) Braces

Indentation

Position of main()

Implicit declarations

extern fubar(int x);
main(int argc, char **argv)

Use of 'int', 'ajint', 'long' and 'ajlong'

EMBOSS assumes an ajlong is at least 64 bits. Use this instead of 'long'. This circumvents any 'long' or 'long long' problems. Of course, if you are using an Alpha box then both your ints and longs will be 64 bits. In this case don't just use 'ajlong' out of laziness as your code will run more slowly on other platforms. Match your datatype to what you need.

Cases where int and long should be used are (e.g.) as parameters to C system library functions.

Global variables

Static variables in functions

Variable declarations

Note that it is easier to read if both datatypes and variables line up.

Always initialise Object pointer variables to NULL. Initialise other datatypes as appropriate in the code. Align initialisations for easy reading.

Consider documenting key variables but do not document individual house-keeping variables, for example, loop counters.

Precedence of operators

Organisation

a) Source File

• includes of system headers
• includes of local headers
• type and constant definitions
• global variables (there should not be any)
• prototypes
• functions

b) Header File

• type and constant definitions
• external object declarations
• external function declarations

It is not necessary to make the declarations 'extern' (although it is arguably safer to do so. Current EMBOSS library code doesn't do this.)

Never use nested includes! (Look at Solaris header files to see how not to do things).

Avoid exporting names outside individual C source files; i.e., declare as static every function that you possibly can.

Always use full ANSI C prototypes.

Structures and unions

Use AjS, AjO, AjP or EmbS, EmbO and EmbP as appropriate.

Use of this convention i.e. 'P' for pointers avoids problems with these abstract datatypes where a pointer could appear to be an object in its own right.

Use of the preprocessor

enum { Red = 0xF00, Blue = 0x0F0, Green = 0x00F };
static const float pi = 3.14159265358;

instead of #defines, which are rarely visible in debuggers.

Macros should avoid side effects. If possible, mention each argument exactly once. Put parentheses around all arguments. When the macro is an expression, put parentheses around the whole macro body. If the macro is the inline expansion of some function then capitalise the name and precede it with 'M'.

Try to write macros so that they are syntactically expressions i.e. so you can put a semi-colon at the end of their use.

The preprocessor can be used for:

• System dependent code (but see the ajsys corollary above.)
• Commenting out code with #if 0
• Conditionals arising from system limits

Function names

• All function names should start with a lower case character.
• All AJAX exported functions should begin with 'aj'.
• All NUCLEUS exported functions should begin with 'emb'
• All application functions should have the application name (followed by an underscore) prepended.
• Once the current code refactoring exercise is complete, names for AJAX and NUCLEUS functions must adhere to strict rules that are defined in the header files.

Argument names

• Function arguments should have meaningful or at least intuitive names but should not be too long.
• Once the current code refactoring exercise is complete, argument names must adhere to strict rules that are defined in the header files.

Loops

Goto

Memory and functions

Do not explicitly use malloc() or calloc(), instead use the AJAX macros e.g. AJALLOC, AJNEW, AJCNEW, AJNEW0, AJCNEW0 etc.

Use constructor functions explicitly, i.e.

    AjPStr  tmpstr = NULL;

    tmpstr = ajStrNew();
    ajStrAssC(&tmpstr,"Hello");

and not:

    AjPStr  tmpstr = NULL;

    ajStrAssC(&tmpstr,"Hello");

• function declaration
• variable declarations
• constructors
• body of function
• destructors

Separate functions by suitable whitespace (4 newlines are recommended).

Applications

A separate application should only be written if it differs by more than one extra major parameter/qualifier to an existing program.

New applications should be put into the 'make check' area of the emboss/Makefile.am until full documentation has been submitted. See the documentation standards document for details.

Code should be tested for memory leakage before committing it to CVS. If you are unclear how to do this then ask.

General guidelines

Duplicated code

Long functions

Long parameter lists

Managing change to code

Managing variables

Switch statements

Over-engineered code

Keep objects clean

Nesting of functions

Object overlap

Use of libraries

2. Code documentation

a) Code documentation standards

• Comments before each function
• Function documentation tag descriptions
• Parameter codes
• Comments before each datatype
• Datatype documentation tag descriptions
• Comments at the head of each library source file
• Comments at the head of each application source file

b) Comments

Good places to put comments are:

• a broad overview at the beginning of an application
• data structure definitions
• at the beginning of a function
• tricky steps within the program or functions (see later wrt function definitions)
• major logical steps within the program or functions

Avoid fancy layout or decoration.

c) GPL licence

d) main() function

/* @prog water ****************************************************************
**
** Smith-Waterman local alignment
**
******************************************************************************/

e) Application functions

/* @funcstatic tcode_readdata ********************************************
**
** Read Etcode.dat data file
**
** @param [w] table1 [AjPFTestcode*] data object
** @param [r] datafile [AjPFile] data file object 
** @return [AjBool] true if successful read
** @@
******************************************************************************/

** @return [void]

See the code documentation standards for a list of supported tags (@param etc).

f) Library functions

3. Application documentation

Full application documentation in a format suitable for end-user biologists is available on-line.

Every EMBOSS application should be well documented and should adhere to the EMBOSS style, see for example documentation the seqret application.

a)APPLICATION DOCUMENTATION PROCESS

To generate the documentation, run the script autodoc.pl on each application you wish to document in turn:

/home/fred/emboss/emboss/scripts/autodoc.pl application_name (for EMBOSS applications)

/home/fred/emboss/emboss/scripts/autodoc.pl -embassy=embassy_package_name application_name (for EMBASSY applications)

You should replace embassy_package_name and application_name with something sensible. The following instructions presume you are working in the EMBASSY pacakage "myemboss" and are writing a program called myprogram.

• Write the program and ACD file
• cd to embassy/myemboss/emboss_doc/master
• cp template.html.save myprogram.html
• edit myprogram.html
  • Change ProgramNameToBeReplaced to myprogram
  • Write some documentation text where indicated
• Run autodoc.pl, e.g. ../../../../scripts/autodoc.pl -embassy=myemboss myprogram

mytest 'Demonstration of sequence reading'
Doing test mytest-ex
/homes/pmr/cvsemboss/embassy/myemboss/emboss_doc/html/mytest.html *created*
/homes/pmr/cvsemboss/embassy/myemboss/emboss_doc/text/mytest.txt *replaced*

The script will run wossname to check that application_name really exists, then generate a template documentation file (for you to fill in) with include directives, plus include files for:

1. One-line description
2. Help table
3. Documentation table
4. "See also" list

1. Usage example; the QA test as if run by a user (mytest.usage)
2. Application input files (mytest.input)
3. Application output files (mytest.output)

1. Comments
2. History

Use an editor of your choice to edit the template, adding documentation text. The template should be adequately commented for you to see how to fill it out. You should see the directives to read the include files, which are created for you by autodoc.pl and by the QA test procedure. Once you complete the template and save it, the application index file (to appear on the web - you don't need to worry about this file) will need to be updated. The entry for the new application will be inserted into the correct (alphabetic) position in the index file.

In brief, the QA test is performed as follows:

• Edit test/qatest.dat
• Find the tests for the EMBASSY package myemboss which all have the line "AB myemboss"
• Add a new test for your application (see below).

Example of entry in qatest.dat

ID myprogram-ex
AB myemboss
AA myprogram
IN
FI stderr
FC = 2
FP 0 /Warning: /
FP 0 /Error: /
FP 0 /Died: /
FI paamir.myprogram
FC = 5
FP /^Usa: tembl-id:PAAMIR\n/
FP /^Length: 2167\n/
//

• cd test/qa
• ../../scripts/qatest.pl myprogram-ex

If it fails, check directory myprogram-ex which contains all the files. Update the test definition and try agaion.

When it works, rerun autodoc.pl as above and it will create the remaining 3 include files (usage example, input and output files).

When all is done, the complete HTML documentation is created in embassy/myemboss/emboss_doc/html/myprogram.html

For EMBOSS applications, work in the doc/programs/master/emboss/apps/ directory instead. Leave out the -embassy=myemboss qualifier from the autodoc.pl commandline. The test definition does not have the "AB myemboss" line, and has "AP myprogram" instead of "AA myprogram". Final documentation goes to doc/programs/html/myprogram.html

There is a little work to do to update the index.html file and the Makefile.am files in the html and text directories - we hope to automate this in time for release 4.0.0.

b) DOCUMENTATION FILES

i) Template html file

This is the template mentioned above with include directives. This should be copied along with the include files to the EMBOSS website on sourceforge (the EMBOSS team have a script for doing this).

ii) Include files

./programs/master/inc/seqret.ione      One-line description.    Taken from ACD file
./programs/master/inc/seqret.ihelp     Help table*              Generated by running the application with "-help".
./programs/master/inc/seqret.itable)   Documentation table**    Generated by running acdtable.
./programs/master/inc/seqret.isee      "See also" list          Generated by running seealso.
./programs/master/inc/seqret.usage     Usage example            Generated via QA tests.
./programs/master/inc/seqret.input     Application input files  Generated from QA tests.
./programs/master/inc/seqret.output    Application output files Generated from QA tests.
./programs/master/inc/seqret.comment   Comments                 Written manually - usually blank.
./programs/master/inc/seqret.history   History                  Written manually - usually blank.

(*  Application parameters and appropriate "in-built" qualifiers)
(** Application parameters)

iii) Raw documentation text

./programs/text/seqret.txt

Documentation (with included text) in plain text format and organised into sections. Used for manual pages and displayed when running application_name -help. This is generated by autodoc.pl.

./programs/html/seqret.html

Final html file, with all data included. Documentation for the web. These might be moved to ./doc/programs/html/apps/ or ./doc/programs/html/embassy/* in the future.

4. Application quality assurance tests

A set of test data consist of input files, application parameters and the corresponding output files. As many sets of test data should be provided as possible, especially for unusual input conditions, to provide as robust a test as possible.

The tests are defined in the file /home/fred/emboss/emboss/test/qatest.dat. There is documentation at the start of that file which describes the records used to define a test.

Each test you define should write its output files to the same directory as the application is run in. It is possible to create sub-directories and write files to them though.

To run the test for a specific application, from the /home/fred/emboss/emboss/test/test/qa directory:

../../scripts/qatest.pl test_name  

If qatest.pl is run on something not defined in qatest.dat it will report "Tests total: 0".
If it succeeds, all files are deleted (unless the test entry included a "DL keep" line and "-kk" was specified on the command line)
If it fails, it will say why and all results are written to an appropriate subdirectory (/test/qa/test_name)

To perform tests, you must must edit the .embossrc file located in the test directory. Make sure to set "emboss_qadata" to appropriate test directory (e.g. /home/fred/emboss/emboss/test)

Talk 8 - Data Input : Using Features

Background information for Practical 8

The notes below are a summary of the on-line feature documentation.

What is a feature?

• a start and end position.
• a name describing the feature.
• the name of the program or database from which it was derived.
• the sense (in a nucleic sequence).
• a score.

A feature table is a groups of features.

Examples of biological data corresponding to features include restriction enzymes cut sites, probabilities of the three states of a protein secondary structure prediction and tables of the start and end positions of things like predicted exons or motif matches. As most sequence analysis programs generate interesting regions in one form or another, there are a huge number of diverse file formats corresponding to features.

Feature formats

The standard formats provide a consistent look and feel to features, helping the user compare the features from different programs more easily. Standardisation also facilitates application interoperability (applications can more easily share their input and output).

The standard file formats will become the default way of reporting sequence features as the EMBOSS project matures.

What are the formats?

Feature tables are stored in one of three ways:

• As part of a sequence file
• As part of a database entry
• As a raw feature table: a file that does not contain the sequence the features refer to.

In all cases, the feature format is identical to that used in the sequence database format of the same name, e.g. EMBL feature format is the same as the (subset of the) EMBL sequence format. This holds for when a raw feature table is output too. The following feature formats are understood by EMBOSS.

Name	Comments / Documentation
embl em	The format used by the EMBL nucleic database.
gff	The General Feature Format defined by the Sanger Centre
swissprot swiss sw	The format used by the SWISSPROT protein database. The feature table keys are also defined
pir	The format used by the PIR protein database.
>nbrf	Only available for input - the same as PIR format

Uniform Feature Object

1. The format of the features in that file.
2. The name of that file.

UFOs can be used for input and output. If no format is specified, the default 'GFF' format is used. You can override the default with a different format when you run the program (see below).

ACD datatypes and built-in command line qualifiers for handling features

ACD provides two datatypes for handling features:

• features for feature input
• featout for feature output

These command-line qualifiers change the behaviour of a features ACD datatype:

  -fformat            string     features format                  Default: ""
  -fopenfile          string     features file name               Default: ""
  -fask               bool       prompt for begin/end/reverse     Default: N
  -fbegin             integer    first base used                  Default: 0
  -fend               integer    last base used, def=max length   Default: 0
  -freverse           bool       reverse (if DNA)                 Default: N

These command-line qualifiers change the behaviour of a featout ACD datatype:

  -offormat           string     output feature format            Default: "" 
  -ofopenfile         string     features file name               Default: "" 
  -ofextension        string     file name extension              Default: ""
  -ofname             string     base file name                   Default: ""
  -ofsingle           bool       separate file for each entry     Default: N
  -ofdirectory        bool       Output feature file directory    Default: ""

Note that the sequence, seqall, seqset & seqsetall datatypes (for sequence input) and seqout,seqoutall & seqoutset datatypes (for sequence output) can also read / write features if their features ACD attribute is set. If set, the sequence output will include feature information either in the same file (if the sequence format supports it) or in a separate file (by default in GFF format).

Reading features

If the feature table is included in the sequence input file (as is generally the case when you reading the sequence from a database), then the feature table will be read with no problem.

To read a raw feature table from file, you must specify the -ufo in-built qualifier on the command line, e.g. '-ufo gff:results.dat'.

Alternatively, the '-fformat' and '-fopenfile' qualifiers can be used to specify the feature format and the file name individually instead of as part of a UFO.

Using '-ufo' or '-fopenfile' to read in a feature table will cause the new feature table to replace any existing feature table that is part of the sequence data.

In-built command line qualifiers for feature input

  -ufo                string     UFO features
  -fformat            string     features format
  -fopenfile          string     features file name

If you wish to combine feature table files from various sources, then the easiest way is to concatenate the feature files (must be in the same format!) into one file and to specify that file using '-ufo'.

Currently, programs that read and use the feature table of an input sequence include diffseq, extractfeat, maskfeat, seqret and showfeat. Currently there aren't any programs that read in a raw feature table (i.e. one that's not part of sequence).

Writing features

If a program is capable of writing out sequences with features (for example run "seqret -feature"), then the feature table will be written out as part of the output sequence file, if the format of the sequence file is one of embl, gff, swissprot or pir (i.e. if the sequence field was designed to hold a feature table).

If the sequence format cannot hold a feature table (e.g.'fasta'), then a file ('unknown.gff') is written with the raw feature table in GFF format.

This behaviour can be overridden by using the command-line qualifiers below. Even if a sequence format that is capable of holding a feature table has been specified, these will enable you to specify a name and format for output to a raw feature table file.

Output sequence command-line qualifiers

  -oufo               string     UFO features
  -offormat           string     features format
  -ofname             string     features file name

Many programs are capable of writing raw feature tables. The default output format for raw features tables is 'gff', but this can be changed by specifying '-offormat' followed by the format name.

Calculated attributes

fbegin (integer)   - start of the features to be used.   
fend (integer)     - end of the features to be used.     
flength (integer)  - total length of sequence            
fprotein (boolean) - feature table is protein            
fnucleic (boolean) - feature table is nucleotide         
fname (string)     - the name of the feature table       
fsize (string)     - number of features                  

type:     (string)  - defines whether the feature is "protein" or "nucleotide". 
                      There is a default based on the type of an input sequence,
                      but a value should always be specified.

nullok:   (boolean) - allows a default name for a feature to be replaced by an empty 
                      string or by -noxxx (where "xxx" is the ACD label for the feature) 
                      on the command line. The application must be able to run without
                      feature input. See below.

format:      (string) Default feature format. 
name:        (string) Default base file name (use of -ofname is preferred).Default: ""
extension:   (string) Default file extension (use of -offormat preferred) 

type:        (string) Defines whether the feature output is "protein" or "nucleotide". 
                      There is a default based on the type of any input sequence, but 
                      a value should always be specified. 
multiple:    (Y/N)    Features for multiple sequences                      Default: N
nullok:      (Y/N)    Allows a default name for a feature to be replaced by an empty string 
                      or by -noxxx (where "xxx" is the ACD label for the feature) on the 
                      command line. The application must be able to run without feature 
                      output.  See below.
nulldefault: (Y/N)    Defaults to 'no file'                                Default: N

The output filename is constructed from the name: and extension: attributes in a $( name).$(extension) format. If the name: attribute is not defined in the ACD file, it will default to the calculated attribute name: of the FIRST sequence that is read in (or $(asequence.name), for a sequence parameter named "asequence").

The nulldefault: attribute overrides the default name generation, and uses an empty string (no feature output) as the default for programs where feature output is only occasionally required. If an empty string is specified on the command line, the standard default value will be generated instead. In combination with the nullok: and missing: attributes, this allows qualifiers to be null by default, and turned on from the command line.

Talk 9 - Data Output : Using Reports

Background information for Practical 9

1. Too many formats
A problem with output from computer programs is that everybody invents their own output format, generally one format per program. While this displays commendable ingenuity and inventiveness it does little for standard parsing. For this reason EMBOSS is moving towards using a set of standard report formats. These should be used whenever appropriate.

Report formats take their origin from the need to deal with EMBL, GenBank and PIR feature tables. It was therefore a natural choice to extend these to cope with other output data. The first thing to consider is that all of the standard sequence feature tables (see Talk 8) are also report formats. These include:

• embl
• genbank
• gff
• pir
• swiss

• dbmotif
• diffseq
• excel
• motif
• nametable
• regions
• seqtable
• simple
• srs
• table
• tagseq

2. Report format theory
You will probably know that feature table information is held as tag/value pairs. To cope with non-standard feature values the AJAX library uses the default '/note' tag. This is transparent to the user/programmer (unless you select EMBL or another output format in which the '/note' tag is visible). Internally, all reports are held, as near as dammit, in GFF (the General Feature Format). It was suggested that this be called the Flexible Open Feature Format (FOFF, pronounced eff off) but this name was sadly rejected. As the features are held in such a flexible format they can easily be converted to any of the report formats given above. This can be selected by using the -rformat built-in qualifier.

GFF has some field that must exist. One is the sequence itself (this may or may not be output depending on the output report format used), the start and end positions of the features, a score (in case the program reports a score for the sequence using whatever algorithm) and a few others. The score and most of the rest of the standard GFF tags can be, and often are, set to zero or NULL automatically by the library. You have to explicitly use them (assign values to them) if you need them, otherwise they'll be ignored.

3. Method to output reports

Each report has a header, body and tail information. The process of outputting using reports is as follows.

1. Define a suitable report datatype in ACD.
2. Collect the ACD report information within the application in a report object (AjPReport).
3. Create a feature table object (AjPFeattable) using ajFeattableNewSeq.
4. For each 'line' of information in the body of the report, create a feature object (AjPFeature) and load the feature object with your data values (by using ajFeatNewII and ajFeatTagAdd).
5. Set any data for the header and tail of the report.
6. Pass the report AjPReport) and feature table (AjPFeattable) objects to the general report writing function (ajReportWrite).
7. Clean-up memory as normal and exit.

4. Example application

This looks hard but isn't. As an example we develop below a simple application that will take a sequence and produce a tabular report format showing its length and molecular weight.

4.1 ACD for a molecular weight application.
Here is an ACD file which will do the job:

application: wreport 
[
  documentation: "Example report program"
]

section: input [ info: "input Section" type: page ]
sequence: sequence  
[
  parameter: "Y"
  type: "Protein"
]

endsection: input


section: advanced [ info: "advanced Section" type: page ]

datafile: aadata  
[
  information: "Amino acid data file"
  help: "Molecular weight data for amino acids"
  default: "Eamino.dat"
]

endsection: advanced


section: output [ info: "output Section" type: page ]

report: outfile  
[
  parameter: "Y"
  rformat: "table"
  multiple: "N"
  precision: "1"
  taglist: "float:molwt int:len"
]
endsection: output

The sequence input has been met before. The datafile datatype is just there so a file of molecular weight data in the EMBOSS data area can be read. It is the report datatype we are interested in ...

The rformat attribute is the equivalent of the "default" attribute in other datatypes. It is the default report format that will be printed if the user doesn't change it with -rformat on the command line.

The multiple attribute says whether multiple reports will be given in the output. This will generally have a value of "N" if you are using the "sequence" datatype and "Y" if you're using the "seqall" datatype.

The precision attribute is for floating point numbers. It gives how many decimal places will be printed in the output.

The most interesting attribute is taglist. This shows, in order, the datatype/column name pairs that will be used in the report. float:molwt therefore means that one of the columns is called 'molwt' and it will contain floating point values. Typical taglist datatypes are :

• "float"
• "int"
• "str"

4.2 C source code
Here it is:

/* @source wreport application
**
** Show sequence length and molwt as a report
**
** @author: Copyright (C) Alan Bleasby (ableasby@embnet.org)
** @@
**
** This program is free software; you can redistribute it and/or
** modify it under the terms of the GNU General Public License
** as published by the Free Software Foundation; either version 2
** of the License, or (at your option) any later version.
**
** This program is distributed in the hope that it will be useful,
** but WITHOUT ANY WARRANTY; without even the implied warranty of
** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
** GNU General Public License for more details.
**
** You should have received a copy of the GNU General Public License
** along with this program; if not, write to the Free Software
** Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA  02111-1307, USA.
******************************************************************************/

#include "emboss.h"




/* @prog wreport **************************************************************
**
** Show sequence length and molwt as a report
**
******************************************************************************/

int main(int argc, char **argv)

{
    AjPSeq       seq    = NULL;
    AjPReport    report = NULL;
    AjPFeattable ftable = NULL;
    AjPFeature   feat   = NULL;
    AjPStr       tmpstr = NULL;

    double       molwt;
    int          len;

    AjPStr       datafn = NULL;
    AjPFile      mfptr  = NULL;


    embInit ("wreport", argc, argv);

    seq       = ajAcdGetSeq ("sequence");
    report    = ajAcdGetReport("outfile");

    /* This bit just reads in an EMBOSS data table of molwt info */
    mfptr     = ajAcdGetDatafile("aadata");


    embPropAminoRead(mfptr);
    /* End of data file reading */

    /* Calculate the values to output */
    len   = ajSeqLen(seq);
    molwt = embPropCalcMolwt(ajSeqChar(seq),0,len-1);


    /* Create a feature table */
    ftable = ajFeattableNewSeq(seq);


    tmpstr = ajStrNew();


    /* Fill head and tail information for the report */
    ajFmtPrintS(&tmpstr,"This is some Header Text");
    ajReportSetHeader(report, tmpstr);

    ajFmtPrintS(&tmpstr,"This is some Tail Text");
    ajReportSetTail(report, tmpstr);


    /* Create feature object and load with the output values */
    feat = ajFeatNewII(ftable,1,len);

    ajFmtPrintS(&tmpstr,"*molwt %.1f", (float)molwt);
    ajFeatTagAdd(feat,NULL,tmpstr);

    ajFmtPrintS(&tmpstr,"*len %d", len);
    ajFeatTagAdd(feat,NULL,tmpstr);


    /* Write report and clean up */
    ajReportWrite(report,ftable,seq);


    ajFeattableDel(&ftable);
    ajStrDel(&tmpstr);
    ajFileClose(&mfptr);

    ajExit ();
    return 0;
}

This is a minimalistic program for clarity (e.g. no notice is taken of any -sbegin or -send values). Taking it step by step, here are the declarations:

    AjPSeq       seq    = NULL;
    AjPReport    report = NULL;
    AjPFeattable ftable = NULL;
    AjPFeature   feat   = NULL;
    AjPStr       tmpstr = NULL;

A sequence (AjPSeq) object is obviously needed. A report object (AjPReport) is declared which is used to pick up the ACD information using ajAcdGetReport. The feature table object (AjPFeattable) will be used to hold the feature objects (AjPFeature) containing the column values. A temporary string object is declared into which the column values will be printed. This string will then be used to load the feature object.

After a bit of code to load in a data file containing amino acid molecular weight data, the molecular weight and length values to be reported are calculated by the lines:

    len   = ajSeqLen(seq);
    molwt = embPropCalcMolwt(ajSeqChar(seq),0,len-1);

Note that the code uses len and molwt for both the variable names and for the column names. This is not necessary but does make the code more clear. The feature table object is then instantiated. Only one feature table is required per report:

    ftable = ajFeattableNewSeq(seq);

Next, some head and tail information is loaded into the report object. This could have been done any time after the ajAcdGetReport but before the report is printed. They are optional but recommended. The temporary string object is used for this.

    ajFmtPrintS(&tmpstr,"This is some Header Text");
    ajReportSetHeader(report, tmpstr);

    ajFmtPrintS(&tmpstr,"This is some Tail Text");
    ajReportSetTail(report, tmpstr);

A feature object can now be created into which the column values can be loaded

    feat = ajFeatNewII(ftable,1,len);

This required 3 parameters. The first is the feature table object, the last two are the sequence start and end positions. Rather naughtily they are hard coded in this example (as mentioned above). A feature object has to be created per line of output in the final report.

Now, the molecular weight and length values are loaded into the feature object:

    ajFmtPrintS(&tmpstr,"*molwt %.1f", (float)molwt);
    ajFeatTagAdd(feat,NULL,tmpstr);

    ajFmtPrintS(&tmpstr,"*len %d", len);
    ajFeatTagAdd(feat,NULL,tmpstr);

All that remains is to print out the report.

    ajReportWrite(report,ftable,seq);

Finally, the dynamic memory is recovered in a clean-up.

    ajFeattableDel(&ftable);

Talk 10 - Objects, Pointers and Memory Management

Background information for Practical 10

No course in EMBOSS programming would be complete without a treatment of pointers. Here you will get a clear explanation of the use of pointers with particular reference to the management of memory for objects in EMBOSS. This coverage of pointers was inspired by Chuck Allison's 'Code Capsules' (find it on the web!)

Pointer basics

   ajint x=0;

If you're O.K. with the idea that 'x is an integer' then in the same way understand that 'a pointer is a memory address'. Spelling that out ...

A POINTER IS MERELY A VARIABLE WHICH HOLDS A MEMORY ADDRESS

If you don't overcomplicate the above idea, you've already gone a long way to understanding pointers.

Example pointer code

main()
{
   ajint x=0;
   /*1*/   printf("Value of x : %d\n", x);
   /*2*/   printf("Memory address of x : %p\n", &x);
   /*3*/   printf("Value of x by indirection : %d\n", *(&x));
}

/* Output:

Value of x : 0
Memory address of x : #1 
Value of x by indirection : 0

(In reality, a hexadecimal number would be printed instead of '#1', but '#1' is easier to follow).
*/

The variable name x is our handle on the reserved memory, it is used to refer to an integer value that happens to live at memory address #1. We usually say that "x is an integer" or "x holds an integer" rather than the more acurate and cumbersome "x is a variable name referring to a reserved area of memory of sufficient size to hold an integer".

In the code:

• /*1*/ First we are printing the value of x.
• /*2*/ Then we are using & to get the memory address of where x is stored and printing it.
• /*3*/ Finally we are using the * operator to print the value stored at this address by indirection.

In practice, a pointer holds the memory address of a specific data object such as an integer, C data structure or even another pointer. You have to specify the type of data pointed at when you declare your pointer. This is not because the memory address of an integer is any different to that of a float, it's so that the compiler knows how the pointer can be used in the source code. For example, the computer must know the type of data pointed at to be able to print a value by indirection.

Pointers to pointers

So how do we declare a pointer variable? Easy ...

   ajint *ptr=NULL;

The * means that ptr is a memory address and the ajint tells us that it's the address of an integer.

When that line of the program is executed, sufficient memory to hold a memory address will be reserved for use by our program. This, like an integer, is normally 4 bytes. The value of these 4 bytes is set to NULL.

Its important to know that its only in the context of a variable declaration that *ptr=NULL means set the value of the pointer to NULL. If *ptr=NULL was found elsewhere in the program it would mean, "set the value held at memory address ptr to NULL". The final thing to mention is that we've used NULL for the pointer and 0 for the integer in the declarations; they achieve the same thing but cannot be used interchangeably as they are not of the same type; you would get serious bitching from the compiler if you tried!

You can see that in the code below:

main()
{
/*1*/  ajint x=0;
/*1*/  ajint *ptr=NULL;
/*2*/  printf("Value of x : %d\n", x);
/*3*/  ptr = &x;
/*4*/ *ptr=5;
/*2*/  printf("Value of x : %d\n", x);
}

/* Output:
Value of x : 0
Value of x : 5
*/

In the code:

• /*1*/ Declare an integer and a pointer to an integer.
• /*2*/ Print the value of x.
• /*3*/ Give ptr the value of the address of x.
• /*4*/ Set the value of x to 5 by indirection.

In the above example, you would normally say that "ptr holds the address of x" or simply "ptr points to x".

It was mentioned above that a pointer can hold the memory address of another pointer. This is obvious when you think that a pointer, like any variable, resides somewhere in memory. So if a pointer that holds the memory address of an integer is a 'pointer to an int', then a pointer that holds the memory address of another pointer is, of course, 'a pointer to a pointer'.This bit of code shows how we declare a pointer to a pointer-to-an-int:

    ajint **ptrto=NULL;

main() 
{ 
/*1*/ ajint x=0;          /* an integer */
/*1*/ ajint *ptr=NULL;    /* a pointer to an integer */
/*1*/ ajint **ptrto=NULL; /* a pointer to a pointer-to-an-integer */

/*2*/ printf("Address of x : %p\n", &x); 
/*2*/ printf("Address of ptr : %p\n", &ptr); 
/*2*/ printf("Address of ptrto : %p\n", &ptrto); 

/*3*/ ptr = &x; 
/*3*/ ptrto = &ptr; 

/*4*/ printf("Value of x : %d\n", x); 
/*4*/ printf("Value of ptr : %p\n", ptr); 
/*4*/ printf("Value of ptrto : %p\n", ptrto); 

/*5*/ printf("Value of x by dereferencing ptr : %d\n", *ptr); 
/*5*/ printf("Value of x by dereferencing ptrto : %d\n", **ptrto); 
} 

/* Output: 
Address of x : #1 
Address of ptr : #2 
Address of ptrto : #3 

Value of x : 0 
Value of ptr : #1     /* i.e. the address of x*/
Value of ptrto : #2   /* i.e. the address of ptr*/

Value of x by dereferencing ptr : 0 
Value of x by dereferencing ptrto : 0 
*/ 

There are no new concepts in the above code, its merely an extension of what you already know about pointers. In the code we:

• /*1*/ We start by declaring 3 variables called x, ptr and ptrto. x is the integer, ptr is a pointer-to-an-integer and ptr to is a pointer to a pointer-to-an-integer.
• /*2*/ We then print the address of each variable; x lives at #1, ptr at #2 and ptrto at #3.
• /*3*/ We assign the address of x to ptr. The address of ptr is assigned to ptrto.
• /*4*/ Print the value of each variable. x has a value of 0, ptr has a value of the address of x, i.e. #1 and ptrto has the value of the address of ptr, i.e. #2.
• /*5*/ Print x out by indirection.

You already know what *ptr means. Further on you see we dereference the ptrto twice, which is what you've got to do if you want to get to the integer from it. The first time you dereference ptrtor you get to ptr, the second time you are effectively dereferencing ptr, which takes you to x. Simple !

This and in fact all problems in pointers are very easily understood if you sketch what's happening on a piece of paper.

Always draw a diagram if you're not sure what's happening with your pointers.

Its gets a bit of a mouthfull when put into words, but here is the explanation of exactly what is happening in **ptrto
ptrto holds the address of ptr (#2). So dereferencing ptrto once will give you the value of ptr (held at #2), i.e. the address of x (#1). By dereferencing ptrto a second time, you effectively dereference ptr, which will give you the value that is held at #1, i.e. the integer value 0

If you've got this far with your head intact then you're closer than you think to having the basics of pointers licked, and you've got more or less all you need to master objects, pointers and memory management in EMBOSS.

Object definition

/* @data AjPPdbtosp *******************************************************
**
** Ajax Pdbtosp object.
** 
** Holds swissprot codes and accession numbers for a PDB code.
** 
** Pdb is the pdb code.
** n is the number of Acc / Spr pairs for this pdb code.
** Acc is the accession number
** Spr is the swissprot code
**
** AjPPdbtosp is implemented as a pointer to a C data structure.
**
** @alias AjSPdbtosp
** @alias AjOPdbtosp
**
** @@
******************************************************************************/
typedef struct AjSPdbtosp
{   	
    AjPStr     Pdb;    
    ajint      n;      
    AjPStr    *Acc;    
    AjPStr    *Spr;    
} AjOPdbtosp, *AjPPdbtosp;

Note how the structure is nicely documented - your object definitions should do the same! There is nothing new here other than Acc and Spr which are both pointers to AjPStr objects. As an AjPStr is itself a pointer (to the AjOStr object proper) you can see that we're dealing with pointers to pointers. In this case, Acc and Spr are going to be used to create two arrays of strings as we can see in the constructor function below:

Constructor function

/* @func ajXyzPdbtospNew ***********************************************************
 **
 ** Pdbtosp object constructor. Fore-knowledge of the number of entries is 
 ** required. This is normally called by the ajXyzPdbtospReadC / ajXyzPdbtospRead 
 ** functions.
 **
 ** @param [r] n [ajint] Number of entries
 **
 ** @return [AjPPdbtosp] Pointer to a Pdbtosp object
 ** @@
 ******************************************************************************/
AjPPdbtosp ajXyzPdbtospNew(ajint n) /*1*/
{ 
 
   AjPPdbtosp ret = NULL;           /*2*/ 
   ajint i=0;
 
   AJNEW0(ret);                     /*3*/
 
   ret->Pdb = ajStrNew();           /*5*/ 
 
   if(n)
   {
      AJCNEW0(ret->Acc,n);          /*4*/ 
      AJCNEW0(ret->Spr,n);          /*4*/ 
      for(i=0; iAcc[i]=ajStrNew();  /*6*/ 
           ret->Spr[i]=ajStrNew();  /*6*/ 
 	}
     }

/*7*/
     ret->n = n;
 
     return ret;
 }
 

We'll go through this line by line.

• /*1*/ The first line declares that the function returns an object pointer of type AjPPdbtosp. The parameter ajint n is the size of the Acc and Spr arrays, i.e. the number of pairs of Acc / Spr values that the object will hold.
• /*2*/The next line declares a variable called ret. This is the object pointer that is going to have memory allocated to it , and will then be returned to the calling function.
• /*3*/ AJNEW0(ret); is the line that allocates an object proper to the pointer ret; ret will now point to an AjOPdbtosp object in memory. By the time AJNEW0(ret); returns, memory space for an AjOPdbtosp object will be reserved, this means enough space for an AjPStr, an ajint and two pointers. Note that we have not got our two arrays or any string objects proper yet!
• AJNEW0 sets all the structure elements to 0, this means n is set to 0 and the pointers (including the AjPStr) are set to NULL.
• AJNEW0 is a macro that's why this line looks a bit weird. It will allocate a single object of the correct type to any pointer that is passed to it - it can be used with any object!
• /*4*/ Compare AJNEW0 to the two AJCNEW0 lines. These allocate an array of n AjPStr object pointers to each Acc and Spr. In other words, ret->Acc and ret->Spr will point to blocks of memory each holding n n AjPStr object pointers each.
  Like AJNEW0, AJCNEW0 will work with objects of any type and also initialises the new variables to 0 or NULL.
• /*5*/ So, we now have our arrays but still no strings yet. ret->Pdb = ajStrNew(); allocates memory for a string object to the pointer Pdb in our new object. Notice that -> is used to dereference the pointer and get to the Pdb element. This is the standard way in C of accessing elements in a data structure when we have a pointer to that data structure.
• /*6*/ The lines ret->Acc[i]=ajStrNew(); and ret->Spr[i]=ajStrNew(); allocates memory for n string objects for each array. It also illustrates how pointer and array notation can be used together. In this case, we are accessing the i'th element of the arrays that ret->Acc and ret->Spr point to. The elements in these arrays are AjPStr (object pointers) and we are allocating a string object to each of them.
• /*7*/ The rest is obvious. The integer in the object is set to the size of the arrays and the pointer to the new object, complete with an allocated string and two arrays of strings, is returned to the calling function by return ret;.
• It is the job of the calling function to free the object itself and all the other allocated memory the elements of the object point to. Which brings us neatly onto the destructor function:

Destructor function

 /* @func ajXyzPdbtospDel ***********************************************************
 **
 ** Destructor for Pdbtosp object.
 **
 ** @param [w] thys [AjPPdbtosp*] Pdbtosp object pointer
 **
 ** @return [void]
 ** @@
 ******************************************************************************/
 
 void ajXyzPdbtospDel(AjPPdbtosp *thys)
 {
     AjPPdbtosp pthis = NULL;
     ajint i;

     if(!thys)	return;
     pthis = *thys;
     if(!pthis)     return;
 
     ajStrDel(&pthis->Pdb);
 
     if(pthis->n)
     {
 	for(i=0; i< pthis->n; i++)
 	{
 	    ajStrDel(&pthis->Acc[i]);
 	    ajStrDel(&pthis->Spr[i]);
 	}
 	AJFREE(pthis->Acc);
 	AJFREE(pthis->Spr);
     }
 
     AJFREE(pthis);
     (*thys)=NULL;
 
     return;
 }

It is your task in Practical 10 to figure out exactly what is going on in this destructor function. If you can do that, then you can be happy that you are on your way to become adept at objects, pointers and memory management in EMBOSS.

Suffice it to say that this function safely clears up all of the memory that was allocated by the constructor, this is achieved by calling appropriate destructor functions and by using AJFREE. AJFREE will free the memory pointed to by its argument. There are 3 places its used, twice to free the arrays and once to free the Pdbtosp object itself. Note! AJFREE as used here will free the arrays but will not free the string objects proper that are pointed to (this is the job of the ajStrDel calls in the preceding code).

The function also sets the object that was passed in to NULL. This is a requirement of all destructor functions for reasons explained in Talk 4.

Calling constructor and destructor functions

  main()
  {
  AjPPdbtosp ptr=NULL;

  ptr = ajXyzPdbtospNew(10);
  ajXyzPdbtospDel(&ptr);

  /* ptr will have been reset to NULL now, and is ready for reuse */
  ptr = ajXyzPdbtospNew(10);
  ajXyzPdbtospDel(&ptr);

  }

EMBOSS memory allocation macros

• AJALLOC(nbytes) equivalent of malloc
• AJALLOC0(nbytes) a calloc of nbytes
• AJCALLOC(count,nbytes) a malloc of count lots of nbytes
• AJCALLOC0(count,nbytes) a calloc of count lots of nbytes
• AJNEW(p) a pointer to an object gets an object allocated for it using malloc
• AJNEW0(p) a pointer to an object gets an object allocated for it using calloc
• AJCNEW(p,c) a pointer to an object gets c objects allocated using malloc
• AJCNEW0(p,c) a pointer to an object gets c objects allocated using calloc

For non-C programmers "malloc" allocates memory but the contents are undefined whereas "calloc" allocates memory setting each location to zero.

Talk 11 - The NUCLEUS Library

Background information for Practical 11

The NUCLEUS library provides higher-level functions and algorithms, mostly for molecular sequence analysis, including sequence comparisons, translation, codon usage and annotation. See the NUCLEUS Library Documentation for the documentation of individual functions and datatypes.

The available NUCLEUS libraries are listed below.

• Sequence Alignment (embaln) - sequence alignment.
• Database Indexing (embdbi) - B+ Tree Indexing plus Disc Cache.
• Database Indexing (embindex) - general database indexing.
• EST Methods (embest - EST alignment functions.
• Protein properties (functions) - amino acid residue/sequence properties.
• Molecular fragments (functions) - Routines for molecular weight matching.
• Pattern Matching Methods (embpat) - General routines for pattern matching.
• Protein data bank (embpdb) - To handle protein structural data.
• Sequence Properties (embprop) - Amino acid residue/sequence properties.
• Show (Display) Methods (embshow) - general routines for sequence display.
• Word Methods (embword) - wordmatch routines
• Comparison Matrices (functions) - General sequence match routines.
• Domain Methods (embdomain) - data structures and algorithms for handling protein domain data.
• Word (n-mer) Methods (embnmer) - General word match routines.
• Data Tables (embdata) - general routines for data tables.
• Signature Methods (embsig) - data structures and algorithms for use with sparse sequence signatures.
• Comments (embcom) - general routines for program complex
• Consensus (embcons) - general routines for program consensus
• Domainatrix Methods (embdmx) - data structures and algorithms used by the protein structure EMBASSY applications.
• Xyz Coordinate Methods (embxyz) - miscellaneous stuff for protein structure.
• Groups (functions) - functions for parsing ACD files.
• Initialization (embinit) - functions for application initialisation.
• Exit Methods (embexit) - functions for exiting cleanly.
• Miscellaneous Methods (embmisc) - miscellaneous routines.
• Reading data files (embread) - data file reading routines.

In contrast to the applications and to the AJAX library, NUCLEUS is not as well developed or documented. In future code refactoring, some of the libraries may be merged together.

In several cases, algorithms and data structures that you would expect to find in NUCLEUS are in fact kept in AJAX. An example of this is the domain handling code, most of which is kept in ajdomain.c/h rather than embdomain.c/h. The reason for this is for purposes of compilation: the ACD file-handling code (which is part of AJAX) must call these functions which therefore must live in AJAX. This may change in a future code refactoring exercise.

Last modified on 2005 by Jon Ison.