Version-1.6.0 Release Notes, October 2005
The main changes in this release, in addition to the usual round of
bug fixes, are the support in Gap4 for very large databases, for
traces from the "454" instrument and for a new SNP Candidates plot.
In addition to these substantial speed improvments have been made.
64-bit file size support
With databases getting ever bigger we finally reached the point where
the gap4 database files needed to be larger than 2Gb in filesize. This
posed a problem for the old 32-bit file offset code (mandated by the
format of the .aux file) so we have made Gap4 64-bit aware.
Firstly note that Gap4 still uses 32-bit file sizes by default and so
is fully backwards compatible. Gap4 may create 64-bit databases by
specifying the "-bitsize 64" command line option. Gap4 will
automatically detect the format on subsequent opens so this option
only need to be specified at creation time. However note that 64-bit
databases are incompatible with the old 32-bit ones and so will not be
read by older Gap4s. Fortunately copy_db has also been modified to
support conversion between 32-bit to 64-bit and vice versa (assuming
the database isn't too large to fit in 32-bits).
Dealing with such massive databases also showed a number of areas
where the speed was too slow. We've made considerable improvements
across multiple places and these changes should also help speed up
normal 32-bit usage too.
The SNP Candidates plot
This new plot has been designed to graphically show the locations
where a consensus column has a strong chance of being made up of
multiple sequences, eg due to it being a SNP or a variance in a
Stage 2 of this plot is to cluster the differences to attempt to pull
apart the sequencing into a sets where each set has no or few internal
differences. These sets may then be viewed directly in the contig
editor by colour coding and sorting the sequences. The editor also has
manual control over moving sequences from one set to another, but
currently saving of edits made this data is not supported.
Finally the consensus sequence for each set may be saved or
alternatively the sets may be split apart to form multiple contigs.
The 454 sequencing machine
This new instrument by 454 Life Sciences is based on pyrosequencing
techniques and so produces a very different style of trace to
traditional Sanger sequencing. The 454 machine SFF format is now
natively supported as is a prototype for the TVF archive (although
both of these format have not been finalised yet and so may be subject
to change). Trev and Gap4 support viewing of "454 flowgrams".
The new flowgrams are substantially smaller than traditional traces
which gives rise to a number of interesting issues with file
management. To compensate for this there are now a variety of tools
for packing multiple small traces into a single large archive. This
can be either a unix tar file (with a faster indexing method than
before) or a new ZTA (ZTR Archive) file. See the io_lib package for
more details and related tools.
We are aware that there are a number of issues to still address with
incorporating pyrosequencing data with traditional Sanger sequencing
data. There is a new replacement for Shuffle Pads which greatly
improves alignments of 454 data produced by Gap4 or Phrap (but isn't
needed if you use 454's own assembler), but there are still issues to
resolve with the use of confidence values. We expect to be
concentrating on this in the oncoming months.
64-bit database size support. Gap4 databases may now grow beyond the
2Gb limit (we've tested 6Gb and in theory it should handle
terabytes, albeit slower). Old 32-bit databases are still supported
and are still the default output format, for backwards
compatibility. Specify "-bitsize 64" on the command line to force
Gap4 to create 64-bit databases.
A new (experimental) plot labelled SNP Candidates. It identifies and
plots likely locations when multiple sequences may have been
assembled together (either SNPs or collapsed repeats) and provides
tools to pull these apart.
The contig editor now has the notion of "sets". A set is basically a
sub-group of a contig, which is useful for exploring the outcome
within a conti of pulling apart a collapsed repeat without actually
doing it. The editor is automatically invoked in this manner by the
SNP candidates output. This is largely experimental, but there are a
number of potential ways to improve interaction. Ideas on a postcard
The editor's Shuffle Pads algorithm has been moved to the main gap4
menu (so it can be scripted non-interactively) and it now does a
substantially better job too (as it is based on the same techniques
used in ReAligner).
Lots of speedups (primarily for the huge 64-bit databases):
Drawing of some consistency graphs;
Experiment file reading on large files (1000 fold on a 1Mb
Dealing with lists of contigs (eg "all contigs") is now O(N)
complexity instead of O(N^2) (On databases with 50K contigs and
2.6million reads this has changed some operations that would take
hours to taking a couple of seconds);
(De)selecting contigs in the contig selector is now much much
Substantial modifications to the gap4 I/O mechanisms (balancing the
"freetree"). The upshot of this is that the worst-case performance
is now dramatically improved and the average performance will be
slightly better too. Complementing a 10Mb contig dropped from about
50 minutes to 1.5 minutes. Disassembling all readings (approx
175000) in this database went from ~3 hours to 2.5 minutes.
Substantial speed up of the contig selector with large databases.
The confidence plot is now substantially faster to plot (typically
The contig editor Report Mutations now has a confidence scale for
the "by difference" option.
The editor Search by Sequence now finds consensus matches too.
Improved mousewheel support in the editor; shift + wheel performs
Dragging a selection in the contig editor now auto-scrolls the
display as appropriate.
Highlight Disagreements now uses ":" to display bases which differ
to the contig but have a low quality value.
Now shows pyrosequencing traces. Eg those from 454.
A new "-trace_scale" option may be used to force a particular Y
scaling instead of auto-scaling. This can be useful when comparing
absolute trace heights across multiple files.
Added a -b option to switch between 32 and 64-bit database sizes.
Substantial speed ups, particularly when dealing with gzipped files
or when extracting data from tar files. This is primarily due to
performing decoding in memory instead of from the disk structures,
but experiment file I/O has had additional (and substantial)
improvements when dealing with very long files (approx 1000 fold
speed increase for reading a 1Mb sequence on the Alpha).
The Exp_info structure now has an "mFILE *fp" member instead of
Some functions are no longer external.
These include many ctf functions, ztr_(de)compress,
ztr_chunk_(read/write), be_read_*, be_write_*,
The default search order for RAWDATA is that the current
directory is searched after the rest of rawdata instead of
Removed support for the old unix "pack" program as a
Preliminary support for 454 flowgrams, including the SFF format.
Added support for hash indexing of tar files or creation of "solid"
archives. This allows traces to be packed into a single archive with
a fast index for extraction. Replaces the old index_tar program.
New programs hash_tar and hash_extract.
Reenabled gzip compression on Windows.
Under MS Windows spin and EMBOSSwin should coexist happily to the
extent that Spin will spot the EMBOSSwin installation and use it.
Removed some redundant old ABI utilities: getABIfloat, getABIhex,
getABIraw, getABIstring. All of these can be achieved via
#1104446 Gap4: user-defined contig editor widths sometimes cause
#1104453 Gap4: 2nd-highest confidence plot failing when the
consensus algorithm is not the phred-style confidence mode.
#1105931 Gap4: crash when invoking the join editor.
#1183748 Gap4: break contig can reset the strand of consensus tags.
#1289095 io_lib: file descriptor leak when opening compressed files.
Gap4: race condition in destroying a sheet widget (eg the editor);
triggerable only by scripts?
Gap4: Report Mutations crashes.
Gap4: rare Find Internal Joins crash.
Gap4: the last base in a sequence was being missed by show edits and
search by edits.
Gap4: mousewheel support on MS Windows.
Gap4: Buffer over-run in the Quality Clip Contig Ends function.
Gap4: editor Search by Sequence sometimes missed matches when the
sequence lengths varied greatly.
Gap4: list contigs windows redisplays after complement contig.
Gap4: display of "editor embedded traces" in the Join Editor.
Gap4: rare corruption in contig joining when joining the last contig
in the database to another.
Gap4: seqInfo (I/O) memory leak.
Gap4: use of memcpy instead of memmove in io_delete_contig(). Was
still working OK, but is compiler dependent.
Gap4: Attempting to free memory twice in Find Internal Joins.
Gap4: Tcl syntax error in Enter Tags.
Gap4: Consensus tags in Enter Tags are not honouring the unpadded
Gap4: rounding errors causing display position bugs when lots of
zooming / unzooming is performed in template display, confidence and
Gap4: stop codons at the ends of contigs (beyond the "unpadded
contig length" position) are displayed in the wrong place.
Gap4: cannot plot more than 10000 stop codons (now essentially
Pregap4: Interactive clipping and pathnames containing spaces.
Mutscan: crashes when inputting text sequence files.
Mutscan: protection against MUT/HET tags occuring outside of MCOV
Spin: "\" vs "/" in EMBOSS_DATA path.
General: windows UNC pathname bugs where //foo/bar was interpreted
General: some strings in ABI files (eg MODL records) have the first
character missed so 3730 is reported as "730".