This is gnugo.info, produced by makeinfo version 4.0b from gnugo.texi.

INFO-DIR-SECTION GNU games
START-INFO-DIR-ENTRY
* GNU go: (gnugo).          The GNU Go program
END-INFO-DIR-ENTRY


File: gnugo.info,  Node: Tuning,  Next: PM Implementation,  Prev: Connection Functions,  Up: Patterns

Tuning the Pattern databases
============================

   Since the pattern databases, together with the valuation of move
reasons, decide GNU Go's personality, much time can be devoted to
"tuning" them. Here are some suggestions.

   If you want to experiment with modifying the pattern database, invoke
with the `-a' option. This will cause every pattern to be evaluated,
even when some of them may be skipped due to various optimizations.

   You can obtain a Smart Go Format (SGF) record of your game in at
least two different ways. One is to use CGoban to record the game. You
can also have GNU Go record the game in Smart Go Format, using the `-o'
option. It is best to combine this with `-a'. Do not try to read the SGF
file until the game is finished and you have closed the game window.
This does not mean that you have to play the game out to its
conclusion. You may close the CGoban window on the game and GNU Go will
close the SGF file so that you can read it.

   If you record a game in SGF form using the `-o' option, GNU Go will
add labels to the board to show all the moves it considered, with their
values. This is an extremely useful feature, since one can see at a
glance whether the right moves with appropriate weights are being
proposed by the move generation.

   First, due to a bug of unknown nature, it occasionally happens that
GNU Go will not receive the `SIGTERM' signal from CGoban that it needs
to know that the game is over. When this happens, the SGF file ends
without a closing parenthesis, and CGoban will not open the file. You
can fix the file by typing:


      echo ")" >>[filename]

at the command line to add this closing parenthesis. Or you could add
the ) using an editor.

   Move values exceeding 99 (these should be rare) can be displayed by
CGoban but you may have to resize the window in order to see all three
digits. Grab the lower right margin of the CGoban window and pull it
until the window is large. All three digits should be visible.

   If you are playing a game without the `-o' option and you wish to
analyze a move, you may still use CGoban's "Save Game" button to get an
SGF file. It will not have the values of the moves labelled, of course.

   Once you have a game saved in SGF format, you can analyze any
particular move by running:


       gnugo -l [filename] -L [move number] -t -a -w

to see why GNU Go made that move, and if you make changes to the
pattern database and recompile the program, you may ask GNU Go to
repeat the move to see how the behavior changes. If you're using emacs,
it's a good idea to run GNU Go in a shell in a buffer (M-x shell) since
this gives good navigation and search facilities.

   Instead of a move number, you can also give a board coordinate to
`-L' in order to stop at the first move played at this location. If you
omit the `-L' option, the move after those in the file will be
considered.

   If a bad move is proposed, this can have several reasons. To begin
with, each move should be valued in terms of actual points on the
board, as accurately as can be expected by the program. If it's not,
something is wrong. This may have two reasons. One possibility is that
there are reasons missing for the move or that bogus reasons have been
found. The other possibility is that the move reasons have been
misevaluated by the move valuation functions. Tuning of patterns is
with a few exceptions a question of fixing the first kind of problems.

   If there are bogus move reasons found, search through the trace
output for the pattern that is responsible. (Some move reasons, e.g.
most tactical attack and defense, do not originate from patterns. If no
pattern produced the bogus move reason, it is not a tuning problem.)
Probably this pattern was too general or had a faulty constraint. Try
to make it more specific or correct bugs if there were any. If the
pattern and the constraint looks right, verify that the tactical
reading evaluates the constraint correctly. If not, this is either a
reading bug or a case where the reading is too complicated for GNU Go.

   If a connecting move reason is found, but the strings are already
effectively connected, there may be missing patterns in `conn.db'.
Similarly, worms may be incorrectly amalgamated due to some too general
or faulty pattern in `conn.db'. To get trace output from the matching
of patterns in `conn.db' you need to add a second `-t' option.

   If a move reason is missing, there may be a hole in the database. It
could also be caused by some existing pattern being needlessly
specific, having a faulty constraint, or being rejected due to a
reading mistake. Unless you are familiar with the pattern databases, it
may be hard to verify that there really is a pattern missing. Look
around the databases to try to get a feeling for how they are
organized. (This is admittedly a weak point of the pattern databases,
but the goal is to make them more organized with time.) If you decide
that a new pattern is needed, try to make it as general as possible,
without allowing incorrect matches, by using proper classification from
among snOoXx and constraints. The reading functions can be put to good
use. The reason for making the patterns as general as they can be is
that we need a smaller number of them then, which makes the database
much easier to maintain. Of course, if you need too complicated
constraints, it's usually better to split the pattern.

   If a move has the correct set of reasons but still is misevaluated,
this is usually not a tuning problem. There are, however, some
possibilities to work around these mistakes with the use of patterns.
In particular, if the territorial value is off because `delta_terri()'
give strange results, the (min)terri and maxterri values can be set by
patterns as a workaround. This is typically done by the endgame
patterns, where we can know the (minimum) value fairly well from the
pattern. If it should be needed, (min)value and maxvalue can be used
similarly. These possibilities should be used conservatively though,
since such patterns are likely to become obsolete when better (or at
least different) functions for e.g. territory estimation are being
developed.

   In order to choose between moves with the same move reasons, e.g.
moves that connect two dragons in different ways, patterns with a
nonzero shape value should be used. These should give positive shape
values for moves that give good shape or good aji and negative values
for bad shape and bad aji. Notice that these values are additive, so
it's important that the matches are unique.

   Sente moves are indicated by the use of the pattern followup value.
This can usually not be estimated very accurately, but a good rule is
to be rather conservative. As usual it should be measured in terms of
actual points on the board. These values are also additive so the same
care must be taken to avoid unintended multiple matches.

   You can also get a visual display of the dragons using the `-T'
option. The default GNU Go configuration tries to build a version with
color support using either curses or the ansi escape sequences. You are
more likely to find color support in rxvt than xterm, at least on many
systems, so we recommend running:

       gnugo -l [filename] -L [move number] -T

in an rxvt window. If you do not see a color display, and if your host
is a GNU/Linux machine, try this again in the Linux console.

   Worms belonging to the same dragon are labelled with the same
letters.  The colors indicate the value of the field `dragon.safety',
which is set in `moyo.c'.

Green:  GNU Go thinks the dragon is alive
Yellow: Status unknown
Blue:   GNU Go thinks the dragon is dead
Red:    Status critical (1.5 eyes) or weak by the algorithm
        in `moyo.c'

   If you want to get the same game over and over again, you can
eliminate the randomness in GNU Go's play by providing a fixed random
seed with the `-r' option.


File: gnugo.info,  Node: PM Implementation,  Next: Symmetry & transformations,  Prev: Tuning,  Up: Patterns

Implementation
==============

   The pattern code in GNU Go is fairly straightforward conceptually,
but because the matcher consumes a significant part of the time in
choosing a move, the code is optimized for speed. Because of this there
are implementation details which obscure things slightly.

   In GNU Go, the ascii `.db' files are precompiled into tables (see
`patterns.h') by a standalone program `mkpat.c', and the resulting `.c'
files are compiled and linked into the main gnugo executable.

   Each pattern is compiled to a header, and a sequence of elements,
which are (notionally) checked sequentially at every position and
orientation of the board. These elements are relative to the pattern
'anchor' (or origin).  One `X' or `O' stone is (arbitrarily) chosen to
represent the origin of the pattern. (We cannot dictate one or the
other since some patterns contain only one colour or the other.)  All
the elements are in co-ordinates relative to this position. So a
pattern matches "at" board position `(m,n,o)' if the the pattern anchor
stone is on `(m,n)', and the other elements match the board when the
pattern is transformed by transformation number `o'. (See below for the
details of the transformations, though these should not be necessary)


File: gnugo.info,  Node: Symmetry & transformations,  Next: Details,  Prev: PM Implementation,  Up: Patterns

Symmetry and transformations
============================

   In general, each pattern must be tried in each of 8 different
permutations, to reflect the symmetry of the board. But some patterns
have symmetries which mean that it is unnecessary (and therefore
inefficient) to try all eight. The first character after the `:' can be
one of `8',`|',`\',`/', `X', `-', `+', representing the axes of
symmetry. It can also be `O', representing symmetry under 180 degrees
rotation.

transformation   I    -    |     .     \    l    r     /
                ABC  GHI  CBA   IHG   ADG  CFI  GDA   IFC
                DEF  DEF  FED   FED   BEH  BEH  HEB   HEB
                GHI  ABC  IHG   CBA   CFI  ADG  IFC   GDA

                 a    b    c     d     e    f    g     h

   Then if the pattern has the following symmetries, the following are
true:


     |  c=a, d=b, g=e, h=f
     -  b=a, c=d, e=f, g=h
     \  e=a, g=b, f=c, h=d
     /  h=a, f=b, g=c, e=d
     O  a=d, b=c, e=h, f=g
     X  a=d=e=h, b=c=f=g
     +  a=b=c=d, e=f=g=h

   We can choose to use transformations a,d,f,g as the unique
transformations for patterns with either `|', `-', `\', or `/' symmetry.

   Thus we choose to order the transformations a,g,d,f,h,b,e,c and
choose first 2 for `X' and `+', the first 4 for `|', `-', `/', and `\',
the middle 4 for `O', and all 8 for non-symmetrical patterns.

   Each of the reflection operations (e-h) is equivalent to reflection
about one arbitrary axis followed by one of the rotations (a-d).  We
can choose to reflect about the axis of symmetry (which causes no net
change) and can therefore conclude that each of e-h is equivalent to
the reflection (no-op) followed by a-d.  This argument therefore
extends to include `-' and `/' as well as `|' and `\'.


File: gnugo.info,  Node: Details,  Next: grid optimization,  Prev: Symmetry & transformations,  Up: Patterns

Implementation Details
======================

  1. An entry in the pattern header states whether the anchor is an `X'
     or an `O'. This helps performance, since all transformations can be
     rejected at once if the anchor stone does not match. (Ideally, we
     could just define that the anchor is always `O' or always `X', but
     some patterns contain no `O' and some contain no `X'.)

  2. The pattern header contains the size of the pattern (ie the
     co-ordinates of the top left and bottom right elements) relative to
     the anchor. This allows the pattern can be rejected quickly if
     there is not room for the pattern to fit around the anchor stone
     in a given orientation (ie it is too near the edge of the board).
     The bounding box information must first be transformed like the
     elements before it can be tested, and after transforming, we need
     to work out where the top-left and bottom-right corners are.

  3. The edge constraints are implemented by notionally padding the
     pattern with rows or columns of `?' until it is exactly 19 (or
     whatever the current board size is) elements wide or high. Then the
     pattern is quickly rejected by (ii) above if it is not at the
     edge. So the example pattern above is compiled as if it was written


          "example"
          .OO????????????????
          *XX????????????????
          o??????????????????
          :8,80

  4. The elements in a pattern are sorted so that non-space elements
     are checked before space elements. It is hoped that, for most of
     the game, more squares are empty, and so the pattern can be more
     quickly rejected doing it this way.

  5. The actual tests are performed using an 'and-compare' sequence.
     Each board position is a 2-bit quantity.  %00 for empty, %01 for
     `O', %10 for `X'.  We can test for an exact match by and-ing with
     %11 (no-op), then comparing with 0, 1 or 2. The test for `o' is the
     same as a test for 'not-X', ie not %10. So and with %01 should
     give 0 if it matches. Similarly `x' is a test that bit 0 is not
     set.



File: gnugo.info,  Node: grid optimization,  Next: Joseki Compiler,  Prev: Details,  Up: Patterns

The "Grid" Optimization
=======================

   The comparisons between pattern and board are performed as 2-bit
bitwise operations. Therefore they can be performed in parallel,
16-at-a-time on a 32-bit machine.

   Suppose the board is layed out as follows :


      .X.O....OO
      XXXXO.....
      .X..OOOOOO
      X.X.......
      ....X...O.

which is internally stored internally in a 2d array (binary)


      00 10 00 01 00 00 00 00 01 01
      10 10 10 10 01 00 00 00 00 00
      00 10 00 00 01 01 01 01 01 01
      10 00 10 00 00 00 00 00 00 00
      00 00 00 00 10 00 00 00 01 00

we can compile this to a composite array in which each element stores
the state of a 4x4 grid of squares :


      ????????  ????????  ???????? ...
      ??001000  00100001  10000100
      ??101010  10101010  10101001
      ??001000  00100000  10000001
     
      ??001000  00100001  ...
      ??101010  10101010
      ??001000  00100000
      ??001000  10001000
     
     ...
     
      ??100010  ...
      ??000000
      ????????
      ????????

   Where '??' is off the board.

   We can store these 32-bit composites in a 2d merged-board array,
substituting the illegal value %11 for '??'.

   Similarly, for each pattern, mkpat produces appropriate 32-bit
and-value masks for the pattern elements near the anchor. It is a
simple matter to test the pattern with a similar test to (5) above, but
for 32-bits at a time.


File: gnugo.info,  Node: Joseki Compiler,  Next: Ladders in Joseki,  Prev: grid optimization,  Up: Patterns

The Joseki Compiler
===================

   GNU Go includes a joseki compiler in `patterns/joseki.c'. This
processes an SGF file (with variations) and produces a sequence of
patterns which can then be fed back into mkpat. The joseki database is
currently in files in `patterns/' called `hoshi.sgf', `komoku.sgf',
`sansan.sgf', `mokuhazushi.sgf' and `takamoku.sgf'.  This division can
be revised whenever need arises.

   The SGF files are transformed into the pattern database `.db' format
by the program in `joseki.c'. These files are in turn transformed into C
code by the program in `mkpat.c' and the C files are compiled and linked
into the GNU Go binary.

   Not every node in the SGF file contributes a pattern. The nodes which
contribute patterns have the joseki in the upper right corner, with the
boundary marked with a square mark and other information to determine
the resulting pattern marked in the comments.

   The intention is that the move valuation should be able to choose
between the available variations by normal valuation. When this fails
the primary workaround is to use shape values to increase or decrease
the value. It is also possible to add antisuji variations to forbid
popular suboptimal moves. As usual constraints can be used, e.g. to
condition a variation on a working ladder.

   The joseki format has the following components for each SGF node:

   * A square mark (`SQ' or `MA' property) to decide how large part of
     the board should be included in the pattern.

   * A move (`W' or `B' property) with the natural interpretation.  If
     the square mark is missing or the move is a pass, no pattern is
     produced for the node.

   * Optional labels (`LB' property), which must be a single letter
     each.  If there is at least one label, a constraint diagram will be
     produced with these labels.

   * A comment (`C' property). As the first character it should have
     one of the following characters to decide its classification:
        - `U' - urgent move

        - `S' or `J' - standard move

        - `s' or `j' - lesser joseki

        - `T' - trick move

        - `t' - minor joseki move (tenuki OK)

        - `0' - antisuji (`A' can also be used)
     The rest of the line is ignored, as is the case of the letter. If
     neither of these is found, it's assumed to be a standard joseki
     move.

     In addition to this, rows starting with the following characters
     are recognized:
        - `#' - Comments. These are copied into the patterns file,
          above the diagram.

        - `;' - Constraints. These are copied into the patterns file,
          below the constraint diagram.

        - `>' - Actions. These are copied into the patterns file, below
          the constraint diagram.

        - `:' - Colon line. This is a little more complicated, but the
          colon line of the produced patterns always start out with
          ":8,s" for transformation number and sacrifice pattern class
          (it usually isn't a sacrifice, but it's pointless spending
          time checking for tactical safety). Then a joseki pattern
          class character is appended and finally what is included on
          the colon line in the comment for the SGF node.

   Example: If the comment in the SGF file looks like

     F
     :C,shape(3)
     ;xplay_attack(A,B,C,D,*)

the generated pattern will have a colon line

     :8,sjC,shape(3)

and a constraint

     ;xplay_attack(A,B,C,D,*)


File: gnugo.info,  Node: Ladders in Joseki,  Prev: Joseki Compiler,  Up: Patterns

Ladders in Joseki
=================

   As an example of how to use autohelpers with the Joseki compiler, we
consider an example where a Joseki is bad if a ladder fails. Assume we
have the taisha and are considering connecting on the outside with the
pattern

     --------+
     ........|
     ........|
     ...XX...|
     ...OXO..|
     ...*O...|
     ....X...|
     ........|
     ........|

   But this is bad unless we have a ladder in our favor. To check this
we add a constraint which may look like

     --------+
     ........|
     ........|
     ...XX...|
     ...OXO..|
     ...*OAC.|
     ....DB..|
     ........|
     ........|
     
     ;oplay_attack(*,A,B,C,D)

   In order to accept the pattern we require that the constraint on the
semicolon line evaluates to true. This particular constraint has the
interpretation "Play with alternating colors, starting with `O', on the
intersections `*', `A', `B', and `C'. Then check whether the stone at
`D' can be captured." I.e. play to this position

     --------+
     ........|
     ........|
     ...XX...|
     ...OXO..|
     ...OOXX.|
     ....XO..|
     ........|
     ........|

and call `attack()' to see whether the lower `X' stone can be captured.
This is not limited to ladders, but in this particular case the
reading will of course involve a ladder.

   The constraint diagram above with letters is how it looks in the
`.db' file. The joseki compiler knows how to create these from labels in
the SGF node. `Cgoban' has an option to create one letter labels, but
this ought to be a common feature for SGF editors.

   Thus in order to implement this example in SGF, one would add labels
to the four intersections and a comment:

     ;oplay_attack(*,A,B,C,D)

   The appropriate constraint (autohelper macro) will then be added to
the Joseki `.db' file.


File: gnugo.info,  Node: DFA,  Next: Tactical Reading,  Prev: Patterns,  Up: Top

The DFA pattern matcher
***********************

   In this chapter, we describe the principles of the gnugo DFA pattern
matcher.  The aim of this system is to permit a fast pattern matching
when it becomes time critical like in owl module (*Note The Owl
Code::). Since GNU Go 3.2, this is enabled by default. You can still
get back the traditional pattern matcher by running `configure
--disable-dfa' and then recompiling GNU Go.

   Otherwise, a finite state machine called a Deterministic Finite
State Automaton (*Note What is a DFA::) will be built off line from the
pattern database. This is used at runtime to speedup pattern matching
(*Note Pattern matching with DFA:: and *Note Incremental Algorithm::).
The runtime speedup is at the cost of an increase in memory use and
compile time.

* Menu:

* Introduction to the DFA::  Scanning the board along a path
* What is a DFA:: A recall of language theory.
* Pattern matching with DFA:: How to retrieve go patterns with a dfa ?
* Building the DFA:: Playing with explosives.
* Incremental Algorithm:: The joy of determinism.
* DFA Optimizations:: Some possible optimizations.


File: gnugo.info,  Node: Introduction to the DFA,  Next: What is a DFA,  Prev: DFA,  Up: DFA

Introduction to the DFA
-----------------------

   The general idea is as follows:

   For each intersection of the board, its neighbourhood is scanned
following a predefined path.  The actual path used does not matter very
much; GNU Go uses a spiral as shown below.

       +---B--------------+
       | C 4 A . . . . . .|
       D 5 1 3 9 . . . . .|
       E 6 2 8 . . X . . .|
       | F 7 . . . . . . .|
       | . +-> . . . . . .|
       | . . . . . . . . .|
       | . O . . . X . . .|
       | . . . . . . . . .|
       | . . . . . . . . .|
       +------------------+

   In each step of the path, the pattern matcher jumps into a state
determined by what it has found on the board so far. If we have
successfully matched one or several patterns in this step, this state
immediately tells us so (in its "attribute").  But the state also
implicitly encodes which further patterns can still get matched: The
information stored in the state contains in which state to jump next,
depending on whether we find a black, white or empty intersection (or
an intersection out of board) in the next step of the path. The state
will also immediately tell us if we cannot find any further pattern (by
telling us to jump into the "error" state).

   These sloppy explanations may become clearer with the definitions in
the next section (*Note What is a DFA::).

   Reading the board following a predefined path reduces the two
dimentional pattern matching to a linear text searching problem.  For
example, this pattern

     ?X?
     .O?
     ?OO

scanned following the path

      B
     C4A
     5139
     628
      7

gives the string "OO?X.?*O*?*?"  where "?" means 'don't care' and "*"
means 'don't care, can even be out of board'.

   So we can forget that we are dealing with two dimensional patterns
and consider linear patterns.


File: gnugo.info,  Node: What is a DFA,  Next: Pattern matching with DFA,  Prev: Introduction to the DFA,  Up: DFA

What is a DFA
-------------

   The acronym DFA means Deterministic Finite state Automaton (See
`http://www.eti.pg.gda.pl/~jandac/thesis/node12.html' or `Hopcroft &
Ullman "Introduction to Language Theory"' for more details).  DFA are
common tools in compilers design (Read `Aho, Ravi Sethi, Ullman
"COMPILERS: Principles, Techniques and Tools"' for a complete
introduction), a lot of powerfull text searching algorithm like
`Knuth-Morris-Pratt' or `Boyer-Moore' algorithms are based on DFA's
(See `http://www-igm.univ-mlv.fr/~lecroq/string/' for a bibliography of
pattern matching algorithms).

   Basically, a DFA is a set of "states" connected by labeled
"transitions".  The labels are the values read on the board, in gnugo
these values are EMPTY, WHITE, BLACK or OUT_BOARD, denoted respectively
by '.','O','X' and '#'.

   The best way to represent a dfa is to draw its transition graph: the
pattern "????..X" is recognized by the following DFA:

        .,X,O     .,X,O    .,X,O    .,X,O     .      .      X
     [1]------>[2]----->[3]----->[4]----->[5]--->[6]--->[7]--->[8 OK!]
     Start

   This means that starting from state [1], if you read '.','X' or 'O'
on the board, go to state [2] and so on until you reach state [5].
From state [5], if you read '.', go to state [6] otherwise go to error
state [0].  And so on until you reach state [8].  As soon as you reach
state [8], you recognize Pattern "????..X"

   Adding a pattern like "XXo" ('o' is a wildcard for not 'X') will
transform directly the automaton by synchronization product (*Note
Building the DFA::).  Consider the following DFA:

     Start .,O   .,X,O    .,O,X   .,X,O      .      .       X
     [1]---->[2]----->[3]----->[4]------>[5]--->[6]---->[7]--->[8 OK!]
      |                ^        ^         ^
      |            .,O |        |         |
      |            ----         |         |
      |           |          X  |         |
      |           |          ---    .,X,O |
      |           |         |             |
      |     X     |   X     | O,.         |
       --------->[9]------>[A]--->[B OK!]-

   By adding a special "error" state and completing each state by a
transition to error state when there is none, we transform easily a DFA
in a "Complete Deterministic Finite state Automaton" (CDFA).  The
synchronization product (*Note Building the DFA::) is only possible on
CDFA's.

     Start .,O   .,X,O    .,O,X   .,X,O      .      .       X
     [1]---->[2]----->[3]----->[4]------>[5]--->[6]---->[7]--->[8 OK!]
      |                ^        ^         ^      |       |      |
      |            .,O |        |         |      |       |      |
      |            ----         |         |      |       |      |
      |           |          X  |         |      |X,O    | .,O  |X,.,O
      |           |          ---    .,X,O |      |       |      |
      |           |         |             |      |       |      |
      |     X     |   X     | O,.         |     \ /     \ /    \ /
       --------->[9]------>[A]--->[B OK!]-      [0  Error state !]

   The graph of a CDFA is coded by an array of states: The 0 state is
the "error" state and the start state is 1.

     ----------------------------------------------------
      state  |   .    |   O    |   X    |   #    |  att
     ----------------------------------------------------
           1 |      2 |      2 |      9 |      0 |
           2 |      3 |      3 |      3 |      0 |
           3 |      4 |      4 |      4 |      0 |
           5 |      6 |      0 |      0 |      0 |
           6 |      7 |      0 |      0 |      0 |
           7 |      0 |      0 |      8 |      0 |
           8 |      0 |      0 |      0 |      0 | Found pattern "????..X"
           9 |      3 |      3 |      A |      0 |
           A |      B |      B |      4 |      0 |
           B |      5 |      5 |      5 |      0 | Found pattern "XXo"
     ----------------------------------------------------

   To each state we associate an often empty list of attributes which
is the list of pattern indexes recognized when this state is reached.
In '`dfa.h'' this is basically represented by two stuctures:

     `
     /* dfa state */
     typedef struct state
     {
       int next[4]; /* transitions for EMPTY, BLACK, WHITE and OUT_BOARD */
       attrib_t *att;
     }
     state_t;
     
     /* dfa */
     typedef struct dfa
     {
       attrib_t *indexes; /* Array of pattern indexes */
       int maxIndexes;
     
       state_t *states; /* Array of states */
       int maxStates;
     }
     dfa_t;'


File: gnugo.info,  Node: Pattern matching with DFA,  Next: Building the DFA,  Prev: What is a DFA,  Up: DFA

Pattern matching with DFA
-------------------------

   Recognizing with a DFA is very simple and thus very fast (See
'`scan_for_pattern()'' in the '`engine/matchpat.c'' file).

   Starting from the start state, we only need to read the board
following the spiral path, jump from states to states following the
transitions labelled by the values read on the board and collect the
patterns indexes on the way.  If we reach the error state (zero), it
means that no more patterns will be matched.  The worst case complexity
of this algorithm is o(m) where m is the size of the biggest pattern.

   Here is an example of scan:

   First we build a minimal dfa recognizing these patterns: "X..X",
"X???", "X.OX" and "X?oX".  Note that wildcards like '?','o', or 'x'
give multiple out-transitions.

     ----------------------------------------------------
      state  |   .    |   O    |   X    |   #    |  att
     ----------------------------------------------------
           1 |      0 |      0 |      2 |      0 |
           2 |      3 |     10 |     10 |      0 |
           3 |      4 |      7 |      9 |      0 |
           4 |      5 |      5 |      6 |      0 |
           5 |      0 |      0 |      0 |      0 |    2
           6 |      0 |      0 |      0 |      0 |    4    2    1
           7 |      5 |      5 |      8 |      0 |
           8 |      0 |      0 |      0 |      0 |    4    2    3
           9 |      5 |      5 |      5 |      0 |
          10 |     11 |     11 |      9 |      0 |
          11 |      5 |      5 |     12 |      0 |
          12 |      0 |      0 |      0 |      0 |    4    2
     ----------------------------------------------------

   We perform the scan of the string "X..XXO...." starting from state 1:

   Current state:     1, substring to scan : X..XXO....

   We read an 'X' value, so from state 1 we must go to state 2.

   Current state:     2, substring to scan : ..XXO....

   We read a '.' value, so from state 2 we must go to state 3 and so on
...

     Current state:     3, substring to scan : .XXO....
     Current state:     4, substring to scan : XXO....
     Current state:     6, substring to scan : XO....
     Found pattern 4
     Found pattern 2
     Found pattern 1

   After reaching state 6 where we match patterns 1,2 and 4, there is
no out-transitions so we stop the matching.  To keep the same match
order as in the standard algorithm, the patterns indexes are collected
in an array and sorted by indexes.


File: gnugo.info,  Node: Building the DFA,  Next: Incremental Algorithm,  Prev: Pattern matching with DFA,  Up: DFA

Building the DFA
----------------

   The most flavouring point is the building of the minimal DFA
recognizing a given set of patterns.  To perform the insertion of a new
pattern into an already existing DFA one must completly rebuild the
DFA: the principle is to build the minimal CDFA recognizing the new
pattern to replace the original CDFA with its "synchronised product" by
the new one.

   We first give a formal definition: Let L be the left CDFA and R be
the right one.  Let B be the "synchronised product" of L by R.  Its
states are the couples (l,r) where l is a state of L and r is a state
of R.  The state (0,0) is the error state of B and the state (1,1) is
its initial state.  To each couple (l,r) we associate the union of
patterns recognized in both l and r.  The transitions set of B is the
set of transitions (l1,r1)--a-->(l2,r2) for each symbol 'a' such that
both l1--a-->l2 in L and r1--a-->r2 in R.

   The maximal number of states of B is the product of the number of
states of L and R but almost all this states are non reachable from the
initial state (1,1).

   The algorithm used in function '`sync_product()'' builds the minimal
product DFA only by keeping the reachable states.  It recursively scans
the product CDFA by following simultaneously the transitions of L and
R. A hast table (`gtest') is used to check if a state (l,r) has already
been reached, the reachable states are remapped on a new DFA. The CDFA
thus obtained is minimal and recognizes the union of the two patterns
sets.

   It is possible to construct a special pattern database that
generates an "explosive" automaton: the size of the DFA is in the worst
case exponential in the number of patterns it recognizes.  But it
doesn't occur in pratical situations: the dfa size tends to be
"stable".  By "stable" we mean that if we add a pattern which greatly
increases the size of the dfa it also increases the chance that the
next added pattern does not increase its size at all.  Nevertheless
there are many ways to reduce the size of the DFA. Good compression
methods are explained in `Aho, Ravi Sethi, Ullman "COMPILERS:
Principles, Techniques and Tools" chapter Optimization of DFA-based
pattern matchers'.


File: gnugo.info,  Node: Incremental Algorithm,  Next: DFA Optimizations,  Prev: Building the DFA,  Up: DFA

Incremental Algorithm
---------------------

   The incremental version of the DFA pattern matcher is not yet
implemented in gnugo but we explain here how it will work.  By
definition of a deterministic automaton, scanning the same string will
reach the same states every time.

   Each reached state during pattern matching is stored in a stack
`top_stack[i][j]' and `state_stack[i][j][stack_idx]' We use one stack
by intersection `(i,j)'.  A precomputed reverse path list allows to
know for each couple of board intersections `(x,y)' its position
`reverse(x,y)' in the spiral scan path starting from `(0,0)'.

   When a new stone is put on the board at `(lx,ly)', the only work of
the pattern matcher is:

     `
      for(each stone on the board at (i,j))
         if(reverse(lx-i,ly-j) < top_stack[i][j])
           {
              begin the dfa scan from the state
              state_stack[i][j][reverse(lx-i,ly-j)];
           }
     '

   In most situations reverse(lx-i,ly-j) will be inferior to
top_stack[i][j]. This should speedup a lot pattern matching.


File: gnugo.info,  Node: DFA Optimizations,  Prev: Incremental Algorithm,  Up: DFA

Some DFA Optimizations
----------------------

   The dfa is constructed to minimize jumps in memory making some
assumptions about the frequencies of the values: the EMPTY value is
supposed to appear often on the board, so the the '.'  transition are
almost always successors in memory.  The OUT_BOARD are supposed to be
rare, so '#' transitions will almost always imply a big jump.


File: gnugo.info,  Node: Tactical Reading,  Next: Life and Death Reading,  Prev: DFA,  Up: Top

Tactical reading
****************

   The process of visualizing potential moves done by you and your
opponent to learn the result of different moves is called "reading".
GNU Go does three distinct types of reading: "tactical reading" which
typically is concerned with the life and death of individual strings,
"Owl reading" which is concerned with the life and death of dragons,
and "life reading" which attempts evaluate eye spaces. In this Chapter,
we document the tactical reading code, which is in `engine/reading.c'.
For a summary of the reading functions see *Note Reading Functions::.

* Menu:

* Reading Basics::               Reading Basics
* Hashing::                      Hashing of positions
* Persistent Cache::             Persistent Reading Cache
* Ko::                           Ko handling
* A Ko Example::                 A Ko Example
* Another Ko Example::           Another Ko Example
* Alternate Komaster Schemes::   Alternate Komaster Schemes
* Superstrings::                 Superstrings
* Reading Functions::            Utilities from `engine/reading.c'
* Debugging::                    Debugging the reading code


File: gnugo.info,  Node: Reading Basics,  Next: Hashing,  Prev: Tactical Reading,  Up: Tactical Reading

Reading Basics
==============

   In GNU Go, tactical reading is done by the functions in
`engine/reading.c'. Each of these functions has a separate goal to fill,
and they call each other recursively to carry out the reading process.

   The reading code makes use of a stack onto which board positions can
be pushed. The parameter `stackp' is zero if GNU Go is examining the
true board position; if it is higher than zero, then GNU Go is
examining a hypothetical position obtained by playing several moves.

   The most important public reading functions are `attack' and
`find_defense'. These are wrappers for functions `do_attack' and
`do_find_defense' which are declared statically in `reading.c'. The
functions `do_attack' and `do_find_defense' call each other recursively.

   The return codes of the reading (and owl) functions and owl can be
0, 1, 2 or 3. Each reading function determines whether a particular
player (assumed to have the move) can solve a specific problem,
typically attacking or defending a string.

   The nonzero return codes are called these names in the source:

        #define WIN  3
        #define KO_A 2
        #define KO_B 1

   A return code of `WIN' means success, 0 failure, while `KO_A' and
`KO_B' are success conditioned on ko. A function returns `KO_A' if the
position results in ko and that the player to move will get the first
ko capture (so the opponent has to make the first ko threat). A return
code of `KO_B' means that the player to move will have to make the
first ko threat.

   Many of the reading functions make use of "null pointers".  For
example, a call to `attack(str, &apos)' will return `WIN' if the string
at `str' can be captured. The point of attack (in case it is
vulnerable) is returned in `apos'. However many times we do not care
about the point of attack. In this case, we can substitute a null
pointer: `attack(str, NULL)'.

   Depth of reading is controlled by the parameters `depth' and
`branch_depth'. The `depth' has a default value `DEPTH' (in
`liberty.h'), which is set to 16 in the distribution, but it may also
be set at the command line using the `-D' or `--depth' option.  If
`depth' is increased, GNU Go will be stronger and slower. GNU Go will
read moves past depth, but in doing so it makes simplifying assumptions
that can cause it to miss moves.

   Specifically, when `stackp > depth', GNU Go assumes that as soon as
the string can get 3 liberties it is alive. This assumption is
sufficient for reading ladders.

   The `branch_depth' is typically set a little below `depth'.  Between
`branch_depth' and `depth', attacks on strings with 3 liberties are
considered, but branching is inhibited, so fewer variations are
considered.

   Currently the reading code does not try to defend a string by
attacking a boundary string with more than two liberties. Because of
this restriction, it can make oversights. A symptom of this is two
adjacent strings, each having three or four liberties, each classified
as `DEAD'. To resolve such situations, a function `small_semeai()' (in
`engine/semeai.c') looks for such pairs of strings and corrects their
classification.

   The `backfill_depth' is a similar variable with a default 12. Below
this depth, GNU Go will try "backfilling" to capture stones.  For
example in this situation:


     .OOOOOO.    on the edge of the board, O can capture X but
     OOXXXXXO    in order to do so he has to first play at a in
     .aObX.XO    preparation for making the atari at b. This is
     --------    called backfilling.

   Backfilling is only tried with `stackp <= backfill_depth'. The
parameter `backfill_depth' may be set using the `-B' option.

   The `fourlib_depth' is a parameter with a default of only 7.  Below
this depth, GNU Go will try to attack strings with four liberties. The
`fourlib_depth' may be set using the `-F' option.

   The parameter `ko_depth' is a similar cutoff. If `stackp<ko_depth',
the reading code will make experiments involving taking a ko even if it
is not legal to do so (i.e., it is hypothesized that a remote ko threat
is made and answered before continuation).  This parameter may be set
using the `-K' option.

   A partial list of the functions in `reading.c' (*note Reading
Functions:: for a fuller list).

   * `int attack(int str, int *move)'
          This basic function determines if the string at `str' can be
          attacked, and if so, `*move' returns the attacking move,
          unless `*movei' is a null pointer. (Use null pointers if you
          are interested in the result of the attack but not the
          attacking move itself.) Returns 1 if the attack succeeds,
          otherwise 0. Returns `KO_A' or `KO_B' if the result depends
          on ko: returns `KO_A' if the attack succeeds provided
          attacker is willing to ignore any ko threat. Returns `KO_B'
          if attack succeeds provided attacker has a ko threat which
          must be answered.

   * `find_defense(int str, int *move)'
          Attempts to find a move that will save the string at `str'. It
          returns true if such a move is found, with `*move' the
          location of the saving move (unless `*move' are null
          pointers). It is not checked that tenuki defends, so this may
          give an erroneous answer if `!attack(str)'.  Returns `KO_A'
          or `KO_B' if the result depends on ko. Returns `KO_A' if the
          string can be defended provided `color' is willing to ignore
          any ko threat. Returns `KO_B' if `color' has a ko threat
          which must be answered.

   * `safe_move(int str, int color)' :
          The function `safe_move(str, color)' checks whether a move at
          `str' is illegal or can immediately be captured. If
          `stackp==0' the result is cached. If the move only can be
          captured by a ko, it's considered safe. This may or may not
          be a good convention.


File: gnugo.info,  Node: Hashing,  Next: Hash Calculation,  Prev: Reading Basics,  Up: Tactical Reading

Hashing of Positions
====================

   To speed up the reading process, we note that a position can be
reached in several different ways.  In fact, it is a very common
occurrence that a previously checked position is rechecked, often
within the same search but from a different branch in the recursion
tree.

   This wastes a lot of computing resources, so in a number of places,
we store away the current position, the function we are in, and which
worm is under attack or to be defended.  When the search for this
position is finished, we also store away the result of the search and
which move made the attack or defense succeed.

   All this data is stored in a hash table, sometimes also called a
transposition table, where Go positions are the key and results of the
reading for certain functions and groups are the data. You can increase
the size of the Hash table using the `-M' or `--memory' option *note
Invoking GNU Go::.

   The hash table is created once and for all at the beginning of the
game by the function `hashtable_new()'. Although hash memory is thus
allocated only once in the game, the table is reinitialized at the
beginning of each move by a call to `hashtable_clear()' from
`genmove()'.

* Menu:

* Hash Calculation::            Calculation of the hash value
* Hash Organization::           Organization of the hash table
* Hash Structures::             Structures in `hash.h'
* Hash Functions::              Hash functions and macros


File: gnugo.info,  Node: Hash Calculation,  Next: Hash Organization,  Prev: Hashing,  Up: Hashing

Calculation of the hash value
-----------------------------

   The hash algorithm is called Zobrist hashing, and is a standard
technique for go and chess programming. The algorithm as used by us
works as follows:

  1. First we define a "go position".  This positions consists of
        * the actual board, i.e. the locations and colors of the stones

        * A "ko point", if a ko is going on.  The ko point is defined as
          the empty point where the last single stone was situated
          before it was captured.

     It is not necessary to specify the color to move (white or black)
     as part of the position. The reason for this is that read results
     are stored separately for the various reading functions such as
     `attack3', and it is implicit in the calling function which player
     is to move.

  2. For each location on the board we generate random numbers:
        * A number which is used if there is a white stone on this
          location

        * A number which is used if there is a black stone on this
          location

        * A number which is used if there is a ko on this location

     These random numbers are generated once at initialization time and
     then used throughout the life time of the hash table.

  3. The hash key for a position is the XOR of all the random numbers
     which are applicable for the position (white stones, black stones,
     and ko position).


File: gnugo.info,  Node: Hash Organization,  Next: Hash Structures,  Prev: Hash Calculation,  Up: Hashing

Organization of the hash table
------------------------------

   The hash table consists of 3 parts:

   * An area which contains so called "Hash Nodes". Each hash node
     contains:
        - A go position as defined above.

        - A computed hash value for the position

        - A pointer to Read Results (see below)

        - A pointer to another hash node.

   * An area with so called Read Results.  These are used to store
     which function was called in the go position, which string was
     under attack or to be defended, and the result of the reading.

     Each Read Result contains:
        - the function ID (an int between 0 and 255), the position of
          the string under attack and a depth value, which is used to
          determine how deep the search was when it was made, packed
          into one 32 bit integer.

        - The result of the search (a numeric value) and a position to
          play to get the result packed into one 32 bit integer.

        - A pointer to another Read Result.

   * An array of pointers to hash nodes.  This is the hash table proper.


   When the hash table is created, these 3 areas are allocated using
`malloc()'.  When the hash table is populated, all contents are taken
from the Hash nodes and the Read results. No further allocation is done
and when all nodes or results are used, the hash table is full.
Nothing is deleted from the hash table except when it is totally
emptied, at which point it can be used again as if newly initialized.

   When a function wants to use the hash table, it looks up the current
position using `hashtable_search()'. If the position doesn't already
exist there, it can be entered using

   `hashtable_enter_position()'.

   Once the function has a pointer to the hash node containing a
function, it can search for a result of a previous search using
`hashnode_search()'.  If a result is found, it can be used, and if not,
a new result can be entered after a search using
`hashnode_new_result()'.

   Hash nodes which hash to the same position in the hash table
(collisions) form a simple linked list.  Read results for the same
position, created by different functions and different attacked or
defended strings also form a linked list.

   This is deemed sufficiently efficient for now, but the representation
of collisions could be changed in the future.  It is also not
determined what the optimum sizes for the hash table, the number of
positions and the number of results are.

