# Hatari Debugger User's Manual

Version 2.6.1, August 2025

Debugger manual written by: Eero Tamminen

See also: Hatari manual

Hatari on the WWW: https://www.hatari-emu.org/

## Index

## The debugger

Hatari has a built-in debugging interface which can be used for analyzing code that runs in the emulated system.

On Unix (Linux / macOS) debugger uses Hatari's parent console window, so run Hatari from the command line to use the debugger. On Windows "-W" option is needed for the console window. You can add a shortcut / icon on your desktop that does it. On Linux such shortcut would execute something like this (replace "xterm" with your favorite terminal program):

    xterm -T "Hatari debug window" -e hatari

To run debugger commands from a file at Hatari startup, one can use the "--parse <file>" command line option. This is useful e.g. for debugging TOS or some demo startup code, or if you always want to use some specific debugger setup (breakpoints etc).

Note that when debugger scripts are run, current directory is set to the currently running script's directory i.e. all file operations are relative to it. After script finishes, earlier current directory is restored. To set current directory from a setup script, e.g. for scripts run at breakpoints, "-f" option needs to be give for the "cd" command.

### Invoking the debugger

Debugger can be invoked with the AltGr + Pause key combination.

Command line "-D" option can be used to toggle whether m68k exceptions will invoke the debugger. Which exceptions cause this, can be controlled with the "--debug-except" option.

Giving "-D" option at Hatari startup is not advised because TOS HW checks generate some exceptions at every TOS boot. It is better to toggle exception catching later from the debugger with the "setopt -D" command.

Alternatively, "--debug-except" option can be prefixed with "prg:" (e.g. "--debug-except prg:all") to enable catching of (specified) exceptions when Atari program given on Hatari command line gets autostarted.

### Debugger configuration options

When saving Hatari configuration, current debugger settings are also saved to the Hatari configuration file.

Settings are following:

nNumberBase
  Debugger number base. Set with the debugger "setopt [bin|dec|hex]" command
nSymbolLines
  Number of lines to show when listing debug symbols
nMemdumpLines
  Number of memory dump lines to show
nFindLines
  Number of find (search) lines to show
nDisasmLines
  Number of disassembly lines to show
nBacktraceLines
  Number of items to show in stack/bactraces
nExceptionDebugMask
  Mask of exceptions which invoke debugger when exceptions catching is enabled (-D). Set with the "--debug-except" option
nDisasmOptions
  Disassembler output options, set with the "--disasm" option
bDisasmUAE
  Whether disassembly uses CPU core internal disassembler ("uae"), or the external disassembler ("ext" one with output options). Set with the "--disasm" option
nSymbolsAutoLoad
  Specifies whether (GEMDOS HD) program symbols are autoloaded when program is executed, when debugger is invoked after that, or whether symbol autoloading is disabled. Set with the "--symload" option and the "symbols autoload [on|off]" command
bMatchAllSymbols
  Whether symbol name TAB-completion matches just symbols from a relevant code section, or all of them. Toggled with the "symbols match" command

These are their defaults:

    [Debugger]
    nNumberBase = 10
    nSymbolLines = -1
    nMemdumpLines = -1
    nFindLines = -1
    nDisasmLines = -1
    nBacktraceLines = 0
    nExceptionDebugMask = 515
    nDisasmOptions = 15
    bDisasmUAE = TRUE
    nSymbolsAutoLoad = 1
    bMatchAllSymbols = FALSE

Settings on how many lines are shown can be changed only from the configuration file. This needs to changed/set only when Hatari debugger is built without readline support, as it is then unable to determine terminal size ("-1" = use terminal height).

Note that "--debug-except" and "--disasm" options can be given either on Hatari command line, or (like all other Hatari command line options), with the debugger "setopt" command.

### General debugger use

At the debugger prompt, type "help" to get a list of all the available commands and their shortcuts:

    Generic commands:
               cd (  ) : change directory
             echo (  ) : output given string(s)
         evaluate ( e) : evaluate an expression
             help ( h) : print help
          history (hi) : show last CPU and/or DSP PC values + instructions
             info ( i) : show machine/OS information
             lock (  ) : specify information to show on entering the debugger
          logfile ( f) : open or close log file
            parse ( p) : get debugger commands from file
           rename (  ) : rename given file
            reset (  ) : reset emulation
       screenshot (  ) : save screenshot to given file
           setopt ( o) : set Hatari command line and debugger options
        stateload (  ) : restore emulation state
        statesave (  ) : save emulation state
            trace ( t) : select Hatari tracing settings
        variables ( v) : List builtin symbols / variables
             quit ( q) : quit emulator

    CPU commands:
          address ( a) : set CPU PC address breakpoints
       breakpoint ( b) : set/remove/list conditional CPU breakpoints
           disasm ( d) : disassemble from PC, or given address
             find (  ) : find given value sequence from memory
          profile (  ) : profile CPU code
           cpureg ( r) : dump register values or set register to value
          memdump ( m) : dump memory
           struct (  ) : structured memory output, e.g. for breakpoints
         memwrite ( w) : write bytes to memory
          loadbin ( l) : load a file into memory
          savebin (  ) : save memory to a file
          symbols (  ) : load CPU symbols & their addresses
             step ( s) : single-step CPU
             next ( n) : step CPU through subroutine calls / to given instruction type
             cont ( c) : continue emulation / CPU single-stepping

    DSP commands:
       dspaddress (da) : set DSP PC address breakpoints
         dspbreak (db) : set/remove/list conditional DSP breakpoints
        dspdisasm (dd) : disassemble DSP code
       dspmemdump (dm) : dump DSP memory
       dspsymbols (  ) : load DSP symbols & their addresses
       dspprofile (dp) : profile DSP code
           dspreg (dr) : read/write DSP registers
          dspstep (ds) : single-step DSP
          dspnext (dn) : step DSP through subroutine calls / to given instruction type
          dspcont (dc) : continue emulation / DSP single-stepping

#### Entering arguments to debugger commands

After writing (with TAB completion) one of the above command names, pressing TAB will (for most commands) show all the available subcommands.

To give numbers in other number bases than the default/selected one, they need to be prefixed with a character indicating this. For decimals this prefix is "#" (#15), for hexadecimals "$" ($F), and for binary values it is "%" (%1111).

By default debugger expects all numbers without a prefix to be decimals, but the default number base can be changed with the "setopt" command, by giving it the desired default number base (bin/dec/hex). When using the hexadecimal number base, remember still to prefix hexadecimal numbers with '$' if they could be confused with register names (a0-7, d0-7)! Otherwise results from expressions and conditional breakpoints can be unexpected.

#### Calculations and immediate evaluation

Instead of giving plain values, one can also use (arithmetic) expressions. Such expression can contain calculations with values of CPU and DSP registers, symbols and Hatari variables, in addition to plain numbers.

Most commands evaluate those by default, but for some of them (e.g. conditional breakpoints) such expressions need to be indicated by surrounding them with single quotes. For example to give a sum of A0 and D0 register values to a command, use 'a0+d0'.

Within expressions, parenthesis are used to indicate indirect addressing, not to change the order of precedence. Width of the addressing can be specified after the parenthesis. For example to get a long value pointed by stack pointer + 2, use '(a7+2).l'.

Values of expressions are always evaluated before being given to a command. Besides arithmetic, they can be used also to give symbol / address / register / variable values to commands that do not otherwise interpret them. If command complains that it did not recognize e.g. a register name, just surround it with single quotes and it will be "evaluated" before being given to the command.

(To support single quotes within string arguments, orphan single quotes are passed as-is, and empty single quote pairs are collapsed to a single ' character.)

Virtual V0-V7 "registers" can be used to store intermediate results for calculations. For example, to get a sum of "_counter" symbol address contents one could use following in suitable breakpoint:

    # store counter sum to V0 virtual register
    r v0='(_counter)'
    # store count of how many values are added
    r v1='v1+1'

And then later on, calculate the average:

    # round the counter sum (add half count to sum)
    r v2='v0 + v1/2'
    # and calculate the rounded average (rounded sum / count)
    e v2/v1

(Another virtual register was used for rounding here, in case one wants to continue summing the "_counter" values with the original value.)

With command argument completion (see readline), result from the last "evaluate" command can be inserted by typing '$' and pressing TAB.

#### Hatari outputs and their controls

When debugging something, there can be so much output that you want to store emulator output for more detailed inspection later.

Hatari has 5 different kinds of outputs and their controls:

- Tracing, controlled by "--trace" and "--trace-file" options
- Logging, controlled by "--log-level" and "--log-file" options
- Debugger CPU+DSP disassembly / memdump / register commands output, controlled by "logfile" command
- Xconout redirection [1] + Hatari command line help, hard-coded to "stdout"
- Rest of debugger (breakpoints etc) + Hatari output, hard-coded to "stderr"

With 3 first options defaulting to "stderr".

To catch all of them, it is better just to redirect all "stdout" and "stderr" output from Hatari to a file:

    hatari --parse debugger.ini --trace os_base 2>&1 | tee out.log

("2>&1" redirects stderr to stdout for piping, and "tee" command both saves and shows the output, so that one can still see the saved output in more or less real-time.)

[1] E.g. VT52 console redirection to Hatari standard output is enabled with "os_base" and "os_all" trace settings and "--conout 2" option.

### Inspecting emulation state

In the beginning, probably the most interesting commands are "m" and "d" for dumping and disassembling memory regions. To do the same for the DSP, use "dm" and "dd" commands.

    > help memdump
    'memdump' or 'm' - dump memory
    Usage:  m [b|w|l] [start address-[end address| count]]
    	dump memory at address or continue dump from previous address.
    	By default memory output is done as bytes, with 'w' or 'l'
    	option, it will be done as words/longs instead.  Output amount
    	can be given either as a count or an address range.

    > help disasm
    'disasm' or 'd' - disassemble from PC, or given address
    Usage:  d [start address-[end address]]
            If no address is given, this command disassembles from the last
            position or from current PC if no last position is available.

    > disasm pc
    (PC)
    $00aa6e : 2f08                                 move.l    a0,-(sp)
    $00aa70 : 0241 0fff                            andi.w    #$fff,d1
    $00aa74 : 207c 00fe 78c0                       movea.l   #$fe78c0,a0
    $00aa7a : 2070 1000                            movea.l   (a0,d1.w),a0
    $00aa7e : 4ed0                                 jmp       (a0)

Both commands accept in addition to numeric addresses also register and symbol names, like in above example. If address is not specified, commands continue showing from the next address after the previously shown data.

"disasm" command default address will be reset to program counter (PC) address every time debugger is re-entered. If history is enabled and it includes addresses just before PC, disassembly will instead start from a slightly earlier address to give more context.

Use "setopt --disasm help" to see options controlling the disassembly output.

Use the "info" command to see state of specific sets of HW registers (e.g. "info videl") and Atari OS structures (e.g. "info gemdos").

One can also show contents of arbitrary program structs with the "struct" command. Parts of the structure can be skipped (with 's') and they can be shown in different number bases:

    > help struct
    'struct' - structured memory output, e.g. for breakpoints
    Usage:  struct <name> <address>[name]:<type>[base][:<count>[/<split>]] ...]

            Show <name>d structure content at given <address>, with each
    	[name]:<type>[base][:<count>] arg output on its own line, prefixed
    	with offset from struct start address, if [name] is not given.
    	Output uses multiple lines when type count <split> is given.
    	Supported <type>s are 'b|c|w|l|s' (byte|char|word|long|skip).
    	Optional [base] can be 'b|o|d|h' (bin|oct|dec|hex).
    	Defaults are hex [base], and [count] of 1.

For example:

    > struct "TOS info" 0xe00000 :s:2 version:wh:1 :s:20 os_date:lh os_conf:wb :s:14 :c:4
    TOS info: $e00000
    + version : 0206
    + os_date : 03172024
    + os_conf : 0000000011111111
    + $2c     : ETOS

By saving such command to a file, it can be used with breakpoint ":file" option, to show contents of a given structure, whenever that breakpoint matches.

Prefixing "info" and "struct" commands with "echo \ec" command would clear screen before their output. This could help noticing changes in real-time output.

#### Selecting what information is shown on entering the debugger

By using the "lock" command, one can ask Hatari to show specific information whenever debugger is entered, e.g. due to a breakpoint. For example to see disassembly from current PC address, use "lock disasm".

"regaddr" subcommand shows disassembly or memory dump of an address pointed by a given register ("lock regaddr disasm a0"). Of the DSP registers, only Rx ones are valid for this subcommand.

"file" subcommand can be used to get (arbitrary number of) commands parsed and executed from a given debugger input file whenever debugger is entered. With this one can output any needed information:

    lock file debugger.ini

To disable showing of this extra information, use "lock default". Without arguments "lock" command will show the available options (like the "info" command does).

### Debug symbols

By default, Hatari loads debug symbols for programs executed from GEMDOS HD whenever debugger is first invoked after that, if program includes debug symbols, or there's a separate *.sym file for them.

Symbolic names for program function and data addresses can be used in arithmetic expressions and conditional breakpoint expressions. They show up in the "disasm" command output, and one can trace calls to them with "trace cpu_symbols" (and DSP symbols with "trace dsp_symbols"). If profiling was enabled with symbols present, debugger shows also a backtrace (for the program code executed since debugger was last invoked) whenever debugger is invoked.

To load debugging symbols in other situations, use the debugger "symbols" command (and "dspsymbols" for DSP).

#### Demangling (C++) symbols

C++ (and other similar languages) store symbols in binaries in a "mangled" format used by tools like linkers, and symbol names need to be "demangled" (expanded) to a more readable / user-friendly format. This can be done using a tool coming with the C++ compiler (in "binutils-m68k-atari-mint" package). ScummVM example:

    $ gst2ascii scummvm.ttp | m68k-atari-mint-c++filt > scummvm.sym

(When symbols are in a file named as "<prgname>.sym", Hatari debugger will load symbols from it, not from the program file.)

Note: most demangled C++ symbols, such as method signatures, contain special characters which prevent them from being used as arguments to breakpoints and other debugger commands. To solve that, debugger commands can accept also a part of a C++ method symbol name, if it is a complete sub-part of that name, and it matches only a single symbol.

For example, given the following C++ symbols:

    - void ScummEngine::startScriptQuick()
    - int ScummEngine::startScriptQuick2(int, int)
    - int ScummEngine::startObject(int, char*)

Following partial names would be accepted:

    - startScriptQuick
    - ScummEngine::startObject

But these would not:

    - ScummEngine (multiple matches)
    - startObj    (incomplete symbol part match)

This leaves out destructor, template and operator overloading method names because those contain characters already reserved for breakpoint and debugger expression operations (and without those characters, partial name would produce multiple matches).

For rest of the demangled symbols, one needs to use symbol addresses instead. Addresses for the loaded symbols (that match user specified substring), can be listed like this:

    symbols name XMLParser::parse

#### Symbols for a program under GEMDOS HD emulation

If currently running program contains debug symbol table, and it is started from GEMDOS HD emulated drive, its symbol names / addresses are automatically loaded when debugger is invoked, and removed when that program terminates.

Above happens only if there are no symbols loaded when the program starts. If there are, one can load program symbol data manually with the following command, after program has been loaded to the memory by TOS (see setting breakpoint at program startup):

    symbols prg

The options needed to add a suitable symbol table to the program, depend on which toolchain is used to build it:

Devpac:
  "OPT D+,X+"
AHCC:
  "-g" and "-l" options for linking
GCC:
  "-g" for compilation, and no strip option for linking
VBCC:
  "-g" (can only be used at linking phase), when VBCC configuration file uses "-bataritos" option for the linker

Generated symbols can be viewed (and converted to debugger ASCII format) with a tool installed with Hatari:

    $ gst2ascii program.tos > program.sym

(By default "gst2ascii" filters out same symbols as Hatari debugger does.)

For C++ programs, pipe the that output through m68k-atari-mint-c++filt demangler (see above) before directing it to a file.

##### Overriding program symbols

Symbols in a program can be overridden by providing similarly named ".sym" file in the same directory. For example, if there is a file called "program.sym", debugger will try to load symbols from that instead of "program.prg" (when debugger is first invoked after given program started).

There are few reasons why one might want to use ".sym" file:

- Debugging C++ programs with demangled (user-friendly) symbol names
- Binary itself can be stripped (be much smaller) when symbols are provided in a separate ".sym" file / download
- Addding missing (function) symbols, or removing (loop) symbols, to improve profiling results

#### For a program on a (disk) image

If the program is not run from a GEMDOS HD emulated drive, but from a cartridge, floppy or HD image, corresponding program is needed also as a normal host file which location can be given to the debugger:

    symbols /path/to/the/program.tos

#### ASCII debug symbol files

If Hatari complains that program does not have debug symbol table, or its symbols are in some unsupported format, there are two options:

- Convert the symbols to ASCII format understood by the Hatari debugger. Writing converters for other ASCII formats is easy, and Hatari already contains converters for DSP LOD files, nm output, and a few other formats.
- Create the ASCII symbols file by hand while debugging a program.

NOTE: nm output for GCC generated a.out binaries includes labels also for loops, not just functions. While loop labels are fine for debugging, they should be removed before profiling. Besides causing misleading profile results, loop labels can seriously slow down profiling (call graph tracking is automatically enabled for profiling when debug symbols are loaded, and operations done on each matched symbol address cause huge overhead if that match is for something happening every few instructions).

ASCII symbols file format is following:

    e01034 T random
    e01076 T kbdvbase
    e0107e T supexec

Where 'T' means text (code), 'D' means data and 'B' means BSS section type of address. The hexadecimal address, address type letter and the symbol name are separated by white space. Empty lines and lines starting with '#' (comments) are ignored.

Debugger will automatically "relocate" the symbol addresses when it loads them from a program binary, but with ASCII symbol files relocation offset needs to be specified separately, unless the symbol names are for fixed addresses (like is the case e.g. with EmuTOS):

    symbols program.sym TEXT

If there are symbols for DATA and BSS sections, they are also assumed to have TEXT address offsets. This is the case both with "gst2ascii" tool, and "nm" output for (at least GCC generated) binaries.

(TEXT, DATA and BSS are virtual debugger variables which values come from the currently loaded program's basepage. They're set after the program is loaded by TOS, see "info basepage" output.)

#### Debugging resident programs

When debugging resident (TSR) programs (terminated with a Ptermres() GEMDOS call), it's common to have a 'trigger' program that invokes some functionality in the TSR being debugged. With symbol autoloading enabled, executing the 'trigger' program from Hatari GEMDOS HD could replace symbols for the TSR with ones from the 'trigger' program.

Symbol replacement can be avoided in two ways:

- Using "symbols autoload off" debugger command, or
- Loading TSR symbols from an ASCII file (see above). This is the way to go if TSR is not a normal Atari program, or some symbols need to be added or filtered out (e.g. for profiling)

### Breakpoints

There are two ways to specify breakpoints for Hatari. First, there are the simple address breakpoints which trigger when the CPU (or DSP) program counter hits a given address. Use "a" (or "da" for the DSP) to create them, for example:

    a $e01034
    a some_symbol

Note that address breakpoints are just wrappers for conditional breakpoints so "b" command is needed to list or remove them.

Then there are the conditional breakpoints which can handle much more complex break condition expressions; they can track changes to register and memory values with bitmasks, include multiple conditions for triggering a breakpoint and so on. Use "b" (or "db" for the DSP) to manage them.

Help explains the general syntax:

    > help b
    'breakpoint' or 'b' - set/remove/list conditional CPU breakpoints
    Usage:  b <condition> [&& <condition> ...] [:<option>] | <index> | help | all

    Set breakpoint with given <conditions>, remove breakpoint with
    given <index>, remove all breakpoints with 'all' or output
    breakpoint condition syntax with 'help'.  Without arguments,
    lists currently active breakpoints.

Unless "breakpoint" command is given one of the pre-defined subcommands ("all", "help"), an index for a breakpoint to remove, or no arguments (to list breakpoints), its arguments are interpreted as a new breakpoint definition.

Each conditional breakpoint can have (currently up to 4) conditions which are separated by "&&". All of the breakpoint's conditions need to be true for a breakpoint to trigger.

#### Breakpoint options

Normally when a breakpoint is triggered, emulation is stopped and debugger invoked. Breakpoint options can be used to affect what happens when a breakpoint is triggered. These options are given after the conditions, and are prefixed with a (space and) ':' character.

<count>
  Break only on every <count> hit. For example, to stop on every other time PC is at given address, use:

      a $1234 :2

file <file>
  Execute debugger commands from given <file> when this breakpoint is hit. This provides complete control over what information is show when the debugger is hit, one can even chain breakpoints (as explained in Chaining breakpoints later on). This should be used when "info" and "lock" options are not enough.
info <name>
  Show on breakpoint hits the same information as "info" command would show (see Inspecting emulation state above). With "lock" option and command there is more control over what information is shown, whereas with "info" option, every breakpoint can show different information, and one does not need to change what is shown on entering the debugger. This option also enables trace option.
lock
  Show the same information on breakpoint hit as what is shown when entering the debugger (see the "lock" command in Inspecting emulation state above). This enables also trace option as one would anyway see this information when entering debugger.
noinit
  Avoid debugger initialization (profiling data reset and disassembly address being set to current PC) on breakpoint hit. This enables trace option as entering debugger would anyway re-initialize debugger state. This option is mainly intended for breakpoints that use either ":file" or ":lock" option to show backtraces with "profile stack" command during profiling. See Usage examples section for an example.
once
  Delete the breakpoint when it is hit, i.e. trigger it only once. Useful when just wanting to reach a specific address. Or when on an instruction that jumps back to a start of the loop, and one wants to finish the loop, following could use:

      b pc > "pc" :once
      continue

print <string>
  Print given string when breakpoint is hit. When many breakpoints can hit in succession, this helps in differentiating between them.
quiet
  Inhibit showing of extra information when breakpoint is either set or hit i.e. show only the information that breakpoint itself outputs.
trace
  Continue emulation without stopping after printing the value that triggered the breakpoint and doing other possible option actions. This is most useful when investigating memory or register value changes (explained below).

Arguments to "file", "info" and "print" options are terminated by the next option (to ':' character), or end of input.

Note: while multiple options can be given for conditional breakpoints, address breakpoints accept only one (with no argument).

#### Breakpoint conditions

"b help" explains very briefly the breakpoint condition syntax:

    > b help
    condition = <value>[.mode] [& <mask>] <comparison> <value>[.mode]

    where:
            value = [(] <register/symbol/variable name | number> [)]
            number/mask = [#|$|%]<digits>
            comparison = '<' | '>' | '=' | '!'
            addressing mode (width) = 'b' | 'w' | 'l'
            addressing mode (space) = 'p' | 'x' | 'y'

For CPU breakpoints, mode is the address width; it can be byte ("b"), word ("w") or long ("l", default). For DSP breakpoints, mode specifies the address space: "P", "X" or "Y". Note that on DSP only R0-R7 registers can be used for memory addressing. For example;

    db (r0).x = 1 && (r0).y = 2

If the value is in parenthesis like in "($ff820)" or "(a0)", then the used value will be read from the memory address pointed by it. Note that this conditional breakpoint expression value is checked at run-time whereas evaluated arithmetic expressions (mentioned in Entering arguments to debugger commands above) are evaluated already when adding a breakpoint. For example, to break when a value in an address (later) pointed by A0 matches the value currently in D0, one would use:

    b (a0) = 'd0'

When interested only on certain bits in the value, one can use '&' and a numeric mask on either side of comparison operator to mask the corresponding value, like this:

    b ($ff820).w & 3 = (a0)  &&  (a1) = d0 & %1100

Comparison operators should be familiar and obvious, except for '!' which indicates inequality ("is not") comparison. For example:

    b d0 > $20  &&  d0 < $40  &&  d0 ! $30

##### Tracking breakpoint conditions

As a convenience, if the both sides of the comparison are exactly the same (i.e. condition is redundant as it is always either true or false), the right side of the comparison is replaced with its current value. With that, one can give something like this:

    b pc > "pc"

As:

    b pc > pc

That in itself is not so useful, but for inequality ('!') comparison, conditional breakpoint will additionally track and output all further changes for the given address/register expression. This can be used for example to find out all value changes in a given memory address, like this:

    b ($ffff9202).w ! ($ffff9202).w :trace

Because tracking breakpoint conditions will print the evaluated value when it changes, they're typically used with the trace option to track changes e.g. to some I/O register.

##### Breakpoint condition notes

- Any '!' condition should be given as the first condition. Because breakpoint evaluation is stopped ("short-circuited") when any of the conditions fails, the tracked value would not be updated correctly unless tracking condition is given as the first one.
- Hatari will internally update some register values without immediately updating the corresponding I/O address range memory addresses. For example the Busy bit for the internal Blitter control register is (internally) cleared when Blitter activity stops, but the actual I/O address for that control register gets updated only when something actually writes or reads that I/O address. Many HW registers behave like this (status registers in FDC, ACIA, MFP, Blitter...).
  For breakpoints that track just a single I/O register memory address, or multiple ones of which none are modified by Hatari, only by emulated code, this is not a problem, they get triggered as expected.
  However, a breakpoint that tracks multiple I/O registers where some of them are updated by Hatari (for example one checking that other Blitter registers are not updated while control register indicates Blitter to be active/busy), will not work as expected!

#### Breakpoint variables

In addition to loaded symbols, the debugger supports also setting conditional breakpoints on values of some "virtual" variables listed by "variables" (v) command. For example:

- Aes/Bios/Gemdos/LineA/LineF/Vdi/XbiosOpcode variables can be used to catch AES, BIOS, GEMDOS, Line-A, Line-F, VDI and XBIOS OS-calls. By default they contain the 0xffff value, so to trace e.g. all AES calls (instead of a specific one) one needs to use something like this:

      b  AesOpcode ! AesOpcode  &&  AesOpcode < 0xffff  :trace

- To stop when TOS starts loading next program, set a breakpoint for Pexec(0,...) OS call:

      b GemdosOpcode = 0x4B && OsCallParam = 0x0

- To stop emulation after program has been loaded, but before it runs, set next breakpoint on its first instruction i.e. when program counter matches the TEXT (code) segment address (taken from program basepage):

      b  pc = TEXT :once

  Note1: Set this breakpoint only after boot reaches desktop, and trigger it only once. Otherwise during (re)boot there would be a warning for every instruction (until TOS sets a valid basepage).
  Note2: address breakpoint cannot be used for this because Hatari variable values are evaluated at run-time only for conditional breakpoints.
- To view current program DATA and BSS segment contents, use the corresponding variables:

      m  DATA
      m  BSS

- To stop at a specific cycle within a frame (that is, PC relative to the current VBL/HBL in cycles), set breakpoints to specific "VBL", "FrameCycles", "HBL" and "LineCycles" variable values. For example, to break after 20 HBLs, use:

      b  HBL = 'HBL+20'

- To stop on every symbol, break on:

      b  PConSymbol = 1

Hint: "info" command "aes", "bios", "gemdos", "vdi" and "xbios" subcommands can be used to list the corresponding OS-call opcodes. For example, to see the GEMDOS opcodes, use:

    info gemdos 1

#### Chaining breakpoints and other actions

As the file pointed by the breakpoint ":file" option (see Breakpoint options) can contain any debugger commands, it can also be used to do automatic "chaining" of debugger and breakpoint actions so that after one breakpoint is hit, another one is set.

For example, with these input files:

- "pexec.ini":

      # continue to "program.ini" on Pexec(0, ....)
      b GemdosOpcode = 0x4B && OsCallParam = 0x0 :trace :once :file program.ini

- "program.ini":

      # continue to "trace.ini" when program execution starts
      b pc = TEXT :trace :once :file trace.ini

- "trace.ini":

      # load symbols, trace gemdos & program function calls
      symbols prg
      trace gemdos,cpu_symbols
      # continue to "disable.ini" after 4 VBLs
      b VBL = 'VBL+4' :trace :once :file disable.ini

- "disable.ini":

      # stop tracing and remove breakpoints
      trace none
      b all

And then start Hatari with the first debugger input file:

    hatari --parse pexec.ini /path/to/your/program.tos

1. "pexec.ini" sets a breakpoint to parse debugger commands from "program.ini" when TOS starts loading the given program (it is first Pexec(0) after boot)
2. "program.ini" sets a breakpoint to parse debugger commands from "trace.ini" when program code begins executing. These two steps are needed because TEXT variable is not valid until TOS has booted
3. "trace.ini" input file loads symbols for the run program, sets Hatari to trace several things (see Tracing section below) in the emulated system for few VBLs until breakpoint runs commands from the "disable.ini" file
4. "disable.ini" input file will disable tracing and remove all (remaining) breakpoints

Note:

- Because debugger input files cannot "continue" emulation, ":trace" option needs to be used for the breakpoint(s) when emulation should continue after the breakpoint action(s).
- In simpler breakpoint chain (like above), where new breakpoint just replaces the previous one, ":once" option tells debugger that breakpoint is not needed after it is hit.

Hint: It is better to test each input file separately before testing the whole chain. Besides the ":file" breakpoint option, one can test these debugger input files also with the debugger "file" command, "file" option for the "lock" command, and with the Hatari "--parse" command line option.

#### Breakpoints in emulation state saves

When emulation state is saved, currently active breakpoints are saved to a separate file with additional ".debug" extension. If breakpoints refer to other files, that file starts with a "cd" command that sets Hatari working directory to what it was when state was saved.

This way the relative paths referred by breakpoint ":file" options (and possible further files referred from those files), should work fine. However, if any of the related files are moved, rest of the them need to be moved too, and the "cd" command path in the ".debug" state save file updated accordingly.

### Stepping through code

After analyzing the emulation state and/or setting new breakpoints, emulation can continued with the "c" command. One can continue for a given number of CPU instructions (or DSP instructions when "dc" is used), or continue forever (until a non-tracing breakpoint triggers) if the nstruction count is omitted.

To continue just to the next instruction, use "s" (step) command to execute just one instruction, or "n" (next), to skip subroutine + exception calls and DBCC branching backwards (i.e. loops). "ds" and "dn" commands do the same for DSP (except that "dn" does not skip loops).

One can also continue with the "n" until instruction of certain type is encountered, by giving it the instruction type:

- "branch" matches branch instructions:
  CPU: BCC, BRA, DBCC, JMP
  DSP: DO/ENDO JCC, JCLR, JMP, JSET, REP
- "subcall" matches subroutine calls:
  CPU: BSR, JSR
  DSP: JSCC, JSCLR, JSSET, JSR
- "subreturn" matches return from subroutine:
  CPU: RTD, RTR, RTS
  DSP: RTS
- "exception" matches exceptions:
  CPU: BKPT, ILLG, STOP, TRAP, TRAPV
- "exreturn" matches return from exception:
  CPU: RTE
  DSP: RTI
- "return" matches both subroutine and exception returns

"subreturn" differs from others by running until current subroutine ends, even if other subroutines are called before that. This is particularly useful for profiling more complex functions; set breakpoint on function start, enable profiling and run until that functions returns, to get its full profile. Example: "n subreturn", or "dn subreturn".

Notes:

- "next subreturn" works only for real subroutines i.e. code called with BSR/JSR and returning with RT[DRS]. In (GCC) optimized compiled C-code, calls to functions in same C / object file may get inlined or just called with JMP. If that is the case, and making the function non-static does not help, move it to another C-file
- "exreturn" and "return" run only until first instruction of given type is encountered. They cannot implement similar functionality as "subreturn", because it is not possible for "next" command to track CPU / DSP exception invocations, and therefore it cannot do call depth tracking required for this either
- Tracking CPU CHK2 and FPU FBCC, FDBCC & FTRAPCC exception / branch instructions is not supported currently

### Tracing

(Hatari needs to be built with ENABLE_TRACING define set for tracing to work. By default it is.)

For example, to continue with real-time disassembling, that can be enabled with "trace cpu_disasm" (or "trace dsp_disasm" for DSP) at the debugger prompt, before continuing.

Disable tracing with "trace none" when entering the debugger again. "trace help" (or TAB) can be used to list all the (about 70) supported traceable things, from HW events to OS functions.

At run-time trace flags can be enabled and disabled individually by starting the trace flags with -/+, like this:

    trace gemdos,aes,vdi   # trace just these
    trace +xbios,bios      # trace additionally these
    trace -aes,-vdi        # remove tracing of these

('+' is optional for addition except at start of the trace flags list.)

Notes:

- If GEMDOS HD emulation is not enabled, GEMDOS call tracing needs to be enabled at Hatari command line, it is not possible to enable it after TOS has initialized GEMDOS.
- AES, BIOS, GEMDOS and XBIOS traces show arguments for (most of) the calls, VDI trace shows only function calls (parsing the arguments would be too complicated).
- Tracing options can be set even from a program within the emulation, by enabling the (deprecated) "--bios-intercept" option and calling XBios 255 from the program with a suitable trace options string.
- Note that the trace output file can be set only when Hatari starts, it cannot be changed from within the debugger (or emulation).

If there is not a trace option for something one would like to track, it should be possible do that with tracing breakpoints, as explained above. For example, following tracks Line-A calls:

    b  LineAOpcode ! LineAOpcode  &&  LineAOpcode < 0xffff  :trace

### Profiling

Hatari supports both Atari programs profiling themselves, and its debugger includes powerful profiling functionality, especially when its use is automated with conditional breakpoints.

Profiling tells where the running code spends most of its (emulated) time. It can be used to find out where a program is (apparently) stuck, or what are the largest performance bottlenecks for a program.

#### NF_CYCLES NatFeat for Atari programs

Motorola CPUs do not have CPU cycle counter instruction, but Atari programs can query (lower 32-bits of) Hatari's internal cycles counter value. Returned counter value differences tell how much time (cycles) elapsed between successive queries.

NF_CYCLES "Native Feature" can be enabled with the "--natfeats on" option. Examples of using Hatari Native Features are included to Hatari "tests/natfeats" directory.

Note: 32-bit cycle counter will wrap in few minutes even on 8-32Mhz CPUs, so caller needs to handle counter wrapping.

#### Collecting profile data in the debugger

Profiling is used by first enabling the profiler (use "dp" for DSP):

    > profile on
    Profiling enabled.

And profiling will start once emulation is continued:

    > c
    Returning to emulation...
    Allocated CPU profile buffer (27 MB).

At debugger invocation, the collected profiling information is processed, a backtrace and a summary of in which parts of memory the execution happened, and how long it took, are shown:

    Allocated CPU profile address buffer (57 KB).
    ROM TOS (0xE00000-0xE80000):
    - active address range:
      0xe00030-0xe611a4
    - active instruction addresses:
      14240 (100.00% of all)
    - executed instructions:
      4589668 (100.00% of all)
    - used cycles:
      56898472 (100.00% of all)
      = 7.09347s
    Cartridge ROM (0xFA0000-0xFC0000):
      - no activity

    = 7.09347s

(DSP RAM will be shown only as single area in profile information.)

If program symbols were loaded, profiler will also show a program backtrace (covering the profiled part). This is especially useful with EmuTOS, because EmuTOS does not immediately terminate programs on crashes, but first waits for a key. By invoking debugger at that point, one will see profiler backtrace for the crash.

#### Investigating the profile data

When back in the debugger, the collected profile data can be inspected:

    > h profile
    'profile' - profile CPU code
    Usage:  profile <subcommand> [parameter]

    	Subcommands:
    		- on
    		- off
    		- counts [count]
    		- cycles [count]
    		- i-misses [count]
    		- d-hits [count]
    		- symbols [count]
    		- addresses [address]
    		- callers
    		- caches
    		- stack
    		- stats
    		- save <file>
    		- loops <file> [CPU limit] [DSP limit]

    	'on' ¨ 'off' enable and disable profiling.  Data is collected
    	until debugger is entered again, at which point profiling
    	statistics ('stats') summary is shown.

    	Most active PC addresses can be queried, sorted either by
    	execution 'counts', used 'cycles', i-cache misses or d-cache hits.
    	First can be limited just to named addresses with 'symbols'.
    	Optional count will limit how many items will be shown.

    	'caches' shows histogram of CPU cache usage.

    	'addresses' lists the profiled addresses in order, with the
    	instructions (currently) residing at them.  By default this
    	starts from the first executed instruction, or from the
    	specified start address.

    	'callers' shows (raw) caller information for addresses which
    	had symbol(s) associated with them.  'stack' shows the current
    	profile stack (this is useful only with :noinit breakpoints).

    	Profile address and callers information can be saved with
    	'save' command.

    	Detailed (spin) looping information can be collected by
    	specifying to which file it should be saved, with optional
    	limit(s) on how many bytes first and last instruction
    	address of the loop can differ (0 = no limit).

For example, to see which memory addresses were executed most and what instructions those have at the end of profiling, use:

    > profile counts 8
    addr:           count:
    0xe06f10        12.11%  555724  move.l    $4ba,d1
    0xe06f16        12.11%  555724  cmp.l     d1,d0
    0xe06f18        12.11%  555724  bgt.s     $e06f06
    0xe06f06        12.11%  555708  move.b    $fffffa01.w,d1
    0xe06f0a        12.11%  555708  btst      #5,d1
    0xe06f0e        12.11%  555708  beq.s     $e06f1e
    0xe00ed8         1.66%  76001   subq.l    #1,d0
    0xe00eda         1.66%  76001   bpl.s     $e00ed8
    8 CPU addresses listed.

Then, to see what the executed code and its costs look like around top addresses:

    > profile addresses 0xe06f04
    # disassembly with profile data:
    # <instructions percentage>% (<sum of instructions>, <sum of cycles>, <sum of i-cache misses>, <sum of d-cache hits>)

    $e06f04 :             bra.s     $e06f10                    0.00% (48, 576, 0, 0)
    $e06f06 :             move.b    $fffffa01.w,d1            12.11% (555708, 8902068, 0, 0)
    $e06f0a :             btst      #5,d1                     12.11% (555708, 6685268, 0, 0)
    $e06f0e :             beq.s     $e06f1e                   12.11% (555708, 4457312, 0, 0)
    $e06f10 :             move.l    $4ba,d1                   12.11% (555724, 11125668, 0, 0)
    $e06f16 :             cmp.l     d1,d0                     12.11% (555724, 4461708, 0, 0)
    $e06f18 :             bgt.s     $e06f06                   12.11% (555724, 4455040, 0, 0)
    $e06f1a :             moveq     #1,d0                      0.00% (16, 64, 0, 0)
    Disassembled 8 (of active 14240) CPU addresses.

Unlike normal disassembly, "profile addresses" command shows only memory addresses which instructions were executed during profiling. Cache hit/miss information is provided only when using cycle-accurate 680x0 emulation.

With symbol information loaded, symbol names are shown above the corresponding addresses. And "profile symbols" command provides a list of how many times the code execution passed through the defined symbol addresses.

#### Profile data accuracy

Profile data accuracy depends on Hatari emulation accuracy. Profile data accuracy, from most to least accurate, with default Hatari emulation options, is following:

- CPU and DSP instruction counts: accurate.
- 68000 cycle counts: tested to be accurate to within 1%.
- MegaSTE 16Mhz 68000 (16KB) cache: hit/miss/cycle counts should also be accurate.
- DSP cycle counts (and the variance information): should be accurate.
- 030 (256B+256B) cache: hit/miss counts are assumed to be accurate.
- 030 cycle counts: can be off by tens of percents.
- FPU cycle counts: can be off by 2x.
- TT-RAM access: perf improvement over ST-RAM is not emulated.
- 040/060 cycle counts: reported, but not accurate, nor really emulated - ignore.

#### Caller information

With symbols loaded (see Debug symbols) before continuing emulation/profiling, additional caller information will be collected for all the code symbol addresses which are called as subroutines. This information includes callstack, call counts, calling instruction type (subroutine call, branch, return etc), and costs for those calls, both including costs for further subroutine calls and without them.

When debugger is re-entered, current callstack is output before profiling information:

    > a _P_LineAttack
    CPU condition breakpoint 1 with 1 condition(s) added:
            pc = $30f44
    $030f44 : 48e7 3820                            movem.l   d2-d4/a2,-(sp)
    > c
    ...
    CPU breakpoint condition(s) matched 1 times.
            pc = $30f44
    Finalizing costs for 12 non-returned functions:
    - 0x32a3c: _P_GunShot (return = 0x32b7e)
    - 0x32b18: _A_FireShotgun (return = 0x3229a)
    - 0x3223a: _P_SetPsprite (return = 0x32e86)
    - 0x32e4e: _P_MovePsprites (return = 0x38070)
    - 0x37f44: _P_PlayerThink (return = 0x36ea0)
    - 0x36e44: _P_Ticker (return = 0x260e0)
    - 0x25dcc: _G_Ticker (return = 0x1e4c6)
    - 0x1e29e: _TryRunTics (return = 0x239fa)
    - 0x238e8: _D_DoomLoop (return = 0x2556a)
    - 0x24d7a: _D_DoomMain (return = 0x44346)
    ...

("profile stack" command can be used in breakpoints with :noinit option to show backtraces during caller profiling.)

Note: rest of this subsection is about caller information format which is mainly of interest for people writing profiling post-processing tools. Come back here if there is some problem with callgraphs produced by those tools.

Other information collected during profiling is shown with following command:

    > profile callers
    # <callee>: <caller1> = <calls> <types>[ <inclusive/totals>[ <exclusive/totals>]], <caller2> ..., <callee name>
    # types: s = subroutine call, r = return from subroutine, e = exception, x = return from exception,
    #        b = branch/jump, n = PC moved to next instruction, u = unknown PC change
    # totals: calls/instructions/cycles/misses
    0xe00030: 0xffffff = 1 e, _main
    0xe000fe: 0xe00a0c = 1 b, memdone
    0xe0010a: 0xe04e34 = 1 s 1/5/72 1/5/72, _run_cartridge_applications
    0xe00144: 0xe04dbe = 1 s 4/118/1512 1/27/444, _init_acia_vecs
    0xe001ea: 0xe00ec6 = 1 b, _int_acia
    0xe0038c: 0xe04c28 = 1 s 1/191/2052 1/191/2052, _init_exc_vec
    0xe003a6: 0xe04c2e = 1 s 1/388/4656 1/388/4656, _init_user_vec
    ...

For example, when one does not know all the places from which a certain function is called, or in what context a certain interrupt handler can be called during the period being profiled, profile caller information will tell that:

    callee: caller: calls: calltype:
      |       |       |   /
    0x379:  0x155 = 144 r, 0x283 = 112 b, 0x2ef = 112 b, 0x378 = 72 s
    583236/359708265/1631189180 72/4419020/19123430, dsp_interrupt
               |                       |                 |
        inclusive costs         exclusive costs     callee name
      (of calls from 0x378)

    Calltypes:
    - b: jump/branch
    - n: PC  just moved to next address
    - r: subroutine return
    - s: subroutine call

(Most "calls" to "dsp_interrupt" were subroutine call returns (=r) to it from address 0x155.)

With the execution counts in normal profiling data, caller information can actually be used to have complete picture of what exactly the code did during profiling. Main/overview work for this analysis is best done automatically, by the profiler data post-processor (documented below).

#### Caller data accuracy

Everything about profile data accuracy applies also to caller costs, but there are additional things to take into account, mainly because profiler cannot determine when exceptions are being handled:

- If there are exception(s) during a subroutine call, costs for the exception handling will also be accounted for that subroutine. This should not be a problem unless those costs are very large, i.e. check how much CPU those exception handlers take.
- Indicated exception handler call type can be incorrect.
- Profiled code doing return address related stack manipulations confuses call tracking and produces incorrect results (profiler has special code to handle EmuTOS AES switcher because of this). Typically this produces large list of functions that are finalized at profile end, so it should be easy to detect.
- Complicated recursive calls seem to sometimes cause inclusive costs (ones including costs of further subroutine calls) to be incorrect, e.g. >100%.
- On DSP, profiler heuristics assume (for speed reasons) that conditional subroutine calls never call the very next instruction (as that would be very bad/inefficient code).

#### Saving profile data to a file

It is useful to save the profile data to a file:

    > profile save program-profile.txt

With the saved profile disassembly (and optional caller information) one can more easily investigate what the program did during profiling, search symbols & addresses in it, and compare the results to profiles saved from earlier versions of the code.

One could even create own post-processing tools for investigating the profiling data more closely, e.g. to find CPU/DSP communication bottlenecks.

### Profile data post-processing

Saved profile data can be post-processed with (Python) script installed by Hatari, to:

- Get lists of functions/symbols with highest costs.
- Get callgraphs of what functions/symbols cause those costs and what kind of call hierarchy the profiled code has.
- Export profile data in Valgrind's Callgrind format for viewing it in Kcachegrind GUI.

#### Providing symbols for the post-processor

When the data is post-processed, one should always provide the post-processor symbols for the profile code! Relying just on the symbols in the profile disassembly can cause costs to be assigned to wrong symbol, if code within a function was not called through its symbol address, but by jumping directly somewhere inside the function.

If code is at a fixed location, one should tell post-processor to handle symbol addresses as absolute (-a):

    $ hatari_profile.py -a etos1024k.sym emutos-profile.txt

Normal programs are relocated and one should instead give the symbols as TEXT (code) section relative ones (-r):

    $ hatari_profile.py -r program.sym program-profile.txt

If symbols are included to the binary, first they need to be extracted to the ASCII format understood by the post-processor:

    $ gst2ascii -b -a -d program.prg > program.sym

(Options given to "gst2ascii" filter out symbols for other things than what are in the program code section.)

If there are some extra symbols that one does not want to see separately in profiles, because they are not real functions, but e.g. loop labels, one can either remove them manually from the ASCII *.sym file, or filter them out with "grep":

    $ gst2ascii -b -a -d program.prg | grep -v -e useless1 -e useless2 > program.sym

For C++ programs, see earlier section(s) on how to best provide symbols for them.

#### Post-processor provided statistics

Above post-processor examples just parse + verify the given data and produce output like this:

    Hatari profile data processor

    Parsing TEXT relative symbol address information from program.sym...
    [...]
    3237 lines with 1550 code symbols/addresses parsed, 0 unknown.

    Parsing profile information from program-profile.txt...
    [...]
    9575 lines processed with 368 functions.

    CPU profile information from 'program-profile.txt':
    - Hatari v1.6.2+ (May  4 2013), WinUAE CPU core

To get statistics (-s) and list of top (-t) CPU users in profile, add "-st" option:

    $ hatari_profile.py -st -r program.sym program-profile.txt
    [...]
    CPU profile information from 'program-profile.txt':
    - Hatari v1.6.2+ (May  4 2013), WinUAE CPU core

    Time spent in profile = 34.49539s.

    Calls:
    - max = 187738, in __toupper at 0x52b88, on line 8286
    - 1585901 in total
    Executed instructions:
    - max = 1900544, in flat_remap_mips+14 at 0x47654, on line 7020
    - 64499351 in total
    Used cycles:
    - max = 15224620, in flat_remap_mips+18 at 0x47658, on line 7022
    - 553392132 in total
    Instruction cache misses:
    - max = 184308, in _BM_T_GetTicks at 0x43b90, on line 4772
    - 4941307 in total

    Calls:
      11.84%      187698  __toupper
      11.48%      182105  _BM_T_GetTicks
      11.48%      182019  _I_GetTime
    [...]
    Executed instructions:
      34.83%    22462729  flat_generate_mips
      14.08%     9080215  flat_remap_mips
       8.55%     5515945  render_patch_direct
       5.09%     3283328  _TryRunTics
    [...]
    Used cycles:
      23.62%   130702768  flat_generate_mips
      12.42%    68735832  flat_remap_mips
       9.77%    54041148  _TryRunTics
       5.80%    32111536  correct_element
    [...]
    Instruction cache misses:
      37.03%     1829764  _TryRunTics
      11.20%      553314  _BM_T_GetTicks
       9.44%      466319  _NetUpdate
       9.27%      457899  _HGetPacket
    [...]

To see also symbol addresses and what is per call cost, add -i option:

    $ hatari_profile.py -st -i -r program.sym program-profile.txt
    [...]
    Executed instructions:
      34.83%    22462729  flat_generate_mips   (0x04778a, 774576 / call)
      14.08%     9080215  flat_remap_mips      (0x047646, 313110 / call)
       8.55%     5515945  render_patch_direct  (0x047382, 29977 / call)
       5.09%     3283328  _TryRunTics          (0x042356, 19660 / call)
    [...]
    Used cycles:
      23.62%   8.14728s  130702768  flat_generate_mips  (0x04778a, 0.28094s / call)
      12.42%   4.28461s   68735832  flat_remap_mips     (0x047646, 0.14775s / call)
       9.77%   3.36863s   54041148  _TryRunTics         (0x042356, 0.02017s / call)
       5.80%   2.00165s   32111536  correct_element     (0x04a658, 0.00001s / call)
    [...]
    Instruction cache misses:
      37.03%     1829764  _TryRunTics          (0x042356, 10956 / call)
      11.20%      553314  _BM_T_GetTicks       (0x043b90, 3 / call)
       9.44%      466319  _NetUpdate           (0x041bcc, 5 / call)
       9.27%      457899  _HGetPacket          (0x041754, 5 / call)
    [...]

(For cycles the "per call" information is in seconds, not as a cost count.)

If profile file contains caller information, "-p" option should be added to see it, as that will also help in detecting symbol issues (see Interpreting the numbers):

    $ hatari_profile.py -st -p -r program.sym program-profile.txt
    [...]
    9575 lines processed with 368 functions.
    [...]
    Of all 1570498 switches, ignored 581 for type(s) ['r', 'u', 'x'].

    CPU profile information from 'badmood-level-load-CPU.txt':
    - Hatari v1.6.2+ (May  4 2013), WinUAE CPU core
    [...]
    Calls:
      11.84%  11.84%      187698    187698  __toupper
      11.48%  11.48%      182105    182105  _BM_T_GetTicks
      11.48%  22.95%      182019    364038  _I_GetTime
    [...]
    Executed instructions:
      34.83%  34.86%  34.86%    22462729  22484024  22484024  flat_generate_mips
      14.08%  14.10%  14.10%     9080215   9091270   9091676  flat_remap_mips
       8.55%                     5515945                      render_patch_direct
       5.09%   5.11%  94.96%     3283328   3294022  61247717  _TryRunTics
    [...]
    Used cycles:
      23.62%  23.69%  23.69%   130702768 131100604 131100604  flat_generate_mips
      12.42%  12.46%  12.46%    68735832  68928816  68930904  flat_remap_mips
       9.77%   9.80%  95.66%    54041148  54238744 529368824  _TryRunTics
       5.80%   5.82%   5.82%    32111536  32193664  32193664  correct_element
    [...]
    Instruction cache misses:
      37.03%  37.14%  98.57%     1829764   1835261   4870573  _TryRunTics
      11.20%  11.24%  11.24%      553314    555191    555191  _BM_T_GetTicks
       9.44%   9.49%  29.13%      466319    468782   1439340  _NetUpdate
       9.27%   9.29%   9.37%      457899    459197    463217  _HGetPacket
    [...]

Now there's a message telling that some of the calls were ignored because according to their "call type", they were actually returns from exceptions, not real calls (this is mainly important for callgraph generation, discussed below).

#### Interpreting the results

In addition to accuracy issues mentioned in previous Profiling sections, function/symbol level costs have gotchas of their own.

The first cost percentage and count column are sums for costs accounted for all the addresses that were in profile data file between the indicated symbol's address and the address of the next symbol (= "between-symbols" cost).

NOTE: If symbols file does not contain addresses for all the relevant symbols, results from this can be misleading; instruction costs get assigned to whatever symbol's address happened to precede those instructions. One does not see which caller is causing it from the caller info or callgraphs either, as missing a symbol for an entry point for such a time sink, means that calls to it had not been tracked by profiler...

The next two percentages (and cost counts) are total cost of calls to given subroutine, based on profiler runtime branch tracking (see caller information documented above). First value is ("exclusive") cost for just that subroutine (from its entry, until execution returns to where it was called from), without costs for branches to further subroutines. Latter value is ("inclusive") cost covering also costs for all the subroutines it calls.

Reasons why "between-symbols" costs, and subroutine call costs can differ, are following:

- Subroutine terminates before next symbol address: "exclusive" cost is smaller than "between-symbols" cost because of missing symbol information (these are indicated with '*' in statistics).
- Subroutine is called more through jumps/branches than through subroutine calls (JSR/BSR): "inclusive" call count may be smaller than "between-symbols" call count which includes costs also for function entries through branches/jumps.
- Subroutine jumps/branches to another function instead of using subroutine call, or function contains additional (non-function) labels: "exclusive" cost is larger than "between-symbols" cost.
- Exception happening during subroutine call: "exclusive" cost is (slightly) larger than "between-symbols" cost.

In the first case, one should check saved profile disassembly to find out whether there are missing symbols for executed function entry points. One can notice function entry points as address gaps and/or instructions retrieving arguments from stack. Exit points can be seen from RTS instructions.

Second case can also be seen from the profile disassembly. Call count is same as count for how many times first instruction is executed (worst case: large loop on subroutine's first instruction).

While subroutine costs should be more accurate and relevant, due to code optimizations, many of the functions are not called as subroutines (on m68k, using JSR/BSR), but just jumped or branched to. Because of this, it is useful to compare both subroutine and "between-symbols" costs. One should be able to see from the profile disassembly which of the above cases is cause for the discrepancy in the values.

NOTE: Before starting to do any serious optimizations based on profile information, one should always verify from profile disassembly where exactly the costs are in a function, to make sure optimization efforts actually can have an impact on performance.

#### Generating and viewing callgraphs

Callgraphs require that saved profile data contains caller function address information, i.e. symbols for the code should be loaded before starting profiling it (see loading symbol data).

Separate callgraphs will be created for each of the costs (0=calls, 1=instructions, 2=cycles) with the -g option:

    $ hatari_profile.py -p -g -r program.sym program-profile.txt
    [...]
    Generating 'program-profile-0.dot' DOT callgraph file...

    Generating 'program-profile-1.dot' DOT callgraph file...

    Generating 'program-profile-2.dot' DOT callgraph file...
    [...]

Callgraphs are saved in GraphViz "dot" format. Dot files can be viewed:

- With "dotty" program included with GraphViz
- With XDot Python GUI (best option on Linux), or some platform specific viewer
- By converting dot file to PostScript or SVG format before viewing it with viewers for those:

      $ dot -Tsvg program-profile-1.dot > program-profile-1.svg

  (problem with most PS/PDF and SVG viewers is that either they do not allow zooming large callgraphs enough or they use huge amounts of memory and get very slow)

Produced callgraph will look like this:

Interpreting the callgraph:

- Diamond shaped nodes are symbols called as subroutines. Values listed in them are subroutine call costs; inclusive (total) cost with exclusive (own) cost in parenthesis, followed by inclusive cost count. Exclusive cost is shown only if it differs from inclusive one.
- Ellipse shaped nodes are for other symbols (functions called using jumps/branches, loop labels etc). Values listed in them are between-symbols costs, i.e. normally they are included to inclusive (total) costs shown in subroutine call node somewhere higher in call hierarchy.
- Nodes which exclusive (own) or between-symbols costs exceed default or explicitly given threshold value, have gray background.
- Both nodes, which inclusive or between-symbols cost exceeds the threshold value, and the arrows to & from them, are marked red.
- Arrow types indicate call types; normal arrows subroutine calls, circles branches/jumps, backarrows returns. Exception calls and returns are indicated with dashed lines, unknown calls with dotted lines.
- Arrow text tells from which address (within the caller) the call originated. If symbol had multiple callers, text includes count of calls from that particular address, and its percentage is of all calls done to that symbol.

#### Making large callgraphs readable

If profile is for larger and more varied amount of code (e.g. program startup), the resulting callgraph can be so huge that it not really readable anymore.

When code includes interrupt handlers, they can get called at any point, which can show in callgraph as "explicit" calls from the interrupted functions. To get rid of such incorrect calls, give interrupt handler names to "--ignore-to" option:

    $ hatari_profile.py -p -g --ignore-to handler1,handler2 -r program.sym program-profile.txt

In large callgraph most of the functions are not really interesting, because their contribution to the cost is insignificant. One can remove large number of them with "--no-leafs" and "--no-intermediate" options, those options act only on nodes which costs are below given threshold. Leaf nodes are ones which do not have any parents and/or children. Intermediate ones have only single parent and children (node calling itself is not taken into account).

Threshold for this is given with the "--limit" (-l) option. With that it typically makes also sense to change the node emphasis threshold with --emph-limit (-e) option:

    $ hatari_profile.py -p -g -l 0.5 -e 2.0 -r program.sym program-profile.txt

When not interested from how many different addresses a given function calls another function, use "--compact" option. If multiple calls between two nodes are still present with it, the reason is them happening through different call paths which were removed from the callgraph after "--compact" option was applied:

    $ hatari_profile.py -p -g -l 1.0 -e 2.0 --no-leafs --no-intermediate --compact -r program.sym program-profile.txt

If even this does not help, all nodes below the given cost threshold limit can be removed with the "--no-limited" option, but this often does not leave much of a call hierarchy. Instead you may consider removing all nodes except for subroutine call ones, with the "--only-subroutines" option.

When having trouble locating nodes of special interest, one can either color them differently with the "--mark" option, or exclude everything else from the callgraph except those nodes and their immediate callers & callees, with the "--only" option:

    $ hatari_profile.py -p -g --only func1,func2 -r program.sym program-profile.txt

Last option for reading the callgraph is using "-k" option to export the data for use in (Linux) Kcachegrind UI. Kcachegrind generates callgraphs on the fly, and just for the area around the selected function, so navigating in callgraph may be easier. It also shows the related profile disassembly, which can make verifying matters easier:

    $ hatari_profile.py -p -k -r program.sym program-profile.txt
    [...]
    Generating callgrind file 'program-profile.cg'...
    [...]
    $ kcachegrind program-profile.cg

### Usage examples

Here is a list of some common debugging tasks and how to do them with the Hatari debugger:

Stopping on program startup, e.g. to examine its data
  Run Hatari with options asking it to load symbols for a program when it's executed (from GEMDOS HD) and parse a debugger input file (e.g. "break.ini") that sets breapoint on the first program instruction (i.e. at start of its TEXT section):

      $ echo "b pc = TEXT" > break.ini
      $ hatari --symload exec --parse prg:break.ini ...

Tracing specific things in the system
  To trace e.g. all GEMDOS calls and I/O operations, use:

      trace  gemdos,io_all

  Please see Tracing section for more information on tracing, what is possible with it and what are its limitations.
Stopping when certain PC address is passed Nth time
  To stop e.g. after function/subroutine at $12345 is called for the 6th time:

      a  $12345 :6

Stopping when specific exception happens
  If one wants a specific breakpoint to trigger on a specific exception, one can check when its handler address is called by the CPU. At the start of memory is the CPU exception table for exception vectors. So, to stop e.g. at bus error with some extra information, one can use following:

      history  on
      b  pc=($8)

  After bus error invokes debugger, "history" command can then be used to see (executed memory addresses with their current) instructions leading to the error. The most interesting vector addresses are: $8 (Bus error), $C (Address error), $10 (Illegal instruction), $14 (Division by zero).
  NOTE: Normally, to invoke debugger for larger set of exceptions, one would use Hatari's "--debug-except" option to specify on which exceptions debugger is invoked, and "setopt -D" to enable that on run-time.
Stopping when register has a specific value
  To stop when e.g. D1 register contains value 5, set a breakpoint on:

      b  d1 = 5

Stopping when a register value changes
  To stop when e.g. D1 register value changes, set a breakpoint on:

      b  d1 ! d1

Stopping when part of register value changes
  To stop when e.g. D1 register lowest byte changes, set a breakpoint on masked lowest 8 bits:

      b  d1 & 0xff ! d1 & 0xff

Stopping when register value is within some range
  To stop when e.g. D1 register value is within range of 10-30, set a breakpoint on:

      b  d1 > 9  &&  d1 < 31

Stopping when memory location has a specific value
  To stop when e.g. bit 1 of the Video Shifter Sync Mode byte at I/O address $ff820a is set i.e. video frequency is 60Hz, set a breakpoint on:

      b  ($ff820a).b & 2 = 2

Stopping when a memory value changes
  To stop when above bit changes, set a breakpoint on its value being different from the current value ('!' compares for inequality):

      b  ($ff820a).b & 2 ! ($ff820a).b & 2

Tracing all changes in specific memory location
  To see the new values and continue without stopping, add the ":trace" breakpoint option:

      b  ($ff820a).b & 2 ! ($ff820a).b & 2  :trace

Finding specific data from memory
  Find "ETOS" character string matches from ROM:

      find c 0xe00000 E T O S

  Find all potential (word aligned) RTS instructions from RAM:

      find w 0x0 0x4e75

Values in different number base
  To see values in different number bases, use "evaluate" command. It can be used also with more complex expressions:

      > e ($ff8802).b & 0x7
        value at ($ff8802).b = $27
      = %111 (bin), #7 (dec), $7 (hex)

Viewing OS structure and IO register values
  To see e.g. basepage for currently running program:

      info  basepage

  To see e.g. all Falcon Videl register values, use:

      info  videl

Showing OS attribute information for specific OS calls
  To see e.g. VDI attributes for all v_gtext VDI calls:

      b  VdiOpcode = $8  :quiet :info vdi

Stopping at specific screen position
  To stop e.g. when VBL is 100, HBL is 40 and line cycles is 5, use the corresponding debugger variables:

      b  VBL = 100  &&  HBL = 40  &&  LineCycles = 5

Stopping after value increases/decreases by certain amount
  To stop e.g. after D0 value has increased by 10, set breakpoint on:

      b  d0 = 'd0 + 10'

Examining specific system call return value
  To check e.g. what is the Fopen() GEMDOS call return value, check with "info gemdos 1" its opcode, set a breakpoint for that and step to next (n) instruction from the trap call when breakpoint is hit. GEMDOS call return value is then in register D0:

      > trace  gemdos
      > b  GemdosOpcode = $3D
      > c
      [...continue until breakpoint...]
      1. CPU breakpoint condition(s) matched 1 times.
              GemdosOpcode = $3D
      > n
      GEMDOS 0x3D Fopen("TEST.TXT", read-only)
      > e  d0
      = %1000000 (bin), #64 (dec), $40 (hex)

Seeing code leading to a breakpoint
  To see CPU instructions executed before debugger was entered, history tracking needs to be enabled before it. Whenever debugger is entered, one can then request given number (here 16) of past PC addresses and their (current) instructions to be shown:

      history  cpu
      c
      [breakpoint is hit and debugger entered]
      history  16

Getting instruction execution history for every breakpoint
  To see last 16 instructions for both CPU and DSP whenever (a normal or tracing) breakpoint is hit:

      history  on
      lock  history 16
      c

Single stepping so that new register values are shown after each step

      lock  registers
      s
      [new register values]
      s
      [new register values]
      ...

Showing current stack contents
  To see first 64 bytes on top of the stack, use:

      m  'a7-64'-a7

Seeing specific information each time debugger is entered
  To see above information whenever some breakpoint is hit, debugger is entered manually etc, write that command to e.g. stack.ini file and then use:

      lock  file stack.ini

  Please see also Chaining breakpoints section for more examples on what can be done with the debugger input files.
Adding cycle information to disassembly
  Profiling collects cycle usage information for all executed instructions:

      profile  on
      c
      [after a while, use AltGr+Pause to get back to debugger]
      d
      [or when wanting to see just the executed instructions]
      profile addresses

  Please see Profiling section for more info.
Finding where a program or the OS is stuck
  Profiling tells from which addresses CPU is executing the instructions:

      profile  on
      c
      [after a while, use AltGr+Pause to get back to debugger]
      profile  addresses

  Please see Profiling section for more info.
Profiling specific function and everything it calls
  Set breakpoint for the function entrypoint and when it is hit, enable profiling and continue to the end of the function:

      address myfunction
      c
      [back in debugger at myfunction]
      profile on
      next subreturn
      [back in debugger when function returns]

  Please see Profiling section on and how to save and analyze profiling information after that, and Stepping section for some "next subreturn" command limitations.
Seeing program callstack when breakpoint is hit
  Profiler caller data, shown automatically when debugger is entered, includes callstack information (with some limitations) if program debug symbols have been loaded.
Seeing call backtraces whenever given function is called
  After enabling profiling and setting breakpoint for a function symbol like this:

      profile  on
      b  pc = _my_function  :quiet :noinit :file showstack.ini

  With "showstack.ini" containing following command:

      profile  stack

  Each call to "my_function" address quietly triggers a breakpoint (without resetting profiling callstack information) which shows the function backtrace. If this is enabled right when program starts, those backtraces will go up to main().
Seeing how program functions/symbols call each other
  Profile data post-processing can provide execution callgraphs.

Hint: for most of the above commands, one just needs to prefix them with "d" (or "dsp" when using full command names) to do similar operation on the DSP.

### Command line editing and libreadline

Hatari debugger is much nicer to use with the command line history, editing and especially the completion support for the command, command argument and symbol names. Like many other programs (Bash etc), debugger uses libreadline to handle this.

#### Hatari building

When building Hatari, please make sure to have the GNU readline development files installed (on Debian / Ubuntu these come from the "libreadline*-dev" packages). Otherwise TAB completion and other nice features do not get enabled when Hatari is configured.

#### Keyboard shortcuts

In addition to the normal line editing keys, libreadline supports several keyboard shortcuts ('^' indicates Control-key):

- Cursor movement (sometimes preferable over using arrow keys):

- ^B - backward
- ^F - forward
- ^arrow - move by word
- ^A - start of line
- ^E - end of line
- ^P - previous line
- ^N - next line

- Cut / paste (relative to cursor position):

- ^W - cut previous word (space separated)
- ^K - cut ("kill") to end of line
- ^U - cut to start of line
- ^Y - paste ("yank") deleted content

- Other text changes:

- ^D - delete character
- ^T - transpose (switch) last 2 characters
- ^_ - undo last change

- Miscanellous:

- ^R - history ("reverse") search
- ^L - clear screen and redraw line

(Above shortcuts are inherited from Emacs text editor, and work also in all shells using readline. See "man readline" on how to configure them.)