/'* \file fb-doc_parser.bi
\brief Header file for the #Parser class

This file contains the declaration of the \ref Parser, a class for 
loading and scanning FB source code. It loads and parses the FB 
source code and calls the matching functions in the \em EmitterIF.

'/


/'* \brief The parser

Class to handle FreeBasic source code. A Parser allways work on 
exactly one input stream, comming from a file or from StdIn. The 
Parser does

 - read the source from an input channel (see StdIn(), File_())
 - call function \ref  EmitterIF::_Init_() before parsing
 - pre-parse the source for relavant code fractions (see pre_parse()) 
   and call the comments emitter-handler on the way.
 - fine-scan any relavant code fractions (see tokenize()) and call 
   the matching emitter-handler for this code (done by the private 
   functions ie like _DEFINE_(), _FUNCTION_(), _VAR_(), ...)
 - provide functions to extract elements form the source (like 
   SubStr(), Bracket(), VarIni(), ...)
 - call matching emitter functions to generaate the output stream
 - call function \ref  EmitterIF::_Exit_() after parsing

When fb-doc follows the source tree (option \em --tree), the 
function \ref EmitterIF::Incl_() creates a new Parser for each file.

'/
TYPE Parser
Public:
/'* \brief The tokens used by the parser

Enumerators used to classify the type of a token found in the 
FreeBasic source code.

'/
  ENUM ParserTokens
    MSG_STOP = -1 '*< end of file / token list reached
    MSG_ERROR    '*< continue parsing
    TOK_EOS   '*< end of statement (either '\n' or ':')
    TOK_BRCLO '*< right parenthesis
    TOK_COMMA '*< a comma
  
    TOK_KLOPN '*< left other bracket
    TOK_KLCLO '*< right other bracket
    TOK_DOT   '*< a dot
    TOK_TRIDO '*< three dots
    TOK_QUOTE '*< a string constant (including quotes)
    TOK_LATTE '*< the '#' character
    TOK_BROPN '*< left parenthesis
    TOK_EQUAL '*< the '=' character

    TOK_ABST  '*< the ABSTRACT keyword
    TOK_ALIA  '*< the ALIAS keyword
    TOK_AS    '*< the AS keyword
    TOK_BYRE  '*< the BYREF keyword
    TOK_BYVA  '*< the BYVAL keyword
    TOK_CDEC  '*< the CDECL keyword
    TOK_CLAS  '*< the CLASS keyword
    TOK_COMM  '*< the COMMON keyword
    TOK_CONS  '*< the CONST keyword
    TOK_DECL  '*< the DECLARE keyword
    TOK_DEFI  '*< the \#DEFINE keyword
    TOK_DIM   '*< the DIM keyword
    TOK_END   '*< the END keyword
    TOK_ENUM  '*< the ENUM keyword
    TOK_EXDS  '*< the EXTENDS keyword
    TOK_EMAC  '*< the \#ENDMACRO keyword
    TOK_EXRN  '*< the EXTERN keyword
    TOK_EXPO  '*< the EXPORT keyword
    TOK_INCL  '*< the \#INCLUDE keyword
    TOK_LIB   '*< the LIB keyword
    TOK_MACR  '*< the \#MACRO keyword
    TOK_NAMS  '*< the NAMESPACE keyword
    TOK_ONCE  '*< the ONCE keyword
    TOK_OVER  '*< the OVERLOAD keyword
    TOK_PASC  '*< the PASCAL keyword
    TOK_PRIV  '*< the PRIVATE keyword
    TOK_PROT  '*< the PROTECTED keyword
    TOK_RDIM  '*< the REDIM keyword
    TOK_PTR   '*< the PTR or POINTER keyword
    TOK_PUBL  '*< the PUBLIC keyword
    TOK_SCOP  '*< the SCOPE keyword
    TOK_SHAR  '*< the SHARED keyword
    TOK_STAT  '*< the STAvarTIC keyword
    TOK_STCL  '*< the STDCALL keyword
    TOK_TYPE  '*< the TYPE keyword
    TOK_UNIO  '*< the UNION keyword
    TOK_VAR   '*< the VAR keyword
    TOK_VIRT  '*< the VIRTUAL keyword
  
    TOK_FUNC  '*< the FUNCTION keyword
    TOK_FILD  '*< the FIELD keyword
    TOK_OPER  '*< the OPERATOR keyword
    TOK_PROP  '*< the PROPERTY keyword
    TOK_SUB   '*< the SUB keyword
    TOK_CTOR  '*< the CONSTRUCTOR keyword
    TOK_DTOR  '*< the DESTRUCTOR keyword

    TOK_BYTE  '*< the BYTE data type
    TOK_DOUB  '*< the DOUBLE data type
    TOK_INT   '*< the INTEGER data type
    TOK_LONG  '*< the LONG data type
    TOK_LINT  '*< the LONGINT data type
    TOK_STRI  '*< the STRING data type
    TOK_SHOR  '*< the SHORT data type
    TOK_SING  '*< the SINGLE data type
    TOK_UBYT  '*< the UBYTE data type
    TOK_ULNG  '*< the ULONG data type
    TOK_ULIN  '*< the ULONGINT data type
    TOK_UINT  '*< the UINTEGER data type
    TOK_USHO  '*< the USHORT data type
    TOK_WSTR  '*< the ZSTRING data type
    TOK_ZSTR  '*< the ZSTRING data type
    TOK_WORD  '*< a word

    TOK_COMSL '*< line end comment (single line)
    TOK_COMML '*< multi line comment
  END ENUM

/'* \name Parser STRINGs

STRING variables to exchange data with th emitters.

\{
'/
  AS STRING _
    Buf, _        '*< the input buffer
    ErrMsg, _     '*< an error message
    Tok, _        '*< the token list
    BlockNam      '*< the name of a block (ENUM, UNION, TYPE)

'* \}


/'* \name Parser token pointers

Pointers to the token list, used to specify the result of the 
parsing process. The tokens specify the construct in progress and 
they are used by the emitters to create their output.

Four main tokens specify the kind of the construct:

 - StaTok: the current construct
 - NamTok - subtokens: DimTok, IniTok (may be zero in parameter lists)
 - TypTok - subtokens: ShaTok, PtrTok, Co1Tok, Co2Tok (is zero for VAR, may be zero for CONST)
 - FunTok - subtokens: ParTok, AliTok, DivTok, By_Tok (is zero for variables)

Other tokens are only valid if the main token is not zero.

\{
'/
  AS INTEGER PTR _
    StaTok, _  '*< the pre-parsed token
    NamTok, _  '*< the name token of the construct
    DimTok, _  '*< the token of the left parenthesis of a dimension
    IniTok, _  '*< the start token of an initializer ('=')
    BitTok, _  '*< the start token of an bitfiled declaration (':')
    TypTok, _  '*< the token of the type keyword
    ShaTok, _  '*< the token of the SHARED keyword
    PtrTok, _  '*< the token of the first PTR keyword
    Co1Tok, _  '*< the token of the first CONST keyword (if any)
    Co2Tok, _  '*< the token of the second CONST keyword (if any)
    FunTok, _  '*< type of the pre-parsed token
    CalTok, _  '*< the token of the calling convention keyword (if any)
    AliTok, _  '*< the token of the ALIAS keyword (if any)
    As_Tok, _  '*< the token of the AS keyword (if no SUB)
    ParTok, _  '*< the token of the left parenthesis of a parameter list
    By_Tok, _  '*< the token of the declaration specifier (BYVAL / BYREF)
    DivTok, _  '*< the token of different purposes like the STATIC keyword in a member function declaration (if any) or the LIB keyword in a normal declaration
    Tk1, _     '*< the first token in a statment
    TkEnd, _   '*< the last token in the list
    UserTok    '*< a token for customised usage in emitter-handlers

'* \}

/'* \name Parser counters, integers and pointers

Diverse variables used on different events. They inform the emitters 
about the state of the parser.

\{
'/
  AS INTEGER _
    Ocha, _       '*< the ouput file number (channel)
    Fin, _        '*< the last position in the input buffer Parser::Buf
    PtrCount, _   '*< the number of PTR keywords
    SrcBgn, _     '*< start position to export FB source
    InLines, _    '*< the current line in the input buffer Parser::Buf
    LevelCount, _ '*< the level in nested blocks (one based)
    ListCount     '*< the current entry in a list (zero based)

'* \}

  DECLARE CONSTRUCTOR(BYVAL AS EmitterIF PTR)

/'* \name Filehandlers

Filehandlers are used to load some FreeBasic source code input in to 
the buffer Parser::Buf from different input channels (may be the 
StdIn pipe, a single file or all files in a folder). Afterwards the 
source code gets parsed and translated output created by the emitter 
gets returned.

\{
'/
  DECLARE SUB File_(BYREF AS STRING)
  DECLARE SUB StdIn()
'* \}

/'* \name Properties to extract original source code

Properties are used to extract original text from the input buffer 
at a certain position. This is the current position of the parser, 
or the position  of the given parameter.

\{
'/
  DECLARE PROPERTY CurTok() AS INTEGER PTR
  DECLARE PROPERTY BitIni() AS STRING
  DECLARE PROPERTY VarIni() AS STRING
  DECLARE PROPERTY SubStr() AS STRING
  DECLARE PROPERTY SubStr(BYVAL AS INTEGER PTR) AS STRING
  DECLARE PROPERTY Bracket(BYVAL AS INTEGER PTR) AS STRING
'* \}

/'* \name Parsers for lists and blocks

External demuxers are called from emitter-handlers when they sit on 
a list of statments, ie in an ENUM block or in a DIM statement with 
more than one variable.

\{
'/
  DECLARE SUB parseListNam(BYVAL AS EmitFunc)
  DECLARE SUB parseListNamTyp(BYVAL AS EmitFunc)
  DECLARE SUB parseListPara(BYVAL AS EmitFunc)
  DECLARE SUB parseBlockEnum(BYVAL AS EmitFunc)
  DECLARE SUB parseBlockTyUn(BYVAL AS EmitFunc)
'* \}

Private:

/'* \brief The modi for end of statement searching

Enumerators used in internal in the parser to specify where to stop 
to tokenize the input buffer

'/
  ENUM EoS_Modi
    TO_END_BLOCK  '*< tokenize to the end of the structure
    TO_END        '*< stop at the end of the token list
    TO_EOL        '*< stop at the next line end
    TO_COLON      '*< stop at the next line end or colon
  END ENUM

/'* \name Internal values

Variables used in pre-parsing process.

\{
'/
  AS INTEGER _
    Po, _     '*< the current position in the input buffer Parser::Buf
    ToLast    '*< the type of the token before the current one

  AS INTEGER PTR _
    Tk, _     '*< the current token of the parser
    A, _      '*< start of a word in pre-parsing
    L         '*< length of a word in pre-parsing

  AS EmitterIF PTR Emit  '*< the emitter interface

'* \}

/'* \name Internal parsers

Functions for evaluating a construct in the FreeBasic source code 
after a relevant keyword was found in the pre-parsering process. 
After the fine-parsing process the matching emitter-handler gets 
called. (Or the function _Error_() in case of a syntax problem. It's 
up to the emitter if and where the user sees the error message.)

\{
'/
  DECLARE SUB pre_parse()
  DECLARE PROPERTY USubStr() AS STRING
  DECLARE FUNCTION getToken() AS INTEGER
  DECLARE FUNCTION tokenize(BYVAL AS EoS_Modi) AS INTEGER
  DECLARE FUNCTION Errr(BYREF AS STRING) AS INTEGER

  DECLARE FUNCTION demuxNam(BYVAL AS INTEGER = TOK_WORD, BYVAL AS INTEGER = 0) AS INTEGER
  DECLARE FUNCTION demuxTyp(BYVAL AS INTEGER = 0) AS INTEGER
  DECLARE FUNCTION demuxDecl() AS INTEGER

  DECLARE FUNCTION skipOverColon() AS INTEGER
  DECLARE FUNCTION skipOverBrclo() AS INTEGER
  DECLARE FUNCTION skipOverComma() AS INTEGER

  DECLARE FUNCTION _VAR_() AS INTEGER
  DECLARE FUNCTION _TYPE_() AS INTEGER
  DECLARE FUNCTION _ENUM_() AS INTEGER
  DECLARE FUNCTION _UNION_() AS INTEGER
  DECLARE FUNCTION _FUNCTION_() AS INTEGER
  DECLARE FUNCTION _DECLARE_() AS INTEGER
  DECLARE FUNCTION _INCLUDE_() AS INTEGER
  DECLARE FUNCTION _DEFINE_() AS INTEGER
  DECLARE FUNCTION _MACRO_() AS INTEGER
'* \}

END TYPE
