Report a bug
If you spot a problem with this page, click here to create a Bugzilla issue.
Improve this page
Quickly fork, edit online, and submit a pull request for this page. Requires a signed-in GitHub account. This works well for small changes. If you'd like to make larger changes you may want to consider using a local clone.


Compiler implementation of the D programming language.

Source lexer.d

static immutable ubyte[256] cmtable;
Do our own char maps
class Lexer;
this(const(char)* filename, const(char)* base, size_t begoffset, size_t endoffset, bool doDocComment, bool commentToken);
Creates a Lexer for the source code base[begoffset..endoffset+1]. The last character, base[endoffset], must be null (0) or EOF (0x1A).
const(char)* filename used for error messages
const(char)* base source code, must be terminated by a null (0) or EOF (0x1A) character
size_t begoffset starting offset into base[]
size_t endoffset the last offset to read into base[]
bool doDocComment handle documentation comments
bool commentToken comments become TOK.comment's
final TOK peekNext();
Look ahead at next token's value.
final TOK peekNext2();
Look 2 tokens ahead at value.
final void scan(Token* t);
Turn next token in buffer into a token.
final Token* peekPastParen(Token* tk);
tk is on the opening (. Look ahead and return token that is past the closing ).
final uint escapeSequence();
Parse escape sequence.
final TOK wysiwygStringConstant(Token* t, int tc);
final TOK hexStringConstant(Token* t);
Lex hex strings: x"0A ae 34FE BD"
final TOK delimitedStringConstant(Token* t);
Lex delimited strings: q"(foo(xxx))" // "foo(xxx)" q"[foo(]" // "foo(" q"/foo]/" // "foo]" q"HERE foo HERE" // "foo\n"

Input p is on the "

final TOK tokenStringConstant(Token* t);
Lex delimited strings: q{ foo(xxx) } // " foo(xxx) " q{foo(} // "foo(" q{{foo}"}"} // "{foo}"}""

Input p is on the q

final TOK escapeStringConstant(Token* t, int wide);
final TOK charConstant(Token* t);
final void stringPostfix(Token* t);
Get postfix of string literal.
final TOK number(Token* t);
Read in a number. If it's an integer, store it in tok.TKutok.Vlong. integers can be decimal, octal or hex Handle the suffixes U, UL, LU, L, etc. If it's double, store it in tok.TKutok.Vdouble.
TKnum TKdouble,...
final TOK inreal(Token* t);
Read in characters, converting them to real.
Exponent overflow not detected. Too much requested precision is not detected.
final void poundLine();
parse: #line linnum [filespec] also allow __LINE__ for linnum, and __FILE__ for filespec
final uint decodeUTF();
Decode UTF character. Issue error messages for invalid sequences. Return decoded character, advance p to last character in UTF sequence.
final void getDocComment(Token* t, uint lineComment, bool newParagraph);
Parse doc comment embedded between t.ptr and p. Remove trailing blanks and tabs from lines. Replace all newlines with \n. Remove leading comment character from each line. Decide if it's a lineComment or a blockComment. Append to previous one for this token.
If newParagraph is true, an extra newline will be added between adjoining doc comments.
static const(char)* combineComments(const(char)* c1, const(char)* c2, bool newParagraph);
Combine two document comments into one, separated by an extra newline if newParagraph is true.