Lexical
In D, the lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes what those tokens are. The D lexical grammar is designed to be suitable for high speed scanning, it has a minimum of special case rules, there is only one phase of translation, and to make it easy to write a correct scanner for. The tokens are readily recognizable by those familiar with C and C++.Phases of Compilation
The process of compiling is divided into multiple phases. Each phase has no dependence on subsequent phases. For example, the scanner is not perturbed by the semantic analyzer. This separation of the passes makes language tools like syntax directed editors relatively easy to produce. It also is possible to compress D source by storing it in 'tokenized' form.- source character set
The source file is checked to see what character set it is, and the appropriate scanner is loaded. ASCII and UTF formats are accepted. - script line
If the first line starts with #! then the first line is ignored. - lexical analysis
The source file is divided up into a sequence of tokens. Special tokens are replaced with other tokens. Special token sequences are processed and removed. - syntax analysis
The sequence of tokens is parsed to form syntax trees. - semantic analysis
The syntax trees are traversed to declare variables, load symbol tables, assign types, and in general determine the meaning of the program. - optimization
Optimization is an optional pass that tries to rewrite the program in a semantically equivalent, but faster executing, version. - code generation
Instructions are selected from the target architecture to implement the semantics of the program. The typical result will be an object file, suitable for input to a linker.
Source Text
D source text can be in one of the following formats:- ASCII
- UTF-8
- UTF-16BE
- UTF-16LE
- UTF-32BE
- UTF-32LE
Format | BOM |
---|---|
UTF-8 | EF BB BF |
UTF-16BE | FE FF |
UTF-16LE | FF FE |
UTF-32BE | 00 00 FE FF |
UTF-32LE | FF FE 00 00 |
ASCII | no BOM |
There are no digraphs or trigraphs in D.
The source text consists of white space, end of lines, comments, special token sequences, tokens, all followed by end of file.
The source text is split into tokens using the maximal munch
technique, i.e., the
lexical analyzer tries to make the longest token it can. For example
>>
is a right shift token,
not two greater than tokens.
End of File
EndOfFile: physical end of the file \u0000 \u001AThe source text is terminated by whichever comes first.
End of Line
EndOfLine: \u000D \u000A \u000D \u000A EndOfFileThere is no backslash line splicing, nor are there any limits on the length of a line.
White Space
WhiteSpace: Space Space WhiteSpace Space: \u0020 \u0009 \u000B \u000C EndOfLine CommentWhite space is defined as a sequence of one or more of spaces, tabs, vertical tabs, form feeds, end of lines, or comments.
Comments
Comment: /* Characters */ // Characters EndOfLine /+ Characters +/D has three kinds of comments:
- Block comments can span multiple lines, but do not nest.
- Line comments terminate at the end of the line.
- Nesting comments can span multiple lines and can nest.
a = /+ // +/ 1; // parses as if 'a = 1;' a = /+ "+/" +/ 1"; // parses as if 'a = " +/ 1";' a = /+ /* +/ */ 3; // parses as if 'a = */ 3;'Comments cannot be used as token concatenators, for example,
abc/**/def
is two tokens, abc and def,
not one abcdef token.
Tokens
Token: Identifier StringLiteral CharacterLiteral IntegerLiteral FloatLiteral Keyword / /= . .. ... & &= && | |= || - -= -- + += ++ < <= << <<= <> <>= > >= >>= >>>= >> >>> ! != !== !<> !<>= !< !<= !> !>= !~ ( ) [ ] { } ? , ; : $ = == === * *= % %= ^ ^= ~ ~= ~~
Identifiers
Identifier: IdentiferStart IdentiferStart IdentifierChars IdentifierChars: IdentiferChar IdentiferChar IdentifierChars IdentifierStart: _ Letter UniversalAlpha IdentifierChar: IdentiferStart DigitIdentifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D. (This is the C99 Standard.) Identifiers can be arbitrarily long, and are case sensitive. Identifiers starting with __ (two underscores) are reserved.
String Literals
StringLiteral: WysiwygString AlternateWysiwygString DoubleQuotedString EscapeSequence HexString WysiwygString: r" WysiwygCharacters " Postfixopt AlternateWysiwygString: ` WysiwygCharacters ` Postfixopt WysiwygCharacter: Character EndOfLine DoubleQuotedString: " DoubleQuotedCharacters " Postfixopt DoubleQuotedCharacter: Character EscapeSequence EndOfLine EscapeSequence: \' \" \? \\ \a \b \f \n \r \t \v \ EndOfFile \x HexDigit HexDigit \ OctalDigit \ OctalDigit OctalDigit \ OctalDigit OctalDigit OctalDigit \u HexDigit HexDigit HexDigit HexDigit \U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit \& NamedCharacterEntity ; HexString: x" HexStringChars " Postfixopt HexStringChar HexDigit WhiteSpace EndOfLine Postfix c w dA string literal is either a double quoted string, a wysiwyg quoted string, an escape sequence, or a hex string.
Wysiwyg quoted strings are enclosed by r" and ". All characters between the r" and " are part of the string except for EndOfLine which is regarded as a single \n character. There are no escape sequences inside r" ":
r"hello" r"c:\root\foo.exe" r"ab\n" // string is 4 characters, 'a', 'b', '\', 'n'An alternate form of wysiwyg strings are enclosed by backquotes, the ` character. The ` character is not available on some keyboards and the font rendering of it is sometimes indistinguishable from the regular ' character. Since, however, the ` is rarely used, it is useful to delineate strings with " in them.
`hello` `c:\root\foo.exe` `ab\n` // string is 4 characters, 'a', 'b', '\', 'n'Double quoted strings are enclosed by "". Escape sequences can be embedded into them with the typical \ notation. EndOfLine is regarded as a single \n character.
"hello" "c:\\root\\foo.exe" "ab\n" // string is 3 characters, 'a', 'b', and a linefeed "ab " // string is 3 characters, 'a', 'b', and a linefeedEscape strings start with a \ and form an escape character sequence. Adjacent escape strings are concatenated:
\n the linefeed character \t the tab character \" the double quote character \012 octal \x1A hex \u1234 wchar character \U00101234 dchar character \® ® dchar character \r\n carriage return, line feedUndefined escape sequences are errors.
Hex strings allow string literals to be created using hex data:
x"0A" // same as "\x0A" x"00 FBCD 32FD 0A" // same as "\x00\xFB\xCD\x32\xFD\x0A"Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2.
Adjacent strings are concatenated with the ~ operator, or by simple juxtaposition:
"hello " ~ "world" ~ \n // forms the string 'h','e','l','l','o',' ', // 'w','o','r','l','d',linefeedThe following are all equivalent:
"ab" "c" r"ab" r"c" r"a" "bc" "a" ~ "b" ~ "c" \x61"bc"The optional Postfix character gives a specific type to the string, rather than it being inferred from the context. This is useful when the type cannot be unambiguously inferred, such as when overloading based on string type. The types corresponding to the postfix characters are:
Postfix | Type |
---|---|
c | char[ ] |
w | wchar[ ] |
d | dchar[ ] |
"hello"c // char[] "hello"w // wchar[] "hello"d // dchar[]
Character Literals
CharacterLiteral: ' SingleQuotedCharacter ' SingleQuotedCharacter Character EscapeSequenceCharacter literals are a single character or escape sequence enclosed by single quotes, ' '.
Integer Literals
IntegerLiteral: Integer Integer IntegerSuffix Integer: Decimal Binary Octal Hexadecimal Integer IntegerSuffix: L u U Lu LU uL UL Decimal: 0 NonZeroDigit NonZeroDigit DecimalDigits Binary: 0b BinaryDigits 0B BinaryDigits Octal: 0 OctalDigits Hexadecimal: 0x HexDigits 0X HexDigits NonZeroDigit: 1 2 3 4 5 6 7 8 9 DecimalDigits: 0 NonZeroDigit _ BinaryDigits: BinaryDigit BinaryDigit BinaryDigits BinaryDigit: 0 1 _ OctalDigits: OctalDigit OctalDigit OctalDigits OctalDigit: 0 1 2 3 4 5 6 7 _ HexDigits: HexDigit HexDigit HexDigits HexDigit: 0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F _Integers can be specified in decimal, binary, octal, or hexadecimal.
Decimal integers are a sequence of decimal digits.
Binary integers are a sequence of binary digits preceded by a '0b'.
Octal integers are a sequence of octal digits preceded by a '0'.
Hexadecimal integers are a sequence of hexadecimal digits preceded by a '0x' or followed by an 'h'.
Integers can have embedded '_' characters, which are ignored. The embedded '_' are useful for formatting long literals, such as using them as a thousands separator:
123_456 // 123456 1_2_3_4_5_6_ // 123456Integers can be immediately followed by one 'L' or one 'u' or both.
The type of the integer is resolved as follows:
Decimal Literal | Type |
---|---|
0 .. 2147483647 | int |
2147483648 .. 9223372036854775807 | long |
Decimal Literal, L Suffix | Type |
0L .. 9223372036854775807L | long |
Decimal Literal, U Suffix | Type |
0U .. 4294967295U | uint |
4294967296U .. 18446744073709551615U | ulong |
Decimal Literal, UL Suffix | Type |
0UL .. 18446744073709551615UL | ulong |
Non-Decimal Literal | Type |
0x0 .. 0x7FFFFFFF | int |
0x80000000 .. 0xFFFFFFFF | uint |
0x100000000 .. 0x7FFFFFFFFFFFFFFF | long |
0x8000000000000000 .. 0xFFFFFFFFFFFFFFFF | ulong |
Non-Decimal Literal, L Suffix | Type |
0x0L .. 0x7FFFFFFFFFFFFFFFL | long |
0x8000000000000000L .. 0xFFFFFFFFFFFFFFFFL | ulong |
Non-Decimal Literal, U Suffix | Type |
0x0U .. 0xFFFFFFFFU | uint |
0x100000000UL .. 0xFFFFFFFFFFFFFFFFUL | ulong |
Non-Decimal Literal, UL Suffix | Type |
0x0UL .. 0xFFFFFFFFFFFFFFFFUL | ulong |
Floating Literals
FloatLiteral: Float Float FloatSuffix Float ImaginarySuffix Float FloatSuffix ImaginarySuffix Float: DecimalFloat HexFloat DecimalFloat: DecimalDigits . DecimalDigits . DecimalDigits DecimalDigits . DecimalDigits DecimalExponent . DecimalDigits . DecimalDigits DecimalExponent DecimalDigits DecimalExponent DecimalExponent e DecimalDigits E DecimalDigits e+ DecimalDigits E+ DecimalDigits e- DecimalDigits E- DecimalDigits HexFloat: HexPrefix HexDigits . HexPrefix HexDigits . HexDigits HexPrefix HexDigits . HexDigits HexExponent HexPrefix . HexDigits HexPrefix . HexDigits HexExponent HexPrefix HexDigits HexExponent HexPrefix: 0x 0X HexExponent p DecimalDigits P DecimalDigits p+ DecimalDigits P+ DecimalDigits p- DecimalDigits P- DecimalDigits FloatSuffix: f F L ImaginarySuffix: iFloats can be in decimal or hexadecimal format, as in standard C.
Hexadecimal floats are preceded with a 0x and the exponent is a p or P followed by a decimal number serving as the exponent of 2.
Floating literals can have embedded '_' characters, which are ignored. The embedded '_' are useful for formatting long literals to make them more readable, such as using them as a thousands separator:
123_456.567_8 // 123456.5678 1_2_3_4_5_6_._5_6_7_8 // 123456.5678 1_2_3_4_5_6_._5e-6_ // 123456.5e-6Floating literals with no suffix are of type double. Floats can be followed by one f, F, or L suffix. The f or F suffix means it is a float, and L means it is a real.
If a floating literal is followed by i, then it is an ireal (imaginary) type.
Examples:
0x1.FFFFFFFFFFFFFp1023 // double.max 0x1p-52 // double.epsilon 1.175494351e-38F // float.min 6.3i // idouble 6.3 6.3fi // ifloat 6.3 6.3Li // ireal 6.3It is an error if the literal exceeds the range of the type. It is not an error if the literal is rounded to fit into the significant digits of the type.
Complex literals are not tokens, but are assembled from real and imaginary expressions in the semantic analysis:
4.5 + 6.2i // complex number
Keywords
Keywords are reserved identifiers.Keyword: abstract alias align asm assert auto body bool break byte case cast catch cdouble cent cfloat char class const continue creal dchar debug default delegate delete deprecated do double else enum export extern false final finally float for foreach function goto idouble if ifloat import in inout int interface invariant ireal is long mixin module new null out override package pragma private protected public real return scope short static struct super switch synchronized template this throw true try typedef typeid typeof ubyte ucent uint ulong union unittest ushort version void volatile wchar while with
Special Tokens
These tokens are replaced with other tokens according to the following table:
Special Token | Replaced with... |
---|---|
__FILE__ | string literal containing source file name |
__LINE__ | integer literal of the current source line number |
__DATE__ | string literal of the date of compilation "mmm dd yyyy" |
__TIME__ | string literal of the time of compilation "hh:mm:ss" |
__TIMESTAMP__ | string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy" |
Special Token Sequences
Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.There is currently only one special token sequence, #line.
SpecialTokenSequence # line Integer EndOfLine # line Integer Filespec EndOfLine Filespec " Characters "This sets the source line number to Integer, and optionally the source file name to Filespec, beginning with the next line of source text. The source file and line number is used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.
For example:
int #line 6 "foo\bar" x; // this is now line 6 of file foo\barNote that the backslash character is not treated specially inside Filespec strings.