www.digitalmars.com
Last update Tue Jun 6 16:38:19 2006

Lexical

In D, the lexical analysis is independent of the syntax parsing and the semantic analysis. The lexical analyzer splits the source text up into tokens. The lexical grammar describes what those tokens are. The D lexical grammar is designed to be suitable for high speed scanning, it has a minimum of special case rules, there is only one phase of translation, and to make it easy to write a correct scanner for. The tokens are readily recognizable by those familiar with C and C++.

Phases of Compilation

The process of compiling is divided into multiple phases. Each phase has no dependence on subsequent phases. For example, the scanner is not perturbed by the semantic analyzer. This separation of the passes makes language tools like syntax directed editors relatively easy to produce. It also is possible to compress D source by storing it in 'tokenized' form.
  1. source character set
    The source file is checked to see what character set it is, and the appropriate scanner is loaded. ASCII and UTF formats are accepted.
  2. script line
    If the first line starts with #! then the first line is ignored.
  3. lexical analysis
    The source file is divided up into a sequence of tokens. Special tokens are replaced with other tokens. Special token sequences are processed and removed.
  4. syntax analysis
    The sequence of tokens is parsed to form syntax trees.
  5. semantic analysis
    The syntax trees are traversed to declare variables, load symbol tables, assign types, and in general determine the meaning of the program.
  6. optimization
    Optimization is an optional pass that tries to rewrite the program in a semantically equivalent, but faster executing, version.
  7. code generation
    Instructions are selected from the target architecture to implement the semantics of the program. The typical result will be an object file, suitable for input to a linker.

Source Text

D source text can be in one of the following formats: UTF-8 is a superset of traditional 7-bit ASCII. One of the following UTF BOMs (Byte Order Marks) can be present at the beginning of the source text:

Format BOM
UTF-8 EF BB BF
UTF-16BE FE FF
UTF-16LE FF FE
UTF-32BE 00 00 FE FF
UTF-32LE FF FE 00 00
ASCII no BOM

There are no digraphs or trigraphs in D.

The source text consists of white space, end of lines, comments, special token sequences, tokens, all followed by end of file.

The source text is split into tokens using the maximal munch technique, i.e., the lexical analyzer tries to make the longest token it can. For example >> is a right shift token, not two greater than tokens.

End of File

EndOfFile:
	physical end of the file
	\u0000
	\u001A
The source text is terminated by whichever comes first.

End of Line

EndOfLine:
	\u000D
	\u000A
	\u000D \u000A
	EndOfFile
There is no backslash line splicing, nor are there any limits on the length of a line.

White Space

WhiteSpace:
	Space
	Space WhiteSpace

Space:
	\u0020
	\u0009
	\u000B
	\u000C
	EndOfLine
	Comment
White space is defined as a sequence of one or more of spaces, tabs, vertical tabs, form feeds, end of lines, or comments.

Comments

Comment:
	/* Characters */
	// Characters EndOfLine
	/+ Characters +/
D has three kinds of comments:
  1. Block comments can span multiple lines, but do not nest.
  2. Line comments terminate at the end of the line.
  3. Nesting comments can span multiple lines and can nest.
Comment processing conceptually happens before tokenization. This means that embedded strings and comments do not prevent recognition of comment openings and closings:
a = /+ // +/ 1;		// parses as if 'a = 1;'
a = /+ "+/" +/ 1";	// parses as if 'a = " +/ 1";'
a = /+ /* +/ */ 3;	// parses as if 'a = */ 3;'
Comments cannot be used as token concatenators, for example, abc/**/def is two tokens, abc and def, not one abcdef token.

Tokens

Token:
	Identifier
	StringLiteral
	CharacterLiteral
	IntegerLiteral
	FloatLiteral
	Keyword
	/
	/=
	.
	..
	...
	&
	&=
	&&
	|
	|=
	||
	-
	-=
	--
	+
	+=
	++
	<
	<=
	<<
	<<=
	<>
	<>=
	>
	>=
	>>=
	>>>=
	>>
	>>>
	!
	!=
	!==
	!<>
	!<>=
	!<
	!<=
	!>
	!>=
	!~
	(
	)
	[
	]
	{
	}
	?
	,
	;
	:
	$
	=
	==
	===
	*
	*=
	%
	%=
	^
	^=
	~
	~=
	~~

Identifiers

Identifier:
	IdentiferStart
	IdentiferStart IdentifierChars

IdentifierChars:
	IdentiferChar
	IdentiferChar IdentifierChars

IdentifierStart:
	_
	Letter
	UniversalAlpha

IdentifierChar:
	IdentiferStart
	Digit
Identifiers start with a letter, _, or universal alpha, and are followed by any number of letters, _, digits, or universal alphas. Universal alphas are as defined in ISO/IEC 9899:1999(E) Appendix D. (This is the C99 Standard.) Identifiers can be arbitrarily long, and are case sensitive. Identifiers starting with __ (two underscores) are reserved.

String Literals

StringLiteral:
	WysiwygString
	AlternateWysiwygString
	DoubleQuotedString
	EscapeSequence
	HexString

WysiwygString:
	r" WysiwygCharacters " Postfixopt

AlternateWysiwygString:
	` WysiwygCharacters ` Postfixopt

WysiwygCharacter:
	Character
	EndOfLine

DoubleQuotedString:
	" DoubleQuotedCharacters " Postfixopt

DoubleQuotedCharacter:
	Character
	EscapeSequence
	EndOfLine

EscapeSequence:
	\'
	\"
	\?
	\\
	\a
	\b
	\f
	\n
	\r
	\t
	\v
	\ EndOfFile
	\x HexDigit HexDigit
	\ OctalDigit
	\ OctalDigit OctalDigit
	\ OctalDigit OctalDigit OctalDigit
	\u HexDigit HexDigit HexDigit HexDigit
	\U HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit HexDigit
	\& NamedCharacterEntity ;

HexString:
	x" HexStringChars " Postfixopt

HexStringChar
	HexDigit
	WhiteSpace
	EndOfLine

Postfix
	c
	w
	d
A string literal is either a double quoted string, a wysiwyg quoted string, an escape sequence, or a hex string.

Wysiwyg quoted strings are enclosed by r" and ". All characters between the r" and " are part of the string except for EndOfLine which is regarded as a single \n character. There are no escape sequences inside r" ":

r"hello"
r"c:\root\foo.exe"
r"ab\n"			// string is 4 characters, 'a', 'b', '\', 'n'
An alternate form of wysiwyg strings are enclosed by backquotes, the ` character. The ` character is not available on some keyboards and the font rendering of it is sometimes indistinguishable from the regular ' character. Since, however, the ` is rarely used, it is useful to delineate strings with " in them.
`hello`
`c:\root\foo.exe`
`ab\n`			// string is 4 characters, 'a', 'b', '\', 'n'
Double quoted strings are enclosed by "". Escape sequences can be embedded into them with the typical \ notation. EndOfLine is regarded as a single \n character.
"hello"
"c:\\root\\foo.exe"
"ab\n"			// string is 3 characters, 'a', 'b', and a linefeed
"ab
"			// string is 3 characters, 'a', 'b', and a linefeed
Escape strings start with a \ and form an escape character sequence. Adjacent escape strings are concatenated:
\n			the linefeed character
\t			the tab character
\"			the double quote character
\012			octal
\x1A			hex
\u1234			wchar character
\U00101234		dchar character
\&reg;			® dchar character
\r\n			carriage return, line feed
Undefined escape sequences are errors.

Hex strings allow string literals to be created using hex data:

x"0A"			// same as "\x0A"
x"00 FBCD 32FD 0A"	// same as "\x00\xFB\xCD\x32\xFD\x0A"
Whitespace and newlines are ignored, so the hex data can be easily formatted. The number of hex characters must be a multiple of 2.

Adjacent strings are concatenated with the ~ operator, or by simple juxtaposition:

"hello " ~ "world" ~ \n	// forms the string 'h','e','l','l','o',' ',
			// 'w','o','r','l','d',linefeed
The following are all equivalent:
"ab" "c"
r"ab" r"c"
r"a" "bc"
"a" ~ "b" ~ "c"
\x61"bc"
The optional Postfix character gives a specific type to the string, rather than it being inferred from the context. This is useful when the type cannot be unambiguously inferred, such as when overloading based on string type. The types corresponding to the postfix characters are:

Postfix Type
c char[ ]
w wchar[ ]
d dchar[ ]
"hello"c          // char[]
"hello"w          // wchar[]
"hello"d          // dchar[]

Character Literals

CharacterLiteral:
	' SingleQuotedCharacter '

SingleQuotedCharacter
	Character
	EscapeSequence
Character literals are a single character or escape sequence enclosed by single quotes, ' '.

Integer Literals

IntegerLiteral:
	Integer
	Integer IntegerSuffix

Integer:
	Decimal
	Binary
	Octal
	Hexadecimal
	Integer

IntegerSuffix:
	L
	u
	U
	Lu
	LU
	uL
	UL

Decimal:
	0
	NonZeroDigit
	NonZeroDigit DecimalDigits

Binary:
	0b BinaryDigits
	0B BinaryDigits

Octal:
	0 OctalDigits

Hexadecimal:
	0x HexDigits
	0X HexDigits

NonZeroDigit:
	1
	2
	3
	4
	5
	6
	7
	8
	9

DecimalDigits:
	0
	NonZeroDigit
	_

BinaryDigits:
	BinaryDigit
	BinaryDigit BinaryDigits

BinaryDigit:
	0
	1
	_

OctalDigits:
	OctalDigit
	OctalDigit OctalDigits

OctalDigit:
	0
	1
	2
	3
	4
	5
	6
	7
	_

HexDigits:
	HexDigit
	HexDigit HexDigits

HexDigit:
	0
	1
	2
	3
	4
	5
	6
	7
	8
	9
	a
	b
	c
	d
	e
	f
	A
	B
	C
	D
	E
	F
	_
Integers can be specified in decimal, binary, octal, or hexadecimal.

Decimal integers are a sequence of decimal digits.

Binary integers are a sequence of binary digits preceded by a '0b'.

Octal integers are a sequence of octal digits preceded by a '0'.

Hexadecimal integers are a sequence of hexadecimal digits preceded by a '0x' or followed by an 'h'.

Integers can have embedded '_' characters, which are ignored. The embedded '_' are useful for formatting long literals, such as using them as a thousands separator:

123_456		// 123456
1_2_3_4_5_6_	// 123456
Integers can be immediately followed by one 'L' or one 'u' or both.

The type of the integer is resolved as follows:

Decimal Literal Type
0 .. 2147483647 int
2147483648 .. 9223372036854775807 long
Decimal Literal, L Suffix Type
0L .. 9223372036854775807L long
Decimal Literal, U Suffix Type
0U .. 4294967295U uint
4294967296U .. 18446744073709551615U ulong
Decimal Literal, UL Suffix Type
0UL .. 18446744073709551615UL ulong
Non-Decimal Literal Type
0x0 .. 0x7FFFFFFF int
0x80000000 .. 0xFFFFFFFF uint
0x100000000 .. 0x7FFFFFFFFFFFFFFF long
0x8000000000000000 .. 0xFFFFFFFFFFFFFFFF ulong
Non-Decimal Literal, L Suffix Type
0x0L .. 0x7FFFFFFFFFFFFFFFL long
0x8000000000000000L .. 0xFFFFFFFFFFFFFFFFL ulong
Non-Decimal Literal, U Suffix Type
0x0U .. 0xFFFFFFFFU uint
0x100000000UL .. 0xFFFFFFFFFFFFFFFFUL ulong
Non-Decimal Literal, UL Suffix Type
0x0UL .. 0xFFFFFFFFFFFFFFFFUL ulong

Floating Literals

FloatLiteral:
	Float
	Float FloatSuffix
	Float ImaginarySuffix
	Float FloatSuffix ImaginarySuffix

Float:
	DecimalFloat
	HexFloat

DecimalFloat:
	DecimalDigits .
	DecimalDigits . DecimalDigits
	DecimalDigits . DecimalDigits DecimalExponent
	. DecimalDigits
	. DecimalDigits DecimalExponent
	DecimalDigits DecimalExponent

DecimalExponent
	e DecimalDigits
	E DecimalDigits
	e+ DecimalDigits
	E+ DecimalDigits
	e- DecimalDigits
	E- DecimalDigits

HexFloat:
	HexPrefix HexDigits .
	HexPrefix HexDigits . HexDigits
	HexPrefix HexDigits . HexDigits HexExponent
	HexPrefix . HexDigits
	HexPrefix . HexDigits HexExponent
	HexPrefix HexDigits HexExponent

HexPrefix:
	0x
	0X

HexExponent
	p DecimalDigits
	P DecimalDigits
	p+ DecimalDigits
	P+ DecimalDigits
	p- DecimalDigits
	P- DecimalDigits

FloatSuffix:
	f
	F
	L

ImaginarySuffix:
	i
Floats can be in decimal or hexadecimal format, as in standard C.

Hexadecimal floats are preceded with a 0x and the exponent is a p or P followed by a decimal number serving as the exponent of 2.

Floating literals can have embedded '_' characters, which are ignored. The embedded '_' are useful for formatting long literals to make them more readable, such as using them as a thousands separator:

123_456.567_8		// 123456.5678
1_2_3_4_5_6_._5_6_7_8	// 123456.5678
1_2_3_4_5_6_._5e-6_	// 123456.5e-6
Floating literals with no suffix are of type double. Floats can be followed by one f, F, or L suffix. The f or F suffix means it is a float, and L means it is a real.

If a floating literal is followed by i, then it is an ireal (imaginary) type.

Examples:

0x1.FFFFFFFFFFFFFp1023		// double.max
0x1p-52				// double.epsilon
1.175494351e-38F		// float.min
6.3i				// idouble 6.3
6.3fi				// ifloat 6.3
6.3Li				// ireal 6.3
It is an error if the literal exceeds the range of the type. It is not an error if the literal is rounded to fit into the significant digits of the type.

Complex literals are not tokens, but are assembled from real and imaginary expressions in the semantic analysis:

4.5 + 6.2i		// complex number

Keywords

Keywords are reserved identifiers.
Keyword:
	abstract
	alias
	align
	asm
	assert
	auto

	body
	bool
	break
	byte

	case
	cast
	catch
	cdouble
	cent
	cfloat
	char
	class
	const
	continue
	creal

	dchar
	debug
	default
	delegate
	delete
	deprecated
	do
	double

	else
	enum
	export
	extern

	false
	final
	finally
	float
	for
	foreach
	function

	goto

	idouble
	if
	ifloat
	import
	in
	inout
	int
	interface
	invariant
	ireal
	is

	long

	mixin
	module

	new
	null

	out
	override

	package
	pragma
	private
	protected
	public

	real
	return

	scope
	short
	static
	struct
	super
	switch
	synchronized

	template
	this
	throw
	true
	try
	typedef
	typeid
	typeof

	ubyte
	ucent
	uint
	ulong
	union
	unittest
	ushort

	version
	void
	volatile

	wchar
	while
	with

Special Tokens

These tokens are replaced with other tokens according to the following table:

Special Token Replaced with...
__FILE__ string literal containing source file name
__LINE__ integer literal of the current source line number
__DATE__ string literal of the date of compilation "mmm dd yyyy"
__TIME__ string literal of the time of compilation "hh:mm:ss"
__TIMESTAMP__ string literal of the date and time of compilation "www mmm dd hh:mm:ss yyyy"

Special Token Sequences

Special token sequences are processed by the lexical analyzer, may appear between any other tokens, and do not affect the syntax parsing.

There is currently only one special token sequence, #line.

SpecialTokenSequence
	# line Integer EndOfLine
	# line Integer Filespec EndOfLine

Filespec
	" Characters "
This sets the source line number to Integer, and optionally the source file name to Filespec, beginning with the next line of source text. The source file and line number is used for printing error messages and for mapping generated code back to the source for the symbolic debugging output.

For example:

int #line 6 "foo\bar"
x;			// this is now line 6 of file foo\bar
Note that the backslash character is not treated specially inside Filespec strings.