new Tokenizer(proto)
Constructs a new Tokenizer.
Parameters:
| Name | Type | Description |
|---|---|---|
proto |
string | Proto to tokenize |
Members
-
index :number
-
Current index.
-
line :number
-
Current line.
-
readingString :boolean
-
Whether currently reading a string or not.
-
source :string
-
Source to parse.
-
stack :Array
-
Stacked values.
-
stringEndsWith :string
-
Whatever character ends the string. Either a single or double quote character.
Methods
-
next() → {string}
-
Gets the next token and advances by one.
Throws:
If it's not a valid proto file- Type
- Error
Returns:
Token or
nullon EOF- Type
- string
-
peek() → {string}
-
Peeks for the next token.
Throws:
If it's not a valid proto file- Type
- Error
Returns:
Token or
nullon EOF- Type
- string
-
toString() → {string}
-
Returns a string representation of this object.
Returns:
String representation as of "Tokenizer(index/length)"
- Type
- string