In this post, we will learn the tokens in Java. Let’s learn how tokens are formed first.
Basically, the tokens are formed from the Lexeme. In simple language, the lexeme is an individual logical unit that has its own meaning and purpose. Let’s take a few examples to understand it clearly.
Example 1:
- int
- float
- return
- break
We all know that these are the reserved words in Java and we called them as Java Keywords.
Example 2:
The basic operators in Java are:
- +
- –
- *
- /
In both the examples, we have taken some keywords and operators and they all have different meanings. int and float are data types in Java and return and break are reserved words and they have different purposes.
Similarly, the operators +, -, *, and / have a different purpose.
In the first example, all the lexemes are a group of Keywords and in the second example, all the lexemes are a group of operators.
Hence, all the Lexemes under a certain group are called the Tokens. In simple words, the smallest part of the program which is identified by the compiler is the tokens.
Let’s find some Lexemes and Tokens in the example below:
int a = b + c - d * e;
From the statement above we can find 11 numbers of Lexemes. Which are:
- int
- a
- =
- b
- +
- c
- –
- d
- *
- e
- ;
And we can separate them in a group:
- Data Type:
int - Identifiers:
a b c d e - Operators:
= + - * - Line Terminator or Separator:
;
These groups are called Tokens.
In Java programming language, the available tokens are: