Lexical Analysis in simple words – token generation, blank space, symbol table, lexical error

Lexical Analysis in simple words – token generation, blank space, symbol table, lexical error

Lexical Analysis in Simple Words

Lexical Analysis is the first phase of a compiler that breaks down the source code into smaller meaningful units called tokens. This process is done by a program called the Lexical Analyzer (Lexer).

 How Lexical Analysis Works?

 The source code is read character by character.
 Meaningful groups of characters are converted into tokens.
 Unnecessary characters like spaces, tabs, and comments are removed.
 A symbol table is created to store identifiers and keywords.
 If an unknown symbol is found, a lexical error is reported.

 Example of Token Generation

Input Code:

Lexical Analyzer Output (Tokens):

Each part of the code is classified into a specific token type.

 Handling Blank Spaces & Comments

Whitespace (spaces, tabs, new lines) is ignored by the lexical analyzer.
Comments (//, /* ... */) are removed, as they don’t affect program execution.

Example:

 The lexer ignores "// This is a comment" and processes only:

 Symbol Table

 A symbol table stores identifiers (variable names, function names) and their attributes like data type, memory location, scope, etc.
 It helps the compiler track variables efficiently.

Example Symbol Table:

 Lexical Errors

Errors occur when the lexer finds an unknown or invalid character.

Example:

Lexical Error: “Unexpected character ‘@’ found.”

 Summary

Lexical Analysis breaks code into tokens.
Blank spaces & comments are ignored.
 A symbol table stores variable names and types.
Lexical errors occur due to unknown characters.

Would you like a step-by-step breakdown of a real Lexical Analysis process?

Lexical Analysis in simple words – token generation, blank space, symbol table, lexical error

LEXICAL ANALYSIS Structure of compiler – Functions and …

Lexical Analysis – Compiler Construction

compiler design lecture notes

Chapter 3 Lexical Analysis

CS 335: Lexical Analysis

Here is a simple explanation of Lexical Analysis and its key components, written in easy language (suitable for beginners or non-technical readers):

---

Lexical Analysis in Simple Words

Lexical Analysis is the first step of a compiler.
It reads the code written by a programmer and breaks it into small parts called tokens.

---

1. What is a Token?

A token is a small, meaningful unit in a programming language.
Think of it like words in a sentence.

Example:

This line will be broken into tokens like:

These tokens help the compiler understand the code step-by-step.

---

2. Role of Blank Spaces

• Blank spaces, tabs, and new lines are called whitespace.

• The lexer (lexical analyzer) usually ignores blank spaces unless they are part of a string or affect indentation (like in Python).

• Their job is to separate tokens so the compiler knows where one ends and another begins.

Example:

Even with extra spaces, the tokens will still be:
int, a, =, 10, ;

---

3. Symbol Table

A symbol table is like a notebook the compiler uses to remember:

• Variables (like x, a, total)

• Function names

• Data types

• Scope and other details

Every time a new identifier is found (like a variable name), it is stored in the symbol table.

---

4. Lexical Errors

These are mistakes in the structure of the code, usually when:

• The token is not valid

• Unexpected characters are found

• A variable starts with a number (like 2abc)

• Illegal symbols are used (@, # in wrong places)

Example:

This is a lexical error.

---

Summary Table:

---

Would you like a visual chart, example code walkthrough, or a PDF summary of this topic?

Lexical Analysis in simple words – token generation, blank space, symbol table, lexical error

Compilers Download