🐧 LINUX ADMINISTRATION

🧠 Introduction to Linux

🌍 What is Linux?

Linux is the kernel of a family of open-source, Unix-like operating systems.
It was created by Linus Torvalds in 1991 and has grown into one of the most influential open-source projects in history.
Known for its stability, security, and versatility, Linux is used across servers, mainframes, embedded systems, and desktops.

⚙️ What is a Kernel?

The kernel is the core component of any operating system.
It acts as a bridge between hardware and software, managing:
- CPU scheduling
- Memory allocation
- Input/output (I/O) operations
- Process management
The kernel ensures efficient and secure access to system resources.

💻 What is an Operating System (OS)?

An Operating System is software that manages hardware and software resources.
It includes:
- The kernel
- System libraries
- Utilities
- User interfaces (graphical shells or command-line interfaces)

🏗️ What is Unix?

Unix is a multi-user, multitasking operating system developed in the 1970s at Bell Labs by Ken Thompson and Dennis Ritchie.
Its modular design—kernel, shell, and utilities—formed the basis for Linux and many modern OSes.

🧩 Overview of GNU/Linux

GNU/Linux refers to a family of open-source operating systems combining:

The Linux kernel, and
The GNU Project’s tools and libraries.

The GNU Project, initiated by Richard Stallman in 1983, provided essential user-space utilities, compilers, and libraries that make Linux a complete, functional system.

🧱 Components of GNU/Linux

1️⃣ Linux Kernel

Developed by Linus Torvalds (1991).
Manages hardware resources, process scheduling, and system calls.
Licensed under the GNU GPL, allowing modification and redistribution.
Highly modular and supports a wide range of hardware (servers, phones, IoT devices).

2️⃣ GNU Software

Provided by the Free Software Foundation (FSF) to complement the kernel.
Key tools include:
- GNU Coreutils – Essential commands (ls, cp, mv, rm).
- GNU C Library (glibc) – Interfaces programs with the kernel.
- GNU Compiler Collection (GCC) – Compilers for C, C++, etc.
- Bash (Bourne Again Shell) – Command-line interface.

These tools enable core system tasks like file management, development, and administration.

3️⃣ Additional Components

Desktop Environments: GNOME, KDE Plasma, XFCE.
Package Managers:
- Debian-based → apt
- Red Hat-based → dnf / yum
- Arch-based → pacman
Software Applications: Firefox, LibreOffice, Apache, Nginx, etc.

⚡ Key Characteristics of Linux

Feature	Description
Open Source	Freely available and modifiable under the GNU GPL license.
Multitasking & Multiuser	Supports multiple users and processes simultaneously.
Stability	Reliable performance, ideal for servers and critical systems.
Security	Strong permissions and user privilege model.
Variety of Distributions	Many tailored variants for different needs (Ubuntu, Debian, Fedora, Arch).
Command Line Power	CLI provides efficiency and flexibility for advanced users.
Community Driven	Continuous improvement via global collaboration.

🐧 Linux Distributions

💡 What is a Distribution?

A Linux distribution (distro) is a complete operating system built around the Linux kernel, bundled with:

GNU tools and libraries
A package manager
Desktop environment or shell
Pre-installed software and configurations

Each distribution serves different goals — from ease of use to high security or minimalism.

⚙️ Components of a Distribution

Linux Kernel – Core of the system.
GNU Software – Shell, utilities, libraries.
Package Manager – Handles software installation and updates.
Desktop Environment – GUI (e.g., GNOME, KDE, XFCE).
Applications – Tools for daily use or development.
Configuration Defaults – Tailored usability and security settings.

🧭 Categories of Linux Distributions

Category	Description	Examples
General-Purpose	Balanced for usability and functionality.	Ubuntu, Linux Mint, Fedora
Server-Focused	Optimized for performance and uptime.	Debian, CentOS Stream, Rocky Linux
Developer-Centric	Customizable and cutting-edge.	Fedora Workstation, Arch, Gentoo
Lightweight	For old or resource-limited systems.	Puppy Linux, Lubuntu, MX Linux
Security-Focused	Privacy and penetration testing.	Kali Linux, Tails, Qubes OS
Specialized	For specific fields (media, IoT, education).	Ubuntu Studio, Raspberry Pi OS

🔄 Release Models

Type	Description	Example
Rolling Release	Continuous updates; always current.	Arch Linux, openSUSE Tumbleweed
Fixed Release	Stable versions released periodically.	Ubuntu, Debian

🏆 Popular Linux Distributions (2025)

🟠 Ubuntu

Debian-based; maintained by Canonical.
Uses apt package manager and GNOME desktop.
Regular releases + LTS versions every 2 years.
Great for desktops, servers, and cloud environments.

🔵 Debian

Renowned for stability and free software commitment.
Offers Stable, Testing, and Unstable branches.
Ideal for servers and users prioritizing reliability.

🟣 Fedora

Sponsored by Red Hat; cutting-edge and open-source focused.
Frequent updates and innovative technologies (e.g., Wayland, PipeWire).
Best for developers and tech enthusiasts.

⚫ Arch Linux

Minimalist, rolling-release model using pacman.
Offers total control and customization.
Includes the Arch User Repository (AUR) for extra packages.

🟢 Linux Mint

Ubuntu-based, user-friendly, and ideal for Windows migrants.
Uses Cinnamon, XFCE, or MATE desktops.
Comes with essential tools pre-installed.

🟤 CentOS Stream / Rocky Linux

Enterprise-grade distros built for stability.
CentOS Stream previews Red Hat updates.
Rocky Linux offers binary compatibility with RHEL.

🔐 Kali Linux

Debian-based, for security testing and forensics.
Bundled with tools like Metasploit, Nmap, and Burp Suite.

🕵️ Tails

Privacy-oriented; routes all traffic through Tor.
Leaves no trace on host systems — ideal for anonymity.

🍓 Raspberry Pi OS

Debian-based and optimized for Raspberry Pi hardware.
Used for IoT, robotics, and education projects.

🏗️ Linux Architecture

Linux architecture is composed of several key layers that work together to manage system resources and provide functionality.

1️⃣ Kernel

The core of the OS; manages CPU, memory, and device communication.
Provides process isolation and hardware abstraction.
Types of kernels:
- Monolithic
- Microkernel
- Hybrid
- Exokernel

2️⃣ System Libraries

Precompiled shared code that provides system functionality.
Acts as an interface between user applications and the kernel.
Example: glibc, the GNU C Library.

3️⃣ Shell

User interface for executing commands.
Translates commands into system calls for the kernel.
Examples: bash, zsh, fish.

4️⃣ Hardware Layer

Physical components: CPU, RAM, Disk, Network, I/O devices.
Interacts directly with the kernel and system libraries.

5️⃣ System Utilities

Tools for configuration and management (e.g., top, ps, ifconfig, apt).
Used for installing software, managing users, and monitoring performance.

🌟 Advantages of Linux

✅ Open Source – Free to use, modify, and distribute.
✅ Secure – Strong user permissions and minimal malware.
✅ Stable – Rarely crashes or requires rebooting.
✅ Community Support – Active developer and user base.
✅ Flexible – Install only the components you need.
✅ Fast & Lightweight – Runs efficiently even on older hardware.
✅ Privacy-Focused – Protects user data and system integrity.
✅ Compatible – Supports many file formats and hardware types.
✅ Frequent Updates – Easy software and system upgrades.

💽 Virtualization in Linux

🧩 What is Virtualization?

Virtualization enables multiple operating systems to run simultaneously on one physical machine by creating virtual machines (VMs) — isolated systems with their own CPU, memory, and storage.

It enhances:

Resource utilization
Isolation
Scalability
Flexibility

🧰 Components of Virtualization

Hypervisor
- Software managing virtual machines.
- Type 1 (Bare-Metal): Runs directly on hardware (KVM, Xen, VMware ESXi).
- Type 2 (Hosted): Runs on a host OS (VirtualBox, VMware Workstation).
Virtual Machine
- A self-contained system with its own OS and applications.
- Operates independently from the host system.
Virtualization Technologies
- Full Virtualization: Complete hardware emulation (QEMU, VirtualBox).
- Paravirtualization: Requires OS modification for better performance (Xen).
- Containerization: Lightweight, shares the host kernel (Docker, Podman).

🎯 Purpose of Virtualization in Linux

🧪 Development & Testing: Run multiple OSes for testing environments.
🖥️ Server Consolidation: Host many services on one machine.
🔒 Isolation: Improve security and stability.
🔄 Cross-Platform Use: Run Windows/macOS on Linux systems.
☁️ Cloud & DevOps: Power containerized and virtualized cloud infrastructures.

Popular Virtualization Tools:

VirtualBox
KVM (Kernel-based Virtual Machine)
VMware
QEMU
Docker / Podman (for containers)

🐚 The Shell

The Shell is a program that provides a text-based interface for users to communicate with the Linux kernel and run programs. It interprets user input, executes commands, and displays output.

It acts as an intermediary between the user, the Linux kernel, and system utilities — either interactively (via terminal commands) or non-interactively (through scripts).

⚙️ Key Functions

Command Execution: Runs commands like ls, cp, or sudo apt update.
Scripting: Automates repetitive tasks using shell scripts.
Environment Management: Manages environment variables like $PATH and $HOME.
Input/Output Redirection: Redirects output (e.g., ls > files.txt) and enables piping (ls | grep txt).
Job Control: Manages background/foreground processes (bg, fg).

🧠 Types of Shells

Bourne Shell (sh): Original Unix shell (1977), simple but limited.
Bash (Bourne Again Shell): Most common shell; improved sh version (default in many distros).
Z Shell (zsh): Enhanced features, great autocompletion, supports plugins (e.g., Oh My Zsh).
KornShell (ksh): Combines sh and csh features, often used in enterprise systems.
C Shell (csh/tcsh): Syntax similar to C; less common today.
Fish (Friendly Interactive Shell): Modern, user-friendly, colorful, and auto-suggestive.
Others: dash (lightweight, scripting-focused) and xonsh (Python-based).

📁 Core Categories and Commands

📂 File and Directory Management

Used to navigate and manipulate files and directories.

ls – List directory contents
- Usage: ls, ls -l, ls -a
- Example:
```
ls -la
```
cd – Change directory
- Usage: cd path/to/dir, cd .., cd ~
- Example:
```
cd /var/log
```
pwd – Print working directory
- Example:
```
pwd
```
mkdir – Create directory
- Example:
```
mkdir my_folder
```
rm – Remove files or directories
- Usage: rm file, rm -r folder
- Example:
```
rm -r old_folder
```
cp – Copy files or directories
- Example:
```
cp document.txt backup/
```

mv – Move or rename

Example:

mv file.txt new_name.txt
mv file.txt /tmp/

touch – Create or update a file
- Example:
```
touch notes.txt
```

🧾 File Viewing and Editing

Used to view or modify file content.

cat – View file contents
```
cat /etc/hosts
```
less – Scroll through files
```
less /var/log/syslog
```
nano – Simple text editor
```
nano script.sh
```
vim – Advanced text editor
```
vim config.txt
```

🖥️ System Information and Monitoring

Used to monitor processes and resource usage.

top – View system processes
```
top
```
htop – Enhanced process viewer
```
htop
```
df – Check disk space
```
df -h
```
du – Check directory sizes
```
du -sh /home/user
```
free – Check memory usage
```
free -h
```

🌐 Networking

Used for connectivity testing and network management.

ping – Test network reachability
```
ping 8.8.8.8
```
curl – Fetch data from URLs
```
curl example.com
```
wget – Download files
```
wget https://example.com/file.iso
```
ip – Manage network interfaces
```
ip addr show
```

🔐 Permissions and Ownership

Manage file and directory access.

chmod – Change file permissions
```
chmod +x script.sh
```
chown – Change ownership
```
chown user:users document.txt
```

⚙️ Process Management

Control and terminate processes.

ps – List running processes
```
ps aux | grep bash
```
kill – Terminate by PID
```
kill -9 1234
```

🧰 Miscellaneous Commands

man – Show manual pages
```
man ls
```
sudo – Run commands as superuser
```
sudo apt update
```
clear – Clear terminal
```
clear
```
history – Show command history
```
history | grep apt
```

📍 Relative & Absolute Paths

Paths define where files or directories are located in the filesystem.

🛣️ Absolute Path

Starts from the root /.
Example:
```
/home/user/documents/file.txt
```

🧭 Relative Path

Based on current directory.
Example:
```
./documents/file.txt
```

📘 Special Path Symbols

Symbol	Meaning	Example
`.`	Current directory	`./script.sh`
`..`	Parent directory	`cd ..`
`~`	Home directory	`cd ~/Documents`
`/`	Root directory	`ls /`

👥 User Management

Linux user management involves creating, modifying, and deleting users and groups, controlling access, and managing permissions.

🧩 Key Concepts

Users: Entities with accounts and UIDs (found in /etc/passwd).
- Root (UID 0): Full control.
- Regular Users: Limited access.
- System Users: Service accounts (e.g., daemon, nobody).
Groups: Collections of users sharing permissions (/etc/group).
Permissions: Controlled by chmod, chown, and displayed with ls -l.
Home Directories: Usually /home/username.
Configuration Files:
- /etc/passwd, /etc/shadow, /etc/group, /etc/gshadow
Sudo: Allows privileged commands. Example:
```
sudo apt update
```

🧑‍💻 Basic User Management Commands

1️⃣ Creating Users

sudo useradd -m -s /bin/bash john
sudo adduser jane
sudo passwd john

2️⃣ Modifying Users

sudo usermod -aG sudo jane
sudo chsh -s /bin/zsh john

3️⃣ Deleting Users

sudo userdel -r john
sudo deluser --remove-home jane

4️⃣ Group Management

sudo groupadd developers
sudo groupmod -n devteam developers
sudo groupdel developers
sudo gpasswd -a john devteam

5️⃣ Viewing User & Group Info

id john
whoami
groups john
cat /etc/passwd | grep john
cat /etc/group | grep devteam

6️⃣ Permission Management

chmod 750 /home/john/data
chown john:developers /home/john/project

🛡️ Best Practices

✅ Use adduser (more user-friendly than useradd)
🔒 Enforce strong passwords (passwd)
🚫 Limit root access; use sudo
👥 Manage permissions via groups
🧾 Regularly audit /etc/passwd and /etc/group
💾 Backup home directories before deletion
⚙️ Edit /etc/sudoers safely using visudo
🧠 Use descriptive usernames and group names

📦 Package Management in Linux

🧰 Overview

Package management is the process of installing, updating, configuring, and removing software packages in Linux.
Different Linux distributions use different package managers to handle software packages and dependencies.

⚙️ Repositories

Package managers use repositories (remote servers) that store collections of packages.
These repositories can be official or third-party.

🧩 Package Management Tools by Distribution

Distribution	Package Manager	Command Examples
Ubuntu / Debian	`apt`, `dpkg`	`apt install`, `apt update`, `dpkg -i`
CentOS / RHEL / Fedora	`yum`, `dnf`, `rpm`	`yum install`, `dnf update`, `rpm -ivh`
Arch Linux	`pacman`	`pacman -S package`
openSUSE	`zypper`	`zypper install package`

🧮 Basic Package Management Commands

🟢 Install a Package

sudo apt install nginx
sudo yum install nginx

🧱 Remove a Package

sudo apt remove nginx
sudo yum remove nginx

🔄 Update Repositories

sudo apt update
sudo yum update

⬆️ Upgrade Installed Packages

sudo apt upgrade
sudo yum upgrade

🔍 Searching and Inspecting Packages

apt search apache2
apt show apache2
yum info nginx
pacman -Qi nginx

📦 List Installed Packages

apt list --installed
yum list installed

🧰 Get Package Info

apt show nginx
yum info nginx

🧱 Install Local Package Files

sudo dpkg -i package.deb
sudo rpm -ivh package.rpm

🔧 Fix Broken Dependencies (Debian/Ubuntu)

sudo apt --fix-broken install

🔁 Package Caching and Cleanup

🧹 Clear Cached Packages

sudo apt clean
sudo yum clean all

📜 View Package History (YUM/DNF)

yum history
dnf history

🧩 Common Package Managers

🐧 Debian/Ubuntu-based Systems

Use APT (Advanced Package Tool).

sudo apt update             # Refresh package index
sudo apt install nginx      # Install a package
sudo apt remove nginx       # Remove a package
sudo apt purge nginx        # Remove package with config files
sudo apt upgrade            # Upgrade all upgradable packages
sudo apt autoremove         # Remove unused dependencies

🐤 Red Hat/CentOS-based Systems

Use YUM (or DNF for newer systems).

sudo yum install nginx
sudo yum remove nginx
sudo yum update
sudo yum search nginx
sudo yum list installed

🧱 Arch-based Systems

Use Pacman.

sudo pacman -Syu            # Update and upgrade
sudo pacman -S nginx        # Install
sudo pacman -R nginx        # Remove
sudo pacman -Ss nginx       # Search
sudo pacman -Q              # List installed packages

📁 File Management in Linux

📘 Overview

File management in Linux involves creating, viewing, editing, moving, copying, and deleting files and directories using command-line tools.

🧰 Common File Commands

Action	Command	Example
List files	`ls`	`ls -la`
Change directory	`cd`	`cd /var/log`
Print working directory	`pwd`	`pwd`
Create a file	`touch`	`touch file.txt`
View file contents	`cat`, `less`, `head`, `tail`	`cat file.txt`
Copy files	`cp`	`cp file1 file2`
Move or rename	`mv`	`mv old new`
Remove files	`rm`	`rm file.txt`
Create directories	`mkdir`	`mkdir new_folder`
Remove directories	`rmdir`, `rm -r`	`rm -r folder`
View file type	`file`	`file data.bin`

🪣 Creating Files

touch file.txt        # Create an empty file

🧮 File Viewing and Editing

📖 Viewing Files

cat file.txt          # View full file
less file.txt         # Scroll through file
head -n 10 file.txt   # View first 10 lines
tail -n 10 file.txt   # View last 10 lines
tail -f logfile.log   # Follow updates in real-time

🔎 Viewing File Details

ls -l                         # Long listing format
ls -lh                        # Human-readable sizes
ls -a                         # Show hidden files
ls -R                         # Recursive listing
file filename                 # Show file type
stat filename                 # Display file metadata

✍️ Editing Files

nano file.txt
vi file.txt
vim file.txt

🧱 Copying and Moving Files

📂 Copy File or Directory

cp file1 file2                # Copy file
cp file1 /dir/                # Copy to directory
cp -r dir1 dir2               # Copy directory recursively

🔀 Move or Rename File

mv file1 /dir/                # Move file
mv oldname.txt newname.txt    # Rename file

🧹 Removing Files and Directories

❌ Remove File

rm file.txt           # Delete a file
rm -i file.txt        # Prompt before deleting

🗑️ Remove Directory

rmdir empty_dir       # Remove empty directory
rm -r folder_name     # Remove a directory recursively
rm -rf folder_name   # Force remove

🧾 File Attributes and Permissions

🔍 View Permissions

ls -l

🔑 Change Permissions

chmod 755 script.sh
chmod u+x script.sh

👤 Change Ownership

chown user:group file.txt  # Change owner and group

📊 File Size and Disk Usage

📏 Check File Size

ls -lh file.txt

💾 Check Disk Usage

du -sh *
df -h

🌟 Filename Expansion (Globbing)

🧠 Overview

Globbing allows you to use special characters (wildcards) to match multiple filenames in the shell.

🔤 Common Wildcards

Wildcard	Description	Example
`*`	Matches any number of characters	`ls *.txt`
`?`	Matches a single character	`ls file?.txt`
`[ ]`	Matches any single character inside brackets	`ls file[1-3].txt`
`{ }`	Matches a comma-separated list of patterns	`ls {file1,file2}.txt`

🧩 Examples

🗂️ Match All Text Files

ls *.txt

🧮 Match Files with Single-Character Difference

ls file?.txt

🧱 Match a Range of Filenames

ls file[1-5].txt

🧰 Match Multiple Specific Files

ls {notes,summary,report}.txt

⚙️ Escaping Special Characters

If you want to prevent globbing, use quotes or backslashes.

ls "*.txt"     # Treats *.txt as literal text
ls \*.txt      # Escapes the asterisk

📝 TEXT FILE MANAGEMENT

🧰 Overview

Text file management in Linux is a fundamental aspect of system administration, development, and everyday computing in GNU/Linux distributions.

Text files are ubiquitous in Linux, serving as:
- Configuration files (e.g., /etc/passwd)
- Scripts (e.g., .sh files)
- Logs (e.g., /var/log/syslog)
- Data storage (e.g., .csv or .txt)
Linux provides a rich set of command-line tools for:
- Creating, viewing, editing, searching, manipulating, and comparing text files
- Emphasizing efficiency, automation, and precision
Text files are plain ASCII or UTF-8 encoded files containing human-readable content, without binary data or formatting (unlike .docx).
Linux treats text files as streams of characters, enabling powerful piping and redirection.
Example:
```
command > file.txt    # Redirect output to a file
```
Common Encodings: UTF-8 (default in modern distributions), ASCII.
Convert encodings using iconv:
```
iconv -f ISO-8859-1 -t UTF-8 file.txt > converted.txt
```
Line Endings:
- Linux: LF (\n)
- Windows: CRLF (\r\n) Convert formats using dos2unix:
```
dos2unix file.txt
```

Piping and Redirection is essential for text management:

> → Overwrite
>> → Append
< → Input
| → Pipe output to another command

Example:

echo "Hello" > file.txt                   # Creates/overwrites file.txt
cat file1.txt file2.txt > combined.txt    # Concatenate files into one

🏗️ Creating Text Files

Creating text files is straightforward, often involving redirection or text editors.

🔸 Using Redirection

🗣️ `echo`

Used for creating simple text files.

echo "Line 1\nLine 2" > notes.txt

Creates notes.txt with two lines.

🧾 `printf`

Provides more precise formatting.

printf "Value: %d\n" 42 > data.txt

🕒 Using `touch`

Creates empty files or updates timestamps.

touch empty.txt

Creates an empty file named empty.txt.

✍️ Using Text Editors

🪶 Nano – Beginner-Friendly CLI Editor

Usage:
```
nano file.txt
```
Shortcuts:
- Save → Ctrl + O
- Exit → Ctrl + X
- Search → Ctrl + W
Example:
```
sudo nano /etc/hosts
```
Edit system hosts file (requires superuser privileges).

⚙️ Vim – Powerful Modal Editor

Usage:
```
vim file.txt
```
Modes:
- Insert mode → i
- Save & exit → :wq
- Search → /pattern
Advanced:
- Record macros → q
- Customize via .vimrc (e.g., syntax checking)
Example:
```
vim script.sh
```
Edit a shell script and enable line numbers using :set number.

🧠 Emacs – Extensible Editor (GUI/CLI)

Usage:

emacs file.txt

Shortcuts:
Save → Ctrl + X, then Ctrl + S
Exit → Ctrl + X, then Ctrl + C
Features:
Org-mode for structured text
Integration with tools like Magit for Git

🪟 Other Editors

vi – Minimal version of Vim
gedit – GUI editor for GNOME
kate – GUI editor for KDE

🔄 Redirection (Managing Data Streams) in Linux

Redirection allows you to change the default flow of data for a process.
Normally:

Standard Input (stdin) → comes from your keyboard
Standard Output (stdout) → goes to your screen (terminal)
Standard Error (stderr) → also goes to your screen

With redirection, you can reroute these data streams to or from files, devices, or other commands.

⚙️ The Three Standard Streams

Stream Name	File Descriptor	Default Device	Purpose
stdin	`0`	Keyboard	Input to a program
stdout	`1`	Terminal	Normal output
stderr	`2`	Terminal	Error messages

🔄 Basic Redirection Operators

1. Output Redirection

Send output to a file instead of the screen.

command > file.txt

Creates or overwrites file.txt with the output.
Example:
```
ls > list.txt
```
Saves the directory listing into list.txt.

2. Append Output

Append output to the end of a file instead of overwriting it.

command >> file.txt

Example:
```
echo "New entry" >> log.txt
```

3. Input Redirection

Use a file as input to a command instead of the keyboard.

command < input.txt

Example:
```
sort < unsorted.txt
```

4. Redirect stderr (Errors)

Send error messages to a file.

command 2> errors.txt

Example:
```
ls /fakepath 2> error.log
```

5. Redirect stdout and stderr together

Send both normal output and errors to the same file.

command > all.log 2>&1

or shorthand:

command &> all.log

Example:

find / -name "*.conf" > results.log 2>&1

6. Suppress Output (Discard It)

Redirect output to /dev/null, a special device that discards data.

command > /dev/null 2>&1

Commonly used to run commands quietly.

🔗 Combining Redirection with Pipes

Pipes (`|`)

A pipe connects the output of one command to the input of another.

command1 | command2

Example:
```
ps aux | grep firefox
```
Sends the list of running processes into grep to filter results.

You can chain multiple pipes:

cat /etc/passwd | grep bash | sort

📘 Advanced Redirection Examples

1. Separate stdout and stderr

command > output.txt 2> error.txt

2. Swap stdout and stderr

command 3>&1 1>&2 2>&3

(Advanced shell scripting use case)

3. Here Documents (Multiline Input)

Feed inline text as input to a command.

cat <<EOF
This is a
multiline text input
EOF

🧩 Practical Use Cases

Task	Command
Save output of `ls`	`ls > files.txt`
Append date to log	`date >> log.txt`
Suppress all output	`command > /dev/null 2>&1`
Capture errors only	`command 2> errors.log`
Combine outputs	`command > all.log 2>&1`
Chain commands	`cat file.txt \| grep keyword \| sort`

🔐 Summary

Symbol	Meaning
`>`	Redirect stdout (overwrite)
`>>`	Redirect stdout (append)
`<`	Redirect stdin
`2>`	Redirect stderr
`&>`	Redirect stdout and stderr together
`	`
`/dev/null`	Discard output

🧭 In Short

Redirection in Linux = Controlling where data flows
(From → Keyboard, To → Screen, or vice versa → Files, Programs, Devices)

🧩 Data Processing Through Command Chaining in Linux

Command chaining in Linux is a powerful technique that allows multiple commands to be executed sequentially, with the output of one command serving as the input to another. This approach enables elegant, efficient data processing pipelines that can transform raw data into actionable insights without intermediate files or complex scripting.

The Unix philosophy emphasizes building small, focused tools that do one thing well and can be combined to solve complex problems. Command chaining is the practical embodiment of this philosophy, enabling developers and system administrators to create sophisticated data processing workflows from simple, reusable components.

By chaining commands, users can:

Execute complex workflows efficiently.
Pass data between commands.
Control execution flow based on success or failure.
Create powerful one-liners for data manipulation.

Core Concepts of Command Chaining

The Pipe Operator (|)

The pipe operator takes the standard output (stdout) of one command and passes it as the standard input (stdin) to another command.

command1 | command2 | command3

Sequential Execution Operators

Semicolon (;) – Executes commands sequentially regardless of success or failure
```
 command1 ; command2 ; command3
```
Logical AND (&&) – Executes the next command only if the previous succeeds
```
command1 && command2 && command3
```
Logical OR (||) – Executes the next command only if the previous fails bash command1 || command2 || command3

Command Grouping with parentheses ()

(command1 ; command2) | command3

🧰 Summary Table

Operator	Description	Example
`;`	Executes commands sequentially, regardless of success or failure.	`cmd1; cmd2`
`&&`	Executes the next command only if the previous succeeds.	`cmd1 && cmd2`
`\|\|`	Executes the next command only if the previous fails.	`cmd1 \|\| cmd2`
`\|`	Pipes output of one command as input to another.	`cmd1 \| cmd2`
`\|&`	Pipes both stdout and stderr to next command.	`cmd1 \|& cmd2`
`&`	Runs a command in the background.	`cmd1 &`

🧩 Fundamental Data Processing Filters

🔍 `grep` – Pattern Matching

Used to search for text patterns within files or command outputs.

cat access.log | grep "ERROR"
grep -in "warning" system.log
grep -v "DEBUG" application.log

✂️ sed – Stream Editor

Used for find-and-replace, deleting, or selecting specific lines in text streams.

sed 's/old/new/' file.txt        # Replace first occurrence per line
sed 's/old/new/g' file.txt      # Replace all occurrences per line
sed '/^#/d' config.txt          # Delete lines starting with '#'
sed -n '10,20p' data.txt        # Print lines 10 to 20

🧮 awk – Text Processing Language

Used for structured text manipulation, field extraction, and computations.

awk '{print $1, $3}' data.txt                 # Print 1st and 3rd fields
awk '$2 > 100 {print $1, $2}' records.txt     # Conditional printing
awk '{sum += $1} END {print sum}' numbers.txt # Sum all values in column 1
awk '$3 ~ /error/ {print}' logs.txt           # Match pattern in 3rd field

🔢 sort – Ordering Data

Used to arrange lines in ascending, descending, or key-based order.

sort -n numbers.txt      # Numeric sort
sort -r file.txt         # Reverse sort
sort -k2 -n data.txt     # Sort by 2nd column numerically
sort -u file.txt         # Unique sort (remove duplicates)

🚫 uniq – Removing Duplicates

Filters out or counts duplicate lines (often used after sort).

uniq file.txt     # Remove consecutive duplicates
uniq -c file.txt  # Count occurrences
uniq -u file.txt  # Show unique lines only
uniq -d file.txt  # Show duplicate lines only

⚙️ Shell Command Pipelines and Chaining Examples

🧩 Example: Simple Pipeline

cat data.txt | grep "error" | sort | uniq -c | sort -nr

Explanation:

cat data.txt — Reads the contents of the file.
grep "error" — Filters lines containing “error”.
sort — Sorts the filtered lines.
uniq -c — Counts unique occurrences.
sort -nr — Sorts numerically in reverse order.

🪄 Examples of Command Chaining

1️⃣ Sequential Execution with `;`

Commands run one after another, regardless of success or failure.

echo "Starting process"; ls; echo "Process complete"

2️⃣ Conditional Execution with `&&` and `||`

&& runs the next command only if the previous succeeds.
|| runs the next command only if the previous fails.

mkdir reports && echo "Directory created successfully" || echo "Failed to create directory"

📊 Data Processing Example: CSV File Handling

Given a CSV file named sales.csv:

Region,Product,Sales
East,Phone,150
West,Laptop,200
East,Laptop,100
West,Phone,250

Extract and summarize sales by product:

cat sales.csv | tail -n +2 | cut -d',' -f2,3 | sort | \
awk -F',' '{arr[$1]+=$2} END {for (i in arr) print i, arr[i]}' | sort -k2 -nr

Output:

Phone 400
Laptop 300

🧩 Combining `|`, `&&`, and `||` for Smart Pipelines

cat data.txt | grep "SUCCESS" > success.log && echo "Success log updated" || echo "No successful entries found"

Saves matching lines to success.log.
Prints a success message if matches are found, otherwise prints an error message.

🔁 Example: Log File Analysis

Find the top 5 most frequent errors in system logs:

grep "ERROR" /var/log/syslog | cut -d' ' -f5- | sort | uniq -c | sort -nr | head -5

🔧 Example: Cleaning and Transforming Data

Convert raw text data to uppercase, normalize spacing, and remove empty lines:

cat raw_data.txt | tr -d '\r' | sed 's/[[:space:]]\+/ /g' | grep -v '^$' | awk '{print toupper($0)}' > clean_data.txt

🧱 Example: Background Processing with `&` and Chaining

Run multiple sorting jobs in parallel and wait for all to finish:

cat bigdata1.csv | sort > sorted1.csv & 
cat bigdata2.csv | sort > sorted2.csv &
wait && echo "All sorting complete"

🧮 Example: Conditional Data Export

Compress a file only if it exists:

[ -f analysis.txt ] && cat analysis.txt | gzip > analysis.gz || echo "File not found!"

🌍 Environment Variables in Linux

🧰 Overview

Environment variables are dynamic named values that affect the behavior of processes and the shell environment in Linux.
They store essential system information and user preferences — such as the location of executables, default editor, user identity, and current working directory.

Environment variables are key to system configuration, shell scripting, and application execution.

🧩 Types of Environment Variables

Type	Scope	Description
System-wide	Global	Available to all users and processes (e.g., `/etc/environment`, `/etc/profile`).
User-specific	Local	Defined for a single user (e.g., `~/.bashrc`, `~/.bash_profile`).
Session variables	Temporary	Exist only for the current shell session.

🧠 Common Environment Variables

Variable	Description
`PATH`	Directories where executable programs are located
`HOME`	Path to the user’s home directory
`USER`	Current logged-in username
`SHELL`	Path to the user’s shell
`PWD`	Current working directory
`OLDPWD`	Previous working directory
`LANG`	System language and locale
`EDITOR`	Default text editor
`LOGNAME`	Name of the logged-in user
`MAIL`	Path to user’s mail directory
`TERM`	Terminal type
`HISTFILE`	Location of command history file
`HISTSIZE`	Number of commands stored in history

🔍 Viewing Environment Variables

🧾 List All Variables

printenv
env
set

🔎 Display a Specific Variable

echo $PATH
echo $HOME
echo $SHELL

💡 Example

echo "My current shell is: $SHELL"
echo "Logged in as: $USER"
echo "Home directory: $HOME"

🧱 Setting Environment Variables

1️⃣ Temporary (Session Only)

Variables defined in this way exist only until the terminal session is closed.

MY_NAME="Irvine"
export COUNTRY="Kenya"
echo $MY_NAME
echo $COUNTRY

2️⃣ Persistent (Across Sessions)

To make variables available permanently, define them in shell configuration files.

🏠 User-Level (Persistent for Current User)

Add to:

~/.bashrc or ~/.bash_profile (for Bash)
~/.zshrc (for Zsh)

Example:

export EDITOR="vim"
export PATH="$PATH:/opt/custom/bin"

Apply changes immediately:

source ~/.bashrc

⚙️ System-Level (Persistent for All Users)

Add to:

/etc/environment
/etc/profile
/etc/bash.bashrc

Example:

PATH="$PATH:/usr/local/sbin"
export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-amd64"

🧩 Unsetting or Removing Variables

🧹 Remove Variable from Current Session

unset MY_NAME
echo $MY_NAME     # No output

🧽 Remove from Persistent Configuration

Edit your ~/.bashrc or /etc/environment and delete the relevant line.

🔄 Modifying `PATH` Variable

The PATH variable tells the shell where to search for executables.

➕ Add Directory to PATH (Temporarily)

export PATH=$PATH:/opt/tools/bin

♻️ Add Directory to PATH (Permanently)

Add this line to ~/.bashrc:

export PATH="$PATH:/opt/tools/bin"

Then reload:

source ~/.bashrc

🧪 Verify PATH

echo $PATH | tr ':' '\n'

🧮 Using Variables in Commands

🧩 Example 1: Substitution

FOLDER="/var/log"
echo "Listing files in $FOLDER"
ls $FOLDER

🧩 Example 2: Combining Variables

FIRST="Linux"
SECOND="Admin"
echo "$FIRST $SECOND"
# Output: Linux Admin

🧩 Example 3: Default Values

echo ${EDITOR:-nano}
# Prints current EDITOR or defaults to 'nano' if not set

🧰 Exporting Variables to Subshells

When you run a new shell or script, variables must be exported to be accessible in child processes.

MY_VAR="Hello"
bash -c 'echo $MY_VAR'     # No output (not exported)

export MY_VAR="Hello"
bash -c 'echo $MY_VAR'     # Output: Hello

📜 Environment Files Summary

File	Scope	Description
`/etc/environment`	System-wide	Global variables for all sessions
`/etc/profile`	System-wide	Login shell initialization
`/etc/bash.bashrc`	System-wide	Interactive shell configuration
`~/.bash_profile`	User-specific	Executed at login
`~/.bashrc`	User-specific	Executed for interactive non-login shells
`~/.profile`	User-specific	Legacy file for login shells

⚙️ Example: Custom Environment Setup

Suppose you want to set up a development environment with Java and Python paths.

1️⃣ Edit your ~/.bashrc:

export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-amd64"
export PYTHONPATH="$HOME/projects/lib"
export PATH="$PATH:$JAVA_HOME/bin:$PYTHONPATH"

2️⃣ Reload your configuration:

source ~/.bashrc

3️⃣ Verify:

echo $JAVA_HOME
echo $PYTHONPATH
which java
which python3

🧠 Tips and Best Practices

Always use export for variables needed by subprocesses.
Avoid overwriting critical variables like $PATH — always append or prepend instead.
Use lowercase for temporary or script-only variables (temp_dir="/tmp").
Use uppercase for environment-wide variables (PATH, EDITOR, JAVA_HOME).
Quote variable values containing spaces (export GREETING="Hello World").
Reload your configuration after changes:
```
source ~/.bashrc
```

🧩 Example Use in Scripts

📜 Script: `env_test.sh`

#!/bin/bash
echo "User: $USER"
echo "Home Directory: $HOME"
echo "Current Shell: $SHELL"
echo "PATH: $PATH"

▶️ Run:

bash env_test.sh

✅ Output:

User: irvine
Home Directory: /home/irvine
Current Shell: /bin/bash
PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin