How to Process Text with Bash Using Grep, Sed, and Awk Commands

Introduction

Text processing in Bash using the grep, sed, and awk commands lets you filter and manipulate input to perform specific tasks, such as file content processing. You can combine the text processing commands with loops, conditional statements, and expressions for advanced processing.

This article explains how to process text in Bash using the grep, sed, and awk commands.

Create a New Bash Script

Follow the steps below to create a new Bash script that loads an input file with multiple lines of content to a variable for further text processing.

Create a new log.txt file to store multiple lines of text.
console
```
$ nano log.txt
```

Add the following contents to the log.txt file.

text

2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
2024-10-25 09:30:45 WARNING: The Disk space running low.
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 ERROR: File not found: www.yml.
2024-10-24 11:15:00 WARNING: High memory usage detected.
2024-10-23 14:25:01 INFO: A scheduled backup is complete.
2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.
2024-10-23 16:20:30 INFO: root User logged out successfully.
2024-10-23 18:45:10 WARNING: CPU temperature exceeding normal level.

Save and close the file.

Create a new processing.sh Bash script that inputs the log.txt file as a variable.
console
```
$ nano processing.sh
```
Add the following contents to the processing.sh file.
bash
```
#!/bin/bash

LOG="log.txt"
```
Save and close the file.

In the above script, the LOG variable takes the log.txt as an input. You'll process the variable value using the grep, sed, and awk commands in the next sections.

Use the `grep` Command to Process Text File

The grep command searches specific text patterns in files, such as the log.txt file you created, and outputs the results for additional processing in a Bash script. Follow the steps below to use the grep command and display the results.

Open the processing.sh script you created earlier.
console
```
$ nano processing.sh
```
Add the following command at the end of the file.
bash
```
grep -i "ERROR" "$LOG"
```
Save and close the file.

The above grep command searches for the term ERROR in the log.txt file and displays all matching results.

Run the script using Bash.

console

$ bash processing.sh

Output:

2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
2024-10-24 10:45:55 ERROR: File not found: www.yml.
2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.

Use the `awk` Command to Process Text File

The awk command analyzes structured data and searches for specific patterns in a file. You can use the awk command to customize file contents, split lines into fields or columns for processing, and match specific patterns. Follow the steps below to process the log.txt file columns using the awk command.

Open the processing.sh Bash script.
console
```
$ nano processing.sh
```
Add the following contents to the file.
bash
```
awk '{print $1, $3}' "$LOG"
```
Save and close the file.

The above awk command prints the first and third columns in the log.txt file to display the date and error type.

Run the script using Bash.

console

$ bash processing.sh

Output:

2024-10-25 INFO:
2024-10-25 ERROR:
2024-10-25 WARNING:
2024-10-24 INFO:
2024-10-24 ERROR:
2024-10-24 WARNING:
2024-10-23 INFO:
2024-10-23 ERROR:
2024-10-23 INFO:
2024-10-23 WARNING:

Use `sed` Command to Process Text File

Stream Editor (sed) is a text processing command that processes and transforms text into a file. The sed command searches, modifies, and replaces contents in standard input or a specific file using Bash. Follow the steps below to use the sed command to search and replace contents in the log.txt file.

Open the processing.sh Bash script.
console
```
$ nano processing.sh
```
Add the following command at the end of the file.
bash
```
sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG"
```
Save and close the file.

The above sed command replaces ERROR with New ERROR and WARNING with New WARNING values in the log.txt file.

Run the script using Bash.

console

$ bash processing.sh

Output:

2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
2024-10-25 09:00:01 New ERROR: Unable to connect to the database on port 3306.
2024-10-25 09:30:45 New WARNING: The Disk space running low.
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found: www.yml.
2024-10-24 11:15:00 New WARNING: High memory usage detected.
2024-10-23 14:25:01 INFO: A scheduled backup is complete.
2024-10-23 15:00:00 New ERROR: Apache: The Network connection lost.
2024-10-23 16:20:30 INFO: root User logged out successfully.
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding normal level.

Combine `grep`, `sed`, and `awk` Commands

You can combine grep, sed, and awk commands to analyze and manipulate large inputs and perform advanced text processing tasks using Bash. Follow the steps below to combine the grep, sed, and awk commands and process the log.txt file.

Create a new combine.sh Bash script.
console
```
$ nano combine.sh
```

Add the following contents to the file.

                            bash
                            
#!/bin/bash

LOG="log.txt"

echo "Processing the log file: $LOG"

echo "Date       Time        Log Level   Message"
echo "------------------------------------------"

grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'

echo -e "\nLog processing complete."

Save and close the file.

In the above Bash script, the combined grep, sed, and awk commands use pipes | to redirect the output of each command and display the modified contents of the log.txt file in a well-structured format that includes all log entries.

Run the script using Bash.

console

$ bash combine.sh

Output:

Processing the log file: log.txt
Date       Time        Log Level   Message
------------------------------------------
2024-10-25 08:15:23 INFO: Apache Web Server Application
2024-10-25 09:00:01 New ERROR: Unable to connect
2024-10-25 09:30:45 New WARNING: The Disk space
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found:
2024-10-24 11:15:00 New WARNING: High memory usage
2024-10-23 14:25:01 INFO: A scheduled backup is
2024-10-23 15:00:00 New ERROR: Apache: The Network
2024-10-23 16:20:30 INFO: root User logged out
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding

Log processing complete

Use `grep`, `sed`, and `awk` with Conditional Bash Statements

You can use grep, sed, and awk commands to process text further with if-else, case, and if-then conditional statements to enhance output readability and handle errors. Follow the steps below to use the conditional statement to process the log.txt file using the search commands in a Bash script.

Create a new conditional-processing.sh Bash script.
console
```
$ nano conditional-processing.sh
```

Add the following contents to the conditional-processing.sh file.

                            bash
                            
                        
#!/bin/bash

LOG="log.txt"

if [[ -f "$LOG" ]]; then
    echo "Processing the log file: $LOG"

    if grep -qE 'ERROR|WARNING' "$LOG"; then

        echo "Date       Time        Log Level   Message"
        echo "------------------------------------------"

        grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'

        echo -e "\nLog processing complete."
    else
        echo "No ERRORs or WARNINGs found in the log file."
    fi
else
    echo "The Log file: $LOG is not available"
fi

echo "...............Completed.............."

Save and close the file.

In the above file:

The outer if-else conditional statement validates the input variable.
The inner if-else conditional statement runs a grep condition to find the ERROR or WARNING fields in the file. The combined grep, sed, and awk commands run when the condition evaluates to true and perform additional processing to display the modified output.

Run the script using Bash.

console

$ bash conditional-processing.sh

Output:

Processing the log file: log.txt
Date       Time        Log Level   Message
------------------------------------------
2024-10-25 08:15:23 INFO: Apache Web Server Application
2024-10-25 09:00:01 New ERROR: Unable to connect
2024-10-25 09:30:45 New WARNING: The Disk space
2024-10-24 10:15:12 INFO: root User logged in.
2024-10-24 10:45:55 New ERROR: File not found:
2024-10-24 11:15:00 New WARNING: High memory usage
2024-10-23 14:25:01 INFO: A scheduled backup is
2024-10-23 15:00:00 New ERROR: Apache: The Network
2024-10-23 16:20:30 INFO: root User logged out
2024-10-23 18:45:10 New WARNING: CPU temperature exceeding

Log processing complete.
...............Completed..............

Use `grep`, `sed`, and `awk` with Bash Loops

Loops such as while, for, and until let you run advanced text processing tasks when using the grep,sed, and awk commands. Loops run continuous tasks when a condition returns true or false and are useful when reading multiple lines in files. Follow the steps below to use a while loop to display the contents of the log.txt file using the output from the grep,sed, and awk commands as the input.

Create a new loop-combined.sh script.
console
```
$ nano loop-combined.sh
```

Add the following contents to the loop-combined.sh file.

                            bash
                            
                        
#!/bin/bash

LOG="log.txt"

if [ -z "$1" ]; then
    echo "Enter a command option such as `ERROR` or `WARNING`. Example Usage: loop-combined.sh ERROR."
    exit 1
fi


LOG_LEVEL=$1
echo "Filtering '$LOG_LEVEL' logs..."

echo -e "Date       Time        Log Level   Message"
echo -e "------------------------------------------"


grep "$LOG_LEVEL" "$LOG" | 
sed "s/$LOG_LEVEL/${LOG_LEVEL^^}/g" |
awk '{print $1, $2, $3, $4, $5, $6}' |  

while read -r line; do
    echo "New entry: $line"
done

Save and close the file.

The above Bash script prompts a user to enter a command option to run the while loop that displays the contents of the log.txt file using the piped output from the grep,sed, and awk commands on each new line.

Run the script using Bash and set ERROR as a command option to test the while loop.

                            console
                            
$ bash loop-combined.sh ERROR

Output:

Filtering 'ERROR' logs...
Date       Time        Log Level   Message
------------------------------------------
New entry: 2024-10-25 09:00:01 ERROR: Unable to connect
New entry: 2024-10-24 10:45:55 ERROR: File not found:
New entry: 2024-10-23 15:00:00 ERROR: Apache: The Network

Run the script again and set WARNING as the command option.

                            console
                            
$ bash loop-combined.sh WARNING

Output:

Filtering 'WARNING' logs...
Date       Time        Log Level   Message
------------------------------------------
New entry: 2024-10-25 09:30:45 WARNING: The Disk space
New entry: 2024-10-24 11:15:00 WARNING: High memory usage
New entry: 2024-10-23 18:45:10 WARNING: CPU temperature exceeding

Conclusion

You have used the grep, sed, and awk commands in Bash to process text. Combining the commands enables advanced text processing and filtering when handling large data from inputs such as log files. Combine the search commands with loops, functions, and advanced conditional expressions in Bash to create efficient scripts that perform specific tasks such as log monitoring.

Tags:

Bash

Linux Commands

How to Process Text with Bash Using Grep, Sed, and Awk Commands

Introduction

Create a New Bash Script

Use the `grep` Command to Process Text File

Use the `awk` Command to Process Text File

Use `sed` Command to Process Text File

Combine `grep`, `sed`, and `awk` Commands

Use `grep`, `sed`, and `awk` with Conditional Bash Statements

Use `grep`, `sed`, and `awk` with Bash Loops

Conclusion

Comments

Products

Features

Solutions

Marketplace

Resources

Company

Tech Talks

Vultr Blogs

How to Process Text with Bash Using Grep, Sed, and Awk Commands

Introduction

Create a New Bash Script

Use the grep Command to Process Text File

Use the awk Command to Process Text File

Use sed Command to Process Text File

Combine grep, sed, and awk Commands

Use grep, sed, and awk with Conditional Bash Statements

Use grep, sed, and awk with Bash Loops

Conclusion

Comments

Tech Talks

Vultr Blogs

Use the `grep` Command to Process Text File

Use the `awk` Command to Process Text File

Use `sed` Command to Process Text File

Combine `grep`, `sed`, and `awk` Commands

Use `grep`, `sed`, and `awk` with Conditional Bash Statements

Use `grep`, `sed`, and `awk` with Bash Loops