How to Process Text with Bash Using Grep, Sed, and Awk Commands

Introduction

Text processing in Bash using the grep, sed, and awk commands lets you filter and manipulate input to perform specific tasks, such as file content processing. You can combine the text processing commands with loops, conditional statements, and expressions for advanced processing.

This article explains how to process text in Bash using the grep, sed, and awk commands.

Create a New Bash Script

Follow the steps below to create a new Bash script that loads an input file with multiple lines of content to a variable for further text processing.

1. Create a new log.txt file to store multiple lines of text.

   console Copy

   $ nano log.txt
   

2. Add the following contents to the log.txt file.

   text Copy

   2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
   2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
   2024-10-25 09:30:45 WARNING: The Disk space running low.
   2024-10-24 10:15:12 INFO: root User logged in.
   2024-10-24 10:45:55 ERROR: File not found: www.yml.
   2024-10-24 11:15:00 WARNING: High memory usage detected.
   2024-10-23 14:25:01 INFO: A scheduled backup is complete.
   2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.
   2024-10-23 16:20:30 INFO: root User logged out successfully.
   2024-10-23 18:45:10 WARNING: CPU temperature exceeding normal level.
   

   Save and close the file.

3. Create a new processing.sh Bash script that inputs the log.txt file as a variable.

   console Copy

   $ nano processing.sh
   

4. Add the following contents to the processing.sh file.

   bash Copy

   #!/bin/bash
   
   LOG="log.txt"
   

   Save and close the file.

   In the above script, the LOG variable takes the log.txt as an input. You'll process the variable value using the grep, sed, and awk commands in the next sections.

Use the grep Command to Process Text File

The grep command searches specific text patterns in files, such as the log.txt file you created, and outputs the results for additional processing in a Bash script. Follow the steps below to use the grep command and display the results.

1. Open the processing.sh script you created earlier.

   console Copy

   $ nano processing.sh
   

2. Add the following command at the end of the file.

   bash Copy

   grep -i "ERROR" "$LOG"
   

   Save and close the file.

   The above grep command searches for the term ERROR in the log.txt file and displays all matching results.

3. Run the script using Bash.

   console Copy

   $ bash processing.sh
   

   Output:

   2024-10-25 09:00:01 ERROR: Unable to connect to the database on port 3306.
   2024-10-24 10:45:55 ERROR: File not found: www.yml.
   2024-10-23 15:00:00 ERROR: Apache: The Network connection lost.

Use the awk Command to Process Text File

The awk command analyzes structured data and searches for specific patterns in a file. You can use the awk command to customize file contents, split lines into fields or columns for processing, and match specific patterns. Follow the steps below to process the log.txt file columns using the awk command.

1. Open the processing.sh Bash script.

   console Copy

   $ nano processing.sh
   

2. Add the following contents to the file.

   bash Copy

   awk '{print $1, $3}' "$LOG"
   

   Save and close the file.

   The above awk command prints the first and third columns in the log.txt file to display the date and error type.

3. Run the script using Bash.

   console Copy

   $ bash processing.sh
   

   Output:

   2024-10-25 INFO:
   2024-10-25 ERROR:
   2024-10-25 WARNING:
   2024-10-24 INFO:
   2024-10-24 ERROR:
   2024-10-24 WARNING:
   2024-10-23 INFO:
   2024-10-23 ERROR:
   2024-10-23 INFO:
   2024-10-23 WARNING:

Use sed Command to Process Text File

Stream Editor (sed) is a text processing command that processes and transforms text into a file. The sed command searches, modifies, and replaces contents in standard input or a specific file using Bash. Follow the steps below to use the sed command to search and replace contents in the log.txt file.

1. Open the processing.sh Bash script.

   console Copy

   $ nano processing.sh
   

2. Add the following command at the end of the file.

   bash Copy

   sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG"
   

   Save and close the file.

   The above sed command replaces ERROR with New ERROR and WARNING with New WARNING values in the log.txt file.

3. Run the script using Bash.

   console Copy

   $ bash processing.sh
   

   Output:

   2024-10-25 08:15:23 INFO: Apache Web Server Application started successfully.
   2024-10-25 09:00:01 New ERROR: Unable to connect to the database on port 3306.
   2024-10-25 09:30:45 New WARNING: The Disk space running low.
   2024-10-24 10:15:12 INFO: root User logged in.
   2024-10-24 10:45:55 New ERROR: File not found: www.yml.
   2024-10-24 11:15:00 New WARNING: High memory usage detected.
   2024-10-23 14:25:01 INFO: A scheduled backup is complete.
   2024-10-23 15:00:00 New ERROR: Apache: The Network connection lost.
   2024-10-23 16:20:30 INFO: root User logged out successfully.
   2024-10-23 18:45:10 New WARNING: CPU temperature exceeding normal level.

Combine grep, sed, and awk Commands

You can combine grep, sed, and awk commands to analyze and manipulate large inputs and perform advanced text processing tasks using Bash. Follow the steps below to combine the grep, sed, and awk commands and process the log.txt file.

1. Create a new combine.sh Bash script.

   console Copy

   $ nano combine.sh
   

2. Add the following contents to the file.

   bash Copy

   #!/bin/bash
   
   LOG="log.txt"
   
   echo "Processing the log file: $LOG"
   
   echo "Date       Time        Log Level   Message"
   echo "------------------------------------------"
   
   grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'
   
   echo -e "\nLog processing complete."
   

   Save and close the file.

   In the above Bash script, the combined grep, sed, and awk commands use pipes | to redirect the output of each command and display the modified contents of the log.txt file in a well-structured format that includes all log entries.

3. Run the script using Bash.

   console Copy

   $ bash combine.sh
   

   Output:

   Processing the log file: log.txt
   Date       Time        Log Level   Message
   ------------------------------------------
   2024-10-25 08:15:23 INFO: Apache Web Server Application
   2024-10-25 09:00:01 New ERROR: Unable to connect
   2024-10-25 09:30:45 New WARNING: The Disk space
   2024-10-24 10:15:12 INFO: root User logged in.
   2024-10-24 10:45:55 New ERROR: File not found:
   2024-10-24 11:15:00 New WARNING: High memory usage
   2024-10-23 14:25:01 INFO: A scheduled backup is
   2024-10-23 15:00:00 New ERROR: Apache: The Network
   2024-10-23 16:20:30 INFO: root User logged out
   2024-10-23 18:45:10 New WARNING: CPU temperature exceeding
   
   Log processing complete

Use grep, sed, and awk with Conditional Bash Statements

You can use grep, sed, and awk commands to process text further with if-else, case, and if-then conditional statements to enhance output readability and handle errors. Follow the steps below to use the conditional statement to process the log.txt file using the search commands in a Bash script.

1. Create a new conditional-processing.sh Bash script.

   console Copy

   $ nano conditional-processing.sh
   

2. Add the following contents to the conditional-processing.sh file.

   bash Copy

   #!/bin/bash
   
   LOG="log.txt"
   
   if [[ -f "$LOG" ]]; then
       echo "Processing the log file: $LOG"
   
       if grep -qE 'ERROR|WARNING' "$LOG"; then
   
           echo "Date       Time        Log Level   Message"
           echo "------------------------------------------"
   
           grep -E 'ERROR|WARNING' "$LOG" | sed -e 's/ERROR/New ERROR/gI' -e 's/WARNING/New WARNING/gI' "$LOG" | awk '{print $1, $2, $3, $4, $5, $6, $7}'
   
           echo -e "\nLog processing complete."
       else
           echo "No ERRORs or WARNINGs found in the log file."
       fi
   else
       echo "The Log file: $LOG is not available"
   fi
   
   echo "...............Completed.............."
   

   Save and close the file.

   In the above file:

   • The outer if-else conditional statement validates the input variable.
   • The inner if-else conditional statement runs a grep condition to find the ERROR or WARNING fields in the file. The combined grep, sed, and awk commands run when the condition evaluates to true and perform additional processing to display the modified output.

3. Run the script using Bash.

   console Copy

   $ bash conditional-processing.sh
   

   Output:

   Processing the log file: log.txt
   Date       Time        Log Level   Message
   ------------------------------------------
   2024-10-25 08:15:23 INFO: Apache Web Server Application
   2024-10-25 09:00:01 New ERROR: Unable to connect
   2024-10-25 09:30:45 New WARNING: The Disk space
   2024-10-24 10:15:12 INFO: root User logged in.
   2024-10-24 10:45:55 New ERROR: File not found:
   2024-10-24 11:15:00 New WARNING: High memory usage
   2024-10-23 14:25:01 INFO: A scheduled backup is
   2024-10-23 15:00:00 New ERROR: Apache: The Network
   2024-10-23 16:20:30 INFO: root User logged out
   2024-10-23 18:45:10 New WARNING: CPU temperature exceeding
   
   Log processing complete.
   ...............Completed..............

Use grep, sed, and awk with Bash Loops

Loops such as while, for, and until let you run advanced text processing tasks when using the grep,sed, and awk commands. Loops run continuous tasks when a condition returns true or false and are useful when reading multiple lines in files. Follow the steps below to use a while loop to display the contents of the log.txt file using the output from the grep,sed, and awk commands as the input.

1. Create a new loop-combined.sh script.

   console Copy

   $ nano loop-combined.sh
   

2. Add the following contents to the loop-combined.sh file.

   bash Copy

   #!/bin/bash
   
   LOG="log.txt"
   
   if [ -z "$1" ]; then
       echo "Enter a command option such as `ERROR` or `WARNING`. Example Usage: loop-combined.sh ERROR."
       exit 1
   fi
   
   
   LOG_LEVEL=$1
   echo "Filtering '$LOG_LEVEL' logs..."
   
   echo -e "Date       Time        Log Level   Message"
   echo -e "------------------------------------------"
   
   
   grep "$LOG_LEVEL" "$LOG" | 
   sed "s/$LOG_LEVEL/${LOG_LEVEL^^}/g" |
   awk '{print $1, $2, $3, $4, $5, $6}' |  
   
   while read -r line; do
       echo "New entry: $line"
   done
   

   Save and close the file.

   The above Bash script prompts a user to enter a command option to run the while loop that displays the contents of the log.txt file using the piped output from the grep,sed, and awk commands on each new line.

3. Run the script using Bash and set ERROR as a command option to test the while loop.

   console Copy

   $ bash loop-combined.sh ERROR
   

   Output:

   Filtering 'ERROR' logs...
   Date       Time        Log Level   Message
   ------------------------------------------
   New entry: 2024-10-25 09:00:01 ERROR: Unable to connect
   New entry: 2024-10-24 10:45:55 ERROR: File not found:
   New entry: 2024-10-23 15:00:00 ERROR: Apache: The Network

4. Run the script again and set WARNING as the command option.

   console Copy

   $ bash loop-combined.sh WARNING
   

   Output:

   Filtering 'WARNING' logs...
   Date       Time        Log Level   Message
   ------------------------------------------
   New entry: 2024-10-25 09:30:45 WARNING: The Disk space
   New entry: 2024-10-24 11:15:00 WARNING: High memory usage
   New entry: 2024-10-23 18:45:10 WARNING: CPU temperature exceeding

Conclusion

You have used the grep, sed, and awk commands in Bash to process text. Combining the commands enables advanced text processing and filtering when handling large data from inputs such as log files. Combine the search commands with loops, functions, and advanced conditional expressions in Bash to create efficient scripts that perform specific tasks such as log monitoring.