AWK is one of the most powerful command in Linux. You can manage data and generate reports using the awk command. It also allows us to use logical operation, variables, print functions and many more. AWK stands for “Aho, Weinberger and Kernighan” and is mostly used for pattern scanning and processing. It searches one or more files to see if they contain lines that matches the specified pattern and then perform associated actions. It reads from a file or from its standard input and outputs to its standard output. For each line, it matches with given pattern in the given order, if matches perform the corresponding action.
Features
• Views a text file as records and fields
• It has variables, conditionals and loops
• Has arithmetic and string operators
• Can generate formatted reports
• Read and edit text from a string or file
In this tutorial, we will take a look into the AWK Linux command with examples and see what it can do.
Basix Syntax of AWK
The basic syntax of AWK command is shown below:
awk options program input-file
A brief explanation of each option is shown below:
• -F fs : Used to specify a file separator.
• -f file : Used to specify a file that contains awk script.
• -v var=value : USed to declare a variable.
We will use the following text file as the input file for all examples in this article:
cat > contents.txt hitesh engineer sales 30000 jayesh director account 25000 vyom manager purchase 20000 bhavesh engineer sales 30000 rajesh directory sales 40000 niraj clerk account 20000 jay peon purchase 23000 deep clerk sales 20000
Now, let’s check the content of the file named ‘contents.txt’ with the AWK command:
awk '{print}' contents.txt
This will print the content of the file as shown below:
hitesh engineer sales 30000 jayesh director account 25000 vyom manager purchase 20000 bhavesh engineer sales 30000 rajesh directory sales 40000 niraj clerk account 20000 jay peon purchase 23000 deep clerk sales 20000
In the above example, no pattern is given so it will print the whole file.
Now, print all lines which match with the pattern “sales”:
awk '/sales/ {print}' contents.txt
This will print all lines that contains the word “sales” as shown below:
hitesh engineer sales 30000 bhavesh engineer sales 30000 rajesh directory sales 40000 deep clerk sales 20000
Variables in AWK
AWK comes with some built-in variables that are used to break a line of text into individual words or pieces called fields. Some of them are shown below:
• $0 : Used for the entire line.
• $1 : Used for the first field.
• $2 : Used for the second field.
• $n : Used for the nth field.
• NR : Used to specify the total number of current records.
• NF : Used to specify the total number of fields in the record.
• FS : It contains the field separator character and is used to divide fields on the input line.
• RS : It stores the current record separator character.
• OFS : It stores the output field separator and is used to separates the fields when Awk prints them.
• ORS : It stores the output record separator and is used to separates the output lines when Awk prints them.
Now, print the field no 1 and 3 from the file contents.txt use the following syntax:
awk '{print $1,$3}' contents.txt
You should see only first and third field of the file contents.txt:
hitesh sales jayesh account vyom purchase bhavesh sales rajesh sales niraj account jay purchase deep sales
You can use NR with the AWK command to print all the lines along with the line number:
awk '{print NR,$0}' contents.txt
Output:
1 hitesh engineer sales 30000 2 jayesh director account 25000 3 vyom manager purchase 20000 4 bhavesh engineer sales 30000 5 rajesh directory sales 40000 6 niraj clerk account 20000 7 jay peon purchase 23000 8 deep clerk sales 20000
You can use NF to display the last field and $1 to display the first field:
awk '{print $1,$NF}' contents.txt
This will print the first and last field of the file contents.txt:
hitesh 30000 jayesh 25000 vyom 20000 bhavesh 30000 rajesh 40000 niraj 20000 jay 23000 deep 20000
To print the line number from 2 to 5 use the NR variable as shown below:
awk 'NR==2, NR==5 {print NR,$0}' contents.txt
Output:
2 jayesh director account 25000 3 vyom manager purchase 20000 4 bhavesh engineer sales 30000 5 rajesh directory sales 40000
To count the number of the lines in file contents.txt using the NR:
awk 'END { print NR } ' contents.txt
You should see the following output:
8
BEGIN and END Blocks
There are also optional BEGIN and END blocks that can contain commands to execute before and after the file processing, respectively. The BEGIN block is used to perform actions before records are processed while END block is used to perform actions after records are processed.
The basic syntax to use BEGIN and END blocks with AWK command is shown below:
awk 'BEGIN { action; } /search/ { action; } END { action; }' input_file
You can print information about the fields you are printing with BEGIN and END blocks.
The following example will print the message before and after processing the second field of each record in file contents.txt:
awk 'BEGIN { print "Start Process." }; { print $2 }; END { print "End Process." }' contents.txt
Output:
Start Process. engineer director manager engineer directory clerk peon clerk End Process.
You can also use the BEGIN and END blocks to transform the data from the file and convert into a table. The following example will convert the file /etc/passwd in to table:
awk 'BEGIN { FS=":"; print "UserttUIDttGIDttHomettShelln--------------"; } {print $1,"tt",$3,"tt",$4,"tt",$6,"tt",$7;} END { print "---------nFile Complete" }' /etc/passwd
Output:
User UID GID Home Shell -------------- root 0 0 /root /bin/bash daemon 1 1 /usr/sbin /usr/sbin/nologin bin 2 2 /bin /usr/sbin/nologin sys 3 3 /dev /usr/sbin/nologin sync 4 65534 /bin /bin/sync games 5 60 /usr/games /usr/sbin/nologin man 6 12 /var/cache/man /usr/sbin/nologin lp 7 7 /var/spool/lpd /usr/sbin/nologin mail 8 8 /var/mail /usr/sbin/nologin news 9 9 /var/spool/news /usr/sbin/nologin uucp 10 10 /var/spool/uucp /usr/sbin/nologin proxy 13 13 /bin /usr/sbin/nologin www-data 33 33 /var/www /usr/sbin/nologin --------- File Complete
Conditional Search
The AWK command also supports several conditional statements including, if, while loop, for loop and many more. This will helps you to fetch lines that matches a specific condition.
The following example will use “if” condition to print all lines that contain “sales” in the third field:
awk '{ if ($3 ~ /sales/) print}' contents.txt
Output:
hitesh engineer sales 30000 bhavesh engineer sales 30000 rajesh directory sales 40000 deep clerk sales 20000
The following example will use the “for” loop to prints the first three fields of each record, one per line.
awk '{ for (i = 1; i <= 3; i++) print $i }' contents.txt
Output:
hitesh engineer sales jayesh director account vyom manager purchase bhavesh engineer sales rajesh directory sales niraj clerk account jay peon purchase deep clerk sales
The following example will use the “while” loop to prints the first two fields of each record, one per line.
awk '{ i = 1; while ( i <= 2 ) { print $i i++ } }' contents.txt
Output:
hitesh1 engineer2 jayesh1 director2 vyom1 manager2 bhavesh1 engineer2 rajesh1 directory2 niraj1 clerk2 jay1 peon2 deep1 clerk2
Processing Output from Other Commands
You can also use AWK command to parse the output from the other command instead of specifying a filename. The “ip a” command print the information about system IP, Mac address, and other network-related information as shown below:
ip a s wlan0
Output:
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 4c:bb:58:9c:f5:55 brd ff:ff:ff:ff:ff:ff
inet 172.20.10.3/28 brd 172.20.10.15 scope global wlan0
valid_lft forever preferred_lft forever
inet6 2401:4900:1d71:ef26:8846:95b2:4ca8:aa7d/64 scope global temporary dynamic
valid_lft 600574sec preferred_lft 81574sec
inet6 2401:4900:1d71:ef26:4ebb:58ff:fe9c:f555/64 scope global dynamic
valid_lft forever preferred_lft forever
inet6 fe80::4ebb:58ff:fe9c:f555/64 scope link
valid_lft forever preferred_lft forever
Now, you can use AWK command to print only the IP address of the system as shown below:
ip a s wlan0 | awk -F '[/ ]+' '/inet / {print $3}
You should see the following output:
172.20.10.3
AWK Command Advance Examples
Example 1: The AWK command allows you to print the lines with some specified count of characters. For example, print lines with more than 27 characters, use the below command:
awk 'length($0) > 27' contents.txt
Output:
jayesh director account 25000 bhavesh engineer sales 30000 rajesh directory sales 40000
Example 2: Check the cube of the given number
To print the cube of the given number till 5, run the following command:
awk 'BEGIN { for(i=1; i<=5; i++) print "Cube of",i,"is",i*i*i; }'
Output:
Cube of 1 is 1 Cube of 2 is 8 Cube of 3 is 27 Cube of 4 is 64 Cube of 5 is 125
Example 3: Count the number of lines in the specified file
You can check the number of lines in the specified file and print it using the following command:
awk 'END { print NR }' contents.txt
Output:
8
Example 4: Find the longest line in the given file and print the character
You can find the longest line in the given file and print the character of that line using the following command:
awk '{ if (length($0) > max) max = length($0) } END { print max }' contents.txt
Output
29
Example 5: Sort the first column of a given file
To sort and print the first column of the file contents.txt, run the following command:
awk -F: '{ print $1 }' contents.txt | sort
Output:
bhavesh engineer sales 30000 deep clerk sales 20000 hitesh engineer sales 30000 jayesh director account 25000 jay peon purchase 23000 niraj clerk account 20000 rajesh directory sales 40000 vyom manager purchase 20000
Example 6: Print the even-numbered lines
To print only even-numbered lines in the file contents.txt, run the following command:
awk 'NR % 2 == 0' contents.txt
Output:
jayesh director account 25000 bhavesh engineer sales 30000 niraj clerk account 20000 deep clerk sales 20000
Example 7: Change the field separator
You can change the filed separator from space to | and print it with the following command:
awk 'BEGIN{OFS="|"}{print $1,$2,$3,$4}' contents.txt
Output:
hitesh|engineer|sales|30000 jayesh|director|account|25000 vyom|manager|purchase|20000 bhavesh|engineer|sales|30000 rajesh|directory|sales|40000 niraj|clerk|account|20000 jay|peon|purchase|23000 deep|clerk|sales|20000
Conclusion
In this tutorial, you learned how to use the AWK command to match the specified patterns and then perform the associated actions. I hope you have a clear idea of how to use AWK command to manipulate, format, and selectively print text files.
