BASH is a command line language, stands for Bourne Again SHell. It is used to get the computer to do different kind of tasks.

 

 

Shortcuts

 

ctrl + c – Stop the current process

ctrl + z – Get out from current process, resume with fg in the foreground or bg in the background

ctrl + d – Logout the current session, similar to exit (Just close the terminal)

ctrl + w – Erases one word in the current line

ctrl + u – Erases the whole line

ctrl + r – Type to bring up a recent command

ctrl + alt + F<Console number> – Switching console when one is stuck on a process, so CTRL + ALT+F2 will get you to a console #2, default is #7

!! – Repeats the last command

exit – Logout the current session

tab – auto complete a long word found from current folder file’s name

clear – clear the screen

 

 

All Installed Packages

 

dpkg -l – Listing all installed packages (applications)

dpkg -l | grep {app_name} – Listing status of specified app

 

 

App Installation

 

apt == apt-get

sudo apt update – Reflash all package inforamtion from all configured source.

sudo apt show – Show information about the given package(s), either installed or not yet.

sudo apt install {app_name} – Install an applications.

sudo apt autoremove – Remove dependencies from uninstalled apps that no longer used on the system.

Configure which version of installation (the same app/cmd) to be used as the default one on the command line:

sudo update-alternatives --config {app/CMD}

Uninstall a app/CMD installation:

sudo apt-get remove [--purge: delete config, ] {App_name}

 

 

File System

 

Absolute path: /home/user/Documents – start with a forward slash ( / )

Relative path:  user/Documents – not begin with a slash

~ (tilde): a shortcut for user’s home directory, so /home/kara/Documents == ~/Documents

. (dot): a reference to the current directory,  so Documents == ./Documents

.. (dotdot): a reference to the parent directory, go back 2 level  cd ../../

/ (slash): a reference to the root directory, go back to root cd /

 

ls [options] [location] – Directory LiSting

ls -ahl – Listing files in this folder,  All files (Hidden files showed), Human readable file size (KB, MB..), Long listing format (in rows, columns)

ls -lt  – Sorting the Formatted listing by time modification

ls lab[1-5] – Using range to match files lab1 ~ lab5, listing contents for lab1 ~ lab5

ls -lh /home/*/.bash_history – list all .bash_history file in every users home directory

ls [^a-k]* – look for any character which is not start with the following a-k

ls -ld testdir – look at directory

 

cd – Change Directory, change to user home directory

cd dir – Change directory to dir

cd / – Back to root

 

pwd [-LP] – Print (current) Working Directory

 

mkdir [-p: make parent dir | -v: verbose] {folder_name} –  Creating/Making a directory dir

 

file [path] –  To see what kind of file in this path (path to a file)

file /home/user/* – type of every file in a directory.

 

rm [-r: Recursive, remove directories | -f: no prompt for nonexistent files and arguments | -i: prompt before every removal] {file_name} – Remove files (*_files)

rm -rf folder_name – Directly remove a whole folder (without any prompts).

 

mv {source_folder/file1} {source_folder/file2} – Rename file1 to file2

mv {source_file} ../{target_file} – Move source file to (go back one level) target file

mv {source_file} folder_name/{target_file} – Move source file to folder_name directory

mv public_html/*.??g public_html/images/ – Move all files end with .__g (image) into another directory.

 

cp [-i: interactive option, prompted before a file is overwritten | -a: preserving the attributes of dir | -v: verbose, print info of what is done | -r copying recursively ] {source_file} {target_file} – Copy the contents of source file to target file

cp -r {source_dir} {target_dir} – Copy source to target (Automatically create target directory if not yet present)

 

touch [options] <filename> – modify the access and modification times on a file, or create file if not exist.

 

cat – Concatenate FILE(s) to standard output (print it)

cat {file_name} – Display contents of a file

cat > {file_name} – Places the standard input into the file

cat >> {file_name} – Append input to the end of the file

 

tac {path} – cat in reverse, print line of data in reverse order

 

less {file} – A tool to view/search a file

 

ln -s {file_name} {link_name} – Create a symbolic link that link to source file

 

lsof – which stands for list open files.

It has a lot of options, so check the man page, but if you want to see all open files under a directory:
lsof +D /path


 

 

File Permission

 

Linux permission string: drwxrwxrwx as  [Directory/Link] [Owner] [Group] [Others]

Meaning of those characters:

	File	Folder
r	can view the contents of file	ability to read the contents of the directory ls
w	can change the contents of the file	ability to write into the directory (create files and directories)
x	can execute or run the file	ability to enter that directory cd

Change mode (permission) for files:

chmod [-v: display what is done] {777: is 111 111 111 then all on} {file_name}

Octal value are 0~7, represent rwx as binary string:

0	000 as ___ : none
1	001 as __x : execute
2	010 as _w_ : write
3	011 as _wx : execute & write
4	read
5	read & execute
6	read & write
7	all

 

 

System Processes

 

top – Display all running processes

ps axu – Process, To display the currently working processes

kill [-9: force to kill] {78600: process IDs} – Kill processes by its ID

killall {proc} – Kill all the process named proc

sleep <number of second> – Wait for a given number of seconds and then quit

Display a tree of processes:

pstree -p     # with process id

 

Executing jobs

!{command} – re-run last used command

./<cmd> – Run a program in the frontground, . refer to current folder, then /<program> refer to this program

<cmd> & – Run a command program in the background, add & at the end of cmd

fg [n] – Brings the most recent job, or Nth job, to foreground

bg – Check current jobs

jobs – Lists currently running background jobs

$ sleep 20 &                 # sleep 20s in the bg
$ jobs                       # listing jobs in the bg
[1]+ Running sleep 30 &

$ bg
-bash: bg: job 1 already in background    # tell your job 1 is running

Xargs

xargs is a tool to run a particular command for every item in a list. Given a list of item of string, for every item of string, xargs can use the string as a parameter to execute command.

$ ls
12.tt 13.tt
$ ls | xargs -n1 -i mv {} {}f
$ ls
12.ttf  13.ttf

-n1 spcifies that the next command should be executed once for each item. -i indicates to use a replacement string in the next command (using {}).

 

 

System Info

 

hostname – show hostname

hostnamectl – show detail host (OS) info

sudo hostnamectl <hostname> – set hostname

date – Show the current date and time

cal – Show this month’s calendar

uptime – Show current uptime (time machine has been working)

w – Display a list of user who is online

last – show a list of last logged-in users

whoami – Print user name logged in as

finger [user_name: display info about this user] – Display a list of user, or with user’s name to display info about user

uname -a – Print system info, kernel info, OS info, …

cat /proc/cpuinfo – CPU information

cat /proc/meminfo – Memory information

Hardware Information

lshw          # list of hardware and settings
lspci         # list of Peripheral Component Interconnect
lsusb         # usb info 

man {command} – Show the manual for a specific command

man -k <search term> – Search a command using key word

df -h [path] – Show Disk space usage for File system (Not folder), in Human-readable text

du -sh <file | folder> – Listing Disk Usage, for all files & folders, or [-s, in Summary for a single folder/file], and Human-readable

free -h – Show memory and swap usage, in Human-readable text

 

 

Filtering

 

A filter, in the context of the Linux command line, is a program that accepts textual data and then transforms it in a particular way.

head [-number of lines to print] {file-path} – prints the first so many lines of it’s input

head -4 mysampledata.txt – first 4 lines

 

tail [-number of lines to print] {file-path} – Prints the last so many lines of it’s input

 

sort [-options] {file-path} – Sort the input by lines, by default options is A-Za-z

 

nl [-options] {file-path} – Number line, print lines with a # in front of the lines.

nl -s '. ' -w 10 mysampledata.txt – Print with indent and put ‘.’ after a #

 

wc [-l: # of line | -w: # of word] {file-path} – Word count, print newline, word, and byte counts for each file

 

cut [-options] {path} – Cut input string and only output what we want

cut -f 1 -d ' ' mysampledata.txt – Using separator ‘ ‘, only output the 1st column of a line

 

sed <expression> {path} – Stream Editor, do a search and replace on input data, expression using RegExp

sed 's/oranges/bananas/g' mysampledata.txt – Substitute “oranges” to “bananas” globally for all lines

sed -n '2p' file.txt – Print 2nd line

sed -n '10,33p' file.txt – Print line 10 up to line 33

sed -n '1p;3p' – Print line 1 and 3

 

uniq [options] {path} – Unique, remove duplicate lines from the data

 

 

Searching

 

 

Search for pattern in files, or folders, print the entire line for every line which contains a string of characters matching the given pattern:

grep [-i: ignore-case | -r: recursive, search all files in folders | -n: display total number of line] 'regular expression' file1, file2, ...

grep '[aeiou]{2,}' mysampledata.txt – identify any line with two or more vowels in a row

grep '2.+' mysampledata.txt – any line with a 2 on it which is not the end of the line

grep 'Gnome Display Manager' /etc/passwd – Search string includes spaces inside a file

grep '^linux' file.txt – string “linux” will match only if it occurs at the very beginning of a line

grep 'b\?right' file.txt – match both “bright” and “right”, The ? character is escaped with a backslash because we’re using basic regular expressions

grep -E 'b?right' file.txt – same regex using extended regular expression

grep -E 's+right' file.txt – match “sright” and “ssright”, but not “right”

 

Or using pipe |

Command | grep 'pattern' – Search pattern in the output of a command via pipe

 

dpkg -l | grep -i 'improved' – Search pattern of ‘improved’ in output of listing all packages

 

locate [-i: ignore-case | -p: ignore spaces, punctuation]{file_name/pattern} – Find all instances of file in the system

find

find <path> -type <d: directory| f: file> -name <filename> -size <+200M,200K> -exec <cmd\;> – Search folders or files in a target directory or below it with name

find / -type d -name myfolder – Search folders called “myfolder” from root

find . -type f -name myfile – Search files called “myfile” from current folder

find /home -size +200M – Search files with size over 200M

find ./public -type f -name '*.js' -exec ls -sh {} \; – Search .js files then execute cmd, ending the cmd with ;, escape the ; with \. '*.js' let find does the expansion internally. {} means the output of find.

 

which <command> – Locate a command/program. Return path name of the files that would be run by default for this app.

whereis <command> – Show possible locations of app (Only for command app)

 

Searches for all the named processes , that matches with the pattern and, by default, returns their ID:

pgrep {process pattern}

 

 

Redirection & Piping

 

Piping is used for manipulating the data flow, redirecting data between files and programs in some useful ways. There are 3 data streams in every program:

1. STDIN (0): Standard input
2. STDOUT (1): Standard output
3. STDERR (2): Standard error

>, Used to let programs output (whatever it sends to STDOUT) to be saved in a file instead of printed to the screen. If the target file exist, its content will be cleared to new content.

{CMD} > {file}
ls > content-file.txt

>>, Append content to file:

{CMD} >> {file}
ls >> existing-file.txt

<, Feeding data into a program via the STDIN stream:

{CMD} < {file}
wc -l < existing-file.txt

Combining < and >:

wc -l < input-file.txt > output-file         # outuput result of number of words of input file into another output file

2>, Redirecting STDERR (No. 2 of stream, error message), only error message is redirected:

ls -l exist.c not-exist.c 2> errors.txt
# error "ls: cannot access not-exist.c: No such file or directory"
# will be output to file

&, Identify the redirection to a stream, &1 redirect to STDOUT, &2 redirect to STDERR

ls -l dir.dir pp > output 2>&1               # no space after "2>"
cat output
# ls: cannot access 'pp': No such file or directory
# -rw-rw-r-- 1 cc cc 9 Dec  1 12:48 dir.dir

# Can also direct error first
# then append output to the file
ls -l dir.dir pp 2> output 1>> output

|, as Piping, is used for sending data from one program to another. Feed the output from the program on the left as the input to the program on the right.

ls -alh | grep <keyword>              # Find something from output of ls

Combining with redirection:

# take first 3 lines of "ls"
# then last lines
# then put result to file
ls -alh | head -3 | tail -1 > file

# Can scroll the result
ls -l /etc | less

 

 

Compression/Backup

zip

Compress a file with zip

zip 
     [-v: verbosely, show detail
      -9: compress better 
      -1: compress fast
      -r: recursively including files inside folders
      -x: exclude the following files
     ]
     {xxx.zip} {file1 file2 ...} -x {folder/*}

Compress all files/folders in this folder and create new.zip (including files inside folders)

zip -v9r new.zip *

# Exclude all files in "subfolder1" and "subfolder2", need ""
zip -v9r new.zip folder -x "folder/subfolder1/*" "folder/subfolder2/*"

unzip

Listing files inside a .zip file

unzip [-l: list files, not unzip it | -v: list verbosely] {xxx.zip}

Extract all files to a target folder (create it if it is not exist)

unzip {xxx.zip} -d {folder_name}

Extract all files to the current folder (can exclude some files)

unzip {xxx.zip} [-x: exclude files that follows] {files_exclude}

tar

tar stand for Tape ARchive.

[-c] Create tar file named file.tar containing file1, 2, 3

tar -cvf {file.tar} {file1 file2 file3}

[-t] Listing all files inside file.tar

tar -tvf {file.tar}

[-x] Extract the files from file.tar

tar -xvf {file.tar}
tar -xvf {file.tar} -C /target/dir    # extract to /target/dir
tar -xvf {file.tar} -C dir            # extract to current subfolder "dir", "dir" need to be exist

Create a tar with Gzip compression

tar -czvf {file.tar.gz} {files}

Extract a tar using Gzip

tar -xjvf {file.tar.gz}

Create tar with Bzip2 compression

tar -cjvf {file.tar.bz2} {files}

Extract a tar using Bzip2

tar -xjvf {file.tar.bz2} {files}

gzip

Compresses file and renames it to file.gz

gzip {files}

Decompresses file.gz back to file

gzip -d {file.gz}

rsync

Sync files on two directories on the same login section

rsync -uvr {source_dir} {destination_dir}

 

 

Service

 

service --status-all – listing all services (+ is active, – is inactive)

service <service_name> status

service <service_name> start

service <service_name> stop

service <service_name> restart

Systemctl

systemctl list-units | grep running – listing all running services

systemctl status <service_name>

systemctl start <service_name>

systemctl stop <service_name>

systemctl restart <service_name>

Journalctl

journalctl – See all log of the OS

journalctl -u <service_name>.service – See full log of a unit (a service unit) status

journalctl -u <service_name> – can do without .service

journalctl -u <service_name> -b – See full log of service status of current boot

journalctl --disk-usage
# Archived and active journals take up 136.0M on disk.

--vacuum-size=200M Reduce disk usage below specified size
--vacuum-files=INT Leave only the specified number of journal files
--vacuum-time=30d Remove journal files older than specified time

journalctl --vacuum-size=200M
# Archived and active journals take up 136.0M on disk.

 

 

Network

 

ip addr – Show IP info

ssh [-i:use identity such as private_key] [private_key_path] login_name@abc.edu -p <port> – Login to a remote system using ssh

sftp -r {name@linux.cs.pdx.edu:Folder_name/*} ~/sub_folder_in_home/.} – Download files or folders from remote linux to your machine (perform in self machine)

ping {host} – Ping host and output results

nslookup –option1 –option2 {host-to-find} {use-this-dns-server} – Name Server Look Up, can query IP address of hosts (Web server, DNS server, Mail server, … )

dig {domain} – Get DNS information for domain

dig -x {host} – Reverse lookup host

scp (Secure Copy Protocol)

A quick, easy and secure way to copy files between different machines across ssh (linux) connection: local -> remote, remote -> local, remote -> remote

man scp
# user manual

scp ./file.jpg username@12.34.56.78:/home/me/file.jpg      # local to remote
scp username@12.34.56.78:/home/me/file.jpg ./file.jpg      # remote to local

scp jane@host.example.com:/home/jane/table.csv pete@host2.example.com:/home/pete/
# remote file to remote folder

scp ./folder username@12.34.56.78:/home/me/folder      # copy folder to folder

wget

The non-interactive network downloader, download a file:

wget {file_URL}
wget -c {file}      # Continue a stopped download

Firewall

Let firewall to allow using a port

sudo ufw [delete] allow|deny|reject|limit {port/protocol}
sudo ufw status                    # See active rules
sudo ufw allow 9091/tcp            # Add allow rule on tcp port 9091
sudo ufw delete allow 9091/tcp     # Delete the "allow rule of port 9001 on tcp

Port

Show a list of services running on the system on tcp and udp ports

• -l = only services which are listening on some port
• -n = show port number, don’t try to resolve the service name
• -t = tcp ports
• -u = udp ports
• -p = name of the program

sudo netstat -lntup

Find Which program using Which port:

sudo netstat -lntup | grep {port | program_name}

 

 

Task Scheduling – Cron

 

cron stands for Command Run ON. It is a mechanism that let the system to run certain commands at every certain time frame. For example, cron lets the system to run a script automatically every hour.

crontab manges a text file (a schedule) that can control when and what commands are scheduled to run. Each user have their own crontab and within the file, each line represents a schedule for a particular command.

crontab -l     # To view a list of current scheduled tasks
crontab -e     # To edit the schedule file

The actual files store on folder: /var/spool/cron/crontabs/<username>

The cron log files stare on: /var/log/cron or /var/log/syslog

For a line of schedule, time frame format:

*	*	*	*	*	cmd
Minutes (0 – 59)	Hours (0 – 23)	Day of Month (1 – 31)	Months (1 – 12)	Day of week (0 – 7) (0 and 7 are Sunday)	cmd to run

Execute script.bash:

* * * * * /home/script.sh             # every minute
30 3 * * 4 /home/script.sh            # every Thursday at 3:30 AM
* 2/5 * * * /home/script.sh    # 2/5 means starting at 2 then every 5 (2,7,12,17,22...)

Check whether cron is running (cron need to be running for cmd to be executed):

ps aux | grep cron

# if not then start it
sudo systemctl start cron

 

 

Tools

 

pwd – Output the current terminal path

echo $VARIABLE – Output value of a variable

which {command} – Locate path of a binary executable (a command)

date [+FORMAT] – Print out date, + followed by a format, +%m/%d/%y print out 12/01/18

awk

It is a pattern scanning and processing language.

awk '{print $2}' data.txt – Print 2nd column of each row in a file

awk '{ sum += $1 }; END { print sum }' data.txt – Sum a number 1st column of each row, then print it out

awk '$3 > 10 {print $1 " - " $2 " : " $3}' data.txt – Perform conditional check, if value 3rd column greater than 10