linux shell & bash

shell & bash

shell指允许用户通过文本操作计算机的程序。

interactive shell：从是否通过标准输入输出与用户进行交互的角度分为交互式shell（interactive）和非交互式shell（non-interactive）。

login shell：从是否以一个指定用户及其环境进入shell角度分为登录式shell和非登录式shell，登录式shell会额外source /etc/profile,/etc/profile.d,~/.profile,~/.bash_login等profile相关脚本。

变量$-包含了当前shell的选项参数，一个字母代表一种选项。部分如下：

• H histexpand
• m monitor
• h hashall
• B braceexpand
• i interactive

判断是否是交互式shell：根据$-输出是否包含字母i来判断。

case "$-" in
*i*)    echo This shell is interactive ;;
*)      echo This shell is not interactive ;;
esac

if [[ "$-" == *i* ]]; then
    echo This shell is interactive
fi

当前使用的shell类型（bash, cshell...）：echo $0；注意$SHELL是默认shell，并非当前shell，也就说如果在echo $SHELL不是当前shell的类型。

uniq只会删除连续重复的行，也就说uniq不在输入的所有行上做distinct，只是在剪掉连续出现的行。想要对在输入的所有行上做distinct，可以考虑先sort，然后uniq，但需要注意这样会改变输入中行字符串出现的顺序。

iostat查看系统io，iftop查看网络流的情况。

ssh连接在很短的不活跃时间后自动断开，如需延长这个闲置时间，通过在ssh配置文件中修改ServerAliveInterval ServerAliveCountMax两个量实现，前者指给远程发送心跳包的时间间隔，后者指连续发送心跳包的最大次数。
~/.ssh/config或/etc/ssh/ssh_config中添加两行：

# 每60秒发送一个心跳包
ServerAliveInterval 60
# 最多连续发送3次
ServerAliveCountMax 3

ssh命令行参数提供密码：

sshpass -p PASSWORD ssh -p PORT USER@HOST

sudo -E 保留当前环境变量执行命令

sudo -i 登入root用户shell

查看当前使用的shell：
echo $SHELL

查看系统可用的shel：
cat /etc/shells

更改用户默认的shell：
修改/etc/passwd文件对应用户最后的shell配置。

bash相关配置文件

登录bash shell相关（ubuntu）：/etc/profile -> /etc/bash.bashrc（在/etc/profile中被source） -> /etc/profile.d/ -> ~/.bash_profile -> ~/.bash_login -> .bashrc

交互式bash shell： /etc/bash.bashrc -> .bashrc

如果shell不是bash，/etc/profile中不会source /etc/bash.bashrc，如启动intellij idea IDE时。

shell script脚本

shell脚本（shell script）是指由一系列shell命令组成的脚本程序文件。shell script不是interative shell，也不是login shell。

小心ls、grep等带有颜色的输出，输出字符串看起来一样的，在做uni时被视为不一样，则小心其中有个输出可能带有颜色字符，用cat -A确认。

for, while中计数

counter=0
for/while...
  counter=$((counter+1))

定义环境变量：
export name=value
取消定义：

交换文件中以逗号分割的前两列的纯bash脚本实现：

while IFS=, read x1 x2 remainning
do
   echo ${x2},${x1},$remainning
done < FILE

$PS1中的特殊符号：
\u：用户名
\h：短主机名
\W：working directory(basename)
\s: shell名字（bash、sh）
\v: shell version
\d: 英文日期
\D{fmt}：日期格式（man strftime）
\n: 换行（多行提示符相关）
\w: full path of working directory
\H: full hostname
以及颜色标签[\exxx]

如果不存在则创建目录
[ -d /path/to/dir ] || mkdir -p /path/to/dir

io重定向 io redirect:
stdout stderr输出到同一个文件：
ls -l /usr > out 2>&1 或者 ls -l /usr &> out。
stdout stderr输出到单独文件 ls -l /usr > stdout 2> stderr。
忽略stderr输出 ls -l /usr > stdout 2>/dev/null

date

将epoc seconds格式化输出 date -d @1305741420。注意，需带@前缀。 若想将millis（毫秒）格式化输出则先去掉数值后3位：echo 1305741420 | cut -b1-10 | xargs -I {} date -d @{}

xargs

将xargs的输入（通常是前面管道的输出）作为待执行命令最后一个参数，如果需要拼接固定（或其他形式格式化），可以指定替换串形式，如：echo 1305741420 | xargs -I {} | date -d @{}，则指定{}为替换串，会将命令形式date -d @{}中的{}替换为xargs接收到的输入。

awk

行处理，根据列是否满足条件输出、或做进一步处理，这样的需求完全可以考虑使用awk。
忽略#开头的行 awk '$1 ~ /^[^#]/'
忽略#开头的行，输出第三列（默认以空格分隔）以<开头的行awk '$1 ~ /^[^#]/ && $3 ~ /^</'
设置分隔符为逗号,：awk -F, awk 'BEGIN{FS=,}'

sed

comm

找出已排序的两个文件中的相同行、不同行，差集、交集、并集皆可实现。
输出3列，分别是：FILE1中独有行、FILE2中独有行、共有行。
comm接受-参数，表示不输出第列，可以是多个列
comm -12仅输出两文件共有行
comm -23仅输出FILE1独有行
comm -13仅输出FILE2独有行

join

合并已排序的两个文件中的行。

top

• -b 输出到文件
• -p 指定pid, -pN1,N2,N3
• -O 打印可以输出的字段，相当于对-o的帮助信息
• -n 迭代次数 -n num
• -O 排序字段 [+|-]fieldName

lsblk

分辨ssd/hdd：lsblk -d -o name,rota 输出0代表ssd，1代表hdd。
lsblk -o name可展示出物理盘的从属虚拟硬盘（slave disk），如物理盘虚拟出的lvm盘。

迭代文件中的行

cat 默认会以空格分割字符串，因此如果文件中行包含空格，则cat不能迭代成功，解决方案是在cat文件之前把IFS变量设为换行符，然后用cat迭代，完成后应该把IFS重置为之前的值
对稍大一点（上百M）的文件，速度相当慢。

IFS=$'\n'
for line in `cat file`
do
    ...
done

控制流程：

if ... ; then
...
elif ... ; then
...
else
...
fi

function

#定义函数，关键字function是可不写
function fun(){
    #获取参数
    echo $1 , $2;
    #返回值，可选
    return $(($1 + $2))
}
#调用函数，需要先定义
fun 5 7;
#获取函数返回值
s = $?

输出错误信息到stderr： echo .... >&2

字符串

判断字符串是否包含给定字符串：

if [[ <string> == *"sub-string"* ]]; then
    ...
fi

下载

wget
根据url下载小文件 wget -nd URL。选项-nd指明不创建目录，否则程序将根据url创建层级目录；选项-b指明后台下载；选项-d打印处理过程；-v指明更详细输出。

定时任务

编写crontab文件

crontab <crontab-file>
crontabl -l # 列出任务

linux文件权限rwx

文件：

目录：

连接其他电脑

ssh ...
#在ssh中
export DISPLAY=:0
[sudo] notify-send <msg>  #右上角显示提示信息
[sudo] xmessage <msg> [-timeout <n>(单位：sec)] [-buttons OK,Cancle] #以xmessage丑陋的窗口弹窗显示信息

#某些可以从命令行打开GUI的程序，如redis-desktop-manager
...

命令及功能：

[ -d "~" ] is false
[ -d ~ ] is true
[ -d "$HOME" ] is true
apt-cache show app # check whether app is installed
apt-get update # update apt repository

telnet <CTRL+]> back to telnet then quit or <CTRL+D> to quit telnet

tail -n +NUM FILE # show line NUM to end

export NAME=xxx
unset NAME

# detect encoding of tile content
enca FILE
# convert encoding of file content
enconv FILE
# convert encoding of text
iconv -f from-encoding -t to-encoding
# convert encoding of file name
convmv -f from-encoding -t to-encoding --notest
# convert file content between character sets
recode FROM-ENC..TO-ENC FILE    (e.g. recode gbk..utf8 /a.txt)
# list the supported character sets of recode
recode -l

lsusb
lscpu

# specify an encoding before extracting
unzip -O encoding
unzip -O cp936 -l chap04.zip | head -n -2 | sed -n '4,$p' | awk '{print $4}' | grep 
# read the contents of files in a zip file; extract file into stdout
unzip -c ZIPFILE ENTRY

locate:
 locate -b(basename) xx ; locate -r regex

# manual for command
man xx; man [1-7] xx; man 7 man
# help for shell-built command 
help <CMD> # help cd; help pwd
# document
info xx
# search man page names and descriptions
apropos xx

# block device, uuid, &...
(sudo) blkid /device...  ;  blkid -s UUID /device ;  blkid -s LABEL /device


# To cut out the third field of text from file or stdin, which is delimited by a #:
cut -d # -f3
# first char of column instead of whole column, cut twice
cut -d # -f3 | cut -c 1  # cannot cut -f3 -c 1

# sort file name numerically
ls /dev/sda* | sort -k 1.9n
# list files sorted by size(desending order)
ls -S
# list files sorted by time(desending order)
ls -t

# list files, dereference symbolic link
ls -L ...

# short desciption of ascii characters
man 7 ascii

# directory or file usage
du -hs FILE
du -a <directory> # including files not only directory
du -d <depth> FILE 

# disk usage
df -lh /mount/point /# or mounted /dev/xxx, if /dev/xx not mounted, output is not what we expected

# disk ssd or hdd
lsblk -d -o name,rota  # 0 for ssd, 1 for hdd
cat /sys/block/sdx/quene/rotational

# create an iso image file from a cd/dvd/bd
dd if=/dev/cdrom of=/path/to/image.iso

# eject a disc
eject   eject cdrom    eject /mount/point    eject /dev/cdrom

# mount Windows shared folder, mount samba share (cifs <- sambfs)
mount -t cifs -o username="WinUser",password="secret",uid="localuser",gid="localgroup",dir_mode=0777 //192.168.x.x /mnt/win
# maybe need run the following command before mount
apt install cifs-utils # if no cifs filesystem supporting lib in system (by checking if /sbin/mount.cifs exists)

# mount an iso image file
mount -o loop /path/to/image.iso /path/to/mount/point
# mount partition by non-superuser
gvfs-mount -d /dev/sda1 /mnt/point/ # gvfs-* for gnome user only
# unmount
gvfs-mount -u <location>
# list
gvfs-mount -l

无密码访问远程主机R
# generate key pair on local host
ssh-keygen -t rsa 
#copy public key id_rsa.pub into remote host R
scp x.pub user@host:/home/u/.ssh/
# on remote host
cat x.pub >> .ssh/authrized_keys

# cgroup: control group
# sudo apt-get install cgroup-bin

# share a read-only/readable-writable directory over lan
?????

# find devices on lan
????

# find a device on lan then mount
?????

# list open files for a process
ls /proc/PID/fd
lsof -p PID

# determine / identify processes using files or sockets
fuser name...

# merge two folders
rsync -abviuzP src/ dest/
    -i turns on the itemized format, which shows more information than the default format
    -b makes rsync backup files that exist in both folders, appending ~ to the old file. You can control this suffix with --suffix .suf
    -u makes rsync transfer skip files which are newer in dest than in src
    -z turns on compression, which is useful when transferring easily-compressible files over slow links
    -P turns on --partial and --progress
    --partial makes rsync keep partially transferred files if the transfer is interrupted
    --progress shows a progress bar for each transfer, useful if you transfer big files

# merge two folders with cp, tip: hard link: cp -r --link dir1/* dir2/* merged/ && rm -r dir1/ dir2/
# IMPOSSIBLE: create necessary destination directories when copying file, e.g. cp foo.txt /path/not/existing/foo.txt

# telnet to a ssl port(e.g. 443)
openssl s_client -connect host:port

# shared library dependencies
ldd BINARY

# list symbols from object files
nm OBJ(?.o) LIB(?.so ?.a)

# set shared library search paths(directories), add paths to /etc/ld.so.conf, then run ldconfig
vim /etc/ld.so.conf
ldconfig
# report missing objects
ldd -r /path/to/ELF_binary
ldd /path/to/executable | grep 'not found'

# 
readelf -d <binary> | grep RPATH

# restore trash that deleted as root
su
cd /path/to/parent/of/file/trashed
restore-trash

# change label of partitions
# mlabel or fatlabel for fat32
# ntfslabel for ntfs
# exfatlabel for exFAT
# e2label for ext2/ext3/ext4
# btrfs for BTRFS

# change label of a fat32 filesystem
sudo mlabel -i /dev/sdb1 ::"new_label"
#if fails & print '... add mtools_skip_check=1 to .mtoolsrc', do
#echo "mtools_skip_check=1" >> ~/.mtoolsrc # as the program says
# get or set label of ms-dos filesystem
sudo fatlabel /dev/sdc1 [new_label]
# change label of a ntfs filesystem
sudo ntfslabel /dev/sda7 "new_label"
# change label of a exfat filesystem
sudo exfatlabel /dev/sdc1 "new_label"
# change label of a ext2/3/4 filesystem
sudo e2label /dev/sdc1 "new_label"

# list background jobs
jobs
bg
# foreground jobs
fg

# get gpu information
lspci -v | grep VGA -A 12

# grep
grep PATTERN {FILE | stdin}
 -r # deep in FILE recursively
 -i # case-insensitive
 -v # not matching
 -A NUM, --after-context=NUM # print NUM lines after matching lines
 -B NUM, --before-context=NUM # print NUM lines before matching lines
 -C NUM, --context=NUM
 -H, --with-filename
 -h, --no-filename


sudo ssldump -i lo port 443 

# delete entries from a zip file
zip -d foo.zip foo/entry/to/delete
# delete a directory from a zip file
zip -d foo.zip "foo/dir/*"

# command information
type <command>
# locate a command
which <command> 

# make naming fifo
mkfifo
# make special files
mknod <name> <type>

# change password of current user
passwd
# change another user
sudo passwd <username>

# add group
sudo addgroup <group-name>
# add user
sudo adduser -ingroup <group-name> <user-name>

# login as / become another user, default as root
su - <user-name>

# tell login shell or not
shopt login_shell

# add user in sudoers
# just put the user into 'sudo' group
usermod -a -G sudo <USER-NAME>
su -        # login as root
visudo
# then move to the last line, write <user-name> ALL=(ALL:ALL) ALL  then ctrl-X to exit, Alt-B to backup previous version file

# screen resolution
xdpyinfo | grep dimensions

# groups of user
groups <user-name>...
# list groups of current user
groups

# add user into group
usermod -a -G <group-name> <user-name>  #e.g usermod -a -G sudo jack

# top usage
memory usage summary value scale: E (circle in KB<-->PB)
memory usage of process: swap between size/percentage: ?
sort by memory usage:M
sort by cpu usage:P
toggle command/command-line: c

# memory usage
free -m (by M unit)
free -g (by G unit)

ps aux -p <PID>

screen [-S session-name] # create a screen sesion
# run some long-time-consuming script
screen -d [session-name] # <Ctrl-A d> detach from session
screen -ls # list sessions
screen -r <session-name> # attach to session
screen -d -r <session-name> # attach , detach first if necessary
# which sesseion i am in
echo $STY
# exit: close session then exit

# to leave the process running after closing the terminal
<ctrl+z> <- suspend the process and back to terminal
bg  <- show background jobs, find out the job id of corresponding process
disown -h %<jobib>  <- let the job run in background

pgrep -f <pattern> # search pattern in command line (not only in command)
pgrep -a <pattern> # print full command

# find files whose name ends with .c in (and desending into) current directory
find . -name "*.c" # quotes are must. or: find . -name \*.c  or: find .-name '*.c'
# 在大范围内查找（如根目录/）是很鲁莽的，find在大范围内查找的开销较大

#!!! * in a shell command line will be expanded as file names(hidden files excluded), if it is a wildcard in an argument, should be enclosed by qutoes or escaped
# find . -name *.c => means find . -name x.c y.c z.c  where x.c y.c z.c are only files ending with .c in current directory
# find . -name \*.c or "*.c" '*.c' => means find files in .(and desending), which name ending with .c
# find files on which no read-permission
find <FILE> ! -perm -u+r # user-non-readable
find <FILE> ! -perm -g+r # no permision on group
# find files by size
find . -size +100M # larger than 100M
find . -size -100M # smaller than 100M
# find files by owner
find . -user <USER> # username or user id all allowed
# find files by time in day unit, -ctime, -atime, -mtime;  c: change(attributes been changed), a: access, m: modify
# by time in minute unit, -cmin, -amin, -mmin
# time argument, +<NUM> over NUM day/min, -<NUM> within NUM day/min
find . -ctime +2 -and -mmin 60 # 
# striping leading './' of output of find .   ; find * will not search hidden file/dir(file of name starts with '.')
find -printf '%P\n'


# md5 checksum
md5sum FILE...

# gui sudo
gksudo <cmd line>

# print http headers
curl -v <URL>

# startup services management
sysv-rc-conf

# change owner of symbolic file, put -h (no dereference) option
chown -h user:group FILE

# mount remote directory conneting by ssh
sshfs user@host:/path /local/path
# umount remote directory
fusermount -u /local/path

# determine linux distribution info
ls /etc/*-release
# or
lsb_release -a

# print kernal version
uname -a

# print listening, established connections with specific port
(sudo) lsof -i:<PORT> # lsof -i:8080
(sudo) netstat -tunlp|grep <PORT> # -tunlp: tcp, udp

# list symbols in a shared library
readelf -s /path/to/libx.s

# encode a image or a file in base64
base64 <FILE>

# system time(date)
date
# hardware clock time
hwclock   # sudo hwclock

# dmesg, messages from kernel
dmesg

# awk
# swap columns
awk '{print $2"\t"$1}' <FILE>
# accumulate numbers, one per line (sum)
awk '{s+=$1} END {print $s}' <FILE>
# using shell variable in awk, you cannot use one directly
awk -v home=$HOME 'print(home)'  # careful: it's home, not $home
# output columns satisfy criteria, e.g not begin with #, and 3rd column starts with <
awk '$1 ~ /^[^#]/ && $3 ~ /^</'


# create directory if not exist
[ -d /dir ] || mkdir -p /dir

ss -tlnp

# rename network interface temporarily, 'eth0' -> 'eth1'
ifconfig eth0 down
ip link set eth0 name eth1
ifconfig eth1 up
# rename network interface permanently, edit /etc/udev/rules.d/some-file


# io statistics
iostat
iostat -x 2 5 # extented info, update every 2sec, total 5 updates

# top io wait process
iotop

# list open files
lsof -p <pid>

# copy into gui clipboard
xsel -b

# compare unique file line by line, make lines *sorted* & *unique* in each file
comm FILE1 FILE2   # a hyphen - means stdin, only one hyphen allowed
 -1 supress column 1 - unique lines in FILE1
 -2 supress column 2 - unique lines in FILE2
 -3 supress column 3 - unique lines in FILE1 & FILE2
 -12 tells intersect, -23 tells {FILE1} - {FILE2}, -13 tells {FILE2} - {FILE1}

# file status
stat FILE

# system uptime
uptime

# network monitor
iftop

# change user name
usermod -l new_name user_name
# move home dir, -m -d
usermod -m(move) -d <NEW_DIR> <USER>
# change uid
usermod -u <NEW_UID> <USER>
# append user to group, add group to user, -a -G
usermod -a -G <GROUP,GROUP2...> <USER>
# list user and group id
id
# list id and belonging groups
id <USER>
#change group name
groupmod -n <NEW_GP> <GROUP>