Linux Tips and Tricks

File Management

Deleting zero length files [find]

find -maxdepth 1 -type f -size 0 -exec rm -f {} \;
Increase the value for maxdepth to control the number of directory levels you want to remove files under (or remove the parameter completely to remove all zero length files under all sub-folders).



Finding a string in multiple files [bash, grep]

For example, to find the text "foo" in all files under the current dir (recursively):
grep -r 'foo' *
For more usages of grep to search for strings in files see here



Performing an operation on all files of a certain type [find]

For example, moving all XML files in a folder and sub-folders to somewhere else (/some/folder):
find . -iname "*.xml" -type f -exec mv {} /some/folder \;
a possibly faster alternative is
find . -iname "*.xml" -type f | xargs -n1 -i mv {} /some/folder
or (for just the current folder)
for i in `ls *.xml`; do mv $i /some/folder; done




Replacing a string in multiple files [bash, find, perl]

For example, to replace the text "192.168.1.2" with "10.0.0.13" in all files with the name Root under all sub-dirs from the current dir:
find . -name Root | xargs perl -pi~ -e 's/192.168.1.2/10.0.0.13/'



Removing files/folders with a certain name [bash, find]

For example, to remove all the CVS folders from a CVS checkout, while leaving the other files and folders intact:
find . -name CVS | xargs rm -r



Removing files when rm complains about "Argument list too long" [bash, find]

For example, in a folder containing too many ".log" files to remove via rm, run a command like so:
find /some/folder/ -name "*.log" | xargs rm



Convert file names to lowercase [bash, tr]

A bash script which will convert all uppercase characters in all filenames in current directory to lowercase (can ignore errors for files which are already all lowercase):
	#!/bin/bash
	for file in *
		do newfile=`echo $file|tr [A-Z] [a-z]`
		echo "Moving $file to $newfile"
		mv $file $newfile
	done
	
An alternative is to use the command line rename utility like so
rename 'y/A-Z/a-z/' *



Renaming files [rename]

To find the text BLA in the filename and replace it with the text FOO in all .tgz files:
rename 's/BLA/FOO/' *.zip
To change the file extension of all .JPG files to .jpg:
rename 's/\.JPG/\.jpg/' *.JPG
To strip the spaces out of all filenames:
rename 's/ //' *.*
find . -name CVS | xargs rm -r



Splitting large files into smaller pieces (and reassembling them) [split, cat]

For example, to split a large gzipped tar archive with the name too.big.for.cdrom.backup.tgz into CD-ROM sized pieces of around 657MB you would issue the following command:
split -b 675000000
This creates files named xaa, xab, xac etc. - one for each chunk of 657MB. To restore the original file from these pieces, copy all the files to some location and issue the command
cat xa* >> too.big.for.cdrom.backup.tgz



Find files that are changed after a certain moment of time (e.g. when installing a new program) [touch, find]

First, run
touch /tmp/now
Then run the program, then
find -newer /tmp/now
to get a list of all files that changed.


Networking

See which programs are using which ports [netstat]

Running "netstat" without any options shows all local connections and domain sockets. To filter this to just show services listening for incoming TCP and UDP connections:
netstat -ltup
Services that list their foreign address as "0.0.0.0:*" (IPv4) and "[::]:*" (IPv6) are listening locally and on all network interfaces and should be checked that they aren't accessible from the internet (unless intentional).
More info at How to use netstat on Linux/


Package Management

APT in a nutshell

To install a package named X:
apt-get install X
To do a dummy run of an installation a package named X (nothing will be installed, useful to check that an installation won't have any problems):
apt-get install X -s
To uninstall a package named X, but leave its configuration files in place:
apt-get remove X
To uninstall a package named X and all its related files:
apt-get --purge remove X
To remove orphaned packaged:
apt-get autoremove
Search all installed packages and check whether there are any broken dependencies (useful to run from time to time):
apt-get check
To get the latest package information from the source in the apt sources.list:
apt-get update
To upgrade to the latest package versions obtained from running the above command:
apt-get upgrade
To search for a package:
apt-cache search X
The above command accepts regular expressions as the argument so to search for package starting with X use ^X and ending with X use X$ etc.

To see which package a file belongs to:
apt-file update
apt-file search SOMEFILE
To do the opposite and see which files a package provides:
apt-file list Y
To improve the performance of APT downloads you can select mirrors by using "netselect-apt". When it is run it will check the mirrors from debian and ping them and save a new sources.list file in the current directory with the fastest mirrors.

To do a dump of all the software currently installed run
dpkg --get-selections > SOMEFILE
If you save this to a file you can restore it later (e.g. on a different machine) by running
dpkg --set-selections < SOMEFILE
apt-get dselect-upgrade
To clean up APT's cache, you can manually remove files from
/var/cache/apt
but a better solution is to use APT to do this, first to clean up packages that it considers old:
apt-get autoclean
Or to delete them all:
apt-get clean



SSH

SSH via an intermediate machine

Certain firewall setups allow one to SSH only onto a single machine (e.g. somehost1.com) and from there one can then SSH onto another machine (e.g. somehost2.com) (i.e. direct SSH to somehost2.com is not possible). In order to SSH to somehost2.com without having to first SSH onto somehost1.com, create a command called something like sssh as follows:
ssh -oproxycommand="ssh -q somehost1.com nc %h %p" $*
If you have set up SSH key access to somehost1.com you can now SSH onto somehost2.com from the machine with direct access only to somehost1.com like so:
sssh user@somehost2.com
and only be prompted for somehost2.com's password.



SCP through an intermediate machine

In a similar vein to the above tip, you can create a very similar command to allow you to SCP to somehost2.com via somehost1.com without having to first SCP onto somehost1.com and from there onto somehost2.com. Create a command (call it sscp) like so:
scp -oproxycommand="ssh -q somehost1.com nc %h %p" $*
and use it like so:
sscp someLocalFile user@somehost2.com:/remote/path



X forwarding over SSH

To run GUI programs on one machine using another machine one can use X forwarding by connecting using
ssh -X user@remotecomputer
and once logged in running the command to start the GUI app, the GUI window will open on the local machine. X11Forwarding must be enabled in the file
/etc/ssh/sshd_config
(which is usually the base by default). To speed things up -Y (Trusted X11Forwarding) or -C (compression) can be used instead of -X.



Monitor SSH login failures

It is worthwhile to periodically check the contents of the file
/var/log/auth.log
for failed SSH connections. This might indicate someone attempting a brute force attack etc.

Shell/CLI

Running multiple commands [bash]

To run a number of commands sequentially, put a ";" between each:
./configure; make; make install
To run a command only if the one before succeeds, use "&&" between them instead:
./configure && make && make install
Similarly to run a command only if the one before it fails, use "||":
./configure || echo "Configure failed"



Creating a .tgz file [tar]

For example, to create a gzipped tar archive with the name foo.tgz of all the contents of the current dir:
tar -czvf foo.tgz *



Determining disk usage [du]

The easiest way to determine how much space your files and directories are taking up is to use the following command:
du -sH * .[a-zA-Z]*
For a quick way to find the biggest ones, try:
du -sH * .[a-zA-Z]* | grep "[0-9]MB"



Reload environment variables [bash]

If you set the variables in .bashrc, issue the command:
source .bashrc



Screen cheet sheat [screen]

Create a new screen session with a certain name:
screen -S screenName
Reattach to a screen by name:
screen -x screenName
Create a new screen:
CTRL-A c
Moving between more than one open screen:
CTRL-A 0/1/2/3 etc.
To scroll back through program output in a screen:
CTRL-A ESC
(push ESC again to go back).
To name a screen window/tab:
CTRL-A A name



Using environment variables [bash]

!$ represents the last word of previous command in your history.

$* represents all command line arguments passed to a command.




Redirecting input and output

Use > to take output from a command to a file:
cat bla > results.txt
This will take the results from cat and put them in a file called results.txt, overwriting results.txt if it exists. To append the output instead of overwriting, one can use >>:
cat bla >> results.txt
Use < to take input into a command:
grep -i X < someFile.txt
This will look for the string "X" in someFile. One can place a number before ">" to direct standard input/output where 0=standard input, 1=standard output, 2 =standard error output. This can be used, for example, to see results of a command sent to standard out while ignoring any errors. e.g. if we run "find" but want to ignore errors (e.g. permissions errors etc). we can redirect error output to /dev/null leaving us with what we are interested in (the standard output):
find / -name *.jpg 2>/dev/null



Determine which processes are running on which ports

Issue the command to see a list of open sockets and pid/program names:
netstat -p



Determine memory usage

Issue the command:
cat /proc/meminfo
(top might also be useful)



Adding paths to library path

As root, add the required paths to
/etc/ld.so.conf
and then run
ldconfig



Determine kernel version [uname]

Issue the command:
uname -a



See status of modules loaded by kernel [lsmod]

Issue the command:
lsmod

Various Errors (and their solutions)

X "connection refused" errors [X windows]

The error message is usually something like 'Xlib: connection to ":0.0" refused by server'. This occurs when you try start an X application as a user other than the user which started X so the authentication keys differ. The first thing to try is
xhost +local:local
as the user who started the x session (usually the user you logged in to X as) before you switch to a new user. This can be added to .bashrc to always allow local (i.e. not over a network) X connections by all users. Some Linux distributions allow you to start X applications as root by starting the root shell using
sux
instead of
su