25 Oct 2014

How Linux is Built

30 Jan 2013

The web is stateless, why you need to refresh the page.

As I deal with people who use computers every day, who don't necessarily understand what is happening. I have decided to try and explain it once in nice simple language so that it is easy to
understand why certain web sites / applications / pages seem to have issues showing the most up-to-date information.

The web is stateless. It has no way of remembering what is happening or what has happened. This is the way it was built from the very lowest levels. It is designed to survive a nuclear war.

What you see on your computer is just a picture or view of the web application or web page at a particular point in time. The time you requested the page you are currently viewing. Your web browser has no way of know if any thing has changed on the server or web page. Unless you ask the server by requesting the page again. You need to refresh the page to see any changes that have been made since you last asked the server.

To make things even more complicated your web browser tries to speed up the time it takes to load pages by caching web pages. If you just viewed a web page your web browsers stores it on your device or computer. If you ask for that page again, instead of fetching the page from our server it will show you the page it stored on your device.

Web developers have developed a number of technologies to work around caching, by telling web browser which pages it can cache and what in a web page it can cache. Though some times this does not always work as expected and thing can be come out of sync. The pages you see are a mix of the old data stored on your device and the new content on the servers.

To make things even more complicated it all comes down to how computers understand time. The time on the server is different from the time on a server sitting in the middle of Russia. Which is also
different from the time on your computer. Even for computer in the same country the time they show can differ by hours if they are not set correctly. So how your web browser caches data, and for how long can be quite different then what our server or web page asked it to.

The point is that the only time that matters when you are working with a web server or web application is the time on the server not on your computer. Bidorbuy provides a way to see this using the follow page, http://www.bidorbuy.co.za/jsp/header/BidorbuyTime.jsp. This page can be found on our site at the very bottom of every page, in the footer under the bidorbuy heading.

http://www.webopedia.com/TERM/S/stateless.html
http://en.wikipedia.org/wiki/Stateless_protocol

http://www.mouseworks.net/refreshbrowser.htm
http://www.refreshyourcache.com/en/cache/
--

https://plus.google.com/u/0/117484568487416046905/posts

05 Sep 2012

Working with log files in the /var/log/ directory Part 3 ssh

Continuing from part 1

Part 3 will deal with using these commands in scripts and over ssh to get some information on remote servers. 

As most of what I wanted to say is covered in an other blog I am provding a link to that blog with some of the important parts cut out.

 http://backreference.org/2011/08/10/running-local-script-remotely-with-arguments/ 

Well, ssh runs a shell on the remote host, so why not do
 $ ssh user@remote < local.sh  

And if your script has arguments
$ ssh user@remote 'cat | bash /dev/stdin arg1 arg2 arg3' < local.sh 
Argument is __arg1__  
Argument is __arg2__  
Argument is __arg3__

Now what would you put inside a script. In my case I have a script that I use to do some basic checks. I will cut out our work specific stuff but you should get a really good idea about what you can do and how to do it. This script is a work in progress please fell free to make changes. 

How I run the script. Use the multiple arguments method

ssh root@192.168.0.51 'cat | bash /dev/stdin -a < ./check.sh


What my script does



 /tmp/check.sh -h
Usage: check.sh [-hvapdbs] [-o arg] args
Options avalible in the script -h Help
  -v version
  -s disable sleep time beween checks

Chose which checks you want to perfom 
  -a (all) checks 
  -p not implemented
  -d defualt linux health checks 
  -w webserver checks
  -l list all the check functions

How to use

ssh root@192.168.0.51 'cat | bash /dev/stdin -a -s' < `basename $0` | tee /somedir/checks_output

The script 



#!/bin/bash

# simple example of how to use this script

# simple script that will run some of the basic checks that need
# to be done on the bob servers. 

# simple boolean to enable or disapable sleep/pauses in script out put 
# set waits=$NULL or waits=  to ensure the test evalues false. 
# how to test
# waits=
# [ $waits ] && echo true
# waits=$NULL
# [ $waits ] && echo true
#
# setting $waits to anything other then null evaluates true
# waits=1
# [ $waits ] && echo true
# true
waits=TRUE
pause=2

#Nicely formated title on checks
title() 
{
    Sstart="\n\e[04;31m="
    Send="=\e[00m\n"
    Sfill="================================================================="
    Stitl="$1"
    Sbuild="$Sstart $HOSTNAME -- $Stitl"
    fill=${#Sbuild}
    #echo $fill
    #echo ${Sfill:fill}
    #echo -e "\e[04;31m= $HOSTNAME ==== $1 =============\e[00m\n"

    Sbuild="$Sstart $HOSTNAME -- $Stitl ${Sfill:fill}$Send"
    echo -e "$Sbuild"
}

# check disk usage on server
disk_check()
{
    title "Checking Disk Space"
    #echo -e "\e[04;31m=$HOSTNAME=====Checking Disk Space===================\e[00m"
    df -h
    [ $waits ] && sleep 3

    # check inodes
    title "Checking Available inodes"
    #echo -e "\e[04;31m=$HOSTNAME=======Checking Available inodes===========\e[00m"
    df -i
    [ $waits ] && sleep 3
}

dmesg_check() 
{
    #dmesg
    title "Checking DMESG"
    #echo -e "\e[04;31m=$HOSTNAME======Checking DMESG========================\e[00m"
    [ $waits ] && sleep 2
    echo "dmesg tail -n 60"
    dmesg | tail -n 60
    [ $waits ] && sleep 8
    echo "summary of dmesg"
    dmesg | sed -e "s/\[[^][]*\]//" -e "s/[0-9]\+//g" | sort | uniq -dc | sort -n
    [ $waits ] && sleep 8
}

mesg_check() 
{
title "Messages Log"
    #cut -d" " -f 4- /var/log/messages | sort | uniq -dc
    #cut -d" " -f 5- /var/log/messages |\
    # sort |\
    # sed -e "s/pid=[0-9]* //"\
    # -e "s/EXIT\: rsync status\=0.*/EXIT: rsync status=0/"\
    # -e "s/last message repeated.*/last message repeated/" |\
    # uniq -c |\
    # sort -n

    sed -e "s/\(.*\)\($HOSTNAME\)/\2/" -e "s/\[[^][]*\]//" -e "s/[0-9]\+//g" /var/log/messages | sort |  uniq -c | sort -n
    [ $waits ] && sleep 5
}

DT_check()
{
    # date
    title "Checking Date & Time"
    [ $waits ] && sleep 3
    date
    [ $waits ] && sleep 3
}

network_check()
{
    # check interfaces for droped packets and errors
    title "Checking Network Interfaces"
    [ $waits ] && sleep 3
    ifconfig
    [ $waits ] && sleep 5
    # check firewall
    echo "Showing ip tables"
    iptables -L
    [ $waits ] && sleep 5
}

crontab_check()
{
    # check the jobs schedule 
    title "Checking Crontab Entries"
    #echo -e "\e[04;31m=$HOSTNAME======Checking Crontab Entries==============\e[00m"
    [ $waits ] && sleep 3
    #find /var/spool/cron/* -printf  \\n%p\\n%t\\n -exec cat {} \;
    find /var/spool/cron/* -printf "\\n\\n%p\\nLast Modified : %t\\nLast Accessed : %a\\n\\n" -exec cat {} \;
    [ $waits ] && sleep 3
    #[ $waits ] && sleep 3
    #[ $waits ] && sleep 3
}

keys_check()
{
# check ssh configuration to make sure it has not changed
    title "Checking Known Hosts"
    [ $waits ] && sleep 2
    cat /root/.ssh/known_hosts 
    echo -e "\n"
    echo -e "\nList of authorized IPs"
    cut -d" " -f1 /root/.ssh/known_hosts | sort -n
    [ $waits ] && sleep 5
    title "Checking Authorized Keys"
    [ $waits ] && sleep 2
    cat /etc/ssh/authorized_keys 
    echo -e "\nList of authorized IDs\n"
    grep -v "#" /etc/ssh/authorized_keys | cut -d" " -f3
    [ $waits ] && sleep 5
}

mail_check()
{
# chech maillog for status not sent should not return any output
    title "Checking Mail Logs"
    [ $waits ] && sleep 3
    echo -e "\nChecking mail logs\n"
    grep -P "(stat|status)=(?"\!"[Ss]ent)" /var/log/maillog
    grep -P "(stat|status)=(?"\!"[Ss]ent)" /var/log/maillog | wc
    #grep -P "stat=(?"\!"Sent)" /var/log/maillog
    #grep -P "stat=(?"\!"Sent)" /var/log/maillog | wc 
    #grep -P "status=(?"\!"sent)" /var/log/maillog
    #grep -P "status=(?"\!"sent)" /var/log/maillog | wc
    [ $waits ] && sleep 3
}


apache_check()
{
    # Apache Error Logs
    title "Checking Apache"
    if  [ -e /var/log/httpd/error.log ]
    then
        [ $waits ] && sleep 2
        #sed -e "s/\[[^][]*\]//g" -e "s/[0-9]\+//g" /var/log/httpd/error.log | sort | uniq -c | sort -n
        title "Sumary of httpd error"
        #echo -e "\n Sumary of httpd error messages"
        sed -e "s/\[[^][]*\]//g" -e "s/ referer:.*//" -e "s/[0-9]\+//g" /var/log/httpd/error.log | sort | uniq -dc | sort -n
    #sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" /var/log/httpd/error.log | sort | uniq -dc | sort -nr
    #sed -e "s/\[.*\]\([^:]*\)\(.*\)/\1/" /var/log/httpd/error.log | sort | uniq -u | sort -nr
        #grep "phpmyadmin" /var/log/httpd/error.log
        [ $waits ] && sleep 5
    else 
        echo -e "no apache error logs at /var/log/httpd/error.log\n"
    fi
}



all_checks()
{
disk_check
dmesg_check 
mesg_check 
DT_check
network_check
crontab_check
keys_check
mail_check
apache_check

}

default_checks() 
{
disk_check
dmesg_check
mesg_check
DT_check
network_check
crontab_check
keys_check
mail_check

}


checks()
{
#g/()/

checklist=$(cat <
title() \n
disk_check()\n
dmesg_check() \n
mesg_check() \n
DT_check()\n
network_check()\n
crontab_check()\n
keys_check()\n
mail_check()\n
apache_check()\n
checks()\n

EOF
)
echo -e $checklist

}


USAGE="Usage: `basename $0` [-hvapdbs] [-o arg] args"
USAGED=$(cat <
Options avalible in the script
\t-h  Help\n 
\t-v  version\n
\t-s  disable sleep time beween checks\n

\nChose which checks you want to perfom \n
\t-a  (all) checks \n
\t-p  not implemented\n
\t-d  defualt linux health checks\n
\t-w  webserver checks\n
\t-l  list all the check functions\n

\nHow to use\n
ssh root@192.168.0.51 'cat | bash /dev/stdin -a < ./check.sh | tee /somedir/server_report\n
EOF
)


# Parse command line options.
while getopts hvsabdwtlo: OPT; do
    case "$OPT" in
    h)
        echo $USAGE
        echo -e $USAGED
        exit 0
        ;;
    v)
        echo "`basename $0` version 0.4"
        exit 0
        ;;
    s)
        waits=
        ;;
    o)
        OUTPUT_FILE=$OPTARG
        ;;
    a)
        echo "doing all checks"
        all_checks
        exit 0
        ;;

    b)
        echo "doing bob checks"
        bob_checks
        exit 0
        ;;

    p)
        echo "doing all checks"
        all_checks
        exit 0
        ;;
    d)
        echo "doing default checks"

        default_checks
        exit 0
        ;;
    w)
        echo "doing apache checks"
        apache_check

        exit 0
        ;;
    l)
        echo "showing all checks"
        checks
        exit 0
        ;;

   \?)
        # getopts issues an error message
        echo $USAGE >&2
        exit 1
        ;;
    esac
done

# Remove the switches we parsed above.
shift `expr $OPTIND - 1`

# We want at least one non-option argument.
# Remove this block if you don't need it.
if [ $# -eq 0 ]; then
    echo $USAGE >&2
    exit 1
fi

# Access additional arguments as usual through
# variables $@, $*, $1, $2, etc. or using this loop:
for PARAM; do
    echo $PARAM
done

# EOF






20 Jul 2012

Delete old emails in gmail from a specific date.

I can perform a search with the following criteria in gmail.
before:2011/11/20 AND in:anywhere

How can I go to the next step and delete all those mails.
Google search for delete old mail from gmail don't return any thing useful.

Idealy I would like to create a filter with some search useing just the "before:" operator.
before:2011/11/20
I keep getting a Please "fill in some search criteria" message box.
To fix this you don't use the filters
Gmail creates a link for you to delete all the old mail, it is not very obvious. It appears just above the first message in the list of messages.

"All 20 conversations on this page are selected. Select all conversations that match this search"


If you are running short of space you could always buy some more from google
https://www.google.com/settings/storage/

05 Jul 2012

gmail filters to search for mail from bidorby

This is essentially what I am trying to do in the Google mail.
http://mashable.com/2012/06/22/gmail-filters/

http://support.google.com/mail/bin/answer.py?hl=en&answer=6579#0

So the images show step for step how to search for emails from bidorbuy.co.za and then create a filter form that search to mark those emails as important.

If you click each image you can load a bigger high resolution version of it. That should make thing easier to see.

First I click the gmail search drop down to show the advanced search.

The advanced search will show up like this

Next I type in the advanced search that I want to perform. 
In the From Filter I type in @bidorbuy.co.za. Then I click the search button at the bottom of the page.
that should perform the search and bring up all the mails that google has been automatically sending to the archive. You may need to change the date range that the search is performed over to include a period of more then 1 day. 

To create the filter I click the drop down again, it remembers my previous search so the from filter is pre-populated

At the bottom of the advanced search dialogue, there is an option to "create filter with this search". Clicking on that takes us to the filter dialogue. 

Now I can tell gmail to always make mail from bidorbuy as important and apply that filter to previous mails. 

06 Jun 2012

my solution to http://blockly-demo.appspot.com/blockly/demos/maze/index.html

Great way to teach people programming,
http://blockly-demo.appspot.com/blockly/demos/maze/index.html

My solution uses the premise that you can get out of any maze by just making turns in one direction. In my case left turns. But this will hold true for right turns as well, but a different path will be followed. When I tested it id did not quite work that way.


On my second attempt I found the universal solution that work for turning left and right.

Here is an other way to do the same thing if you know which path you want to take through the maze

http://blog.beens.org/2012/06/google-blockly-great-way-to-introduce.html











15 May 2012

Working with log files in the /var/log/ directory Part 1


The point of this tutorial is to give a better understanding of how to look at the log files in your /var/log directory. Using the built-in command line tools available, if you are not using some special purpose monitoring suit. This tool is in the intermediate level, all though those with lost of *nix experiance will find certain sections tedious as I describe what I typed in detail, for novice first time users. This is not how ever a tutorial on vim or any of the other tools.

I will be looking at apache logs and dmesg or syslog files. But with a good understanding of these tools you can adapt them to other log files.

The basic command you should be familiar with are:
vim a text editor
cat a tool to out put the contents of a text file. 
sed a stream editor
grep a text search tool
cut a tool to cut words or characters from lines of text
head a tool to out put the first n lines of a text file. 
tail a tool to out put the last n lines of a text file.
less a tool to out put the contents of a text file.
sort tool to sort lines of text
uniq tool to find unique lines of from sorted text
tee tool to out put text passing through a pipe
wc count the lines, characters and bytes in file. 

Some other tools that would be worth looking at are:
ssh
bash especially how redirects and pipes work. "> < | ".

At some point you need to go and read the man pages on all those command, and applications.

Part 1 Using Vim

Why use vim, because you can use some very power full text search and replace filters to get some useful info from you log files. The other tools allow you to do the same thing in a much more automated way.

Counting occurrences of a particular line or error message. One of the most useful things you can do is be able to get some stats like a counts of a particular error message.

In vim we use a very similar syntax to what we use in sed.
First we open the log file in a read only mode. You could use view, but in some environments view is compiled with out certain options. Which makes using the visual selection commands impossible.

vim -R /var/log/syslog

Next I looked for some text that I want to get a count of. "local kernel:" seamed to stand out. I entered the visual mode using the "v" key and selected the text I want to search for "local kernel". I yanked the text with "y", which is similar to copying. Next I stared the global search tool by typing in ":g/" now I want to past the text I just copied so I use `ctlr+r+"` to past the text I just copied into the command.

g/local kernel:/

After I pressed enter, global brings up a list of all occurrences of the provided text "local kernel:". It was a long list. To see how long that list is I can use the substitute command. Global allows you to add the substitute command to the end of your search. So I entered the command ":" mode again and pressed the up arrow to get to the previous command and add "s///n" at the end. Which will give me a count for my search string.

g/local kernel:/s///n 
47918 substitutions on 47918 lines 

Now that I know how many times that lines with local kernel is found in the file, I want to remove those lines to see what else might be needing my attention. Again I enter command mode ":" and press the up arrow. I then change my global command to look some thing link this.

:g/local kernel:/d

W10: Warning: Changing a readonly file
47918 fewer lines


That warning about being read only can be ignored, because that is how I want this process to work. I don't want to remove vital information from a production log file. It would probably be a good Idea to copy these files some where else before doing what I am doing but with some 4 GB monster log files this may not be an option. Repeat this process until you are left with and empty log file or unique strings. 

This is process is useful to see how many times certain errors are cropping up. Some times it is easier to focus on fixing on problem that  accounts for 90% of the errors. But using vim in this way gets tiresome very quickly. The faster and more automated way of doing this is to use your command line and eventually to script the processes. 

Part 2 Using just the command line

The tool that makes command line processing of log files to get stats about the number of error messages is uniq. The problem with most log files is that unique on every line because of date information and process information.  So we need a way to get around this problem.

For starters I generate some 404 page not found errors in my apache log file.


 wc /var/log/apache2/error.log
  28  386 3421 /var/log/apache2/error.log

 wc /var/log/apache2/error.log
  81 1075 8863 /var/log/apache2/error.log


wc gives me some stats on the file like how many lines are in it how many charters and lastly how big the file is in bytes. So when I try to run the file through uniq it comes out  the same size. 


uniq /var/log/apache2/error.log | wc
     81    1075    8863


uniq needs the files sorted inorder to be able to give us the count of duplicate lines in a file or to produce a list of only those lines that are uniq. So just buy sorting the file I am already removing some of the duplicates. Now if I want to get some stats on the duplicates I can by telling "uniq -c" to count lines and "uniq -d" to only show duplicates. Read the man page for more info. 


sort /var/log/apache2/error.log | uniq | wc
     76    1010    8358


sort /var/log/apache2/error.log | uniq -dc | wc
      5      70     545


sort /var/log/apache2/error.log | uniq -dc 
      2 [Tue May 15 09:54:10 2012] [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico
      2 [Tue May 15 09:54:13 2012] [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico
      2 [Tue May 15 09:54:14 2012] [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico
      2 [Tue May 15 09:54:23 2012] [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico
      2 [Tue May 15 09:54:27 2012] [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico

As  you can see favicon.ico is not found, but because of the date it is not producing an accurate count of that "File does not exist error". I need some way to remove that information from file before sending it to uniq.

The first and simplest way to do that is with the "cut" command.  Using the magic of pipes I am going to use head and cut together to so I don't get my terminal scrolling while working out what parameters cut needs. "head -n 2" tells head to only show the first 2 lines.  `cut -d" " -f2-` tells cut to use the delimiter of a single " " space and to cut from the 2nd space "-" "-f2-" to the end of the line.


head -n 2 /var/log/apache2/error.log 
[Mon May 14 09:36:07 2012] [error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
[Mon May 14 09:36:07 2012] [error] python_init: Python executable found '/usr/bin/python'.


head -n 2 /var/log/apache2/error.log | cut -d" " -f2-
May 14 09:36:07 2012] [error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
May 14 09:36:07 2012] [error] python_init: Python executable found '/usr/bin/python'.


head -n 2 /var/log/apache2/error.log | cut -d" " -f5-
2012] [error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
2012] [error] python_init: Python executable found '/usr/bin/python'.


head -n 2 /var/log/apache2/error.log | cut -d" " -f6-
[error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
[error] python_init: Python executable found '/usr/bin/python'.

Now there is none of the date string left in the file. I can sort it and try uniq again. The only problem is that even though I have the counts of each error, it is not showing with the most common first or last. I could also remove those lines that only occur once using the "-d" duplicates only option on uniq.



cut -d" " -f6- /var/log/apache2/error.log | sort | uniq -c | wc
     14     130    1273


cut -d" " -f6- /var/log/apache2/error.log | sort | uniq -c
      6 [error] [client 127.0.0.1] File does not exist: /usr/share/doc/apache2-doc/manual/en/index.html21
      5 [error] [client 127.0.0.1] File does not exist: /var/www/html/conf/12
     29 [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico
      6 [error] [client 127.0.0.1] File does not exist: /var/www/html/isos.html12
      7 [error] [client 127.0.0.1] File does not exist: /var/www/html/tmp/12
      4 [error] python_init: Python executable found '/usr/bin/python'.
      4 [error] python_init: Python path being used '/usr/lib/python2.7/:/usr/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
      4 [error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
      4 [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.1 with Suhosin-Patch mod_python/3.3.1 Python/2.7.3 configured -- resuming normal operations
      2 [notice] caught SIGTERM, shutting down
      1 [notice] Graceful restart requested, doing restart
      4 [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
      4 [notice] mod_python: using mutex_directory /tmp 



cut -d" " -f6- /var/log/apache2/error.log | sort | uniq -dc | sort -n
      2 [notice] caught SIGTERM, shutting down
      4 [error] python_init: Python executable found '/usr/bin/python'.
      4 [error] python_init: Python path being used '/usr/lib/python2.7/:/usr/lib/python2.7/plat-linux2:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
      4 [error] python_init: Python version mismatch, expected '2.7.2+', found '2.7.3'.
      4 [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.1 with Suhosin-Patch mod_python/3.3.1 Python/2.7.3 configured -- resuming normal operations
      4 [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
      4 [notice] mod_python: using mutex_directory /tmp 
      5 [error] [client 127.0.0.1] File does not exist: /var/www/html/conf/12
      6 [error] [client 127.0.0.1] File does not exist: /usr/share/doc/apache2-doc/manual/en/index.html21
      6 [error] [client 127.0.0.1] File does not exist: /var/www/html/isos.html12
      7 [error] [client 127.0.0.1] File does not exist: /var/www/html/tmp/12
     29 [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico

That is now very useful. I can still take it further buy removing the information I don't want more selectively using sed. Sed is a very advanced tool, it will take a while to get the hang of it, as learning regular expressions is learning a new language. There are lots of really good tutorials on sed that can be found through google. 

In my case I want to remove every thing inside brackets, as that information is not really useful to me. The first example only removes the first set of brackets, the second removes all brackets. The difference is the "g" at the end of the sed regex. Which tell sed to go global and use the whole line. 


sed -e "s/\[[^][]*\]//" /var/log/apache2/error.log | sort | uniq -dc | sort -n | tail -n 3

      6  [error] [client 127.0.0.1] File does not exist: /var/www/html/isos.html12
      7  [error] [client 127.0.0.1] File does not exist: /var/www/html/tmp/12
     29  [error] [client 127.0.0.1] File does not exist: /var/www/html/favicon.ico

sed -e "s/\[[^][]*\]//g" /var/log/apache2/error.log | sort | uniq -dc | sort -n | tail -n 3
      6    File does not exist: /var/www/html/isos.html12
      7    File does not exist: /var/www/html/tmp/12
     29    File does not exist: /var/www/html/favicon.ico



One thing worth noting here is that some messages in my error log are error level notice and other are error level error. Reading the Apache documentation there are a couple more like waning, and critical.
If I want to only see stats for messages with are of the level error I need to reduce the input I am sending to the whole long pipe. I will use grep  to only process those lines in the file that are containg some special text, like "[notice]" or "[error]". The problem is that with regular expressions "[]" are special characters, so I need to escape them using the "\"


grep "\[notice\]" /var/log/apache2/error.log | sed -e "s/\[[^][]*\]//g" - | sort | uniq -dc | sort -n | tail -n 3
      4   Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.1 with Suhosin-Patch mod_python/3.3.1 Python/2.7.3 configured -- resuming normal operations
      4   mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
      4   mod_python: using mutex_directory /tmp 



grep "\[error\]" /var/log/apache2/error.log | sed -e "s/\[[^][]*\]//g" - | sort | uniq -dc | sort -n | tail -n 3
      6    File does not exist: /var/www/html/isos.html12
      7    File does not exist: /var/www/html/tmp/12
     29    File does not exist: /var/www/html/favicon.ico

The only thing to be a ware of the more complex the pipe become the cpu intensive the work to be done. As your text files get larger into the Giga byte range you may want to be selective in what you do and when. 

Part 3 working with scripts and over a network. 


Part 3 will deal with using these commands in scripts and over ssh to get some information on remote servers.


creating a custom session for gnome to only run awn

This how to is reproduced in full cause it is so use full, follow the links to go to the original. 

Run AWN / Dock in Natty Narwhal, instead of Gnome Panel or Unity Launcher/Panel


If you want to use Avant Window Navigator (AWN) or another dock instead of the default panels/launcher in Natty Narwhal, there are multiple approaches to achieve that.

The best way is to add a custom session option, because it would leave the default session options in their original state, so that you could easily use them as well, and it wouldn't get reset by an upgrade of the respective packages.


Installation

If you don't have AWN installed yet, run these commands in the Terminal, this will add its own PPA to your sources to ensure you always have the most recent version of it:

sudo add-apt-repository ppa:awn-testing/ppa
sudo apt-get update
sudo apt-get install avant-window-navigator-trunk

If you already installed AWN through the official repos, remove it before:

sudo apt-get purge avant-window-navigator
sudo apt-get autoremove

Once it is installed, you will find it under "Applications > Accessories" in classic Gnome, or by typing its name into the Unity Dash.

Sessions

Notice that the 'required components' that were specified in Gconf in previous releases of Ubuntu are now specified by the session settings in "/usr/share/gnome-session/sessions".

Modify the red marked entries to use another dock than AWN.

Custom Session

1. Create the .desktop file:

gksudo gedit /usr/share/xsessions/dock.desktop

[Desktop Entry]
Name=AWN
Comment=This session logs you into GNOME with the dock of your choice
Exec=gnome-session --session=dock
TryExec=gnome-session
Icon=
Type=Application
X-Ubuntu-Gettext-Domain=gnome-session-2.0

Note: What you put as the value for "Name=" here, will be shown as the name of the new session option.

2. Create the session settings:

gksudo gedit /usr/share/gnome-session/sessions/dock.session

[GNOME Session]
Name=Dock
Required=windowmanager;panel;filemanager;
Required-windowmanager=gnome-wm
Required-panel=avant-window-navigator
Required-filemanager=nautilus
DefaultApps=gnome-settings-daemon;

Classic Gnome Session, "Ubuntu Classic"

Modify "/usr/share/gnome-session/sessions/classic-gnome.session":

gksudo gedit /usr/share/gnome-session/sessions/classic-gnome.session

[GNOME Session]
Name=Classic GNOME
Required=windowmanager;panel;filemanager;
Required-windowmanager=gnome-wm
Required-panel=avant-window-navigator
Required-filemanager=nautilus
DefaultApps=gnome-settings-daemon;
IsRunnableHelper=/usr/lib/nux/unity_support_test --compiz
FallbackSessionsID=GNOME2d
GNOME2d=2d-gnome

Unity Session, "Ubuntu"

1. Disable the "Ubuntu Unity Plugin" in "CompizConfig Settings Manager".

2a. Add "avant-window-navigator" to "Startup Applications"

2b. or modify "/usr/share/gnome-session/sessions/ubuntu.session":

gksudo gedit /usr/share/gnome-session/sessions/ubuntu.session

[GNOME Session]
Name=Ubuntu
Required=windowmanager;panel;filemanager;
Required-windowmanager=compiz
Required-panel=avant-window-navigator
Required-filemanager=nautilus
DefaultApps=gnome-settings-daemon;
IsRunnableHelper=/usr/lib/nux/unity_support_test
FallbackSessionsID=FallbackUnity2d;FallbackClassicGnome
FallbackUnity2d=2d-ubuntu
FallbackClassicGnome=classic-gnome
FallbackClassicGnomeMessage=It seems that you do not have the hardware required to run Unity. Please choose Ubuntu Classic at the login screen and you will be using the traditional environment.

Tips

You should place launchers on your desktop in case you need to restart the one, AWN or another dock:

avant-window-navigator

or you want to run the other, Gnome Panel:

gnome-panel

If your dock is also capable to host the Notification Area, like AWN does, then remove those from your Gnome Panel. Otherwise, the respective AWN applet gets killed each time you start Gnome Panel.

I myself find it very handy to have the ability to fire up Gnome Panel occasionally. Mostly for UF support reasons, otherwise not really.

Run Dialog (Alt+F2)

Since the usual Run Dialog's are either provided by Gnome Panel or the Unity Dash, Alt+F2 will stop working when you run only AWN.

AWN's Cairo Menu applet includes a "Launch..." menu item, but those needs to be 'filled' with a replacement for the Run Dialog.

XFRun4

The best replacement in terms of style and integration compared to Gnome Panel's Run Dialog is XFRun4. It's included by the package "xfce4-utils", which is provided by the official repos. To avoid the installation of unnecessary additional packages, if you don't have XFCE installed parallelly, use the "--no-install-recommends" option of 'apt-get':

sudo apt-get install --no-install-recommends xfce4-utils

After the installation, you can either run it through the mentioned "Launch..." menu item in Cairo Menu, or create a keyboard shortcut in CCSM for the command "xfrun4". If you choose the set the key combination to Alt+F2, you will get a conflict dialog, then choose "Disable Run Dialog" (that option is included by the "Gnome Compatibility" plugin).

Synapse

Another great alternative to Gnome Panel's Run Dialog and Unity's Dash is Synapse. It can not only run commands, but also installed applications, which you would otherwise have to run through the menu of classic Gnome, Cairo Menu of AWN, or the Dash of Unity. And similar to the latter makes it use of the Zeitgeist engine to also give you quick access to any kind of recently accessed stuff.

See this guide about further information and how to install it:

http://www.tuxgarage.com/2011/05/synapse-alternative-to-run-dialog-dash.html

Troubleshooting

If Gnome Panel is still getting run at login, alongside to your dock or instead of it:

- Remove the content of "~/.config/gnome-session/saved-session":

rm -f ~/.config/gnome-session/saved-session/*

- Remove the now obsolete Gconf keys:

gconftool-2 --unset /desktop/gnome/session/required_components_list
gconftool-2 --recursive-unset /desktop/gnome/session/required_components

- Check if Gnome Panel is listed/enabled in "Startup Applications", and if so, disable it.

If your dock doesn't get run at login:

- Remove "panel" from the line "Required=" in the concerning session settings:

Required=windowmanager;filemanager;

- Add the command for your dock to "Startup Applications":

avant-window-navigator